Qwen ai Lanza Qwen2.5-7b-Instruct-1m and Qwen2.5-14b-Instruct-1m: Allows implementation with a context length of up to 1 million tokens

Advances in large languages ​​(LLM) models have significantly improved natural language processing (NLP), allowing capacities such as contextual understanding, code generation and reasoning. However, a key limitation persists: the restricted size of the context window. Most LLMs can only process a fixed quantity of text, usually up to 128,000 tokens, which limits their ability to handle tasks that require an extensive context, such as analyzing extensive documents or purifying large code bases. These limitations often require solutions such as fragmentation of the text, which increases computational complexity. Overcoming these challenges requires models that can expand the duration of the context efficiently without compromising performance.

<h3 class="wp-block-heading" id="h-qwen-ai-s-latest-release”>The last launch of Qwen ai

Qwen ai has presented two new models, QWEN2.5-7B-Instruction-1M and QWEN2.5-14B-Instruction-1Mdesigned to withstand context lengths of up to 1 million chips. Developed by the Alibaba Group Qwen team, these models also come with an open source inference frame optimized to handle long contexts. This advance allows developers and researchers to work with larger data sets in a single pass, offering a practical solution for applications that require extended context processing. In addition, the models have improvements in the scattered care mechanisms and the optimization of the kernel, resulting in faster processing times for extended inputs.

Technical details and benefits

The QWEN2.5-1M series retains an architecture based on transformers, incorporating characteristics such as Attention of grouped consultations (GQA), Rotary positional inlays (rope)and RMSnorma for stability in prolonged contexts. The training involved both natural and synthetic data, with tasks such as Fill in the middle (FIM)Paragraph reorganization and recovery based on positions that improve the capacity of the model to handle long -range dependencies. Methods of scarce care as Double fragment care (DCA) Allow efficient inference by dividing sequences into manageable parts. Progressive pre -entry strategies, which gradually climb the context length of 4K to 1 million tokens, optimize efficiency while controlling computational demands. The models are fully compatible with the VLLM open source inference frame, which simplifies integration for developers.

Results and knowledge

The reference results demonstrate the capacities of the QWEN2.5-1M models. In it Access key recovery testThe 7B and 14B variants successfully recovered hidden information of 1 million tokens, which demonstrates their effectiveness in long -term context scenarios. In other reference points, including RULER and Needle in a haystack (Niah)The 14B model surpassed alternatives such as GPT-4O-mini and call-3. Dispersed attention techniques contributed to reducing inference times, achieving accelerations of up to 6.7x In GPU NVIDIA H20. These results highlight the capacity of the models to combine efficiency with high performance, which makes them adequate for real world applications that require a broad context.

Conclusion

The QWEN2.5-1M series addresses the critical limitations of NLP significantly expanding the duration of the context maintaining efficiency and accessibility. By exceeding the limitations that have long hindered the LLMs, these models open new possibilities for applications ranging from the analysis of large data sets to the processing of complete code repositories. With innovations in scarce attention, kernel optimization and pre-entry of prolonged context, QWEN2.5-1M offers a practical and effective tool to address complex and context tasks.

Verify he Paper, Models hugging the face and Technical details. All credit for this investigation goes to the researchers of this project. Besides, do not forget to follow us in <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter and join our Telegrams channel and LINKEDIN GRabove. Do not forget to join our Subbreeddit of more than 70,000 ml.

<a target="_blank" href="https://nebius.com/blog/posts/studio-embeddings-vision-and-language-models?utm_medium=newsletter&utm_source=marktechpost&utm_campaign=embedding-post-ai-studio” target=”_blank” rel=”noreferrer noopener”> (Recommended Reading) Nebius ai Studio expands with vision models, new language, inlays and Lora models ^(Promoted)