NEWSLETTER
In large language models (LLM), extended input sequences processing requires significant computational and memory resources, leading to a slower inference and higher hardware costs. The care mechanism, a central component, exacerbates these challenges even more due to their quadratic complexity in relation to the length of the sequence. In addition, maintaining the previous context using a key value cache (KV) results in high memory overloads, which limits scalability.
A key limitation of LLM is its inability to handle sequences longer than its trained context window. Most models degrade on performance when they face extended entries due to inefficient memory management and increasing cost calculation costs. Existing solutions often depend on fine adjustment, which is intensive in resources and requires high quality long context data sets. Without an efficient method for context extension, tasks such as the summary of documents, the generation of recovery and the generation of text in a long way remain limited.
Several approaches have been proposed to address the problem of long context processing. Flashtentent2 (FA2) optimizes memory consumption minimizing redundant operations during attention calculation, but does not address computational inefficiency. Some models use selective attention to Token, either static or dynamic, to reduce processing overload. KV cache eviction strategies have been introduced to eliminate older tokens selectively, but run the risk of permanently discarding important contextual information. Hip attention is another approach that tries to download tokens used to external memory; However, it lacks efficient cache management, which leads to greater latency. Despite these advances, no method has effectively approached the three key challenges:
- Long context generalization
- Efficient memory management
- Computational efficiency
Researchers from the Kaist and Deepauto.ai introduced InfiniteAn advanced frame that allows efficient long context inference to mitigate memory bottlenecks. The model achieves this through a hierarchical tokens pruning algorithm, which dynamically eliminates less relevant context tokens. This modular pruning strategy selectively retains tokens that contribute more to attention calculations, significantly reducing processing overload. The frame also incorporates adaptive rope adjustments (rotary positional inlays), which allows models to generalize longer sequences without additional training. In addition, infinite uses a new KV cache discharge mechanism, which transfers less frequent access tokens to host memory while guaranteeing efficient recovery. These techniques allow the model to process up to 3 million tokens in a 48 GB GPU, which makes it the most scalable long long -scale inference method.
The central innovation of infinite is its pruning mechanism in several stages, which constantly improves the selection of context in multiple stages. The tokens are first divided into fixed length pieces, and each piece is processed according to their attention calculation contribution. A Top-K selection approach ensures that only the most critical tokens are kept and others are eliminated. The method followed by infinity, unlike other hierarchical pruning models, is completely parallel, which makes it computationally effective. The KV Cache management system optimizes the use of memory by means of less important dynamic context tokens while maintaining the flexibility of recovery. The model also uses multiple methods of interpolation of strings in different layers of attention, which facilitates soft adaptation to long sequences.
The model demonstrates an acceleration of 18.95 × in the decoding of attention for a context of one million token compared to traditional methods without additional training. The KV cache discharge technique reduces GPU memory consumption by up to 96%, which makes it practical for large -scale applications. In reference evaluations such as Longbench and ∞bench, the infinite constantly exceeds avant -garde methods, achieving a 9.99% higher score than INFLM. In addition, decoding yield increases by 3.2 × in consumption GPU (RTX 4090) and 7.25 × in business grade GPU (L40S).
In conclusion, the research team successfully addressed the main bottlenecks of long context inference with infinity. The frame improves LLM's capabilities by integrating the pruning of token hierarchical, the discharge of KV cache and the generalization of the rope. This advance allows previously trained models to process extended sequences without losing the context or increasing computational costs. The method is scalable, efficient in hardware and applicable to several ai applications that require long memory retention.
Verify he Paper, <a target="_blank" href="https://github.com/DeepAuto-ai/hip-attention/” target=”_blank” rel=”noreferrer noopener”>Source Code and <a target="_blank" href="https://auth.liteai.io/realms/public/protocol/openid-connect/auth?response_type=code&client_id=app-frontend-nextjs-prod&redirect_uri=https%3A%2F%2Fchat.deepauto.ai%2Fapi%2Fauth%2Fcallback%2Fkeycloak&code_challenge=4XC7xDsuurzSIZAWwH6e10gDBxJON_7hidm5Goi9fxo&code_challenge_method=S256&scope=openid+profile+email” target=”_blank” rel=”noreferrer noopener”>Live demonstration. All credit for this investigation goes to the researchers of this project. In addition, feel free to follow us <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter And don't forget to join our 75K+ ml of submen.
Recommended open source ai platform: 'Intellagent is a framework of multiple open source agents to evaluate the conversational the complex system' (Promoted)
Asif Razzaq is the CEO of Marktechpost Media Inc .. as a visionary entrepreneur and engineer, Asif undertakes to take advantage of the potential of artificial intelligence for the social good. Its most recent effort is the launch of an artificial intelligence media platform, Marktechpost, which stands out for its deep coverage of automatic learning and deep learning news that is technically solid and easily understandable by a broad audience. The platform has more than 2 million monthly views, illustrating its popularity among the public.
