FlashAttention-3 Released: Achieves Unprecedented Speed ​​and Accuracy Using Advanced Hardware and Low-Precision Computing

Attention flash-3The latest release of the FlashAttention series, FlashAttention is designed to address bottlenecks inherent in the attention layer in Transformer architectures. These bottlenecks are crucial to the performance of large language models (LLMs) and applications that require extensive context processing.

The FlashAttention series, including its predecessors FlashAttention and FlashAttention-2, has revolutionized the operation of attention mechanisms on GPUs by minimizing memory reads and writes. This innovation has been widely adopted by most libraries to accelerate Transformer training and inference, which has significantly contributed to the dramatic increase in LLM context length in recent years. For example, context length has grown from 2-4K tokens in models like GPT-3 to 128K tokens in GPT-4 and even up to 1M tokens in models like Llama 3.

Despite these advancements, FlashAttention-2 was only able to achieve 35% utilization of the theoretical peak FLOPs on the H100 GPU, highlighting a gap between potential and actual performance. FlashAttention-3 seeks to close this gap by leveraging new hardware capabilities in modern GPUs. Specifically, it introduces three main techniques to improve attention speed on Hopper GPUs: exploiting the asynchrony of Tensor and TMA cores to overlap computation and data movement, interleaving block-wise matrix multiplication and softmax operations, and using incoherent processing to take advantage of hardware support for low-precision FP8 computations.

One of the standout features of FlashAttention-3 is its ability to exploit the asynchrony of the Tensor and TMA cores. This allows for overlapping overhead and data movement through warp specialization and interleaving operations. Warp specialization involves separate producer and consumer warps that handle TMA and WGMMA operations. FlashAttention-3 employs overlapping between and within warp groups of GEMM (general matrix multiplication) and softmax operations. This pingpong scheduling technique ensures that while one warp group performs GEMM operations, another can handle softmax calculations, thereby optimizing GPU resource utilization.

FlashAttention-3 makes significant use of low-precision FP8 computations, which double the performance of Tensor Core compared to FP16. This innovation increases computational speed and accuracy by reducing quantization error through incoherent processing. By applying the Hadamard transform with random signs to disperse outliers, FlashAttention-3 effectively reduces quantization error, making it a robust solution for high-performance LLMs.

FlashAttention-3 is 1.5-2x faster than FlashAttention-2 with FP16, achieving up to 740 TFLOPS, 75% of the theoretical peak FLOPs on H100 GPUs. With FP8, FlashAttention-3 achieves nearly 1.2 PFLOPS, a significant jump in performance with 2.6x lower error compared to the baseline FP8 attention.

These advancements are underpinned by the use of NVIDIA’s CUTLASS library, which provides powerful abstractions that enable FlashAttention-3 to take advantage of the capabilities of Hopper GPUs. By rewriting FlashAttention to incorporate these new features, Dao ai Lab has achieved significant efficiency gains, enabling new model capabilities such as extended context lengths and improved inference speeds.

In conclusion, the release of FlashAttention-3 represents a paradigm shift in the design and implementation of attention mechanisms in large language models. Dao ai Lab has demonstrated how targeted optimizations can yield significant performance improvements by closely aligning algorithmic innovations with hardware advancements. As the field continues to evolve, these advancements will be crucial to boosting what is possible with large language models and their applications across multiple domains.

Review the Blog, Paper, and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter.

Join our Telegram Channel and LinkedIn GrAbove!.

If you like our work, you will love our Newsletter..

Don't forget to join our Subreddit with over 46 billion users