Microsoft AI researchers introduce advanced low -bit quantification techniques to allow efficient LLM implementation on edge devices without high computational costs

Edge devices such as smartphones, IoT devices and integrated systems process data locally, improving privacy, latency reduction and improvement of response capacity, and ai is quickly integrating into these devices. But, implementing large language models (LLM) on these devices is difficult and complex due to their high computational and memory demands.

LLMs are huge in power size and requirements. With billions of parameters, they demand a significant capacity of memory and processing that exceeds the capacities of most edge devices. While quantization techniques reduce the size of the model and energy consumption, conventional hardware is optimized for symmetrical calculations, which limits support for mixed precision arithmetic. This lack of native hardware support for low -bit calculations restricts the implementation on mobile and integrated platforms.

The above methods to execute the LLM devices on border use high -bit precision formats such as FP32 and FP16, which improve numerical stability but require significant memory and energy. Some approaches use lower bits quantization (eg, int8 or int4) to reduce resources demands, but compatibility problems arise with existing hardware. Another technique, dechantization, expands compressed models before calculation, but introduces latency and denies efficiency gains. In addition, traditional matrix multiplication (GEMM) requires uniform levels of precision, which causes performance optimization in different complex hardware architectures.

Microsoft researchers introduced a series of advances to allow efficient low -bit quantization for LLM devices in edge devices. Your approach includes three main innovations:

1. Staircase data type compiler
2. T-MAC MPGEMM Library
3. Hardware architecture of LUT Tensioner Tensor Core

These techniques aim to overcome hardware limitations facilitating the multiplication of general mixed precision (MPGEMM) and reducing computational overload. With these solutions, the researchers propose a practical framework that supports an efficient LLM inference without requiring specialized GPUs or high -power accelerators.

The first component of the staircase compiler joins the gap between the representations of the low BIT model and the hardware restrictions. Converts the data not supported into hardware compatible representations while the efficiency is maintained. This approach guarantees that modern deep learning architectures can use personalized data types without sacrificing performance.

The MPGEMM T-MAC Library optimizes mixed precision calculations using a method based on the search table (LUT) instead of traditional multiplication operations. This innovation eliminates the need for dechantization and significantly improves the computational efficiency of the CPU.

In addition, the hardware architecture of the LUT Tensioner nucleus presents a specialized accelerator designed for low -bit quantization. Take advantage of a set of optimized instructions to improve performance while reducing energy consumption.

In evaluations, the staircase data type compiler exceeds conventional deep neuronal (DNN) compilers for up to 14.6 times for specific low -bit calculations. When analyzed on edge devices such as the Surface 7 laptop with the Qualcomm Snapdragon x Elite chipset, the T-Mac Library achieved 48 tokens per second for model 3B Bitnet-B1.58, overcoming existing inferences libraries. In low -end devices, such as the Raspberry Pi 5, he achieved 11 tokens per second, which demonstrates significant efficiency improvements. Meanwhile, the hardware of the LUT tensor nucleus achieved an increase of 11.2 times in energy efficiency and an increase of 20.9 times in computational density.

Several key conclusions of Microsoft research include:

1. Low bits quantization reduces the size of the model, which allows efficient execution on edge devices.
2. The T-MAC Library improves the speed of inference by eliminating traditional multiplication operations.
3. The staircase compiler guarantees a perfect integration of the data of low -bits databases with existing hardware.
4. Optimized techniques reduce energy use, which causes LLMs to be feasible for low energy devices.
5. These methods allow the LLM to function effectively in a wide range of hardware, from high -end laptops to low -power IoT devices.
6. These innovations reach 48 tokens per second in Snapdragon x Elite, 30 tokens per second in flame 7b of 2 bits and 20 tokens per second in 4 bits 7b calls.
7. They also enable applications promoted by ai in mobile, robotic and integrated systems by making the LLMs more accessible.

In conclusion, the study highlights the importance of conscious hardware quantification techniques to implement LLMS on edge devices. The proposed solutions effectively address the long data challenges of memory consumption, computational efficiency and hardware compatibility. When implementing the core of Tensor Ladder, T-Mac and Lut Tensioner, researchers have raided the way for next-generation ai applications that are faster, more efficient in energy and more scalable on several platforms.

Verify he Details and Paper. All credit for this investigation goes to the researchers of this project. Besides, don't forget to follow us <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter and join our Telegram channel and LINKEDIN GRsplash. Do not forget to join our 75K+ ml of submen.

Recommended open source ai platform: “Intellagent is a multiple open source agent frame to evaluate the complex conversational system” (promoted)