Intel Releases Low-Bit Quantized Open LLM Leaderboard to Evaluate Language Model Performance Across 10 Key Benchmarks

The domain of large language model (LLM) quantization has attracted attention due to its potential to make powerful ai technologies more accessible, especially in environments where computational resources are scarce. By reducing the computational load required to run these models, quantization ensures that advanced ai can be employed in a broader range of practical scenarios without sacrificing performance.

Large traditional models require substantial resources, which prevents their implementation in less equipped environments. Therefore, it is crucial to develop and refine quantization techniques, methods that compress models to require fewer computational resources without a significant loss of precision.

Various tools and benchmarks are used to evaluate the effectiveness of different quantification strategies in LLMs. These benchmarks cover a wide spectrum, including general knowledge and reasoning tasks in various fields. They evaluate models in zero-shot and few-shot scenarios, examining how well these quantized models perform on different types of cognitive and analytical tasks without extensive tuning or with minimal example-based learning, respectively.

Intel researchers presented the Low Bit Quantized Open LLM Leaderboard in Hug the face. This leaderboard provides a platform to compare the performance of various quantified models using a consistent and rigorous evaluation framework. Doing so allows researchers and developers to measure progress in the field more effectively and determine which quantification methods produce the best balance between efficiency and effectiveness.

The method used involves rigorous testing through Eleuther ai Language Model Evaluation Harness, which runs models through a battery of tasks designed to test various aspects of the model's performance. Tasks include understanding and generating human responses based on given prompts, problem solving in academic subjects such as mathematics and science, and discerning truths in complex question scenarios. Models are scored based on the accuracy and fidelity of their results compared to expected human responses.

Ten key benchmarks used to evaluate models in the Eleuther ai Language Model Evaluation Harness:

1. AI2 Reasoning Challenge (Shot 0): This set of elementary school science questions presents a challenge set of 2,590 “difficult” questions that both retrieval and co-occurrence methods generally fail to answer correctly.
2. AI2 Easy Reasoning (Trigger 0): This is a collection of easier science questions for primary school, with an Easy Set comprising 5197 questions.
3. HellaSwag (0 shots): It tests common sense inference, which is easy for humans (about 95% accurate) but challenging for state-of-the-art (SOTA) models.
4. MMLU (0 shots): It evaluates the multi-task accuracy of a text model on 57 diverse tasks, including elementary mathematics, US history, computer science, law, and more.
5. TrueQA (trigger 0): Measures a model's tendency to replicate falsehoods online. Technically it is a 6-shot task because each example begins with six pairs of questions and answers.
6. Winogrande (0 shots): A common sense adversarial reasoning challenge at scale, designed to be difficult for models to navigate.
7. FIGHT (shot 0): It focuses on common sense physical reasoning, evaluating models using a specific reference data set.
8. Lambada_Openai (0 shots): A data set that evaluates the text comprehension capabilities of computational models through a word prediction task.
9. OpenBookQA (0 shots): A question answering dataset that mimics open-book exams to assess human understanding of various topics.
10. BoolQ (trigger 0): A question answering task where each example consists of a short passage followed by a binary yes/no question.

In conclusion, these benchmarks collectively assess a wide range of reasoning skills and general knowledge in zero-shot and low-shot environments. The leaderboard results show a wide range of performance across different models and tasks. Models optimized for certain types of reasoning or specific areas of knowledge sometimes struggle with other cognitive tasks, highlighting the trade-offs inherent in current quantification techniques. For example, while some models may excel at narrative comprehension, they may underperform in data-intensive areas such as statistics or logical reasoning. These discrepancies are critical to guide the design of future models and improvements in training approaches.

Sources: