Exposing Vulnerabilities in LLM Automated Benchmarks: The Need for Stronger Anti-Cheat Mechanisms

Automatic benchmarks such as AlpacaEval 2.0, Arena-Hard-Auto, and MTBench have gained popularity for evaluating LLM due to their affordability and scalability compared to human evaluation. These benchmarks use LLM-based automatic annotators, which align well with human preferences, to provide timely evaluations of new models. However, high success rates at these benchmarks can be manipulated by altering the duration or style of production, although measures have been developed to control for these factors. This raises concerns that adversaries may intentionally exploit these benchmarks to increase promotional impact and mislead performance evaluations.

Evaluating open text generation is challenging because a single correct result is needed. Human evaluation is reliable but expensive and time-consuming, so LLMs are often used as evaluators for tasks such as ai feedback, summarization, and hallucination detection. Recent benchmarks such as G-eval and AlpacaEval leverage LLMs to evaluate model performance efficiently. However, contradictory attacks are emerging on LLM-based assessments, which allow manipulation through irrelevant prompts or optimized sequences to skew results. While defenses such as fast writeback exist, adversaries continue to find ways to exploit these vulnerabilities, highlighting the need for more robust assessment methods.

Researchers from Sea ai Lab and Singapore Management University showed that even a “null model” that generates constant, irrelevant responses can manipulate automatic LLM benchmarks such as AlpacaEval 2.0, Arena-Hard-Auto and MT-Bench to achieve high win rates . By exploiting the weaknesses of automatic annotators, such as GPT-4, structured trap responses can achieve win rates of up to 86.5%. Although their study is a proof of concept, it shows the potential for adversaries to use LLMs to design undetectable cheating strategies to obtain unethical promotional benefits. This research emphasizes the urgent need for anti-cheating mechanisms to ensure the reliability of automatic LLM benchmarks.

The study presents a method to manipulate the automatic scorers used to evaluate LLM results. The approach involves two main cheating strategies: structured cheating responses and adversarial prefixes generated by random search. Structured trap responses are designed to align with the assessment criteria, taking advantage of the scoring templates used by automated scorers. Meanwhile, adversarial prefixes are strategically inserted at the beginning of responses to influence the scoring process. These techniques, tested on systems such as AlpacaEval 2.0, significantly increase success rates, demonstrating how evaluation mechanisms can be easily gamed and highlighting vulnerabilities in LLM benchmark systems.

Extensive ablation studies were performed on open source automated annotators, specifically Llama-3-Instruct models (parameters 8B, 70B). These models demonstrated human-level evaluation capabilities comparable to ChatGPT and GPT-4. The structured response technique had minimal impact on the Llama-3-8B model, but Llama-3-70B showed a stronger positional bias, especially in swapped configurations. Random search significantly increased win rates for both models, with Llama-3-8B increasing from 2.9% to 95.4% and Llama-3-70B from 0.4% to 95.1%, highlighting the effectiveness of the method to improve trap performance.

In conclusion, the study reveals that even “null models,” which consistently provide irrelevant answers, can exploit weaknesses in automatic LLM benchmarks and achieve high success rates, such as 86.5% in AlpacaEval 2.0. These benchmarks, including Arena-Hard-Auto and MT-Bench, are cost-effective for evaluating language models but susceptible to manipulation. The study emphasizes the need for stronger anti-cheating mechanisms to ensure the credibility of model evaluations. Future work should focus on automated methods to generate adversarial results and stronger defenses, as current strategies such as controlling the duration and style of results are insufficient.

look at the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 50,000ml

(Next Event: Oct 17, 202) RetrieveX – The GenAI Data Recovery Conference (Promoted)

Sana Hassan, a consulting intern at Marktechpost and a dual degree student at IIT Madras, is passionate about applying technology and artificial intelligence to address real-world challenges. With a strong interest in solving practical problems, he brings a new perspective to the intersection of ai and real-life solutions.