MIRAGE-Bench: An Automatic Multilingual Benchmark for Recovery Augmented Generation Systems

Large language models (LLMs) have emerged as crucial tools for handling complex information search queries due to techniques that improve both retrieval and response generation. Retrieval augmented generation (RAG) is a well-known framework in this area that has garnered much interest as it can produce responses that are more accurate and context-relevant. In RAG systems, an LLM creates a response based on the retrieved content after a retrieval step in which relevant information or passages are collected. By connecting comments to particular passages, this provision allows LLMs to cite sources, helping to minimize false information or hallucinations and making verification simpler and more reliable.

A well-known RAG system is Microsoft's Bing Search, which improves the reliability of responding to referred content by incorporating retrieval and grounding techniques for citing sources. However, due to uneven access to high-quality training data in languages ​​other than English, existing RAG models focus primarily on English, limiting their usefulness in multilingual environments. The effectiveness of LLMs in multilingual RAG environments, where both questions and answers are in languages ​​other than English, such as Hindi, is still unknown.

There are two main types of benchmarks used to evaluate RAG systems. Initial benchmarks, based on heuristics, evaluate models on a number of dimensions using a combination of computational measures. Despite being reasonably priced, these standards still rely on human tastes as a golden truth for comparison, and it can be difficult to determine a clear ranking between models.

The second type, known as field-based benchmarks, uses a high-performing LLM as a professor to evaluate model results through direct model comparisons in a competition-like environment. However, this method can be expensive and computationally demanding, especially when comparing a large number of models in depth, as is the case when evaluating 19 models using OpenAI's GPT-4o, which can be very expensive.

A team of researchers from the University of Waterloo and VECTARA propose a new framework called MIRAGE-BENCH to resolve the limitations of both approaches. It uses a cheaper method to analyze multilingual generation in 18 languages. This unique benchmark was created using a retrieval dataset known as MIRACL, which includes training-relevant Wikipedia sections as well as human-curated questions. MIRAGE-BENCH uses seven essential heuristic factors, including fluency, citation quality, and language detection, among others, to evaluate the caliber and applicability of responses produced by LLM. GPT-4o judges a smaller sample of multilingual queries in situations where more precise evaluations are required.

To function as a substitute judge, MIRAGE-BENCH also incorporates Machine Learning techniques by building a random forest model. To train this classification learning model, heuristic features and the Bradley-Terry model, a statistical technique frequently used in classification, are used. Without needing an expensive LLM judge each time, the trained machine can produce a synthetic league table to score multilingual LLMs. In addition to saving money, this procedure allows the leaderboard to adjust to new or modified evaluation standards. The team has shared that, based on experimental data, the MIRAGE-BENCH methodology regularly places large-scale models at the top and comes very close to expensive GPT-4o-based league tables, achieving a high correlation score.

Using data generated under the guidance of high-performance models such as GPT-4o, MIRAGE-BENCH has been shown to be advantageous for smaller LLMs, such as those with 7 to 8 billion parameters. The efficiency and scalability of multilingual RAG benchmarks eventually improve with this surrogate assessment methodology, opening the door to more comprehensive and inclusive assessments of LLMs in a variety of languages.

The team has shared their main contributions as follows.

1. The creation of MIRAGE-BENCH, which is a benchmark created especially to promote multilingual RAG research and help multilingual development.

1. A trainable learning-to-rank model has been used as a surrogate judge to combine heuristic-based measures with an arena-style leaderboard, successfully striking a balance between computational efficiency and accuracy.
2. The advantages and disadvantages of 19 multilingual LLMs have been discussed in terms of their generation capabilities in multilingual RAG.

look at the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 55,000ml.

(Next live webinar: October 29, 2024) Best platform to deliver optimized models: Predibase inference engine (promoted)