BRAG Launches: High-Performance SLMs (Small Language Models) Trained Specifically for RAG Tasks for Less Than $25 Each

BOAST is a series of high-performance Retrieval Augmented Generation (RAG) models developed by Maximalists ai Researcher. BRAG models are a family of small language models (SLMs) designed to offer high-performance, cost-effective alternatives in ai-powered language processing. These models have been trained at an impressively low cost of less than $25 each, positioning them as efficient and economical solutions in artificial intelligence.

BRAG models were created in response to the need for efficient, high-performance language models that do not require the extensive computational resources typically associated with large-scale models such as those from Nvidia and OpenAI. The primary motivation behind BRAG was to develop a set of models that could match or exceed the performance of leading models such as Cohere’s Command R+, Qwen2, Llama3.1, and Llama3 Instruct, while keeping training costs to a minimum.

The BRAG series includes four models:

1. BRAG-Qwen2-7b-v0.1
2. BRAG-Llama-3.1-8b-v0.1
3. BRAG-Llama-3-8b-v0.1
4. BRAG-Qwen2-1.5b-v0.1

These models are chosen based on their performance in open benchmarks and their ability to balance efficiency and capacity. The models underwent a two-stage fine-tuning process inspired by Nvidia’s ChatQA approach, involving initial training on general instruction datasets followed by RAG-specific datasets.

The BRAG models are particularly notable for their performance relative to their size. The 1.5B models offer an excellent balance between performance and efficiency. In comparison, the 7B and 8B models can handle more complex tasks such as understanding large contexts, interpreting tabular data, and mathematical reasoning. This strategic selection of models and training methodology allowed the maximalists to optimize performance while effectively managing costs.

Training of the BRAG model involved LoRA (low-rank adaptation) and QLoRA (quantized LoRA) techniques. LoRA enables faster training with reduced computational demands by simplifying the adaptation matrices. In contrast, QLoRA compresses the weight parameters to 4-bit precision, which significantly reduces memory usage and facilitates training on consumer-grade GPUs.

The models were evaluated using ChatRAG-Bench, a benchmark designed to assess conversational QA and RAG capabilities across various document types and question formats. Evaluation metrics included F1-Score and Exact Match Accuracy, which provided insight into the models’ ability to generate accurate and contextually relevant responses.

During the training process, several challenges were encountered including handling long documents, interpreting tabular data, and solving domain-specific queries. These issues were mitigated by careful selection of datasets and experimentation with various data combinations. For example, the inclusion of datasets such as DROP, Quoref, and SQuAD helped improve the models’ capabilities to handle complex and diverse data types. The F1 score metric, while widely accepted, was observed to have limitations in capturing semantic nuances and context. This highlighted the need for more holistic and context-aware evaluation metrics to better measure model performance.

In conclusion, Maximalists plan to improve BRAG models by enhancing RAG performance and tabular data handling and introducing citation generation for better interpretability. They also aim to refine query rewriting techniques to improve search accuracy and relevance. BRAG development was supported by credits from Modal Labs, which facilitated cost-effective experimentation. By leveraging innovative training techniques and strategic model selection, BRAG has proven that top-notch performance can be achieved with minimal resource expenditure, paving the way for more accessible and efficient ai solutions.

Review the Models and DetailsAll credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram Channel and LinkedIn GrAbove!. If you like our work, you will love our Newsletter..

Don't forget to join our Over 47,000 ML subscribers on Reddit

Find upcoming ai webinars here

x-300-.png” alt=””/>