OPEN-RAG: a new AI framework designed to improve reasoning capabilities in RAG with open source LLM

Large language models (LLMs) have made enormous progress in various natural language processing (NLP) tasks, but they often suffer from factual inaccuracies, particularly in complex reasoning scenarios involving multi-hop queries. Current retrieval-augmented generation (RAG) techniques, especially those using open source models, struggle to handle the complexity of reasoning about the retrieved information. These challenges lead to noisy results, inconsistent context, and difficulties distinguishing relevant data from distractors.

Researchers from Bangladesh University of Engineering and technology, University of North Texas, York University, Canada, Salesforce Research, Qatar Computing Research Institute (QCRI), Fatima Al-Fihri Predoctoral Fellowship, and the community Cohere For ai present Open-RAG, a novel framework that improves the reasoning capabilities of augmented recovery generation models using open source LLM. Open-RAG transforms a dense LLM into a parameter-efficient sparse mixture of experts (MoE) model, capable of handling complex reasoning tasks, including single-hop and multi-hop queries. By dynamically selecting relevant experts, the model effectively addresses distractors that appear relevant but are misleading. Open-RAG also incorporates a hybrid adaptive retrieval method that helps decide when to retrieve information, balancing performance gains and inference speed.

Structurally, Open-RAG integrates constructive learning, architectural transformation, and reflection-based generation into a cohesive framework. It transforms a dense LLM into a sparse MoE model that combines selective expert activation with parameter efficiency. The framework trains the model not only for direct task performance but also for navigating and contrasting between useful information and distractors. This approach employs reflection tokens, which help monitor the retrieval process and evaluate the relevance and support of the retrieved information. Open-RAG's hybrid adaptive retrieval system also leverages these reflection tokens to decide whether retrieval is necessary at any given time, thereby improving overall efficiency and response accuracy.

Experimental results show that Open-RAG based on Llama2-7B outperforms several state-of-the-art RAG models, such as ChatGPT-RAG, Self-RAG, and Command R+. On several knowledge-intensive tasks, Open-RAG demonstrated superior reasoning capabilities and factual accuracy compared to these proprietary models. For example, it outperformed ChatGPT-RAG on HotpotQA and MuSiQue datasets, which involve complex multi-hop questions. The hybrid adaptive retrieval method was also effective in balancing retrieval frequency and improving overall response quality. Additionally, Open-RAG's ability to selectively activate experts based on query complexity ensures that the computational load remains manageable without sacrificing performance.

Conclusion

In conclusion, Open-RAG represents an important step forward in improving the factual accuracy and reasoning capabilities of RAG models with open source LLM. By combining a parameter-efficient MoE architecture with hybrid adaptive recovery, Open-RAG delivers improved performance on complex reasoning tasks while remaining competitive with state-of-the-art proprietary models. This work not only highlights the potential of open source LLMs to achieve high accuracy and efficiency, but also lays the foundation for future improvements, such as focusing on the performance of long-form generation tasks and further optimizing the model architecture. .

look at the Paper and Project. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 50,000 ml.

(Next Event: October 17, 2024) RetrieveX – The GenAI Data Recovery Conference (Promoted)