Learn about RECITE: a new paradigm to help LLMs generate more accurate factual knowledge without retrieving it from an external corpus

Learning in context is a natural language paradigm that demonstrates the ability of pretrained models to capture new behavior using only a small number of example cues as input. The latest research indicates that Extended Language Models (LLMs), such as GPT-3 and the latest fad, ChatGPT, can achieve outstanding performance when it comes to contextual learning in knowledge-intensive NLP tasks. For example, LLMs have successfully demonstrated their ability to answer arbitrary factual queries with respect to open domain question answering, which essentially refers to generating answers to arbitrary context-free questions. Researchers have found that increased recall can be highly beneficial for knowledge-intensive activities, which can further improve the performance of LLMs. LLMs perform augmentation of retrieval by extracting relevant documents from an external corpus.

However, in recent years, researchers have questioned whether LLMs are capable of producing factual data that is more accurate without the aid of augmented recall generation. A team of researchers from Google Brain and CMU has done some groundbreaking research paper that illustrates exactly this! The team has come up with an entirely new approach called RECITation-augmented gEneration (RECITE), in which, for a given input, RECITE first uses sampling to recall one or more relevant passages from the LLMs’ own memories before generating the final results. . RECITE’s innovative recite-answer approach has demonstrated cutting-edge performance in a variety of knowledge-intensive NLP tasks, including Closed Book Question Answering (CBQA). The team’s research paper was also published at the prestigious ICLR 2023 conference.

The paradigm presented by the Google Brain researchers is based on dividing intensive work on original knowledge into two subtasks: task execution and knowledge recitation. Recitation can be considered as an intermediate knowledge retrieval process, while task execution is the final phase in which final results are generated. The researchers noted that while few-shot prompts can help LLMs perform specific NLP tasks, these tasks are generally not formatted similarly to the original pre-training causal language modeling goal. This often makes it difficult for LLMs to accurately recall information from memory. As a result, this observation gave the researchers the idea to use an additional knowledge recitation step. The knowledge recitation stage was included to simulate the language modeling pre-training task, which ultimately improved the LLMs’ ability to generate factual information.

The researchers’ ultimate goal was to simulate a human’s ability to recall relevant factoids before responding to knowledge-intensive queries. The team tested and refined their recite-response scheme for short-take closed book question answering (CBQA) tasks. These tasks consist of two parts: the reciting evidence module, which requires you to read pertinent passages, and the question and answer module, which asks you to find answers based on the evidence you just recited. The researchers presented a prompt-based recitation learning system that utilizes the LLM’s ability to learn in context. Paired sample questions and recited evidence were provided to assist LLMs in learning such instances in context for reciting the question.

The researchers ran extensive tests on four pretrained models (PaLM, UL2, OPT, and Codex) and three CBQA tasks (Natural Questions, TriviaQA, and HotpotQA) to evaluate their RECITE paradigm. It was found that by using different language models pre-trained with the suggested recite and answer technique, the performance of CBQA on the Natural Questions and TriviaQA data sets could be greatly improved. The researchers also made an interesting observation that while the performance gains in NQ were more consistent across various language models, the reciting and responding improvements in TriviaQA were more significant across smaller language models. The probable cause for this could be that Trivia-style questions often include more contextual information, which lessens the recitation impact for powerful LLMs like PaLM.

Even if the method developed by Google Brain Researchers is impressive, there is still work to be done. To update time-sensitive information, a purely LLM-based solution currently requires training or tuning LLMs on the new corpus, which can be quite computationally expensive. The researchers want to work on this front in the near future. In addition, in line with their future plans, the researchers also plan to validate the effectiveness of augmented recitation generation for additional knowledge-intensive NLP tasks in the closed-book context, such as fact checking.

review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 15k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.