Meet LIMA: a new 65B parameter LLaMa model fine-tuned across 1000 prompts and carefully selected responses

Language models develop general-purpose representations transferable to almost any language generation or interpretation job by being pre-trained to anticipate the next token at a staggering scale. Therefore, different approaches to aligning language models have been proposed to facilitate this transfer, with a particular emphasis on instruction fitting over sizeable data sets with millions of examples and, more recently, reinforcement learning from human feedback. (RLHF) collected over millions of human interactions. annotators, for existing alignment techniques to work on the ChatGPT levels, large specialized data and computing resources are required.

However, they show that with a good language model already trained, very good performance can be obtained simply by tuning 1000 correctly chosen training instances. According to his hypothesis, alignment can be a quick and easy procedure in which the model learns the format or style of engaging users to reveal skills and information already learned during previous training. They collect 1,000 instances of resembling authentic user signals and great responses to verify this idea. They pick 750 of the best questions and answers from online discussion forums like Stack Exchange and wikiHow, and rate them for quality and variety.

They also manually compose 250 question and answer instances while emphasizing a consistent response style along the lines of an AI assistant and optimizing task diversity. Researchers from Meta AI, Carnegie Mellon University, the University of Southern California, and Tel Aviv University train LIMA, a 65B-parameter LLaMa model previously trained and improved on this collection of 1,000 examples. Three hundred difficult test questions compare LIMA with contemporary language models and products. LIMA outperforms OpenAI RLHF-trained DaVinci003, which was trained with RLHF, as well as an Alpaca replica with parameters 65B, which was entered into 52,000 samples, in a human preference study.

Although humans often prefer GPT-4, Claude, and Bard responses to LIMA responses, this is not always the case; LIMA consistently produces equivalent or preferable results in 43%, 46%, and 58% of situations, respectively. They repeat the annotations of human preferences using GPT-4 while the annotator confirms their findings. When LIMA’s responses are evaluated on an absolute scale, 88% meet the message requirements and 50% rate as outstanding. Ablation tests show significant improvements when data quality improves and yields drop significantly when the amount of data increases without simultaneously increasing the variety of requests.

Furthermore, they discover that LIMA can carry out coherent multi-turn speech despite having no dialogue examples. Including 30 handmade dialogue strings in the training can enhance this ability. Overall, these surprising results show the effectiveness of pretraining and its relative value over learning reinforcement approaches and large-scale adaptation of instruction. They demonstrate how a robust pretrained language model can be tuned to provide outstanding competitive results in various indications using 1000 well-selected samples. There are, however, drawbacks to this strategy.

The mental work required to create such instances is enormous and hard to scale up. Second, while LIMA typically provides strong responses, an unlucky sample during decoding or an aggressive prompt can often result in a weak response. LIMA is less resistant than product grade models. However, the data provided in this work show that difficult alignment problems can be easily addressed.

review the Preprint paper. Don’t forget to join our 22k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club