In recent years, language models have become one of the fastest growing fields in Artificial Intelligence. These models have been developed to process, produce and use natural language text to drive some creative and innovative AI applications. Language models are revolutionizing and ushering in a new era of AI expansion. The model developed by OpenAI called GPT-3, which has gained popularity recently, has extraordinary capabilities and shows great performance. It uses a transformative architecture to process text, resulting in a model that can effortlessly generate content and answer questions like a human would. Not only this, the model is even capable of summarizing long texts, completing codes and performing tasks with super good speed and accuracy.
Language models can operate without fail, thanks to the concept of learning in context by which they generalize to unseen tasks. However, learning in context (ICL) shows a slight limitation due to its sensitivity towards the selection of examples in context and the inability to take into account the interrelationship between examples in context. The new approach, called Composing Examples for Learning in Context or simply CEIL, formulates the process of choosing examples in context as a subset selection problem. It doesn’t rely on simple heuristics like the previous methods, but instead shows a lot of interaction between the input and the examples.
Contextual learning can be simply explained as learning in which the model learns something new and unique by looking at examples similar to what the model is trying to predict. This can be explained with the help of an example. While learning the addition of fractions in Mathematics, one learns it by first looking at examples involving the addition of fractions with the same denominator. The idea is to understand the patterns and rules to solve new and invisible problems. In terms of learning in context, for the model to understand and classify positive and negative sentences, it shows several examples and some context about the sentence, such as an app review or a tweet.
Since traditional methods use basic estimates and show suboptimal performance, CEIL is a better approach because it uses the concept of Determining Point Processes (DPP). It does this to model the interaction between the given input and the in-context examples. DPP is a probabilistic model that selects multiple subsets of items from a larger set. The determinants in DPP measure the volume of a subspace of a larger space traversed by a set of vectors. In CEIL, DPP has been used to choose various sets or subsets of examples to train a model. CEIL models all exemplar ensembles by learning their joint probability with a conditional DPP, followed by training to align with the language model score via contrastive loss.
The team behind Compositional Exemplars for In-context Learning (CEIL) has validated the approach on 12 classification data sets and generated 7 different natural language processing tasks. Data ranged from sentiment analysis and paraphrasing detection data to reasoning and response to open-ended questions. CEIL proved to be more efficient and effective than standard methods due to its transferability and compositionality. Consequently, the introduction of Composition Examples for Learning in Context (CEIL) appears to be a game changer in natural language processing.
review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 14k+ ML SubReddit, discord channel, and electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.
Tanya Malhotra is a final year student at the University of Petroleum and Power Studies, Dehradun, studying BTech in Computer Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a data science enthusiast with good analytical and critical thinking, along with a keen interest in acquiring new skills, leading groups, and managing work in an organized manner.
