NEWSLETTER
Large language models (LLM) have become an indispensable part of contemporary life, shaping the future of almost all conceivable domains. They are widely recognized for their impressive performance in all the tasks of variable complexity. However, the instances have arisen where the LLMs have been criticized for generating unexpected and insecure responses. Consequently, the ongoing research aims to align LLM more closely with human preferences while taking advantage of its extensive training data.
Methods such as reinforcement learning human feedback (RLHF) and direct preference optimization (DPO) have proven effective. However, they still require iterative training, which is often not practical. Therefore, researchers focus on modifying inference approaches to coincide with the performance of training -based optimization methods. This article explores the latest research that improves the alignment of human preferences during inference time.
Shanghai ai Laboratory researchers have introduced the optimization of trial time preferences (TPO), a new framework designed to align LLM's results with human preferences during inference. This framework can be conceptualized as an online online learning paradigm, where the policy model continually interacts with a new reward model to refine its results.
TPO incorporates a mechanism to take advantage of interpretable textual feedback for the optimization of preferences instead of conventional numerical score. To achieve this, the authors translate reward signals into textual rewards through criticism. Then, the model generates suggestions of the transformed rewards and updates its outputs to align with the signals during the tests.
During the real test, the newly generated responses are described in each step optimization step, and the ends of the response are classified as “chosen” or “rejected” outputs. Then, the model learns the strength of the best or “chosen” exits and the deficiencies of the rejected responses to compile a “textual loss.” The model then generates suggestions or “textual gradients” for the next iteration. TPO thus improves the iterative output based on interactions with text rewards.
The authors used policy models aligned and not aligned to validate the concept and determine if the model had suffered an optimization of preferences during training. Two key models included in the study were called-3.1-70b-Sft, a non-aligned model that did not suffer an optimization of preferences during training, and call-3.1-70b-instrust, a model aligned trained with preferences optimization. In addition, the experiments covered many data sets to evaluate the following instruction, alignment of preferences, security and mathematical reasoning.
The results of these experiments confirmed that some TPO optimization steps significantly improved the performance in aligned and unaligned models. When comparing the TPO-based inference optimization with traditional training optimization approaches, the researchers found that the call-3.1-70b-Sft-sfto-sft aligned model exceeded its lineup setback of flame setbacks Tpo times. In addition, the application of TPO to a aligned model with only 22 billion parameters achieved an LC score of 53.4% and a 72.2% WR score
Conclusion: The research team introduced TPO, an online learning frame in politics to align LLM's results by human preference. This framework optimized the responses in the inference time and eliminated the discomfort of resentment and weight updates. In addition, Tpo offered high scalability and flexibility, so it is a promising approach to future works of LLM.
Verify he Paper and Girub. All credit for this investigation goes to the researchers of this project. Besides, don't forget to follow us <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter and join our Telegram channel and LINKEDIN GRsplash. Do not forget to join our 70k+ ml of submen.
<a target="_blank" href="https://nebius.com/blog/posts/studio-embeddings-vision-and-language-models?utm_medium=newsletter&utm_source=marktechpost&utm_campaign=embedding-post-ai-studio” target=”_blank” rel=”noreferrer noopener”> (Recommended Read) Nebius ai Studio expands with vision models, new language models, inlays and Lora (Promoted)
ADEEBA Alam Ansari is looking for its double title at the Indian Institute of technology (IIT) Kharagpur, winning a B.tech in Industrial Engineering and an M.tech in Financial Engineering. With great interest in automatic learning and artificial intelligence, it is an avid reader and an inquisitive person. Adeeba firmly believes in the power of technology to empower society and promote well -being through innovative solutions driven by empathy and a deep understanding of real world challenges.
