Stanford researchers present contrastive preference learning (CPL): a new machine learning framework for RLHF that uses the regret preference model

Aligning models with human preferences poses significant challenges in ai research, particularly in sequential and high-dimensional decision-making tasks. Traditional reinforcement learning from human feedback (RLHF) methods require learning a reward function from human feedback and then optimizing this reward using RL algorithms. This two-phase approach is computationally complex and often results in high variability in policy gradients and instability in dynamic programming, making it impractical for many real-world applications. Addressing these challenges is essential for advancing ai technologies, especially for fine-tuning large language models and improving robotic policies.

Current RLHF methods, such as those used to train large language models and image generation models, typically learn a reward function from human feedback and then use RL algorithms to optimize this function. While effective, these methods rely on the assumption that human preferences directly correlate with rewards. Recent research suggests that this assumption is flawed and leads to inefficient learning processes. Furthermore, RLHF methods face significant optimization challenges, including high variance in policy gradients and instability in dynamic programming, which restrict their applicability to simplified settings such as contextual bandits or low-dimensional state spaces.

A team of researchers from Stanford University, the University of Texas at Austin, and the University of Massachusetts at Amherst introduces contrastive preference learning (CPL), a new algorithm that optimizes behavior directly from human feedback using a regret-based human preference model. CPL bypasses the need to learn a reward function and subsequent optimization by learning by iteration by leveraging the maximum entropy principle. This approach simplifies the process by directly learning the optimal policy via a contrastive objective, making it applicable to high-dimensional and sequential decision-making problems. This innovation offers a more scalable and computationally efficient solution compared to traditional RLHF methods, expanding the scope of tasks that can be effectively addressed using human feedback.

CPL is based on the maximum entropy principle, which leads to a bijection between advantage functions and policies. By focusing on policy optimization rather than advantages, CPL uses a simple contrastive objective to learn from human preferences. The algorithm operates in a policy-agnostic manner, allowing it to use arbitrary Markov decision processes (MDPs) and handle high-dimensional state and action spaces. Technical details include the use of a regret-based preference model, where human preferences are assumed to follow regret under the user's optimal policy. This model is integrated with a contrastive learning objective, allowing for straightforward policy optimization without the computational overhead of RL.

The evaluation demonstrates the effectiveness of CPL in learning policies from sequential and high-dimensional data. CPL not only matches, but often outperforms, traditional RL-based methods. For example, on several tasks such as Bin Picking and Drawer Opening, CPL achieved higher success rates compared to methods such as Supervised Fine-Tuning (SFT) and Preference-based Implicit Q-learning (P-IQL). CPL also showed significant improvements in computational efficiency, being 1.6x faster and four times more parameter-efficient compared to P-IQL. Moreover, CPL demonstrated robust performance on different types of preference data, including both dense and sparse comparisons, and effectively utilized high-dimensional image observations, further underscoring its scalability and applicability to complex tasks.

In conclusion, CPL represents a significant advancement in learning from human feedback, addressing the limitations of traditional RLHF methods. By directly optimizing policies through a contrastive objective based on a regret preference model, CPL offers a more efficient and scalable solution to align models with human preferences. This approach is particularly impactful for sequential and high-dimensional tasks, demonstrating improved performance and lower computational complexity. These contributions are poised to influence the future of ai research, providing a robust framework for human-aligned learning across a wide range of applications.

Review the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram Channel and LinkedIn GrAbove!. If you like our work, you will love our Newsletter..

Don't forget to join our Over 47,000 ML subscribers on Reddit

Find upcoming ai webinars here

Aswin AK is a Consulting Intern at MarkTechPost. He is pursuing his dual degree from Indian Institute of technology, Kharagpur. He is passionate about Data Science and Machine Learning and has a strong academic background and hands-on experience in solving real-world interdisciplinary challenges.