Google DeepMind Researchers Propose WARM: A Novel Approach to Tackle Reward Hacking in Large Language Models Using Weight-Averaged Reward Models

In recent times, large language models (LLMs) have gained popularity for their ability to answer user queries in a more human-like manner, which is achieved through reinforcement learning. However, aligning these LLMs with human preferences in reinforcement learning from human feedback (RLHF) can lead to a phenomenon known as reward hacking. This occurs when LLMs exploit flaws in the reward model (RM), achieving high rewards without meeting the underlying goals, as illustrated in Figure 1(b). Bounty hacking raises concerns such as performance degradation, checkpoint selection challenges, potential biases, and most importantly, security risks.

The main challenges identified when designing RM to mitigate reward piracy include changes in distribution and inconsistent preferences in the preference data set. Changes in the distribution arise due to policy drift during RL, leading to a drift of the offline preference data set. Inconsistent preferences arise from noisy binary labels, which introduces low agreement between labelers and affects the robustness of RM. To address these challenges, existing approaches have explored strategies such as KL regularization, active learning, and prediction ensemble (ENS). However, these methods face efficiency issues, reliability concerns, and struggle with preference inconsistencies.

To address these challenges, this article proposes W.eight TOaveraged reward METERModels (WARM) (illustrated in Figure 1(a)), a simple, efficient and scalable strategy to obtain a reliable and robust RM. WARM combines multiple RMs using linear interpolation in weight space, providing benefits such as efficiency, improved reliability in distribution changes, and increased robustness in labeling corruption. Diversity among fitted weights is a key factor for the effectiveness of WARM.

WARM is compared to Prediction Ensemble (ENS), demonstrating its efficiency and practicality by requiring a single model at inference time, eliminating memory and inference overhead. Empirical results indicate that WARM performs similarly to ENS in terms of variance reduction, but shows superiority under distribution changes. The paper presents the concept of linear mode connectivity (LMC) as a key factor in the success of WARM, demonstrating its ability to memorize less and generalize better than the prediction ensemble. There are 3 observations that are made in the experiments and tested empirically in Figures 3 and 4:

• Observation 1 (LMC): The accuracy of the interpolated model is at least as good as the interpolation of the individual accuracies.
• Observation 2 (WA and ENS): Weight averaging and prediction set work similarly.
• Observation 3 (WA and ENS): The accuracy gains of WA over ENS grow as the data move away from the training distribution.

The benefits of WARM extend beyond its primary objectives. It aligns with the updatable machine learning paradigm, enabling parallelization in federated learning scenarios. WARM could contribute to privacy and bias mitigation by reducing the memorization of private preferences. The method shows potential for combining RMs trained on different data sets, supporting iterative and evolving preferences. Further exploration includes extending WARM to direct preference optimization strategies.

Despite its innovation, WARM has limitations compared to prediction ensemble methods, including potential limitations in handling various architectures and uncertainty estimation. WARM does not completely eliminate spurious correlations or biases in preference data, suggesting the need for additional methods for a comprehensive solution. Lastly, WARM focuses on improving reward modeling and should be considered within the broader context of responsible ai to address safety risks from misalignment.

In conclusion, weight-averaged reward models (WARMs) offer a promising solution to challenges in reward modeling, improving alignment in RLHF. The paper's empirical results and theoretical insights position WARM as a valuable contribution toward creating more aligned, transparent, and effective ai systems.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook community, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our Telegram channel

<!– ai CONTENT END 2 –>