The goal of aligning language models with human preferences requires data that reveals these preferences. Ideally, time and money can be spent carefully collecting and tailoring personalized preference data for each subsequent application. However, in practice, a few publicly available preference data sets are often used to train reward models for reinforcement learning from human feedback (RLHF). While new preference data sets are being introduced with increasing frequency, there are currently no efforts to measure and compare these data sets. In this paper, we systematically study preference data sets through three perspectives: scale, label noise, and information content. We propose specific metrics for each of these perspectives and discover different axes of comparison for a better understanding of preference data sets. Our work is a first step towards a data-centric alignment approach by providing insights that aid in the efficiency of training and iterative data collection for RLHF.
