Machine learning (ML) and artificial intelligence (ai) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industrial-scale machine learning applications (music streaming, video streaming, and mobile applications). Based on real-world data from an extensive search relevance annotation program, we show that errors can be predicted with moderate model performance (AUC = 0.65-0.75) and that the model performance generalizes well. across all applications (i.e., a global task-independent model). works in tandem with task-specific models). Unlike previous research, which has often focused on predicting annotation labels from specific task features, our model is trained to predict errors directly from a combination of task features and behavioral features derived from the annotation, in order to achieve a high grade. of generalization. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities significantly increases the number of annotation errors corrected (e.g., 40% efficiency gains for the music streaming application). These results highlight that behavioral error detection models can produce considerable improvements in the efficiency and quality of data annotation processes. Our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop machine learning.
