We propose to use n-best reranking to improve sequence-level knowledge distillation (Kim and Rush, 2016), where we extract pseudo-labels for the training data of the learner model from the top-n best hypotheses and leverage a diverse set of models with different inductive biases, objective functions, or architectures, including some publicly available large language models, to choose the highest-quality hypotheses as labels. The effectiveness of our proposal is validated through experiments on the GermanEnglish and ChineseEnglish translation tasks of WMT'21. Our results demonstrate that using pseudo-labels generated by our n-best reranker leads to a significantly more accurate learner model. Indeed, our best learner model achieves comparable accuracy to a large translation model from (Tran et al., 2021) with 4.7 billion parameters, while having two orders of magnitude fewer parameters.
