Language models prior to the appearance of tasks through sampling of grouped importance

Specialized language models (LMS) focus on a specific task or domain in which they often exceed the generalists of the same size. However, the specialized data necessary for these models to be made in excess are only available in a limited amount for most tasks. In this work, we build specialized models from large generalist training sets. We adjust the training distribution of generalist data with guidance of the specific domain data. We explore several approaches, with a sampling of grouped importance that stands out. This method groups the set of generalist data and the samples of these groups based on their frequencies in the smallest specialized data set. It is scalable, suitable for pretending and continues previously, works well in the configuration of multiple tasks. Our findings demonstrate improvements in different domains in terms of perplexity and precision of language modeling in multiple choice questions. We also present ablation studies that examine the impact of data sizes, group configurations and model sizes.