VS FLOPS parameters: Scale laws for optimal shortage for language mixture models

The scale of the ability of language models has proven consistently a reliable approach to improve performance and unlock new capabilities. The capacity can be defined mainly by two dimensions: the number of parameters of the model and the calculation for example. While the scale generally implies increasing both, the precise interaction between these factors and their combined contribution to the general capacity is not completely understood. We explore this relationship in the context of the dispersed mixture of experts (MOE), which allow to climb the number of parameters without proportionally increasing failures for example. We investigate how the level of scarcity varying, that is, the fraction of the inactive parameters, affects the performance of the model during the previous prison evaluation and of few shots. We find that under different restrictions (for example, parameter size and total training computation), there is an optimal level of shortage that improves both training efficiency and model performance. These results provide a better understanding of the impact of scarcity on scale laws for MOE and complement existing work in this area, offering ideas to design more efficient architectures.