Revealing challenges in language model performance: A study of representation saturation and degeneration.
Language models (LMs) face challenges in self-supervised learning due to representation degeneration. LMs such as BERT or GPT-2 have low ...
Tag: saturation
