Google DeepMind researchers present DiLoCo: a novel low-communication, distributed machine learning algorithm for efficient and resilient training on large language models

The growing capabilities of language models in real-world applications are often hampered by the intricate challenges associated with training them on a large scale using conventional methods such as standard backpropagation. Google DeepMind's latest breakthrough, DiLoCo (Distributed Low-Communication), sets a new precedent in language model optimization. On the paper “DiLoCo: Distributed training of low-communication language models”, the research team presents an innovative distributed optimization algorithm that revolutionizes training approaches by operating on groups of loosely connected devices, achieving a notable increase in performance and reducing communication by 500 times.

Inspired by the principles of federated learning, the researchers devised a variant of the widely recognized federated averaging algorithm (FedAvg), infusing it with elements similar to the FedOpt algorithm. DiLoCo strategically incorporates AdamW as an internal optimizer and leverages Nesterov Momentum as an external optimizer, creating an ingenious fusion that addresses challenges rooted in conventional training paradigms.

The brilliance of DiLoCo lies in its three fundamental pillars:

1. Limited co-location requirements: Each worker needs co-located devices, but the total number required is noticeably smaller, alleviating logistical complexities.

2. Reduced communication frequency: Workers no longer need to communicate at every step, but instead synchronize only at intervals of 𝐻 steps, significantly reducing communication overhead to only hundreds or even thousands.

3. Device Heterogeneity: While devices within a cluster must be homogeneous, DiLoCo allows different clusters to operate using diverse device types, offering unparalleled flexibility.

The DiLoCo training process involves replicating a pre-trained model 𝜃(0) multiple times. Each worker independently trains a replica of the model on its individual data chunk for 𝐻 steps. Subsequently, the workers average their external gradients and an external optimizer updates the copy of the global parameter 𝜃 (1), which is distributed to the workers. This cyclic process is repeated 𝑇 times, allowing the training of each replica at different global locations using multiple accelerators.

In practical experiments on the C4 data set, DiLoCo, which employs eight workers, achieves performance on par with fully synchronous optimization while reducing communication by a staggering 500 times. Additionally, DiLoCo demonstrates exceptional resilience to variations in data distribution among workers and adapts seamlessly to changing resource availability during training.

In essence, DiLoCo emerges as a robust and transformative solution for distributing transformative language model training across multiple poorly connected machines. This innovative approach not only overcomes infrastructure challenges but also exhibits unparalleled performance and adaptability, heralding a major breakthrough in language model optimization.

<!– ai CONTENT END 2 –>