Distributed training becomes reality this year; state-of-the-art by ~2027
Solving the AI energy crisis | Greg Osuri on what it takes to power AI (Changelog)
“There is a lot of work that indicates distributed training will become a reality by end of the year, and by end of the year we will produce a model as good as GPT-3. By end of next year, or maybe going into 2027, there’s a good chance that we’ll be able to produce a state-of-the-art model fully trained distributed — or decentralized rather.” — 00:21:50
Context: After surveying DiLoCo (DeepMind), Prime Intellect’s 32B model, Nous Research’s DisTrO, Pluralis’s asynchronous swarm training, and Gensyn’s fault tolerance.