Decentralized training is starting to work
Akash Accelerate '24: Official Livestream (Akash Network)
“I think we’re seeing a lot of progress in the decentralized training space… especially a lot of the research papers that are coming out, like DiLoCo and whatnot, that propose training on heterogenous, heavily distributed compute — so seems to be promising.” — 01:30:48
Context: Greg responding to Eric Voorhees’ warning that open-source AI’s future depends on solving efficient decentralized training before Meta stops releasing frontier models. (Captions garble the paper name; DiLoCo is the likely referent.)