Decentralized training research is promising
Akash Accelerate '24: Official Recap (Akash Network)
“I think we’ve been seeing a lot of progress in the decentralized training space. I’m very, very positive, especially a lot of the research papers that are coming out, like DiLoCo and whatnot, that propose training on heterogeneous, heavily distributed compute — seems to be promising.” — 01:08:14
Context: Greg responding to Eric Voorhees’ warning that if decentralized training isn’t solved before Meta stops releasing open models, “we have a possibly very dystopian future.”