Distributed training reaches state of the art in 2-3 years
The Infrastructure Behind Agentic Finance, with Shashank Yadav and Greg Osuri of Akash Network (Fraction AI)
“They trained a 72 billion parameter model which is better than Llama 70B… there are a lot of models that are trained in a fully distributed manner… that I believe will achieve SOTA in like two to three years. It’s not there yet.” — 00:26:49
Context: Citing frameworks from Pluralis, Gensyn, and Prime Intellect’s OpenDiLoCo, plus a Bittensor subnet’s 72B model, against the 3.2 Tbps NVLink bandwidth objection; he also mentions running Llama on his own home GPU. (Caption garbles “SOTA” as “sort.”)