Distributed training is a few generations behind and catching up fast
Greg Osuri | Trump's impact on crypto x AI, why DePIN is inevitable, and Akash Network revenue ATH's (Proof of Coverage Media)
“A 10 billion parameter model is somewhere in between GPT-2 and GPT-3… what I heard from the grapevine is we’re going to have a 100 billion parameter model that’s going to begin training in the next few months, even before the end of the year, and after that we’re going to have a 500 billion and a trillion — GPT-4 is a trillion parameter model. So it’s only a few generations we’re away… and that’s catching up really quick. The only limitation is the incentives, and crypto is phenomenal at devising incentives.” — 00:26:03
Context: Citing Prime Intellect’s OpenDiLoCo 10B run; quantified forecast for decentralized training scale.