# Distributed training is a few generations behind and catching up fast

- **Speaker:** Greg Osuri
- **Attribution:** attributed
- **Said:** 2024-11-25
- **Theme:** Decentralized AI — https://www.gregosuri.com/predictions/decentralized-ai/
- **Source:** Greg Osuri \| Trump's impact on crypto x AI, why DePIN is inevitable, and Akash Network revenue ATH's (Proof of Coverage Media)
- **Watch at:** 00:26:03 — https://www.youtube.com/watch?v=Rp_LL6tPDFU&t=1563s
- **Canonical:** https://www.gregosuri.com/predictions/decentralized-ai/2024-11-25-distributed-training-is-a-few-generations-behind-a-qwta/
- **Cite as:** Greg Osuri, "Distributed training is a few generations behind and catching up fast," Greg Osuri \| Trump's impact on crypto x AI, why DePIN is inevitable, and Akash Network revenue ATH's (Proof of Coverage Media), 25 Nov 2024, 00:26:03. https://www.youtube.com/watch?v=Rp_LL6tPDFU&t=1563s

## Quote — verbatim

> A 10 billion parameter model is somewhere in between GPT-2 and GPT-3... what I heard from the grapevine is we're going to have a 100 billion parameter model that's going to begin training in the next few months, even before the end of the year, and after that we're going to have a 500 billion and a trillion — GPT-4 is a trillion parameter model. So it's only a few generations we're away... and that's catching up really quick. The only limitation is the incentives, and crypto is phenomenal at devising incentives.

## Context — site annotation, not spoken

Citing Prime Intellect's OpenDiLoCo 10B run; quantified forecast for decentralized training scale.