# Internet-scale training: 10B now, then 100B, then a trillion parameters

- **Speaker:** Greg Osuri
- **Attribution:** attributed
- **Said:** 2024-11-26
- **Theme:** Decentralized AI — https://www.gregosuri.com/predictions/decentralized-ai/
- **Source:** Greg Osuri - Akash: Decentralized Compute Marketplace - ep 158 (Zima Red)
- **Watch at:** 00:37:27 — https://www.youtube.com/watch?v=GwKZIY2aBe0&t=2247s
- **Canonical:** https://www.gregosuri.com/predictions/decentralized-ai/2024-11-26-internet-scale-training-10b-now-then-100b-then-a-t-zqd2/
- **Cite as:** Greg Osuri, "Internet-scale training: 10B now, then 100B, then a trillion parameters," Greg Osuri - Akash: Decentralized Compute Marketplace - ep 158 (Zima Red), 26 Nov 2024, 00:37:27. https://www.youtube.com/watch?v=GwKZIY2aBe0&t=2247s

## Quote — verbatim

> The first billion parameter model is being trained now on Nous — by the way, using Akash. First 10 billion parameter is being trained by Prime Intellect, all using Akash... Next model is a 100 billion parameter model, and after that you're going to see a trillion parameter model... With enough capital and with enough incentives, I don't see a reason why training over the internet can be a profitable venture.

## Context — site annotation, not spoken

Nous Research and Prime Intellect had just cracked heterogeneous, non-colocated training (DiLoCo/OpenDiLoCo); he notes GPT-4 is believed to be ~1.2T parameters as the target scale.