A 15B-parameter LLM was trained over the public internet, matching centralized accuracy
Sponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future - Greg Osuri (PyTorch)
“A concrete decentralized training milestone is Nous Research trained a 15 billion parameter LLM over the public internet on volunteer machines and donated instances in about 10 days and about 11,000 steps, matching centralized run accuracy using WAN-aware optimizers. This demonstrates that with the right comms and async strategy, frontier class models can train beyond a single hyperscaler.” — 00:13:16
Context: Empirical case study; he also cites Gensyn’s verification protocol (46% overhead, 93% accuracy under 50% churn) as the trust layer for untrusted nodes.