The internet itself can be the training cluster
Sponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future - Greg Osuri (PyTorch)
“An alternative is thousands of nodes — small data centers, campuses, labs, and individual providers cooperating over the internet to train one model. That is not science fiction. We’ve seen precedent at internet scale: Folding@home volunteers once peaked around 2.4 exaflops, outpacing the top supercomputer at that time. With the right algorithms and verification, we can turn the internet itself into a capable, resilient training fabric for PyTorch jobs.” — 00:06:39
Context: After listing hyperscaler ceilings; he cites DiLoCo (~500x sync reduction) and Nous DisTrO (~857x bandwidth reduction) as the enabling breakthroughs.