Distributed training over the internet just became real
Interview With Greg Osuri, Founder Of Akash Network (Secret Network)
“Google wrote a paper called DiLoCo last year, and a company called Prime Intellect implemented [it]… they were able to train a 10 billion parameter model over the internet… another company called Nous Research is going to do a 15 billion parameter model soon… there are five other companies doing decentralized training.” — 00:20:58
Context: The “two months ago” breakthrough that unlocks his whole home-data-center thesis by removing the need for concentrated GPU clusters.