Distributed training just broke the co-location requirement
Ep. 654 The Intersection of AI and Cryptocurrency with Akash Network (CRYPTO 101)
“There’s enormous amount of research… Google really released a paper called [DiLoCo] and a bunch of companies actually implemented this paper and actually proved we can train a 32 billion parameter model fully distributed… you can have a cluster in New York… Singapore… San Francisco and actually contribute to a training run… That was not possible a few years ago.” — 00:26:13
Context: He dates the breakthrough to “about six months ago” (00:25:21) and says if Nous Research’s DisTrO approach proves out at 100B parameters “that will turn a lot of heads” (00:27:07).