Decentralized AI

“People say crypto AI is dead. I’m like, well, no, we’re just getting started. All the junk is gone — that’s supposed to happen… But I think the next version of crypto AI is going to solve distributed training. It’s going to solve distributed training in such ways that you would never see it coming from a million miles — it’s going to hit you hard.” — Greg Osuri, 00:25:22

“We, as a distributed training landscape, are maybe at a GPT-2 level in terms of our technological progress… that’s going to take some time and some resources and incentives. My big question is not the technology — it’s what is the incentive for someone to provide the compute to train a model… and I think that’s where we’re entering the crypto AI space, and where Akash fits in — the infrastructure layer for the entire thing.” — Greg Osuri, 00:14:39

“I think distributed training will be a thing. And of course we’re not going to replace the major labs doing their own thing, but with distillation, all these techniques these days, I don’t know what kind of mo[a]t the major big labs are going to continue having.” — Greg Osuri, 03:20:06

“I feel like 6 months to a year’s time we’re going to see distributed clusters and distributed training going to be a thing because the technology is not there yet. I mean, we’re kind of there, but not really there yet.” — Greg Osuri, 00:34:20

“On the bright side, distributed training actually is taking the center stage. We saw a new model by a subnet in Bittensor called Covenant, which did a 72 billion parameter model, which is not state-of-the-art but definitely better than what Llama 70 billion parameter model was… it’s not hard to see where you have a distributed grid and that grid contributing to a larger distributed network for AI as a possible solution.” — Greg Osuri, 02:14:33

“I’ve been talking to several labs — I don’t want to name who, they’re under NDAs — everybody’s looking at distributed training as a serious potential. If we figure out an incentive structure for distributed training — that’s where crypto has a phenomenal role — I don’t see how distributed training will [not] take main stage in a couple of years. And when that happens, bet you, you want GPUs in a farmland.” — Greg Osuri, 02:15:15

“Frankly, a 70 billion parameter model can be now trained on a fully distributed network and that’s a big deal… that 70B was trained by Meta in their massive data centers. You don’t need a Meta to produce a 70 billion parameter model… I’m very excited for decentralized training to succeed where your home computers can be leveraged, because your energy cost is going to be a bigger variable than anything.” — Greg Osuri, 00:42:10

“They trained a 72 billion parameter model which is better than Llama 70B… there are a lot of models that are trained in a fully distributed manner… that I believe will achieve SOTA in like two to three years. It’s not there yet.” — Greg Osuri, 00:26:49

“I was at ICML last year. I gave a talk about training and distributed training becoming a real thing, because if we have to rely on these companies to give us intelligence, then we lost as a society. And that message seems to resonate quite well with a lot of folks in AI.” — Greg Osuri, 00:35:29

“These moratoriums, these energy challenges, regulatory challenges are going to force distributed training. So I’m extremely bullish in terms of where this is going.” — Greg Osuri, 00:43:38

“If we crack heterogeneous compute, that means you can mix and match older GPUs and newer GPUs, then you have significantly more supply available… As that becomes real, I think you’re going to see decentralized network come to light in ways that was not obvious before.” — Greg Osuri, 00:34:15

“Today, training workloads are evolving to a point that can work on distributed networks, heterogeneous networks, highly fault tolerant. We’re seeing companies like Pluralis, for example, training fairly large or usable models on fully decentralized networks.” — Greg Osuri, 00:39:02