Distributed training becomes a thing in 6 months to a year
Akash Network - Decentralized Cloud Built for AI's Next Frontier (DePIN Connection)
“I feel like 6 months to a year’s time we’re going to see distributed clusters and distributed training going to be a thing because the technology is not there yet. I mean, we’re kind of there, but not really there yet.” — 00:34:20
Context: After walking through the three blockers (communication overhead, fault tolerance, heterogeneous GPUs) and praising Pluralis for solving two of them with a 7B distributed-trained model.