Decentralized training of frontier models is not possible today
DePin, Scams & Decentralized ML (Chris Joannou)
“If you want to train advanced models, you need low latency, very minimal message-passing latency between two neurons. So if you have to do it in a decentralized way, that’s not possible… the trade-off is, yeah, you can get lower cost, but training time is going to be higher. So you’re never going to catch up to a GPT-4.” — 00:43:05
Context: Honest technical framing: communication cost must be lower than compute cost (00:34:30); fine-tuning and inference distribute fine, frontier training doesn’t — yet. He praises Gensyn for attacking this.