Distributed training will be a thing; big labs' moat in doubt
LIVE from NEARCON Day 1 In SF (The Rollup)
“I think distributed training will be a thing. And of course we’re not going to replace the major labs doing their own thing, but with distillation, all these techniques these days, I don’t know what kind of mo[a]t the major big labs are going to continue having.” — 03:20:06
Context: Models getting smaller and smarter make distributed training viable; cites Peter Steinberger’s viral agent product as proof that individuals without funding can now build history-making products.