Dwarkesh Patel research: Pretraining progress comes mainly from data improvements
Pretraining progress is mostly coming from data
AI BriefingDwarkesh Patel published an experimental analysis training representative model recipes and data corpora from each year between 2019 and 2025 under compute budgets of up to 1e19 FLOPs. He found that data improvements produced a 12.0x increase in compute efficiency, while model improvements produced a 3.7x increase, making the contribution from data approximately 3.24 times that of models.
Source: Dwarkesh Patel: Podcast & Blog · dwarkesh.com
Event follow-up1
View all event coverage- Dwarkesh Patel research: Pretraining progress comes mainly from data improvementsDwarkesh Patel: Podcast & Blog · September 9