Learn about this first
Dwarkesh Patel published an experimental analysis training representative model recipes and data corpora from each year between 2019 and 2025 under compute budgets of up to 1e19 FLOPs. He found that data improvements produced a 12.0x increase in compute efficiency, while model improvements produced a 3.7x increase, making the contribution from data approximately 3.24 times that of models.
Latest developments
Dwarkesh Patel research: Pretraining progress comes mainly from data improvements
Event progress1
Dwarkesh Patel publishes a study breaking down progress in pretraining
Dwarkesh Patel: Podcast & Blog: Dwarkesh Patel research: Improvements in data account for most progress in pretraining