Trending
Hot topicsCalmed down

Dwarkesh Patel publishes a study breaking down progress in pretraining

1 Articles1 Reporting sourcesUpdated September 9

Learn about this first

Dwarkesh Patel published an experimental analysis training representative model recipes and data corpora from each year between 2019 and 2025 under compute budgets of up to 1e19 FLOPs. He found that data improvements produced a 12.0x increase in compute efficiency, while model improvements produced a 3.7x increase, making the contribution from data approximately 3.24 times that of models.

Latest developments

Dwarkesh Patel research: Pretraining progress comes mainly from data improvements

Event progress1

  1. Dwarkesh Patel publishes a study breaking down progress in pretraining

    Dwarkesh Patel: Podcast & Blog: Dwarkesh Patel research: Improvements in data account for most progress in pretraining

Finished reading the news? Let’s talk about your business.

Leave your contact details, and I’ll get in touch with you personally.

WECHAT

Add me on WeChat

Press and hold or select the WeChat ID below to copy it.