Hybrid pretraining scaling laws under data constraints
Scaling Laws for Mixture Pretraining Under Data Constraints
AI BriefingApple's machine learning research team systematically studied the trade-offs involved in mixed pretraining with scarce target data and general-purpose data through more than 2,000 language model training experiments. The study found that an excessively low proportion of target data leads to insufficient exposure to the target domain, while an excessively high proportion causes diminishing returns and even overfitting due to too many duplicate samples, providing quantitative guidance for mixed pretraining strategies under data constraints.
Source: Apple Machine Learning Research · machinelearning.apple.com