UK AISI tests show that filtering ESM-2 pretraining data can weaken targeted capabilities, but around $80 in compute can restore them
The UK's AI Security Institute Chem-Bio team pretrained ESM-2 protein language models with 8M to 3B parameters from scratch to test the effects of pretraining data filtering on specialized scientific models.
Why it's recommended:The original article provides experimental data showing that pretraining data filtering can selectively weaken protein model capabilities, but around $80 in compute can restore them, making complementary data access controls necessary.