UK AISI tests show that filtering ESM-2 pretraining data can weaken targeted capabilities, but around $80 in compute can restore them
Pretraining data filtering in open-weight biological models
AI BriefingThe UK's AI Security Institute Chem-Bio team pretrained ESM-2 protein language models with 8M to 3B parameters from scratch to test the effects of pretraining data filtering on specialized scientific models.
Source: UK AI Security Institute: Blog · aisi.gov.uk
Event follow-up1
View all event coverage- UK AISI tests show that filtering ESM-2 pretraining data can weaken targeted capabilities, but around $80 in compute can restore themUK AI Security Institute: Blog · 21 days ago