AWS proposes a priming method that can transform pretrained Transformers into hybrid state-space architectures
The AWS team published the paper “Priming: Hybrid state space models from pre-trained Transformers,” proposing a priming method that reuses pretrained Transformer weights to build a hybrid architecture. It cuts pretraining costs by over 99%, with the entire process consuming less than 0.5% of the source model’s token budget.
Why it's recommended:The AWS team proposed a priming method that reduces the cost of pretraining a hybrid architecture from pretrained Transformers by over 99%, and open-sourced 8B and 32B models along with the code.