Learn about this first
Google has released a TPU fork of TorchTitan, hosted at github.com/google-pytorch/torchtitan. Adapted from Meta’s TorchTitan implementation, the fork enables PyTorch models to train on Google Cloud TPUs. Alongside the native TorchTPU backend, it provides SimpleFSDP, Graph Trainer, Pallas kernels, and an optimization playbook.
Google says that, for most models, achieving good MFU on TPUs mainly involves a series of configuration changes to an existing recipe, rather than writing new code.
AI-generated from reports · updated 27 days agoLatest developments
Google Releases TorchTitan TPU Fork and Provides a Model Optimization Playbook
Event progress1
Introducing TorchTitan on TPU integration and optimization methods
Google Developers Blog: Google releases a TPU fork of TorchTitan and a model optimization playbook