Learn about this first
Google has released native PyTorch backends for vLLM and SGLang based on TorchTPU, enabling developers to deploy models on Google Cloud TPU using PyTorch tools. The vLLM backend was developed in collaboration between Google and Inferact. The SGLang backend was open-sourced by RadixArk’s SGLang team in collaboration with Google and Meta. Google says both engines are running at scale in preview customers’ production environments.
The vllm-torchtpu plugin for vLLM reuses vLLM’s existing model code, allowing most existing models to be deployed without modifying their model definitions. It requires Python 3.12 on a Cloud TPU VM. The recommended hardware is TPU v7x (Ironwood); v6e, v5e, and v5p are still experimentally supported. SGLang-TorchTPU lets the same SGLang engine provide production-grade serving on both GPUs and TPUs. Its Miles TPU training is in preview, and the team aims to provide Day 0 TPU support for new open-source models by Q1 2027.
Both projects offer free TPU compute pools for contributors: vLLM says its pool is the first community-contributed TPU capacity pool, while RadixArk opened the pool for SGLang. vLLM also says Inferact’s core maintainers will help with design and PR reviews. Free compute is available by application only to developers actively contributing to these engines. MiniMax later thanked the SGLang team for bringing the H3 model to TPU and said the backend can deploy open-source models including MiniMax H3, GLM-5.3 Flash, and Kimi K3. Ant Ling said Ling 3.1 Flash and the Ling 3.0 family are also among the first models supported.
AI-generated from reports · updated 26 days agoLatest developments
SGLang-TorchTPU Released; Open-Source Models Including Ling 3.1 Flash Among the First Supported
Event progress2
MiniMax Thanks SGLang-TorchTPU for Supporting the H3 Model
MiniMax: Thanks to SGLang-TorchTPU for Bringing the H3 Model to TPUGoogle Releases Native PyTorch Backends for vLLM and SGLang on TPU
Google Developers Blog: Google Releases Native TorchTPU Inference Engines for vLLM and SGLang