vLLM releases a TorchTPU backend, enabling vLLM to run on Google Cloud TPUs
Introducing vLLM TorchTPU
AI BriefingThe vLLM team, Inferact, and Google jointly released vLLM TorchTPU (vllm-torchtpu), a new backend plugin for running vLLM on Google Cloud TPU. It reuses vLLM’s existing model code, allowing most existing models to be deployed without modifying their model definitions.
Also reported by 2 other sources
- RadixArk, Google, and Meta jointly open-source SGLang-TorchTPU, enabling SGLang to natively provide production-grade PyTorch inference on TPUsLMSYS Blog · 27 days ago
- Google released native TorchTPU inference engines for vLLM and SGLangGoogle Developers Blog · 27 days ago
Source: vLLM official blog · vllm.ai
Event follow-up1
View all event coverage- Google released native TorchTPU inference engines for vLLM and SGLangGoogle Developers Blog · 27 days ago