| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |
Important
vLLM JAX is developed in vllm-project/tpu-inference.
This repository is the future home of the project. Until the move is complete, the code, issues, and pull requests live in tpu-inference. Please open issues and pull requests there.
vLLM JAX (vllm-jax, formerly tpu-inference) is the vLLM hardware plugin for serving JAX models on TPU. It aims to:
- Push the limits of TPU hardware performance in open source.
- Keep vLLM's standard user experience, telemetry, and interface.
PyTorch-native TPU serving is a separate project, vllm-torchtpu. If you are serving PyTorch models on TPU, use that instead.
The Recommended Models and Feature Matrix pages list the models and features validated through unit, integration, and performance testing. The full support matrix is in the tpu-inference README.
Get started with vLLM on TPUs by following the quickstart guide.
Visit our documentation to learn more.
Compatible TPU Generations
- Recommended: v7x, v5e, v6e
- Experimental: v3, v4, v5p
Contributions go to vllm-project/tpu-inference until the move is complete:
- 🐞 Submit Bugs & Suggest Features: Open a new issue.
- 👀 Provide Feedback on Pull Requests: Review open pull requests.
- 📚 Improve Our Documentation: Fix a typo, clarify a confusing section, or write a new recipe.
To contribute code, start with the Contributing Guide and the good first issues.
- For technical questions and feature requests, open a GitHub Issue in tpu-inference
- For discussing with fellow users, use the TPU support topic in the vLLM Forum
- For coordinating contributions and development, use the Developer Slack (#sig-tpu)
- For collaborations and partnerships, contact us at vllm-tpu@google.com
