Repository navigation
docs: add tip for FlashInfer JIT cache install for vLLM model server - #1405
Merged
ananthsub merged 2 commits intoMay 28, 2026
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
ananthsub
marked this pull request as ready for review
May 26, 2026 14:10
ananthsub
added
documentation
cmunley1
approved these changes
May 28, 2026
| ``` | ||
|
|
||
| <Tip> | ||
| **Recommended on workstations without a system CUDA toolkit** (i.e. no `/usr/local/cuda`). Install FlashInfer's pre-built kernel packages so vLLM's sampler and allreduce paths do not try to JIT-compile via `nvcc` at startup. Both companion packages enforce strict version equality with the `flashinfer-python` that vLLM pulled in, so pin them to its exact version: |
Contributor
There was a problem hiding this comment.
Maybe we could recommend a container?
Also feel this could go to bottom of readme if most people won't need it, but then people might miss it, so im not sure.
Contributor
Author
There was a problem hiding this comment.
i dont want to recommend the RL container right now since it's hard to tell which gym commit/version is currently used in the container. over time i agree we could recommend a container or offer a dockerfile to make it easier to prefetch these dependencies
vLLM's sampler and allreduce paths use FlashInfer, which falls back to runtime JIT compilation via nvcc when no pre-built kernel cache is installed. On a fresh `uv pip install vllm --torch-backend=auto` setup without a system CUDA toolkit at `/usr/local/cuda`, this fails with: RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist because PyPI's PyTorch wheels ship the CUDA runtime libs but not the nvcc compiler. Add a Tip block in the vLLM model server page explaining the symptom and pointing users at the recommended fix: install flashinfer-cubin and flashinfer-jit-cache (CUDA-version-matched) so FlashInfer skips JIT entirely. Reported in NVBug 6192981. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
QA reported that the previous unpinned `uv pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu130` resolves to the newest wheel on the index (e.g., 0.6.11.post3+cu130), which then fails the strict version-equality check in flashinfer/jit/env.py against vLLM 0.21.0's pinned flashinfer-python==0.6.8.post1: RuntimeError: flashinfer-jit-cache version (0.6.11.post3+cu130) does not match flashinfer version (0.6.8.post1) Update the vLLM model server page to: - discover the installed flashinfer-python version (FI_VER) and torch CUDA tag (TORCH_CU) at install time - pin both flashinfer-cubin and flashinfer-jit-cache to ${FI_VER} - include the matching `--index-url https://flashinfer.ai/whl/${TORCH_CU}` for the JIT cache wheel Add a second example error block so users hitting this exact mismatch can self-diagnose. Validated end-to-end in /home/ansubramania/dev/Gym-vllm-flashinfer-docs: - Reproduced the same FI_VER=0.6.8.post1 / TORCH_CU=cu129 vLLM 0.21.0 environment. - Confirmed the pinned wheels resolve (PyPI 200 for cubin; cu129 index lists the matching jit-cache wheel). - With pinned companions installed, FLASHINFER_DISABLE_JIT=1, CUDA_HOME unset, and nvcc removed from PATH, flashinfer.sampling.top_k_top_p_sampling_from_logits (the exact API in Phuoc's traceback) returns sampled token ids successfully. - Negative control: uninstall flashinfer-jit-cache and re-run the same call; it raises MissingJITCacheError, confirming the AOT cache is what fixes the failure. Reported in NVBug 6192981. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/vllm-flashinfer-jit-docs
branch
from
May 28, 2026 22:51
a29211c to
89730b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vLLM's sampler and allreduce paths use FlashInfer, which falls back to runtime JIT compilation via nvcc when no pre-built kernel cache is installed. On a fresh
uv pip install vllm --torch-backend=autosetup without a system CUDA toolkit at/usr/local/cuda, this fails with:because PyPI's PyTorch wheels ship the CUDA runtime libs but not the nvcc compiler. Add a Tip block in the vLLM model server page to point users to install flashinfer-cubin and flashinfer-jit-cache so FlashInfer skips JIT entirely.