Repository navigation
[Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance - #51222
Conversation
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Fixes a crash in the V1 GPU Model Runner V2 path for encoder-only EPD instances by avoiding an unnecessary multimodal embedding gather that can raise Encoder cache miss when the EC connector already has the item. This keeps the encoder instance alive and preserves the scheduler optimization that skips redundant re-encoding.
Changes:
- Add
ModelState.encode_mm_inputs(...)as an “encode-only” helper (no gather) and refactorDefaultModelState.get_mm_embeddings(...)to use it. - Update
ModelRunner.execute_model(...)to callencode_mm_inputs(...)and skipget_mm_embeddings(...)foris_encoder_onlyinstances. - Add regression tests covering (a) encode-only behavior and (b) scheduler behavior when the connector already has the cache item.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| vllm/v1/worker/gpu/model_states/interface.py | Adds encode_mm_inputs helper to run the MM encoder and cache outputs without gathering. |
| vllm/v1/worker/gpu/model_states/default.py | Refactors MM embedding path to reuse encode_mm_inputs before gathering. |
| vllm/v1/worker/gpu/model_runner.py | Skips gather/get-mm-embeddings for encoder-only instances; only encodes + publishes via EC connector. |
| tests/v1/worker/test_encoder_runner.py | Adds tests asserting encoder-only encode caches outputs and doesn’t gather. |
| tests/v1/core/test_scheduler.py | Adds test ensuring scheduler skips encoder inputs when the connector already has the item (both roles). |
Suppressed comments (1)
tests/v1/worker/test_encoder_runner.py:217
- Same as above:
prepare_mm_inputsreturns(mm_hashes: list[str], mm_kwargs: list[tuple[str, MultiModalKwargsItem]]), so the empty case should be([], [])rather than([], {}).
state.encoder_runner.prepare_mm_inputs.return_value = ([], {})
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
/ci run |
|
✅ Triggered Buildkite CI #82644 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #82644. |
|
/ci retry |
|
✅ Triggered Buildkite CI #82745 for commit |
|
Hi @gty111, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
markdownlint (MD032) rejects the lists introduced by vllm-project#51300, which fails `pre-commit run --all-files` on every PR. Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
Documentation preview: https://vllm--51222.org.readthedocs.build/en/51222/ |
|
/ci retry |
|
✅ Buildkite CI #82745 failed during CI setup, so its test failure set is incomplete. Use |
|
/ci run |
|
✅ Triggered Buildkite CI #82753 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #82753. |
…only instance (vllm-project#51222) Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Purpose
An EPD encoder instance dies with
RuntimeError: Encoder cache missas soon as itis handed a multi-modal item the EC connector already holds.
The whole instance goes down, not just the request: everything after it gets a
connection error.
Root cause. When
ec_connector.has_cache_item()is true the scheduler stopsscheduling that item for encoding (
scheduler.py:1610) — correct, the embeddingis already published. But the V2 runner then calls
get_mm_embeddings()before its
is_encoder_onlyearly return, and the gather inside it raises forany scheduled item missing from the local encoder cache.
The gather is pure waste here — an encoder instance runs no language model, so
the
inputs_embedsit builds is discarded by the very next line. V1 never hadthe bug because its producer path returns before the gather
(
gpu_model_runner.py:4206).Fix. Give the encoder instance the encode half only. No scheduler change, so
a producer still skips re-encoding what the connector already has.
Reproduce
Two encoder instances sharing one
shared_storage_path— whatdisagg_epd_proxy.pybuilds from several--encode-servers-urls, since itround-robins across them:
Also reached by any instance whose local encoder cache lacks an entry the
connector kept — e.g. a restart with warm shared storage.
Test
Two regression tests, each checked by reverting the behaviour it pins:
tests/v1/worker/test_encoder_runner.py—execute_mm_encodercaches encoderoutputs and never gathers. Making it gather fails the test.
tests/v1/core/test_scheduler.py— neither role re-encodes an item theconnector already holds. This pins the optimisation the obvious "make the
producer encode anyway" fix would destroy.
End-to-end, Qwen2.5-VL-3B, the two-encoder repro above:
Encoder cache miss/fatal errorin E2 logsave_cachesdoes not fire on E2 in either arm, so the skip-re-encodeoptimisation is untouched.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.