💬 Join the EPD / EC-connector Slack channel!

Interested in EPD (Encoder/Prefill/Decode disaggregation) or the EC connector? Click the link above to join the discussion on vLLM Slack — everyone is welcome!

EPD = Encoder / Prefill / Decode disaggregation.

Umbrella tracker for EPD and EC-connector work, grouped by area.
GitHub displays the current title and state of the native PR/issue references below; no manual completion checkboxes are used.

Model Runner V2

• [MM][V2] Enable encoder-only ViT CUDA graph capture #56922 — Enable encoder-only ViT CUDA graph capture in Model Runner V2
• [Refactor][Model Runner V2][Multimodal] Move the encoder-only path out of the shared runner #53176 — Move the encoder-only path out of the shared runner
• [Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance #51222 — Skip gather mm embeddings for encoder-only instance
• [Feature]: Support the EC connector in Model Runner V2 (E/P/D disaggregation) #47999 — Support the EC connector in Model Runner V2 (E/P/D disaggregation) — feature issue
  • [Model Runner v2] E/P/D disaggregation support #38390 — E/P/D disaggregation support

Model Support

• [EPD] Support DeepSeek-V4.1 image metadata-only inputs #62802 — Support DeepSeek-V4.1 image metadata-only inputs
• [Bugfix][MiMo] Declare embedding_fields so an EPD pair can serve images #58938 — Declare embedding_fields for MiMo-V2-Omni so an EPD pair can serve images

Common

• [EC Connector] Add Metrics Collection #54960 — Add EC connector metrics collection — stats hooks, CLI + Prometheus
• [EC Connector] Added Build Connector Worker Meta for EC Connector #49585 — Build connector worker meta for the EC connector
• [EC Connector] Call to EC Connector update_connector_output from scheduler #49579 — Call update_connector_output from the scheduler
• [EC Connector] Add has_pending_push_work  #49582 — Add has_pending_push_work — prepares for RDMA

Configuration

• [EC] Automatically enable embedding inputs on EC/KV consumers #56379 — Automatically enable embedding inputs on EC/KV consumers
• [Config][EC] Normalize producer-only encoder config #53656 — Normalize producer-only encoder config
• [EPD] Allow KV consumers to omit MM embeddings #52697 — Allow KV consumers to omit MM embeddings

Proxy and Control Plane

• [Bugfix][EPD] Preserve upstream status and error details in proxy responses #54140 — Preserve upstream status and error details in proxy responses
• [Feature][EPD] Support dynamic EPD (EC-connector) proxy #54176 — Support dynamic EPD (EC-connector) proxy
• [Performance][EPD] Reduce Python proxy serialization overhead #56657 — Reduce Python proxy serialization overhead
• [Frontend][EPD] Use JSON arrays for multimodal metadata #56090 — Use JSON arrays for multimodal metadata — reduces serialization overhead; prepares router and Rust frontend integration

vLLM Router

• [Feature] Add EPD routing with JSON metadata router#266 — Add E+PD / E/P/D routing with JSON metadata

Rust Frontend

• [Frontend][EPD] Support JSON metadata references in Rust HTTP frontend #55842 — Support JSON metadata references in the Rust HTTP frontend for EPD
• [Rust Frontend][gRPC] Preserve multimodal metadata for remote-prefill decode #54814 — Preserve multimodal metadata for remote-prefill decode in the Rust frontend

Device Support

• [XPU] Use encoder-only model runner for EC producer instances #59320 — Use encoder-only model runner for EC producer instances on XPU
• [XPU] Support EC connector KV Offloading on XPU #49532 — Support EC connector KV offloading on XPU

Video Support and Optimization

• [Multimodal] Use GPU NVDEC for EPD encoder-only instance video media IO #53675 — Use GPU NVDEC for EPD encoder-only instance video media I/O
• [Frontend] Add video embeds input support #54242 — Add video embeds input support

Audio Support

• [EPD] Support metadata-only audio-in-video requests #62573 — Support metadata-only audio-in-video requests
• [EPD] Support metadata-only audio inputs #57887 — Support metadata-only audio inputs (Qwen2-Audio, AudioFlamingo3, Ultravox, Qwen2.5-Omni, Qwen3-Omni)

Mooncake Connector

RFC: Mooncake Project Overall Roadmap — kvcache-ai/Mooncake#1883

• [Doc] Document ECMooncakeConnector for EPD disaggregation #59196 — Document ECMooncakeConnector for EPD disaggregation — usage guide
• [Feature][Multimodal] Add cross-encoder caching to Mooncake P2P #56242 — Reuse encoder outputs across Encoder instances via Mooncake Store; retain P2P delivery to Prefill
• [EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine #41567 — Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine — P2P, prioritized

CPU Offloading Connector

• [EC Connector][CPU EC Connector] Add Metrics Collection #57333 — Add metrics collection to the CPU EC connector
• [Bugfix][EC Connector] Release shared memory on NIXL setup failure #59875 — Release shared memory on NIXL setup failure
• [Refactor][EC Connector] Add backend extension points to ECCPUWorker #54033 — Add backend extension points to ECCPUWorker
• [Docs][EC Connector] CPU EC Connector usage Docs #54522 — CPU EC Connector usage documentation
• [Bugfix][EC Connector] Fall back when MADV_POPULATE_WRITE is unsupported #53190 — Fall back when MADV_POPULATE_WRITE is unsupported
• [EC Connector] EC Offloading Connector use events instead of StepTracker #49994 — Use events instead of StepTracker
• [EC Connector] CPU Offloading EC Connector #47423 — CPU offloading EC connector

NIXL Connector (llm-d / IBM)

• [Bugfix][EC Connector] Fail the request, not the engine, when a remote encoding cannot arrive #55290 — Fail the request instead of the engine when a remote encoding cannot arrive
• [EC Connector] P2P NIXL + CPU EC Connector #47941 — P2P NIXL + CPU EC connector
• [EC Connector] CPU EC Connector #42998 — CPU EC connector — superseded; split into [EC Connector] CPU Offloading EC Connector #47423 (merged) and [EC Connector] P2P NIXL + CPU EC Connector #47941

LMCache Connector

• [ECConnector] LMCache EC connector entrypoint #38668 — LMCache EC connector entrypoint

ZMQ Connector

RFC: Refactor EPD — #52359

• ECZmqConnector #52770 — ECZmqConnector

Optimization

RFC: #55639 — Avoid redundant encoder work when Prefill can reuse KV or embeddings.

• [EPD][Model Loader] Skip language-model checkpoint shards for --mm-encoder-only #58086 — Skip language-model checkpoint shards for --mm-encoder-only
• [EPD] Remove duplicate image preprocessing in EPD and enable preprocess on GPU #50390 — Remove duplicate image preprocessing and enable preprocessing on GPU

Fixes

• [Bugfix][EC Connector] Allocate producer control ports at startup #61764 — Allocate producer control ports at startup
• [Bugfix][EC Connector] Don't create hash directories on cache-miss lookups in ECExampleConnector #60826 — Avoid creating empty hash directories on ECExampleConnector cache-miss lookups
• [Bugfix][EC Connector] Size transfers by measured encoder output widths #57904 — Fix Qwen3-Omni transfer shapes for Mooncake and CPU/NIXL
• [Bugfix][Core] Schedule encoder-only prompts larger than one step #59029 — Schedule encoder-only prompts larger than one step
• [Bugfix][EPD] Skip sampling for encoder-only async steps #58490 — Skip sampling for encoder-only async steps
• [Bugfix][EPD] Preserve explicit multimodal UUIDs with caches disabled #56432 — Preserve explicit multimodal UUIDs with caches disabled.
• [Bugfix][EC Connector] Fix ECExampleConnector load device under TP>1 #40416 — Fix ECExampleConnector load device under TP>1
• [Bugfix][EPD] Fix encoder round-robin fan-out #52491 — Fix encoder round-robin fan-out
• [Bugfix][V1][Multimodal] Ignore stale same-step encoder cache evictions #52482 — Ignore stale same-step encoder cache evictions
• [Bugfix][EC Connector] Don't stop an encoder-instance request before its images are encoded #50275 — Don't stop an encoder-instance request before its images are encoded