[RFC] SGLang EPD Performance / Architecture / Observability Enhancements

1. Motivation

The community introduced EPD (Encoder-Prefill-Decode) disaggregation via #12263 in December 2025. We ported it into our internal fork in January 2026 (based on the v0.5.5 baseline). Since then, our team has made extensive engineering optimizations and stability fixes around EPD for MiMo-V2.5, which have been validated through long-running online inference at large-scale production. These changes have not yet been contributed back upstream — the goal of this RFC is to reintroduce these production-proven changes to the community, rewritten against the current mainline and delivered incrementally.

Over the past several months, we identified and resolved the following bottlenecks one by one:

• Image preprocessing bottleneck: In large-image scenarios, running resize / normalize / patchify on the CPU dominates end-to-end latency. We migrated preprocessing to the GPU to eliminate the CPU bottleneck. (The community independently implemented the same approach in use aync load for encoder_server #15456.)
• High-volume image download bottleneck: Under high QPS, images were downloaded serially one by one, and PIL decoding of large images was constrained by the GIL. We used multiprocessing for parallel download + PIL decode across multiple processes, bypassing the GIL. (The community implemented a similar capability in use aync load for encoder_server #15456 using multithreading.)
• Serial download / forward: Even after the two optimizations above, batches were still processed serially — while batch N was running forward, batch N+1 could only wait. We built a pipeline: multiprocess download + PIL decode → queue → GPU preprocess + ViT forward, allowing concurrent execution and cross-batch overlap.
• Kernel launch overhead of small ViT / Audio ops: With small batches and fixed shapes, the overhead of numerous kernel launches accounts for a significant share of runtime. We used CUDA Graph capture + replay to skip launch overhead. (The community already has a ViTCudaGraphRunner for the Qwen family; we extended it for MiMo ViT and added a new Audio CUDA Graph path.)
• Redundant download + preprocessing at Decode nodes: In EPD mode, the Decode node re-downloads images / videos / audio and re-runs the processor. We changed the flow so that after Prefill finishes, lightweight metadata is forwarded to Decode via the bootstrap server, skipping the entire download + preprocess path.

In addition, we completed work across architecture (per-process DP Encoder, SHM transport enhancements), features (automatic audio-track extraction from video, MiMo-V2.5 EPD adaptation), observability (OTel tracing, MM token propagation, EncoderMetricsCollector), and other optimizations (HTTP session reuse, fine-grained caching, RadixCache.split_by_offset, CUDA sync elimination).

Performance gains (standalone encoder benchmark, MiMo-V2.5, 1K-token image):

QPS improved 2× with comparable latency — the gains come primarily from cross-request batching, parallel image download, download / forward pipelining, and GPU-side preprocessing. The benchmark tool has already been submitted as a standalone PR ([Bench] Add QPS-based encoder benchmark for EPD separation #24700 ).

2. PR Overview