Skip to navigation
Image UnderstandingVLM Performance Optimization

Embedding Cache

Skip the vision encoder for images and clips it has already encoded
View as Markdown

Overview

The embedding cache (or encoder cache) is a CPU-side LRU cache that stores vision encoder outputs. When the same multimodal content, such as an image or video, appears in multiple requests, the cached embedding is reused instead of running the vision encoder again. This reduces GPU load on the encoder and lowers latency for repeated content.

Embedding cache is separate from the KV cache, which reuses attention key/value state after prefill to skip prefill and go straight to decode. For KV cache reuse and routing, see Multimodal KV Routing.

When to Use

Use the embedding cache when your workload includes repeated multimodal content across requests. Common scenarios:

  • Product catalog queries or multi-turn chat where users ask about the same image(s).
  • Document processing pipelines that reference shared diagrams or figures
  • Video QA or benchmark workloads that repeatedly reference the same clips

If your workload consists entirely of unique multimodal content, the cache provides no benefit.

Configuration

Launch a disaggregated multimodal deployment with the embedding cache enabled:

cd $DYNAMO_HOME/examples/backends/vllm
bash launch/disagg_multimodal_e_pd.sh --multimodal-embedding-cache-capacity-gb 10

Set --multimodal-embedding-cache-capacity-gb based on your expected working set of unique multimodal content. A larger cache holds more embeddings but consumes more host memory.

If one image URL can return different content for different sessions, enable session-scoped caching as described in Shared image cache. Dynamo then partitions its image embedding caches by affinity scope and bypasses them when a request has no valid scope.

See the backend-specific documentation (vLLM, TRT-LLM) for more details.

How It Works

In vLLM/TRT-LLM disaggregated flows, the prefill worker owns the CPU-side LRU cache. On a hit, the encode worker is skipped entirely. On a miss, the encode worker produces the embedding, transfers it via NIXL, and the prefill worker saves it to the cache.

In SGLang E/PD, the encode worker owns the cache and skips re-encoding on cache hits before forwarding the cached image or video embeddings downstream.

An image seen before is a hit in the CPU embedding cache and the encode worker is skipped. A new image is a miss: the encode worker runs the vision encoder, sends embeddings over NIXL, and they are saved to the cache before reaching the inference engine.

Support Matrix

BackendAggregatedDisaggregated (E/PD)Notes
vLLMYesYesAggregated uses vLLM-native ec_both; disaggregated uses Dynamo EmbeddingCacheManager
TensorRT-LLM—YesDynamo MultimodalEmbeddingCacheManager in PD worker
SGLang—YesDynamo MultimodalEmbeddingCacheManager in the encode worker

This support requires vLLM 0.17.0 or newer.