[RoadMap][Call For Contribution] Mooncake Project Overall Roadmap #1883

Milestone 1: Inference & RL Framework Integration

Deepen integration with mainstream frameworks and enhance support for large-scale architectures.

• (Agentic-aware): Support agentic-aware KVCache System in inference frameworks.
• (PD Disaggregation): Optimize PD with more features and support SOTA models.
• (Hybrid Model): Optimize performance for New Models.
• (RL Disaggregation): Separate Rollout (sampling) and Update (weight updating) using a globally shared KVCache pool and efficient sample flow for higher PPO/RLHF throughput.

Milestone 2: Store V3 Evolution

Enhance the core storage system for better modularity, elasticity, and high concurrency.

• (TE & Store Decoupling): Completely separate Transfer Engine (TE) and Store into independent packages.
• (Tensor-Native & Multi-Replica): Integrate TP rank and model info into APIs; support replica lists for high-frequency KVCache items.
• (Client/Worker Decoupling): Separate dummy client from real client for flexible, highly available deployment topologies.
• (Engram-like Backend): Introduce an optimized storage abstraction for N-gram key-value operations to handle highly sparse and concurrent random access.
• (Object Grouping): Allow specifying multiple objects as part of the same group during put operations, with visibility and eviction handled at the group level
• (Visibility Enhancement): Further improve visibility by providing more flexible and comprehensive metrics querying capabilities
• (Memory Allocation Optimization): OffsetAllocator is currently optimized for uniform size distributions, but there are now cases with two or more size classes; targeted optimizations are needed to improve memory utilization

Store V3 RFCs:

• Mooncake #1035 — [RoadMap] Mooncake Store V3 Roadmap
• Mooncake #1836 — [RFC] Mooncake Store Multi-Tenant Isolation
• Mooncake #1732 — [RFC] Support Radix Tree Index in Store Master for Prefix Query
• Mooncake #1772 — [RFC] KVPool eviction: evict suffix keys when prefix keys are missing
• Mooncake #1645 — [RFC] Hard Pin and Upsert for Mooncake Store
• Mooncake #1648 — [RFC] Kubernetes Lease-Based HA Without External etcd
• Mooncake #1650 — [RFC] HA OpLog Store Abstraction Layer
• Mooncake #1673 — [RFC] Data Type Support in Mooncake Store
• Mooncake #1372 — [RFC] Support Engram as a storage backend in Mooncake
• Mooncake #1150 — [RFC] Persistence and Recovery Solution for KV Metadata and Segment Information
• Mooncake #1131 — [RFC] Refactor allocation strategy for better performance and flexibility
• Mooncake #1100 — [RFC] Dynamic replica management
• Mooncake #1062 — [RFC] Add Local Cache Mechanism for Mooncake Store Client
• Mooncake #1054 — [RFC] Contribute Local Storage to Distributed Pool
• Mooncake #972 — [RFC] More KVCache metrics in both master/client side
• Mooncake #961 — [RFC] Fake Client for Mooncake Client
• Mooncake #1809 — [RFC] Integrate Mooncake Store into Dynamo KVCM with Native Rust API (Closed)
• Mooncake #1920 — [RFC] Rolling Upgrade Support for Mooncake Store
• Mooncake #1957 — [RFC] Mooncake Store NVMe KV Backend
• Mooncake #1979 — [RFC] Dynamo KVBM + Mooncake Store Integration Design
• Mooncake #1998 — [RFC] Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction
• Mooncake #2081 — [RFC] Simplify Mooncake Store for Maintainability
• Mooncake #2127 — [RFC] Lightweight Best-Effort Group Semantics for Mooncake Store Objects
• Mooncake #2153 — [RFC] Dynamic Tenant Quota and Scoped-Key Isolation

Store V3 Key PRs (recent):

• Mooncake #2080 — [Store] Add DSA-like workload allocation strategy (Merged)
• Mooncake #2112 — [Store] feat: introduce per-tenant storage quota in master
• Mooncake #2180 — [Store] Add opt-in grouped object routing semantics

Milestone 3: Transfer Engine (TENT) & Networking

Focus on declarative data orchestration and pushing hardware limits.

Release TENT V1.1 at 26Q2 @alogfans

• (Declarative Orchestration): Implement a plugin-based multi-transport architecture; enable dynamic path selection via hardware auto-discovery and declarative routing policies.
• (QoS-aware Slice Spraying): Introduce cross-process priority-based slice spraying, minimizing the interference between different tasks; aiming to reduce P99 latency in 8x 400 Gbps network.
• (Ease of Maintenance): Decouple heterogeneous hardware support from transport protocols via a unified GPU abstraction layer and modular plugin architecture to streamline cross-platform deployment. Improve the self-healing ability.
• (Transfer Queue): Implement centralized/distributed transfer queue with priority scheduling and backpressure for high-throughput, low-latency data movement under heavy load.
• (NCCL Support): Use NCCL device API as an optional backend

Transfer Engine RFCs:

• Mooncake #1058 — [RoadMap] Mooncake Transfer Engine NEXT
• Mooncake #1850 — [RFC] End-to-End Transfer Tracing for Mooncake
• Mooncake #1749 — [RFC] Optimizing Mooncake TCP Transport for High-Concurrency Throughput
• Mooncake #1773 — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
• Mooncake #2117 — [RFC] Hierarchical Arch for Intra/Inter Data Center Deployment
• Mooncake #2123 — [RFC] Device Transport — GPU-Initiated Communication for TENT
• Mooncake #2132 — [RFC] TENT Local Transfer Admission Queue
• Mooncake #2132 — [RFC] [RFC]: NCCL device API in Mooncake TENT

Transfer Engine Key PRs (recent):

• Mooncake #2079 — [TENT] Add rule-based transport and device selection
• Mooncake #2090 — [TENT] Add cross-platform dynamic library adapter with GPU vendor abstraction
• Mooncake #2155 — [TE] Improving the RDMA transport failure handling
• Mooncake #2160 — [TE] Add explicit progressBatch API

Milestone 4: OS-Level Optimizations

Improve throughput and reduce latency through low-level system integrations.

• (Decentralized Metadata): Migrate SSD metadata management from clients to a dedicated Master to resolve bottleneck issues.
• (Multi-Level Resource Pooling): Aggregate VRAM, DRAM, NVMe, and Remote resources into a transparent cache pool using CXL and SHM.

Related RFCs:

• Mooncake #1123 — [RFC] Support hugepage memory allocation via mmap to bypass RDMA memory registration limits (Closed)
• Mooncake #2106 — [RFC] GPU VRAM (HBM) Pool Support for Mooncake Store

Milestone 5: Routing & Checkpoint Engine

Enhance global scheduling and model weight handling.

• (KVCache-Aware Routing): Route Decode requests based on underlying KVCache distribution (History Cache) to maximize hit rates and minimize TTFT.
• (Checkpoint Engine): Strengthen parallel distribution of weights via P2P Store; support native Torch format offload for fast MoE loading.

Conductor & Routing RFCs:

• Mooncake #977 — [RFC] Mooncake-Conductor: Design and Implementation of a Global Scheduler
• Mooncake #1767 — [RFC] Mooncake Data Plane Model Distribution, Phase 1
• Mooncake #2189 — [Call For Contributions] Mooncake Conductor Roadmap

Milestone 6: Heterogeneous & Cross-Platform Inference (Domestic GPU Support)

Break hardware silos for collaborative deployment. Expand support for domestic accelerator platforms.

• (Cross-Architecture Collaboration): Enable tasks to flow across different vendors (e.g., Vendor A for high-throughput Prefill, Vendor B for low-latency Decode).
• (Ascend 910B / Atlas 800T): Adapt Transfer Engine and Store for Huawei Ascend NPU, including Ascend Direct protocol and HCCL integration.
• (Kunpeng SuperNode UB Transport): Enable UB transport via Kunpeng UMDK URMA APIs for Kunpeng 950 platforms.
• (Moore Threads MACA/MUSA): Align MACA GPU paths with Mooncake build system; integrate MUSA AI Tensor Engine (MATE) for FA3 attention backend.
• (MetaX / Yunsilicon): Resolve RDMA compatibility with Yunsilicon XScale NIC and ensure stable deployment.
• (AMD ROCm / HIP): Add ROCm HIP support to the Mooncake Python package for AMD GPU transport.
• (AWS EFA): Enhance EFA transport with fi_read, endpoint LRU eviction, and multi-NIC striping.

Related RFCs:

• Mooncake #1773 — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
• Mooncake #1058 — [RoadMap] TENT — Full Heterogeneous Deployment section (Ascend ↔ NVIDIA)
• Mooncake #1035 — [RoadMap] Store V3 — Hardware mentions: AMD, Moore Threads, AWS EFA, Ascend
• Mooncake #1975 — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design
• Mooncake #1965 — [Feature Request] MACA Support for Mooncake EP/PG

Tracked issues (Ascend):

• Mooncake #1854 — [Ascend 910B] rtsHostRegister failed in mooncake-store
• Mooncake #1107 — Mooncake RDMA error on Ascend 910B
• Mooncake #1766 — Failed to modify QP to RTR with Yunsilicon XScale (MetaX)

Heterogeneous Key PRs (recent):

• Mooncake #2118 — feat: add Hygon DCU/DTK and Iluvatar CoreX platform support (Merged)
• Mooncake #2196 — [Store] enables the Ubtransport for Mooncake Store (Kunpeng)
• Mooncake #2113 — [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin

Milestone 7: Unified Resource Management & Multi-Model Serving

Manage KVCache and model weights cohesively on GPUs.

• (Tiered Weight Storage): Manage weights across VRAM (hot), DRAM (warm), and SSD (cold).
• (Fine-Grained GPU Sharing): Implement CUDA MPS + custom memory isolation for multi-model sharing.
• (Dynamic Weight Swapping): Use Mooncake's TE for fast RDMA-based weight migration.
• (Unified View): Manage both KVCache and weight pools globally.

#2282

Milestone 8: Fast Instance Startup

Utilize data transfer capabilities to accelerate initialization.

• (Fast Model Distribution): Pull weights via RDMA from neighboring nodes instead of remote storage.
• (CUDA Context Restore): Snapshot and restore initialized inference processes to skip warmup.
• (Warm Pool): Maintain a pool of pre-warmed processes for stateful scaling.
• (Lazy Pulling): Optimize container images to only pull needed layers.

Related RFCs:

• Mooncake #1767 — [RFC] Mooncake Data Plane Model Distribution, Phase 1

Milestone 9: Elasticity & Self-Healing

Leverage decoupled architecture for robust operations.

• (LLM-Aware Autoscaler): Scale based on global KVCache pressure and Prefix Tree hit rates.
• (Fast GPU Recovery): Recover failed nodes in <10s by loading weights, utilizing the intact KVCache pool.
• (sched_ext BPF Scheduler): Ensure lowest-latency CPU scheduling for GPU feeder threads.
• (TE Fault Tolerance): Automatic fallback to TCP and re-routing on RDMA/node failures: Already supported by TENT.

Related RFCs:

• Mooncake #975 — [RFC] Cleanup processing replicas if the transferring client is down (Closed)
• Mooncake #1920 — [RFC] Rolling Upgrade Support for Mooncake Store

Milestone 10: Agent & Multi-Modal Support

Build the foundation for complex Agent and Multi-Modal workflows.

• (Agent State Cloning): Fast A2A state sharing and cross-node data transport.
• (Agent PD Disaggregation): Separate resources for "thinking" vs. "interactive" agents.
• (EPD Disaggregation): Separate Vision Encoder, Prefill, and Decode for multi-modal scenarios. (Qwen-series supported in sglang)
• (Diffusion Serving): Explore disaggregated architectures for Diffusion models.

Omni Collaboration

• (Omni Transfer Engine Integration): Integrate Mooncake Transfer Engine as a connector for inter-stage data transfer in the omni multi-modal pipeline (AR → Generation → Diffusion).
• (Worker-Level Inter-Stage Transfer): Optimize worker-level data movement between omni pipeline stages via high-speed RDMA or shared memory.
• (Omni Prefix Caching): Support Hidden State Prefix Caching for multi-modal omni models.

Related RFC

• [RFC]: Mooncake and vLLM-Omni Collaboration Roadmap vllm-project/vllm-omni#2904
• [RFC]: Qwen3-Omni Stage Transfer via Mooncake Transfer Engine vllm-project/vllm-omni#3635
• Mooncake #2098 — [RFC] Agent-Aware KV Cache Support in Mooncake (Phase 1)

Diffusion Collaboration

• (Disaggregated Diffusion Deployment): Support decentralized disaggregated deployment for video generation models, decoupling transformer and decoder stages.
• (Queue-based Orchestration): Implement queue-based scheduling for multi-stage diffusion serving requests.

Milestone 11: KVCache Dynamic Migration

• (Live Migration): Support lossless, dynamic hot-migration of massive KVCache states across clients/nodes for Spot Instances and elastic scaling.
• (Migration on Store Instance Exit): Support migrating hot data to other nodes during scale-down or rolling upgrades (optional, enabled manually).
• (Load Balancing)

Related RFCs:

• Mooncake #1159 — [RFC] Replica copy and move support

Migration Key PRs (recent):

• Mooncake #2170 — [Store] add copy/move execution api support in master side (Merged)

Milestone 12: Hardware Support & ICMS

• (New Hardware): Support Supernodes, Rubin, PPU, Matrix384.
• (ICMS): Follow BF4 developments and integrate related capabilities.

Milestone 13: SSD Offload Support

• (SSD offload): Support offloading KV cache to SSD, ensuring higher hit ratio in limited DRAM scenarios.
• (Bandwidth optimization): Optimized bandwidth of SSD loading using io_uring lib.
• (L2→L1 data promotion): If no memory replica and the object is used, promote it to memory.
• (Smart eviction strategy): e.g., Selectively evict cold data with LRU-like strategy.

Related RFCs:

• Mooncake #971 — [RFC] Support distributed read and write operations to SSDs
• Mooncake #952 — [RFC] Support Mooncake file eviction in DFS/3FS (Closed)
• Mooncake #1940 — [RFC] Implement SSD Pool over NVMe-oF in Mooncake Store
• Mooncake #1957 — [RFC] Mooncake Store NVMe KV Backend
• Mooncake #1990 — [RFC] Native SSD KV Storage Backend for Mooncake Store
• Mooncake #2025 — [RFC] L2→L1 data promotion: If no memory replica and the object is used, promote it to memory
• Mooncake #2031 — [RFC] Promote data from L2 -> L1 when it's not in L1

SSD Offload Key PRs (recent):

• Mooncake #2084 — feat(nof): add NVMe-oF SSD cache support
• Mooncake #2167 — [Store] introduce NVMe KV backend core
• Mooncake #2172 — feat(store): add SPDK NoF worker pool
• Mooncake #2176 — [Store] L2->L1 promotion-on-hit: observability metrics + max_per_heartbeat knob

Milestone 14: Mooncake PG

• (Transfer Engine): Integrate TENT and refactor transfer engine management.
• (Fault-Tolerance Backend): Improve FT-backend fault-handling and recovery, explore elastic backend design.
• (PyTorch c10d API): Lazy initialization & Split group for faster initialization.
• (Collective Communication): Support ring- and tree-topology algorithm.
• (Hardware Adaption): Support Ascend, Moore Threads, and AWS EFA.
• (Reliability): (1) Establish comprehensive unit-testing and benchmark suites (2) Agentic Specification for robust AI-Coding generation.

Mooncake PG Key PRs & Issues (recent):

• Mooncake #2157 — [Feature Request] Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan
• Mooncake #2149 — [PG] support external TE (Merged)
• Mooncake #2182 — [PG] Add rank fault tolerance and degraded recovery

Milestone 15: Mooncake EP

• (High Throughput Kernel): Implement high throughput kernel for high batch size inference.
• (Elastic EP): Support elastically EP extension.
• (TMA Engine): TMA engine optimization for Hopper-specific architecture.
• (Benchmark): Comprehensive MoE kernel benchmark under incast traffic.

Mooncake EP RFCs:

• Mooncake #1975 — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design

—

Milestone 16: Documentation

• (Update and Refactor Docs): The documentation has become extensive; restructure it and update outdated content
• (AI-Friendly Knowledge Base): Organize Mooncake’s knowledge into a format that is easy for AI systems to read and consume

To contribute features or provide feedback, please edit this issue directly (if you have write access) or comment below. I will update the roadmap accordingly.