[alec-flowers]

Summary

• resolve TensorRT-LLM's nested kv_events_config after applying engine overrides
• subscribe directly to the vLLM-compatible ZMQ stream from each TensorRT-LLM attention-DP rank
• derive per-rank endpoints as base_port + rank, including Slurm node discovery for multi-node deployments
• bypass the legacy TensorRT-LLM polling and Python ZMQ republishing path in native mode
• keep the existing --publish-kv-events behavior unchanged for legacy configurations; disabled or publisher="null" native configs create no subscriber

This proof of concept pairs with NVIDIA/TensorRT-LLM#16869.

Validation

• focused config-only native subscriber test passed against the compiled Dynamo runtime
• 44 synchronous TRT-LLM publisher tests passed; 6 async tests were not runnable in the cached validation image because pytest-asyncio was absent
• local Qwen3-0.6B TensorRT-LLM smoke test delivered a stored-event batch with two full blocks to the real Dynamo KV router
• Python compilation, repository-pinned Black 23.1.0, isort, and git diff --check passed

The local-only router smoke script is intentionally excluded from this PR.

Summary by CodeRabbit

• New Features

  • Added configurable KV-event publication modes: disabled, polling, and streaming.
  • Added streaming support through direct rank-specific ZMQ connections, including multi-node deployments.
  • Added automatic endpoint and host resolution for supported deployment environments.

• Bug Fixes

  • Added validation for streaming configuration, endpoints, and required TensorRT-LLM cache settings.
  • Improved handling of polling-based KV-event publication and metrics-only operation.

• Tests

  • Expanded coverage for streaming configuration, host discovery, endpoint mapping, and publication modes.

[datadog-official]

 

⚠️ Warnings

🚦 3 Pipeline jobs failed

Docs link check | lychee     

See error

Rejected status code: 503 Service Unavailable while accessing https://marketplace.visualstudio.com/items?itemName=SanjulaGanepola.github-local-actions.

PR | deploy-status-check     

PR | trtllm Deploy Test / agg_router     

See error

Error during helm upgrade: Release 'ci-32009262645' does not exist. Installing it now. Error: release: already exists

ℹ️ Info

🔄 Datadog auto-retried 14 jobs - 12 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 6.67%
• Overall Coverage: 28.06% (-29.86%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d84ba65 | Docs | Datadog PR Page | Give us feedback!}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2f31c1b3-ef6f-49ba-a111-4ff71871e7ed

📥 Commits

Reviewing files that changed from the base of the PR and between e15800b and 9cdd2e0.

📒 Files selected for processing (4)

• components/src/dynamo/trtllm/publisher.py
• components/src/dynamo/trtllm/tests/test_trtllm_fpm_publisher.py
• components/src/dynamo/trtllm/tests/test_trtllm_unit.py
• components/src/dynamo/trtllm/workers/llm_worker.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

TRT-LLM KV-event publication now uses explicit disabled, polling, or streaming modes. Streaming validates configuration, discovers rank hosts, and creates rank-specific ZMQ subscribers. Worker wiring and tests now use the new mode and configuration parameters.

Changes

KV Event Publication

Layer / File(s)	Summary
Resolve and validate publication configuration components/src/dynamo/trtllm/workers/llm_worker.py	The worker resolves nested KV-event settings, validates streaming ZMQ endpoints and the Python V2 cache manager backend, updates runtime state, and passes the resolved mode to the publisher.
Implement rank-aware publication modes components/src/dynamo/trtllm/publisher.py	The publisher supports disabled, polling, and streaming modes. Streaming discovers hosts and creates rank-specific ZMQ subscribers. Polling retains consolidator or direct NATS routing.
Validate modes and streaming configuration components/src/dynamo/trtllm/tests/test_trtllm_fpm_publisher.py, components/src/dynamo/trtllm/tests/test_trtllm_unit.py	Tests cover explicit modes, host discovery, endpoint offsets, nested configuration resolution, backend validation, and metrics-only initialization.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to 9cdd2

Native streaming may map some attention-DP ranks to the wrong hosts in tensor-parallel multi-node deployments, causing KV cache events to stop reaching the router for those ranks. The host-mapping contract should be confirmed or corrected before merge.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description gives detailed technical scope and validation results, but it does not include the required Related Issues section or the requested Where should the reviewer start section. It also use…	Add the required Related Issues section and select either a linked issue with a real issue number or confirm that no related issue exists. Add a Where should the reviewer start section with the key files for review. Map the existing Summary…
Docstring Coverage	⚠️ Warning	Docstring coverage is 43.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: native TensorRT-LLM KV cache event consumption.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description gives detailed technical scope and validation results, but it does not include the required Related Issues section or the requested Where should the reviewer start section. It also uses Summary and Validation instead of the template's Overview and Details headings.

Resolution

Add the required Related Issues section and select either a linked issue with a real issue number or confirm that no related issue exists. Add a Where should the reviewer start section with the key files for review. Map the existing Summary and Validation content to the template's Overview and Details sections, or retain the content under clearly equivalent headings.

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}