[julienmancuso]

Summary

Cherry-pick #16655 (a98d755) into release/1.6.0.

Explicit checkpointRef components using WaitForCheckpoint were missing the snapshot compatibility hash on generated DCDs, preventing restored workers from starting. Apply restore-candidate metadata before the replica gate while keeping pending snapshots at zero replicas.

Resolved one conflict by retaining the release branch's DCD-specific helper and receiver; the fix and regression tests otherwise match the merged change.

Refs DYN-4991

Validation

• Focused checkpoint startup-policy, pending automatic capture, existing DCD state, renderer restore-label, and restore-candidate metadata tests in ./internal/controller/ and ./internal/checkpoint/.
• git diff --check.
• No real cluster restore performed.

[dynamo-review-agent]

This added assertion cannot distinguish the claimed regression: the subtest constructs a DCD directly and exercises the unchanged workload renderer, bypassing applyCheckpointStartupPolicy, so the base revision already emits this compatibility hash. The retained TestComponentWorkloadsReconciler_ApplyCheckpointStartupPolicy/ready_explicit_snapshot_under_wait_policy_stamps_candidate_and_compatibility_hash case invokes the changed boundary and fails on the old WaitForCheckpoint early return while preserving coverage of the candidate and compatibility-hash behavior.

🤖 AI Fix

Remove the compatibility-hash assertion from this renderer subtest and retain the direct WaitForCheckpoint startup-policy case as the regression test.

[devin-ai-integration]

❌ Dynamo PR CI failed — run 38957802911 (attempt 2) on 587de8b92a

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework	Deploy	Snapshot
vLLM	❌ 1 ⏹️ 1 ✅ 2	❌ 1 ✅ 2
SGLang	⏹️ 2	❌ 1 ✅ 2
TRT-LLM	❌ 1 ⏹️ 1	❌ 1 ✅ 2

Failure details

5 deploy jobs failed, all on the K8s side: vllm agg could not create its vCluster (helm release name still in use), trtllm agg timed out with worker pods unschedulable for lack of GPUs, and all 3 Snapshot tests timed out with the checkpoint pod stuck Pending. Each looks like an infrastructure issue, so a rerun of failed jobs is likely enough. The 4 cancelled Deploy jobs (vllm/trtllm agg_router, sglang agg/agg_router) were cancelled seconds after trtllm agg failed at 21:54, which looks like fail-fast.

❌ vllm Deploy Test / agg: vcluster create: helm "cannot re-use a name that is still in use"

Job: vllm Deploy Test / agg · Failed step: Self-bootstrap vCluster (rerun) · Logs: gh run view --job 117204623675 -R ai-dynamo/dynamo --log-failed

+ vcluster create ci-38957802911 --namespace gh-id-38957802911-pr-16687-dt --connect=false --upgrade ...
21:29:34 info Upgrade vcluster ci-38957802911...
21:29:35 fatal error executing helm upgrade ci-38957802911 ... --install ...
Error: cannot re-use a name that is still in use
+ grep -qE 'release: already exists|another operation \(install/upgrade/rollback\) is in progress' /tmp/tmp.R6MZm4CL6g
+ exit 1
##[error]Process completed with exit code 1.

The job failed 38s in while (re)creating the vCluster ci-38957802911: helm upgrade --install rejected the release name as still in use, and this message is not one of the patterns the retry logic matches, so the step exited. No test ran. This is a CI infrastructure signature (a leftover helm release from attempt 1 in the shared namespace); a rerun of failed jobs is likely enough.

❌ trtllm Deploy Test / agg: test_deployment[trtllm-agg-normal] pytest-timeout >1200s

Job: trtllm Deploy Test / agg · Failed step: Run Dynamo Deploy Test · Logs: gh run view --job 117204623433 -R ai-dynamo/dynamo --log-failed

WARNING ManagedDeployment: Connection test failed for pod trtllm-agg-0-trtllmworker-94kpn (attempt 3/3): HTTPConnectionPool(host='localhost', port=13873): Read timed out. (read timeout=5)
WARNING ManagedDeployment:dgd_utils.py:1726 Port forward failed after 3 attempts for pod trtllm-agg-0-trtllmworker-94kpn
ERROR   ManagedDeployment:dgd_utils.py:1628 Unable to get metrics for TRTLLMWorker
FAILED tests/deploy/test_dgd.py::test_deployment[trtllm-agg-normal] - Failed: Timeout (>1200.0s) from pytest-timeout.
=========== 1 failed, 1 skipped, 1 deselected in 1225.51s (0:20:25) ============
...
Message: Resources not ready: trtllm-agg-router: TRTLLMWorker: scheduling condition InsufficientScheduledPods: Insufficient scheduled pods. expected at least: 1, found: 0
FailedScheduling - 0/15 nodes are available: 5 Insufficient nvidia.com/gpu, 5 node(s) had untolerated taint(s), 5 node(s) were unschedulable.

tests/deploy/test_dgd.py::test_deployment[trtllm-agg-normal] hit the 1200s pytest timeout. The TRTLLMWorker never answered its port-forwarded health/metrics port, and the cluster dump shows worker pods stuck on FailedScheduling with Insufficient nvidia.com/gpu. That points to GPU capacity on the shared CI cluster; a rerun of failed jobs is likely enough.

❌ vLLM Snapshot Deploy Test: test_dgd_checkpoint_restore_deploy: deployment not Ready within 600s

Job: vLLM Snapshot Deploy Test · Failed step: Run Snapshot Deploy Test · Logs: gh run view --job 117205901562 -R ai-dynamo/dynamo --log-failed

snapshot-agent-xh6w6: FailedMount - MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition
E       TimeoutError: Deployment vllm-checkpoint-1792186460356 failed to reach Ready=True, state=successful within 601s (timeout=600s)
E       Pod status at timeout:
E         checkpoint-25020c0ea803865640d51e2c4d19ccab-7tgx8/*: Unknown: Pending (no container status)
E         vllm-checkpoint-1792186460356-0-frontend-qwgbh/main: Running
E         vllm-checkpoint-1792186460356-0-worker-g7phv/*: Unknown: Pending (no container status)
Resources not ready: vllm-checkpoint-1792186460356: worker: scheduling condition InsufficientScheduledPods
FAILED tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy - TimeoutError: ... within 601s (timeout=600s)

tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy timed out because the checkpoint pod (and the vLLM worker) stayed Pending (no container status) for the whole wait. Earlier in the log the snapshot-agent pods repeatedly hit FailedMount ... failed to sync configmap cache: timed out waiting for the condition. The DGD also reports InsufficientScheduledPods for the worker. All three Snapshot jobs fail the same way at the same time.

❌ SGLang Snapshot Deploy Test: test_dgd_checkpoint_restore_deploy: deployment not Ready within 900s

Job: SGLang Snapshot Deploy Test · Failed step: Run Snapshot Deploy Test · Logs: gh run view --job 117205961297 -R ai-dynamo/dynamo --log-failed

snapshot-agent-cfrvd: FailedMount - MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition
E       TimeoutError: Deployment sglang-checkpoint-1792186470538 failed to reach Ready=True, state=successful within 900s (timeout=900s)
E       Pod status at timeout:
E         checkpoint-eb6afe9b5fbabb94d3caf727fe589e5c-l5ktt/*: Unknown: Pending (no container status)
E         sglang-checkpoint-1792186470538-0-frontend-2h5hj/main: Running
FAILED tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy - TimeoutError: ... within 900s (timeout=900s)

Same failure as vLLM Snapshot: tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy timed out because the checkpoint pod (and the vLLM worker) stayed Pending (no container status) for the whole wait. Earlier in the log the snapshot-agent pods repeatedly hit FailedMount ... failed to sync configmap cache: timed out waiting for the condition.

❌ TRTLLM Snapshot Deploy Test: test_dgd_checkpoint_restore_deploy: deployment not Ready within 900s

Job: TRTLLM Snapshot Deploy Test · Failed step: Run Snapshot Deploy Test · Logs: gh run view --job 117205982485 -R ai-dynamo/dynamo --log-failed

snapshot-agent-nk5fz: FailedMount - MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition
E       TimeoutError: Deployment trtllm-checkpoint-1792186470780 failed to reach Ready=True, state=successful within 901s (timeout=900s)
E       Pod status at timeout:
E         checkpoint-0105d4c29b6500014078b13a1ae1af43-rlppk/*: Unknown: Pending (no container status)
E         trtllm-checkpoint-1792186470780-0-frontend-2kz8l/main: Running
FAILED tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy - TimeoutError: ... within 901s (timeout=900s)

Same failure as vLLM Snapshot: tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy timed out because the checkpoint pod (and the vLLM worker) stayed Pending (no container status) for the whole wait. Earlier in the log the snapshot-agent pods repeatedly hit FailedMount ... failed to sync configmap cache: timed out waiting for the condition.

For agents

{"pr": 16687, "run_id": 38957802911, "run_attempt": 2, "head_sha": "587de8b92a0a5c947fdb4b2ba59587836098548e", "failures": [{"job": "vllm Deploy Test / agg", "job_id": 117204623675, "failed_step": "Self-bootstrap vCluster (rerun)", "signature": "vcluster create: helm cannot re-use a name that is still in use", "tests": [], "log_cmd": "gh run view --job 117204623675 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm Deploy Test / agg", "job_id": 117204623433, "failed_step": "Run Dynamo Deploy Test", "signature": "pytest-timeout >1200s; worker FailedScheduling Insufficient nvidia.com/gpu", "tests": ["tests/deploy/test_dgd.py::test_deployment[trtllm-agg-normal]"], "log_cmd": "gh run view --job 117204623433 -R ai-dynamo/dynamo --log-failed"}, {"job": "vLLM Snapshot Deploy Test", "job_id": 117205901562, "failed_step": "Run Snapshot Deploy Test", "signature": "TimeoutError: checkpoint deployment not Ready within 600s; checkpoint pod Pending", "tests": ["tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy"], "log_cmd": "gh run view --job 117205901562 -R ai-dynamo/dynamo --log-failed"}, {"job": "SGLang Snapshot Deploy Test", "job_id": 117205961297, "failed_step": "Run Snapshot Deploy Test", "signature": "TimeoutError: checkpoint deployment not Ready within 900s; checkpoint pod Pending", "tests": ["tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy"], "log_cmd": "gh run view --job 117205961297 -R ai-dynamo/dynamo --log-failed"}, {"job": "TRTLLM Snapshot Deploy Test", "job_id": 117205982485, "failed_step": "Run Snapshot Deploy Test", "signature": "TimeoutError: checkpoint deployment not Ready within 900s; checkpoint pod Pending", "tests": ["tests/deploy/test_dynamocheckpoint.py::test_dgd_checkpoint_restore_deploy"], "log_cmd": "gh run view --job 117205982485 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 38957802911. Updated on every full-CI run of this PR.}