Repository navigation
fix(operator): give HX Cyborg conductors their LPU Agent addresses - #15804
Conversation
The LPX renderer wrote the Cyborg lpu_servers ConfigMap only for XT hybrid builds. HX hybrid Cyborg conductors received the Agents' partition table at /configs instead, so cyborg-entrypoint --expand-hosts found no /configs/lpu_servers: an all-local HX deployment failed at startup, and a mixed or all-LPU one could not reach its Agents. Render the same ConfigMap for every hybrid workload and bind the conductor's config volume to it. List one Agent per remote runtime partition in compiler order. HX keeps selected prop-sync chain members as physical partitions, so address each chain through its first member, as XT does after collapsing it. An all-local selection renders an empty list. XT output is unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Josh Hay <jhay@nvidia.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
|
👋 Hi joshhay95! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
github-actions
Bot
added
fix
external-contribution
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (11)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughHybrid Cyborg rendering now covers HX and XT projections. Prop-sync chain positions determine which Agent endpoints appear in the generated ConfigMap. Hybrid workload rendering publishes and mounts that config. ChangesHybrid Cyborg Config Rendering
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to The inspected HX workload receives its Agent-address configuration and mount. No identified issue prevents merging after normal checks. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 6 files. (5 skipped: 5 unsupported.)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Warning Some tools did not complete. Review the errors below. 🔧 golangci-lint (2.13.2)golangci-lint (2.13.2) skipped for deploy/operator/.golangci.yml: configuration requires version: "2". Migrate older configurations using golangci-lint migrate and review the result; see https://golangci-lint.run/docs/product/migration-guide/. Comment |
|
/ok to test babe0a9 |
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
❌ Dynamo PR CI failed — run 37604627410 (attempt 2) on
|
| Other | Jobs |
|---|---|
| DGDR Deploy Test | ❌ 3 ✅ 1 |
Failure details
3 jobs failed, all the same way: the DGDR Deploy Test CPU jobs (profiling, validation, lifecycle) stop in Verify DGDR test prerequisites because the dynamographdeploymentrequests.nvidia.com CRD is not installed in the test vCluster. No DGDR tests ran.
❌ DGDR Deploy Test / CPU / profiling: CRD dynamographdeploymentrequests.nvidia.com not found
Job: DGDR Deploy Test / CPU / profiling · Failed step: Verify DGDR test prerequisites · Logs: gh run view --job 112788191623 -R ai-dynamo/dynamo --log-failed
namespace/dgdr-profiling-37604627410-2 created
secret/hf-token-secret created
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "dynamographdeploymentrequests.nvidia.com" not found
##[error]Process completed with exit code 1.
The namespace and HF token secret were created, but kubectl wait for the DGDR CRD failed with NotFound: the CRD does not exist in the vCluster this job uses.
❌ DGDR Deploy Test / CPU / validation: CRD dynamographdeploymentrequests.nvidia.com not found
Job: DGDR Deploy Test / CPU / validation · Failed step: Verify DGDR test prerequisites · Logs: gh run view --job 112788191652 -R ai-dynamo/dynamo --log-failed
namespace/dgdr-validation-37604627410-2 created
secret/hf-token-secret created
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "dynamographdeploymentrequests.nvidia.com" not found
##[error]Process completed with exit code 1.
Same failure as the profiling job: the DGDR CRD is missing from the vCluster.
❌ DGDR Deploy Test / CPU / lifecycle: CRD dynamographdeploymentrequests.nvidia.com not found
Job: DGDR Deploy Test / CPU / lifecycle · Failed step: Verify DGDR test prerequisites · Logs: gh run view --job 112788192320 -R ai-dynamo/dynamo --log-failed
namespace/dgdr-lifecycle-37604627410-2 created
secret/hf-token-secret created
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "dynamographdeploymentrequests.nvidia.com" not found
##[error]Process completed with exit code 1.
Same failure as the profiling job: the DGDR CRD is missing from the vCluster.
For agents
{"pr": 15804, "run_id": 37604627410, "run_attempt": 2, "head_sha": "babe0a90be6ec514799523f9fb31174e32fe9cd5",
"failures": [
{"job": "DGDR Deploy Test / CPU / profiling", "job_id": 112788191623, "failed_step": "Verify DGDR test prerequisites", "signature": "CRD dynamographdeploymentrequests.nvidia.com not found", "tests": [], "log_cmd": "gh run view --job 112788191623 -R ai-dynamo/dynamo --log-failed"},
{"job": "DGDR Deploy Test / CPU / validation", "job_id": 112788191652, "failed_step": "Verify DGDR test prerequisites", "signature": "CRD dynamographdeploymentrequests.nvidia.com not found", "tests": [], "log_cmd": "gh run view --job 112788191652 -R ai-dynamo/dynamo --log-failed"},
{"job": "DGDR Deploy Test / CPU / lifecycle", "job_id": 112788192320, "failed_step": "Verify DGDR test prerequisites", "signature": "CRD dynamographdeploymentrequests.nvidia.com not found", "tests": [], "log_cmd": "gh run view --job 112788192320 -R ai-dynamo/dynamo --log-failed"}
]}Posted automatically by Devin for run 37604627410. Updated on every full-CI run of this PR.
Why
The LPX renderer writes the Cyborg
lpu_serversConfigMap only for XT (LP20) hybrid builds. For HX (LP30) hybrid builds, the Cyborg conductor'sconfigvolume pointed at the Agents' partition table instead. Cyborg'scyborg-entrypoint --expand-hostsreads/configs/lpu_servers, so:lpx.experimental.localPartitions) fails at startup on the missing file, andWhat
configvolume to it, as for XT.<scaling group>-${GROVE_PCSG_INDEX}-<agent template>-<node offset>. HX keeps selected prop-sync chain members as separate physical partitions, so each chain is addressed through its first member. XT already collapses the chain the same way.lpu_servers, which matches XT.Changes stay under
deploy/operator/internal/dynamo/lpx/. They include updated HX hybrid goldens (mixed GPT-OSS, all-local, and all-LPU) and a focused render test that covers chains and local selections.Testing
go test ./internal/...(fromdeploy/operator, with envtest assets)golangci-lintv1.64.8 on./internal/dynamo/lpx/...and./internal/controller/lpx/...🤖 Generated with Claude Code
Summary by CodeRabbit