Repository navigation
feat(orchestration): served_model_name on model-serving services - #3323
Merged
Merged
Conversation
driver.policy_model's auto-wiring set policy_model_name from a service's `model` field verbatim (SubmitConfig._resolve_and_validate_placements), with no way to know that a user's own extra_args might tell the engine to serve under a different name (e.g. --served-model-name super-bf16 when model: is a local checkpoint path like /checkpoint). The evaluation harness then requested the model by the wrong name and got rejected -- vLLM only recognizes the name(s) it was actually told to serve under. served_model_name on BaseModelServiceConfig (not VllmServiceConfig specifically -- the auto-wiring already treats any BaseModelServiceConfig generically, and any future model-serving service type gets this for free) gives both sides one source of truth: _vllm_base_flags emits --served-model-name from it, and policy_model_name auto-wiring reads the same field. Falls back to `model` when unset, so existing configs are unaffected. Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>
yaoyu-33
added
area:orchestration
marta-sd
previously approved these changes
Sep 11, 2026
Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>
marta-sd
approved these changes
Sep 11, 2026
prokotg
enabled auto-merge (squash)
September 11, 2026 13:32
6 of 9 tasks
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
## Summary Add `served_model_name` to `BaseModelServiceConfig` so vLLM (and any future model-serving service type) can be told to serve a model under a name different from its `model` field, and so `driver.policy_model` auto-wiring picks up that same name. - Previously, `driver.policy_model`'s auto-wiring set `policy_model_name` from `service.model` verbatim (`SubmitConfig._resolve_and_validate_placements`), with no way to account for `extra_args` telling the engine to serve under a different name (e.g. `--served-model-name super-bf16` when `model:` is a local checkpoint path like `/checkpoint`). The evaluation harness then requested the model by the wrong name and vLLM rejected it, since it only recognizes the name(s) it was actually told to serve under. - New `served_model_name: str | None` field on `BaseModelServiceConfig` (not `VllmServiceConfig`-specific, so any future model-serving service type gets this for free) becomes the single source of truth for both sides: `_vllm_base_flags` emits `--served-model-name` from it, and `policy_model_name` auto-wiring reads the same field. - Falls back to `model` when unset, so existing configs are unaffected. ## Changes - `nemo_gym/orchestration/api.py` — `served_model_name` field on `BaseModelServiceConfig`; `policy_model_name` auto-wiring uses `service.served_model_name or service.model` - `nemo_gym/orchestration/executors/slurm_script.py` — `_vllm_base_flags` emits `--served-model-name` (shell-quoted) when set - `tests/unit_tests/test_orchestration_api.py`, `tests/unit_tests/test_slurm_script.py` — coverage for the new field, its default fallback, and quoting ## Test plan - [ ] `pytest tests/unit_tests/test_orchestration_api.py -x` - [ ] `pytest tests/unit_tests/test_slurm_script.py -x` --------- Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
served_model_nametoBaseModelServiceConfigso vLLM (and any future model-serving service type) can be told to serve a model under a name different from itsmodelfield, and sodriver.policy_modelauto-wiring picks up that same name.driver.policy_model's auto-wiring setpolicy_model_namefromservice.modelverbatim (SubmitConfig._resolve_and_validate_placements), with no way to account forextra_argstelling the engine to serve under a different name (e.g.--served-model-name super-bf16whenmodel:is a local checkpoint path like/checkpoint). The evaluation harness then requested the model by the wrong name and vLLM rejected it, since it only recognizes the name(s) it was actually told to serve under.served_model_name: str | Nonefield onBaseModelServiceConfig(notVllmServiceConfig-specific, so any future model-serving service type gets this for free) becomes the single source of truth for both sides:_vllm_base_flagsemits--served-model-namefrom it, andpolicy_model_nameauto-wiring reads the same field.modelwhen unset, so existing configs are unaffected.Changes
nemo_gym/orchestration/api.py—served_model_namefield onBaseModelServiceConfig;policy_model_nameauto-wiring usesservice.served_model_name or service.modelnemo_gym/orchestration/executors/slurm_script.py—_vllm_base_flagsemits--served-model-name(shell-quoted) when settests/unit_tests/test_orchestration_api.py,tests/unit_tests/test_slurm_script.py— coverage for the new field, its default fallback, and quotingTest plan
pytest tests/unit_tests/test_orchestration_api.py -xpytest tests/unit_tests/test_slurm_script.py -x