🔥 Jadwal & Hasil: NVIDIA NeMo/Gym/pull/

adil-a · 2026-08-24T17:59:15Z

Problem

Dataset rows carry agent_ref (the agent that runs them), stamped at data prep. Two problems today:

You cannot re-route a stamped dataset. --agent only fills rows that have no agent_ref. It never overrides one. To run the same data with a different agent, you must regenerate the dataset.
A stale agent_ref crashes mid-run. If a row names an agent not in the running config, dispatch throws a raw ConfigKeyError after earlier rows were already sent. 7 committed datasets in this repo hit this today.

Change

1. New agent_map config: explicit re-routing.

# Before: train.jsonl stamped with agent A. To run it with agent B: regenerate the dataset.
# After:
gym eval run --no-serve -i train.jsonl '+agent_map={agent_a: agent_b}'

agent_map is a lookup table: "when a row says this agent, send it to that agent instead." Keys are agent_ref.name values found in the data. The reserved key _default is a catch-all: it matches every row that has no specific entry, including rows with no agent_ref at all.

Per row, first hit wins:

agent_map[<row's agent_ref.name>] → use it
agent_map._default → use it
row's own agent_ref → use it (no agent_ref either → hard error, same as today)

Worked example — input rows carry agent_a, agent_b, and (none):

You pass	`agent_a` row	`agent_b` row	no-ref row
(nothing)	agent_a	agent_b	error
`+agent_map={agent_a: agent_x}`	agent_x	agent_b	error
`+agent_map={agent_a: agent_x, _default: agent_y}`	agent_x	agent_y	agent_y
`--agent agent_z`	agent_z	agent_z	agent_z

2. --agent X now means agent_map={_default: X}: it routes every row to X.

# Before: --agent B only affected rows missing agent_ref; stamped rows kept their agent silently.
# After:  --agent B routes ALL rows to B. A warning lists any values it overrode.

This restores the semantics #568 shipped. #761 changed it to fill-only with no test, no doc, and no mention in the commit message. No existing test pins fill-only.

3. Agent names are validated before dispatch.

# Before: first bad row → raw omegaconf ConfigKeyError mid-collection, partial output on disk.
# After, before any request is sent:
ValueError: Rows reference agents not present in the running config:
'math_with_judge_simple_agnet' (did you mean 'math_with_judge_simple_agent'?).
Include the agent's config in the run, or re-route with +agent_map/+agent_name.

What does NOT change

Rows with no agent_ref and no --agent: same hard error as before.
Runs that never pass --agent/agent_map: identical routing. Verified by replaying rollout preprocessing + dispatch for all 220 runnable config closures in the repo: 165 healthy inputs are byte-identical; the only diffs are the 7 stale-agent_ref datasets now failing pre-dispatch instead of mid-run.

Tests

98 passed. New contract tests pin the precedence order and the validation errors, so the semantics cannot flip silently again.

Context

Bottom PR of the dataset↔agent decoupling stack (RFC: gym-dataset-preparation, Open Question 1). Later PRs move routing out of dataset rows entirely; this PR makes today's stamped datasets re-routable and makes routing failures loud.

🤖 Generated with Claude Code

copy-pr-bot · 2026-08-24T17:59:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ananthsub · 2026-08-25T07:37:12Z

+        after valid rows have already been dispatched.
+        """
+        requested = {name for row in examples if (name := (row.get(AGENT_REF_KEY_NAME) or {}).get("name")) is not None}
+        available = set(global_config_dict.keys())


This treats every top-level config key as a valid agent. An agent_map target naming a resources or model server passes validation, then receives a /run request that it cannot handle.

This should filter to responses_api_agents

Fixed in 58d45a8: available now filters to instances with a responses_api_agents block, and a target that exists but is not an agent fails pre-dispatch with '<name>' (exists but is not an agent instance). Test added (agent_map target naming an RS instance). This check previously sat one PR up the stack; moved it here, where the validator is introduced.

adil-a · 2026-08-26T02:22:13Z

/ok to test 58d45a8a4fdef043a129ef2436cbcc9be3132b6c

copy-pr-bot · 2026-08-26T02:22:16Z

/ok to test 58d45a8a4fdef043a129ef2436cbcc9be3132b6c

@adil-a, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

adil-a · 2026-08-26T04:26:21Z

/ok to test 58d45a8

adil-a · 2026-08-26T04:58:21Z

/ok to test 58d45a8

adil-a · 2026-08-26T06:20:02Z

/ok to test 8025f00

…ent override Review feedback on #2710: 1. _validate_agent_names accepted any top-level config key, so an agent_map target naming a resources or model server passed validation and then received a /run request it cannot handle. The check now filters to instances with a responses_api_agents block; naming an existing non-agent instance fails pre-dispatch with 'exists but is not an agent instance'. (This check previously landed one PR up the stack; it belongs here, where the validator is introduced.) 2. The reward-profiling quick-start still described +agent_name as fill-only ('supplies a default for rows without one'). Updated: it routes every row, overriding baked agent_ref (shorthand for +agent_map={_default: <name>}), which is this PR's restored #568 override semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

adil-a · 2026-08-26T19:32:16Z

/ok to test f08b1ae

…ent override Review feedback on #2710: 1. _validate_agent_names accepted any top-level config key, so an agent_map target naming a resources or model server passed validation and then received a /run request it cannot handle. The check now filters to instances with a responses_api_agents block; naming an existing non-agent instance fails pre-dispatch with 'exists but is not an agent instance'. (This check previously landed one PR up the stack; it belongs here, where the validator is introduced.) 2. The reward-profiling quick-start still described +agent_name as fill-only ('supplies a default for rows without one'). Updated: it routes every row, overriding baked agent_ref (shorthand for +agent_map={_default: <name>}), which is this PR's restored #568 override semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

adil-a · 2026-08-26T20:06:54Z

/ok to test 0c45b6b

…ent override Review feedback on #2710: 1. _validate_agent_names accepted any top-level config key, so an agent_map target naming a resources or model server passed validation and then received a /run request it cannot handle. The check now filters to instances with a responses_api_agents block; naming an existing non-agent instance fails pre-dispatch with 'exists but is not an agent instance'. (This check previously landed one PR up the stack; it belongs here, where the validator is introduced.) 2. The reward-profiling quick-start still described +agent_name as fill-only ('supplies a default for rows without one'). Updated: it routes every row, overriding baked agent_ref (shorthand for +agent_map={_default: <name>}), which is this PR's restored #568 override semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

adil-a · 2026-08-26T20:08:39Z

/ok to test 10dce44

…ent override Review feedback on #2710: 1. _validate_agent_names accepted any top-level config key, so an agent_map target naming a resources or model server passed validation and then received a /run request it cannot handle. The check now filters to instances with a responses_api_agents block; naming an existing non-agent instance fails pre-dispatch with 'exists but is not an agent instance'. (This check previously landed one PR up the stack; it belongs here, where the validator is introduced.) 2. The reward-profiling quick-start still described +agent_name as fill-only ('supplies a default for rows without one'). Updated: it routes every row, overriding baked agent_ref (shorthand for +agent_map={_default: <name>}), which is this PR's restored #568 override semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

adil-a · 2026-08-26T20:30:22Z

/ok to test 8aeeb09

Two changes to how rows are routed to agents at collection time: 1. agent_map: explicit per-agent re-routing, keyed by the agent_ref.name in the data, with a _default entry applying to unmatched rows (including rows with no agent_ref). Precedence: agent_map[<row value>] > agent_map._default > row agent_ref. agent_name becomes sugar for agent_map={_default: ...} and therefore re-routes every row again, restoring the override semantics #568 shipped and #761 silently reduced to fill-only (no test or doc pinned the fill-only behavior). A warning enumerates overridden values for one cycle. 2. Pre-dispatch validation: before any /run is sent, every distinct agent name in the rows is checked against the running config; unknown names fail with a did-you-mean error instead of a raw omegaconf ConfigKeyError thrown mid-collection after valid rows were already dispatched. New contract tests pin the precedence rules so future semantic changes cannot land silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

…g_dict Pre-dispatch agent validation reads the running config off the server client; the in-process mock lacked the attribute, failing the server suite (shard 7). Only server test that drives the collector directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

…ent override Review feedback on #2710: 1. _validate_agent_names accepted any top-level config key, so an agent_map target naming a resources or model server passed validation and then received a /run request it cannot handle. The check now filters to instances with a responses_api_agents block; naming an existing non-agent instance fails pre-dispatch with 'exists but is not an agent instance'. (This check previously landed one PR up the stack; it belongs here, where the validator is introduced.) 2. The reward-profiling quick-start still described +agent_name as fill-only ('supplies a default for rows without one'). Updated: it routes every row, overriding baked agent_ref (shorthand for +agent_map={_default: <name>}), which is this PR's restored #568 override semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

adil-a mentioned this pull request Aug 24, 2026

feat(rollout): task_source resolution and run-level fan-out #2713

Merged

adil-a force-pushed the data-prep/agent-map branch from cc59042 to 31518cb Compare August 24, 2026 22:43

ananthsub reviewed Aug 25, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public August 26, 2026 04:26 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 04:28 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 04:29 Inactive

adil-a force-pushed the data-prep/agent-map branch from 58d45a8 to 8025f00 Compare August 26, 2026 06:20

copy-pr-bot Bot temporarily deployed to public August 26, 2026 06:20 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 06:22 Inactive

marta-sd force-pushed the data-prep/agent-map branch from 8025f00 to de42916 Compare August 26, 2026 10:08

marta-sd approved these changes Aug 26, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public August 26, 2026 10:09 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 10:10 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 10:11 Inactive

marta-sd force-pushed the data-prep/agent-map branch from de42916 to 397cc06 Compare August 26, 2026 10:25

copy-pr-bot Bot temporarily deployed to public August 26, 2026 10:25 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 10:27 Inactive

adil-a force-pushed the data-prep/agent-map branch from fa997c5 to f08b1ae Compare August 26, 2026 19:31

copy-pr-bot Bot temporarily deployed to public August 26, 2026 19:32 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 19:34 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 19:35 Inactive

adil-a force-pushed the data-prep/agent-map branch from f08b1ae to 0c45b6b Compare August 26, 2026 20:06

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:07 Inactive

adil-a force-pushed the data-prep/agent-map branch from 0c45b6b to 10dce44 Compare August 26, 2026 20:08

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:08 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:10 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:11 Inactive

adil-a force-pushed the data-prep/agent-map branch from 10dce44 to 8aeeb09 Compare August 26, 2026 20:30

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:30 Inactive

copy-pr-bot Bot temporarily deployed to public August 26, 2026 20:32 Inactive

adil-a and others added 3 commits August 26, 2026 16:02

kajalj22 force-pushed the data-prep/agent-map branch from 8aeeb09 to e5fe62b Compare August 26, 2026 21:03

adil-a mentioned this pull request Aug 28, 2026

feat(benchmarks): explicit agent pin on benchmark datasets; scaffold generates the decoupled layout #2724

Merged

qiaochuz-nv mentioned this pull request Oct 12, 2026

fix(grl): start GRL READMEs without the unbound gymnasium_agent and keep agent_ref on --agent #4544

Merged

Conversation

adil-a commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

What does NOT change

Tests

Context

Uh oh!

copy-pr-bot Bot commented Aug 24, 2026

Uh oh!

ananthsub Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

adil-a Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

copy-pr-bot Bot commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

adil-a commented Aug 26, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adil-a commented Aug 24, 2026 •

edited

Loading