Repository navigation
Bound text-only continuation nudges - #3198
Merged
Merged
Conversation
sgharghabi
marked this pull request as draft
September 9, 2026 23:06
sgharghabi
force-pushed
the
fix/finance-agent-bounded-continuations
branch
2 times, most recently
from
September 10, 2026 16:53
d98bee6 to
e1724cc
Compare
sgharghabi
marked this pull request as ready for review
September 10, 2026 18:10
ushnish-de
approved these changes
Sep 10, 2026
bxyu-nvidia
approved these changes
Sep 10, 2026
Contributor
|
/ok to test db9060f |
Add continue_if_not_tool_call (default true) so Gym eval still injects the no-tool-call nudge until max_steps or time. GRPO overlays set false to end the episode on the first text-only turn and keep NeMo-RL trajectories monotonic. Record stop_reason text_only when that path ends the loop. Signed-off-by: Shaya Gharghabi <sgharghabi@nvidia.com>
sgharghabi
force-pushed
the
fix/finance-agent-bounded-continuations
branch
from
September 10, 2026 18:20
db9060f to
5e2aece
Compare
sgharghabi
enabled auto-merge (squash)
September 10, 2026 18:24
Contributor
Author
|
/ok to test 5e2aece |
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
Disable synthetic user nudges for GRPO finance training to preserve NeMo-RL monotonic trajectories. Keep Gym eval on the default (nudge and continue), and record `text_only` as the stop reason. ## What does this PR do? Stops the finance agent from injecting a synthetic user message (`Continue.` / the v2 nudge) on a text-only turn when GRPO training turns the flag off. That injection is a new user turn. On reasoning models, the chat template then treats earlier assistant turns as “before the last user message” and strips their `<think>` blocks. The re-templated prompt is no longer a prefix of the tokens NeMo-RL already recorded, which trips `non-monotonically increasing trajectory` and then the fatal `Non-contiguous messages found`. A 50-turn **tool** loop is fine; a single nudge is not. This PR: - Adds `continue_if_not_tool_call` (bool, **default `true`**): - **Eval** (shipped `finance_agent.yaml` / `finance_agent_v2.yaml`): omit the field; inherit `true`. A text-only turn injects the nudge and the loop continues until `max_steps` / time. - **Training** (NVFlow GRPO overlay): `false`. The first text-only reply ends the episode. No nudge. - Records `stop_reason: text_only` when that path ends the loop, so a zero from “never submitted” is distinguishable from “judged wrong” or `max_turns`. - Leaves `max_steps: 50` as the overall turn budget for real tool-calling episodes. `max_steps` still caps all model turns. `continue_if_not_tool_call` only applies when a turn is prose with no tool call, so training does not burn 50 steps on an all-text loop. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Shaya Gharghabi <sgharghabi@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disable synthetic user nudges for GRPO finance training to preserve NeMo-RL monotonic trajectories. Keep Gym eval on the default (nudge and continue), and record
text_onlyas the stop reason.What does this PR do?
Stops the finance agent from injecting a synthetic user message (
Continue./ the v2 nudge) on a text-only turn when GRPO training turns the flag off.That injection is a new user turn. On reasoning models, the chat template then treats earlier assistant turns as “before the last user message” and strips their
<think>blocks. The re-templated prompt is no longer a prefix of the tokens NeMo-RL already recorded, which tripsnon-monotonically increasing trajectoryand then the fatalNon-contiguous messages found. A 50-turn tool loop is fine; a single nudge is not.This PR:
continue_if_not_tool_call(bool, defaulttrue):finance_agent.yaml/finance_agent_v2.yaml): omit the field; inherittrue. A text-only turn injects the nudge and the loop continues untilmax_steps/ time.false. The first text-only reply ends the episode. No nudge.stop_reason: text_onlywhen that path ends the loop, so a zero from “never submitted” is distinguishable from “judged wrong” ormax_turns.max_steps: 50as the overall turn budget for real tool-calling episodes.max_stepsstill caps all model turns.continue_if_not_tool_callonly applies when a turn is prose with no tool call, so training does not burn 50 steps on an all-text loop.Checklist
pre-commit run --all-files) (so CI lint/format/copyright pass).git commit -s) (so the DCO check passes).