Skip to content

[Stirrup] Lift LLM kwargs from config and clear stale deliverables - #1187

Merged
bxyu-nvidia merged 2 commits into
NVIDIA-NeMo:mainfrom
Kh4L:stirrup-config-and-stale-deliverables
Apr 30, 2026
Merged

bxyu-nvidia merged 2 commits into
NVIDIA-NeMo:mainfrom
Kh4L:stirrup-config-and-stale-deliverables

Conversation

@Kh4L

@Kh4L Kh4L commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Two issues surfaced while validating the agent against an RL training loop:

1. Stale deliverables leak across rollouts

responses_api_agents/stirrup_agent/app.py:347 does task_dir.mkdir(parents=True, exist_ok=True) and then copies the current run's output files into task_dir. It never clears the directory first, and only same-named files are overwritten by shutil.copy2. When an agent revisits the same (task_id, rollout_index) (e.g. across RL training steps that reuse persist_deliverables_dir), files from the prior visit linger in repeat_<n>/. The judge in resources_servers/gdpval/app.py::_verify_rubric calls read_deliverable_files(body.deliverables_dir) and convert_deliverables_to_content_blocks(...) against the whole directory, so stale artifacts from a prior step contaminate the reward signal.

Fix: shutil.rmtree(task_dir, ignore_errors=True) immediately before the mkdir. Each visit gets a clean directory.

2. Hardcoded LLM sampling kwargs and per-call completion cap

nemo_client.DynamicMaxTokensChatCompletionsClient.generate hardcoded temperature=1.0, top_p=0.95, extra_body.chat_template_kwargs.enable_thinking=True, and _MAX_COMPLETION_TOKENS = 64000. Although temperature is plumbed all the way through _run_stirrup_agent, the client never consumed it β€” every call ignored the configured value and ran at 1.0. The 64K cap on per-call max_completion_tokens similarly couldn't be tuned to match a training-side response-length budget.

Fix: expose the four values as constructor params, with the historical hardcoded values as defaults, and surface them on StirrupAgentWrapperConfig:

  • temperature β€” already a config field; now actually reaches the client.
  • top_p: float = 0.95 (new field)
  • enable_thinking: bool = True (new field)
  • max_completion_tokens_cap: int = 64000 (new field)

top_p also accepts a per-request override via body.top_p, mirroring how temperature and max_output_tokens already work.

Test plan

  • New unit tests in responses_api_agents/stirrup_agent/tests/test_nemo_client.py cover the kwargs landing on the wire request, the max_completion_tokens_cap ceiling, and the unchanged defaults.
  • CI green.

Signed-off-by: Serge Panev <spanev@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

bxyu-nvidia
bxyu-nvidia previously approved these changes Apr 30, 2026
Signed-off-by: Serge Panev <spanev@nvidia.com>
@bxyu-nvidia
bxyu-nvidia merged commit 5abd9a0 into NVIDIA-NeMo:main Apr 30, 2026
6 checks passed
cmunley1 pushed a commit that referenced this pull request May 1, 2026
…1187)

## Summary

Two issues surfaced while validating the agent against an RL training
loop:

### 1. Stale deliverables leak across rollouts

`responses_api_agents/stirrup_agent/app.py:347` does
`task_dir.mkdir(parents=True, exist_ok=True)` and then copies the
current run's output files into `task_dir`. It never clears the
directory first, and only same-named files are overwritten by
`shutil.copy2`. When an agent revisits the same `(task_id,
rollout_index)` (e.g. across RL training steps that reuse
`persist_deliverables_dir`), files from the prior visit linger in
`repeat_<n>/`. The judge in
`resources_servers/gdpval/app.py::_verify_rubric` calls
`read_deliverable_files(body.deliverables_dir)` and
`convert_deliverables_to_content_blocks(...)` against the whole
directory, so stale artifacts from a prior step contaminate the reward
signal.

Fix: `shutil.rmtree(task_dir, ignore_errors=True)` immediately before
the `mkdir`. Each visit gets a clean directory.

### 2. Hardcoded LLM sampling kwargs and per-call completion cap

`nemo_client.DynamicMaxTokensChatCompletionsClient.generate` hardcoded
`temperature=1.0`, `top_p=0.95`,
`extra_body.chat_template_kwargs.enable_thinking=True`, and
`_MAX_COMPLETION_TOKENS = 64000`. Although `temperature` is plumbed all
the way through `_run_stirrup_agent`, the client never consumed it β€”
every call ignored the configured value and ran at 1.0. The 64K cap on
per-call `max_completion_tokens` similarly couldn't be tuned to match a
training-side response-length budget.

Fix: expose the four values as constructor params, with the historical
hardcoded values as defaults, and surface them on
`StirrupAgentWrapperConfig`:

- `temperature` β€” already a config field; now actually reaches the
client.
- `top_p: float = 0.95` (new field)
- `enable_thinking: bool = True` (new field)
- `max_completion_tokens_cap: int = 64000` (new field)

`top_p` also accepts a per-request override via `body.top_p`, mirroring
how `temperature` and `max_output_tokens` already work.

## Test plan

- [x] New unit tests in
`responses_api_agents/stirrup_agent/tests/test_nemo_client.py` cover the
kwargs landing on the wire request, the `max_completion_tokens_cap`
ceiling, and the unchanged defaults.
- [ ] CI green.

---------

Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: cmunley1 <cmunley@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants