Repository navigation
cp: docs(token-capture): fix external agent harness guide (4102) into r0.7.0 - #4133
Merged
Merged
Conversation
## What changed and why The "Training on External Agent Harnesses" page received QA feedback (NVBug 6848266) that it was misleading in places, had snippets that could not run, and lacked a first-run workflow. This PR updates `fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx` only. - **Tool calling.** The page told readers to set `tool_parser: hermes`. vLLM also needs automatic tool choice, so following the old text could leave tool calls disabled. The step now shows both `vllm serve` flags (`--enable-auto-tool-choice --tool-call-parser`) and both NeMo RL keys (`http_server_serving_chat_kwargs.enable_auto_tools` and `tool_parser`). - **Runnable consumer example.** The framework-side snippet used undefined names (`source`, `downstream`, `rollout_id`). It is now a complete async `deliver_rollout(result, source, write_durably)` function that rebuilds, writes durably, and then retires. A separate complete example shows that `run_conformance` is async and takes factories. - **Terminal attribution.** The metrics table listed `delivered` as the only healthy value of `terminal_attribution.chain`. An unattributed rollout with exactly one chain is also kept unmasked. The table and the text after it now explain this, matching the masking logic in `nemo_gym/token_id_capture/consumer.py`. - **First-run walkthrough.** A new Claude Code section covers `vllm serve`, a `token_capture.yaml` overlay, `gym env start --config`, `gym eval run --no-serve`, and a `jq` check of `_ng_token_capture` and `mask_sample`. - **Custom harness routing.** A new step explains the `/ng-rollout/<rollout_id>/training-token-capture` URL prefix with a concrete URL, says what happens without the training segment, and links to the model-call capture page. - **Audience.** Framework storage, consumer-side delivery, conformance, and unique rollout ids now sit under an "Advanced framework integration" heading after "Current limitations". The YAML snippets used NeMo RL's `env.nemo_gym` nesting, which does not work as a `gym env start --config` overlay. They now use top-level Gym keys, and one sentence tells NeMo RL users to nest them under `env.nemo_gym`. A live run of the walkthrough found two more problems, both fixed here: - With `--no-serve`, the rollout collector reads capture settings from the `gym eval run` command, not from the running servers. If the overlay goes only to `gym env start`, the model server records calls, but the collector writes text-only rollouts that have no `_ng_token_capture` field and are not masked. The walkthrough now passes `--config token_capture.yaml` to both commands, and the page explains why. Making the collector use the servers' settings is a code change and will be a separate PR. - Claude Code requests 32,000 output tokens on every call. The walkthrough's `vllm serve` now sets `--max-model-len 65536`, which leaves room for that request and fits the model on one 48 GB GPU. Without the flag, vLLM 0.29.0 fails to start on that GPU, because the model's default 262,144-token context needs more KV cache than the GPU has. ## Related issue NVBug 6848266 (internal QA tracker). No GitHub issue exists for it. ## Validation - Ran the walkthrough on a local RTX 6000 Ada against vLLM 0.29.0, the version pinned in `pyproject.toml`, serving `Qwen/Qwen3-4B-Thinking-2507`. Every rollout came back rebuilt and unmasked, with `n_calls: 1`, `chains: 1`, an empty `terminal_attribution.chain`, and `delivered_fraction: 1.0`. The rebuilt messages carry generation token ids with matching log probabilities. Rewards were 0 because the model's `<think>` text ended up in the answer the verifier extracted. That is a model and environment pairing issue, not a capture issue. The tasks did not need tools, so tool calling itself was not exercised. - Ran the `deliver_rollout` and `run_conformance` examples against Gym's built-in file store as a stand-in for `my_package.capture`. The conformance checks passed, and the delivery example masked an empty capture as expected. - `gym env resolve` composes the overlay with the training model config and the Claude Code environment as intended. - Tested the `jq` command on sample rows. - `fern check` reported no errors. Its one warning is the redirects check, which is skipped without Fern authentication and also appears on `main`. - `pre-commit run --files` on the changed page. ## Rollout evidence The Claude Code walkthrough described under Validation was run against a live vLLM server. No other behavior changed, because this PR edits documentation only. ## Compatibility Documentation only. No API, config, or benchmark-result changes. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
svcnvidia-nemo-ci
added
area:training
Contributor
Author
|
/ok to test d66bceb |
ananthsub
approved these changes
Oct 6, 2026
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4133-d66bcebacbe4.docs.buildwithfern.com/nemo/gym |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beep boop [🤖]: Hi @ananthsub 👋,