Repository navigation
feat(docs) recipes - #2471
Merged
Merged
feat(docs) recipes#2471
Conversation
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Adds the base (pretraining) evaluation recipes for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 under scripts/more/base/, alongside the instruct recipes already in scripts/more/. - local_..._base_endpoint.yaml 21 short-context tasks against your endpoint - local_..._base_ruler.yaml RULER 64k-1M, needs a long-context endpoint - README.md setup, per-task notes, known issues They are kept in their own subdirectory because they work differently from the instruct recipes: these are nemo-evaluator-launcher configs run with `nel run --config ...`, not Gym CLI scripts, since the base benchmarks are lm-evaluation-harness and nemo-skills tasks rather than Gym environments. reproducibility.md gains a pointer so they are discoverable from the index. Both configs run against an endpoint you provide; neither deploys anything. RULER needs a separately served long-context endpoint, because the checkpoint declares max_position_embeddings 262144 and anything past 256k is refused unless the server was started to allow it. The README carries both serve commands. Validation: - 20 of the 21 short-context tasks succeed; of the 17 with a reference value, all 17 match it within 0.02 absolute and 13 within 0.006. - The 9 log-likelihood tasks were run at full sample counts against two independently served endpoints (vLLM 0.19.1 TP=4 and 0.26.0 TP=2); maximum deviation between them was 0.0071. Two known issues are documented rather than worked around: adlr_minerva_math_nemo_4_shot fails in lm-evaluation-harness:26.03 on a single dataset document, and the RULER 512k/1M lengths run beyond the checkpoint's 262144 position range. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Add a "Resuming an interrupted run" section: RESUME=1 on every gym-native recipe, the nel-next equivalents (nel eval run --resume / nel eval resume), and the two benchmark-specific traps -- GDPval's multistage journal marking a stage complete even when all of its rollouts failed, and CritPt's replay path for a spent Artificial Analysis quota. Also correct the usage paths throughout: commands are shown as scripts/more/<recipe>.sh rather than ./<recipe>.sh, since every recipe must be run from the repo root to resolve its dataset and prepare script. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Rewrite the recipe as a single `gym eval run` matching the published run's config field by field: num_repeats 5 on the stock dataset, concurrency 350, verify_timeout_seconds 21600, thinking on, skip_special_tokens false, sequential_reasoning_allowed false, no token cap. Concurrency is the substantive fix. Scoring buffers 70 distinct problems per call and a rollout waiting to be scored holds its slot, while repeats of a problem are placed in different batches -- so any value between 71 and 349 spreads the live rollouts across batches that can never fill, and the run wedges until verify_timeout expires. The previous header recommended PARALLEL=140, which fails exactly this way. 350 keeps every row in flight. Drop the interleaved-dataset generator: it existed to make a 350-row file fill its first batch early, which is unnecessary once every row is live. reproducibility.md gains the fallback for endpoints that cannot take 350 -- five single-repeat runs at 70, averaging mean/reward, which produces the same five scoring calls and the same mean. Also corrects the multi-key note: keys fail over when one is rate-limited, they do not spread load. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Group the recipes by the model they evaluate rather than leaving them flat: reproducibility.md what can be reproduced, plus the benchmark tables base/README.md the pretraining suite (unchanged) instruct/README.md install, running, per-benchmark setup, resuming instruct/gym/ the ten Gym recipes instruct/nemo-evaluator/ the three NeMo Evaluator configs base/ already owned its own setup, so instruct/ now owns instruct's; the two families genuinely install differently. Deliberately no README inside gym/ or nemo-evaluator/ -- those are code folders, and a fourth level would put the prerequisites in two places and split "how do I reproduce X" across four files. nel-next/ becomes nemo-evaluator/: nel-next is internal shorthand, and the configs belong to NVIDIA-NeMo/Evaluator. Folder names stay lowercase and free of spaces because they appear in every command in the docs. Fixes a real path bug this move exposed. The SWE-bench configs carried instruction_template: nel-next/prompts/swebench_instruction.md which nemo-evaluator resolves against the working directory (`Path.cwd() / p`), while the docs tell you to run from the repo root -- where that path does not exist. It is now a full path from the repo root. The neighbouring `_base_: prompts/...` needs no change: `_find_base` searches the referring config's own directory first. Also carried in: the recipe usage lines now show the path to run from the repo root instead of ./<recipe>.sh, and pinchbench sets NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=10 itself rather than .env.example telling every benchmark to export it -- only pinchbench's published run raised it. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
…ones Follow-up to the initial drop, aligning scripts/more/base/ with the conventions the instruct recipes settled on: - Rename the configs to match their siblings: base-suite.yaml and ruler.yaml, instead of carrying the upstream Evaluator filenames. The `local_` prefix encoded an execution mode that no longer applies now both configs are deployment: none, and the model name is already implied by the directory. - Use POLICY_API_KEY for the model under test, matching the instruct recipes and instruct/gym/.env.example. It was INFERENCE_API_KEY, so anyone filling in one .env would have met two names for the same thing. - Match the endpoint placeholder style used by the instruct NeMo Evaluator configs: https://<your-endpoint>/v1/completions and <your-model-id>. The route stays /v1/completions, which is the part that actually differs for a base model. - Fix a stale cross-reference that sent readers to the Nemotron 3 Ultra recipes in another repository for the instruct recipes, which now live next door in ../instruct/. - Trim the README preamble that repeated what reproducibility.md now explains about the base/instruct split, and link there instead. - State that no judge is required. Every task here is scored deterministically by lm-evaluation-harness or nemo-skills, so unlike several instruct recipes there is no JUDGE_API_KEY and no judge endpoint to configure. The README now says exactly which endpoints are needed instead: one completions endpoint that honours echo with logprobs, plus a second long-context one for RULER. Both configs re-verified with nel run --dry-run: they compose, all tasks resolve, and neither emits a deployment step. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Document how the published runs served the model: the hosted endpoint, and the vLLM command these recipes were validated with, pointing at the HF model card as the authoritative serving guidance. Note the single deployment difference for the NeMo Evaluator recipes so one endpoint covers everything here. Remove "Resuming an interrupted run": RESUME is already in the env-var list at the top, and the CritPt replay command is already in the CritPt section with a more direct cache path. The GDPval multistage-journal caveat was the only fact with no other home, so it moves into the GDPval section; nel's --resume moves onto the nel eval run line. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
…he config
`agents.list[N].timeoutSeconds` is not a key OpenClaw 2026.6.5 accepts -- the
version pinned in Dockerfile.benchmark. Its schema is strict, so the one unknown
key invalidates the whole config file:
[reload] config reload skipped (invalid config):
agents.list.1: Unrecognized key: "timeoutSeconds"
The gateway then never registers the benchmark agent, so the agent never runs:
`Total API requests: 0`, no transcript, and every task scores 0 -- while the run
exits 0 with a full set of rollouts, so it reads as a very bad model rather than
a broken harness.
Introduced in f9a873d (#2211). Verified by A/B on one task, same image and
endpoint, toggling only this line:
with the line reward 0.0, transcript_event_count 0
without the line reward 1.0, transcript_event_count 7
The two sibling writes added by the same commit are kept -- OpenClaw accepts
both, and the passing run had them in place:
providers.custom.timeoutSeconds
agents.defaults.timeoutSeconds
The second of those already applies the timeout to every agent, so #2211's
intent survives; the per-agent write was redundant as well as fatal. Hunk header
adjusted from +322,110 to +322,109 so the patch still applies.
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The docs said LIMIT counts problems and that anything below 70 cannot be scored. Both are wrong. LIMIT counts rollouts, and the dataset-level num_repeats is materialized before --limit applies, with repeats grouped rather than interleaved. Checked directly: LIMIT=3 materializes three rows, all Challenge_1_main -- one distinct problem, not three. So the 70th distinct problem does not appear until row 346 of 350, and no limited run ever reaches a scoreable batch. The old wording invited LIMIT=70 as a smoke test, which instead hangs until verify_timeout_seconds expires. Also splits the recipe's combined note in two: the LIMIT behaviour and the concurrency floor are separate facts and were sharing a sentence. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
…idates the config" This reverts commit c6121ee. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The hosted endpoint may not be reachable for readers outside NVIDIA, and the model id it named does not resolve on that endpoint's public model list, so following it would fail at the first call. The section now points at the model card for authoritative serving guidance and keeps the validated vLLM command. Also drops the "Gym does not install on 3.12" aside; the version requirement above it already says what is needed. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
Signed-off-by: ritaneves <rfernandesne@nvidia.com>
Each Gym recipe now checks the repo out at the commit its tech report number was produced with before running, so a fresh clone reproduces the published figure rather than whatever main happens to be that day. `nemotron_recipes` is excluded from the checkout, so a running recipe never overwrites itself, and HEAD does not move. Pinning is on by default. PIN_GYM=0 runs against the current checkout and GYM_PIN overrides the commit. The checkout writes to tracked files, so the README and .env.example both say to commit or stash first. Most recipes pin v0.5.0 (e446e4f), the commit the reference runs used; gdpval pins 57c15a2 and pinchbench 86290ee, taken from the rundirs of their own reference runs. The NeMo Evaluator install is pinned the same way, replacing @main with 230c8411 -- the public commit matching the build the Terminal-Bench and SWE-bench numbers came from. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The pin used `git checkout <pin> -- .`, which stages everything it writes. After running any recipe the user's index held 265 staged reverts, ready to be swept into whatever they committed next. It also only wrote files that exist at the pin, so several hundred newer ones survived alongside them: the tree was pin-plus-current rather than the pin, and running two recipes with different pins layered one on the other. `git restore --source` writes the working tree without touching the index, and deleting tracked files absent from the pin makes the result exact. Running gpqa (e446e4f), then gdpval (57c15a2), then gpqa again now leaves the tree byte-identical to that pin every time with nothing staged, and `git restore .` returns the checkout to where it started. Also drops the `# v0.5.0` label from eight recipes: the pin is not that tag. v0.5.0 is 6e50df7, 26 commits later; the pinned tree calls itself 0.5.0rc0. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
… note GNU comm collates in the caller's locale, so sorting the two file lists with LC_ALL=C and then comparing them under a UTF-8 locale made it print "file 1 is not in sorted order" on every run. It happened to still produce the right set here, but comm only warns and carries on, so that is luck rather than guarantee. Running comm itself under LC_ALL=C makes the collation match the sort. Verified on Linux (git 2.43, GNU coreutils): three pins in sequence, each leaves the tree byte-identical to that pin with nothing staged, and no warnings. The README note also went denser than it needed to be while getting accurate. Same facts, closer to the length it was. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The pin block deleted tracked files that do not exist at the pinned commit, so the tree matched the pin exactly. That guarded a real mechanism -- Gym walks the tree in agent_registry.py and globs benchmarks/*/config.yaml, so leftover newer directories are visible to pinned code -- but no run has actually failed because of it, and the GPQA smoke completed fine with all of them present. Shipping `xargs -0 rm -f` into a user's checkout to insure against that is the wrong trade: one bad pathspec deletes their work. Restoring the pinned files and leaving newer ones alone is the smaller, more honest operation, so the README now says plainly that files added after the pin stay where they are. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The previous commit said files added to Gym after the pinned commit stay where they are. They do not: `git restore --source` removes paths that match the pathspec but do not exist in the source, so the pin already lands on an exact tree. Measured on Linux -- 362 tracked files absent from the pin, 0 left after the restore line alone, which is also why the cleanup that commit removed was redundant rather than merely risky. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
tau3 could not run pinned. It calls `gym env prefetch` to build the tau2 agent venv, and that subcommand landed in e590c4f on 2026-08-04 -- after all three pins. The next line hard-exits when the venv is missing, so the recipe died before reaching `gym eval`. `gym eval prepare` does not create the venv either (rc=0, 97 rows, venv still absent), and `gym env packages` only prints packages. Pinning tau3 to e590c4f, the first commit that has it. Nothing tau3 depends on moved in between: benchmarks/tau2 is unchanged and responses_api_agents/tau2 differs by one line in a test file. The pinned tree still reports 0.5.0rc0. The pin block was also in the wrong place in this recipe -- below the prefetch call and the two uv pip installs, so the agent venv was built against the unpinned tree and the pin then swapped the code beneath it. Every other recipe pins before its first gym call; this one now does too. Smoke tested at the new pin: rc=0, 2 rollouts, mean reward 1.0. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
The note said the recipes would move to nemotron_recipes/ before merging to main. That move has happened, so the note now describes the current location as if it were still pending. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Contributor
|
/ok to test |
@gchlebus, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
Contributor
|
/ok to test 01c1e34 |
gchlebus
previously approved these changes
Aug 11, 2026
…n's effect Three corrections found while smoke testing the recipes. The model id is `-BF16`, matching the published repository and the convention base/README.md already uses. It appeared in four places -- the model card link, the opening line, and both halves of the vllm command. The serve line needed it too: that argument is the same HuggingFace repo id as the link, so the bare form does not resolve. GDPval's second prerequisite was headed "Root, for the PDF conversion step", which scopes root to something a reader can reasonably decide they do not care about. Root is also needed for the sandbox, which starts with `--home /root` and cannot be entered by an ordinary user. That failure comes first and is the more severe one: every task dies and the run exits 0 having produced nothing, whereas the conversion failure only costs score. Both now sit under a plain "Root." The pin overwrites 265 files and deletes 361, so "overwrites tracked files" left out the larger half. `git restore .` still puts everything back, as the same sentence says. Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com>
gchlebus
approved these changes
Aug 11, 2026
gchlebus
added a commit
that referenced
this pull request
Oct 15, 2026
## What does this PR do? Adds `nemotron_recipes/super-3.5/`: Gym recipes reproducing the Nemotron 3.5 Super VL tech-report results (22 benchmarks), with `run.sh`, setup scripts, env templates and `reproducibility.md`. Follows the Lightning 3.5 recipes (#2471). No changes outside this folder. ## Validation Every benchmark was set up from scratch and fully evaluated, with results compared against the tech report; they reproduce the published numbers within run-to-run variation. ## Rollouts N/A: no environment or agent code changes. ## Compatibility and benchmark impact None: new folder only. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). N/A: recipes and docs only. - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com> Signed-off-by: Grzegorz Chlebus <gchlebus@nvidia.com> Co-authored-by: Grzegorz Chlebus <gchlebus@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Grzegorz Chlebus <gchlebus@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.