Repository navigation
docs(evaluate): Benchmarks page + expanded environment list - #1826
Merged
Merged
Conversation
) - Add fern/versions/latest/pages/evaluation/benchmarks.mdx: new dedicated page covering what benchmarks are, how they relate to training environments (same resources server, fixed evaluation overlay), CLI discovery commands, benchmark selection criteria, category overview, and cards to Browse Environments / Diagnose Results / Add a Benchmark. - Update evaluation/index.mdx: replace inline Benchmarks prose and checklist with a 3-card block pointing to the new page; add Benchmarks to Next Steps. - Update evaluation/environment-list.mdx: add CLI discovery callout at top (gym list benchmarks/environments/agents, gym search); add 14 missing resources servers (arena_judge, math_proof_judgement, frontierscience_judge, equivalence_llm_judge, bunsenbench_chemistry_mcq, critpt, ether0, inverse_if, abstention, openenv, verifif, genrm_compare, tales, grl_tetris); rename category "Math" to "Math & Science"; add Benchmarks card to footer. Closes #1761 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-docs-evaluation-benchmarks-page-1761.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
- Rewrite Gym Evaluation Loop with actual CLI commands (gym list, gym env validate, gym eval run/aggregate/profile) so the steps are actionable - Slim Models and Agent Harnesses sections to one sentence + cards each; remove repeated prose already covered in Architecture/concepts pages - Add Agent Skills card pointing to /agent-server/agent-skills so the feature is discoverable from the evaluation workflow - Fix Reward Profiling card: link to /reference/cli-commands#eval-profile instead of the top-level CLI reference - Remove duplicate Benchmarks card from Next Steps; differentiate remaining card descriptions so they don't read as repeats - Remove redundant Agent Harnesses paragraph Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
…gate cards Add Build Verifiers and Aggregate Metrics cards to the Interpreting Results section of the Benchmarks page, and add a Reward Profiling card. Move Browse Environments and Add a Benchmark to a new Next Steps footer section where they belong, away from the results-interpretation context. Also strengthen the Interpreting Results prose: note that scores are only as reliable as the verifier behind them, and call out gym eval aggregate explicitly alongside gym eval profile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
- aggregate-metrics: fill empty description field; add Benchmarks card to footer so the benchmarks→aggregate-metrics link has a return path - index: clarify step 3 owns --resume (gym eval run flag), step 4 is solely about sharding and merging with gym eval aggregate - benchmarks: update frontmatter description to mention interpreting results, which is now a full page section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
sephmard
marked this pull request as ready for review
June 30, 2026 02:57
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Seph Mard <smard@nvidia.com>
github-project-automation
Bot
moved this from Dev Todo
to Done
in NeMo Gym 0.4.0 - July 1
agronskiy
pushed a commit
that referenced
this pull request
Jun 30, 2026
Closes #1761. Part of #1436. ## Summary **New page — `evaluation/benchmarks.mdx`** Dedicated Benchmarks reference page covering: what a benchmark is (evaluation-configured overlay on top of an environment's resources server); how benchmarks relate to environments with a real config example (`aime24`); CLI discovery (`gym list benchmarks`, `gym search`); benchmark selection criteria table; overview of all six benchmark categories (80+); and an Interpreting Results section with cards linking to Build Verifiers, Aggregate Metrics, Diagnose Results, and Reward Profiling. **`evaluation/index.mdx`** - Renamed page title from "Evaluation" to "Evaluate" to match the nav label - Replaced inline Benchmarks prose and checklist with a 3-card block pointing to the new page - Rewrote the Gym Evaluation Loop as 7 numbered steps with actual CLI commands - Added Agent Skills card (`/agent-server/agent-skills`) - Slimmed Models and Agent Harnesses sections to 1 sentence each - Fixed step 3/4 split: `--resume` now correctly attributed to `gym eval run`, not `gym eval aggregate` **`evaluation/environment-list.mdx`** - CLI discovery callout added at top - Category renamed "Math" → "Math & Science" - 14 missing resources servers added across all category sections: `arena_judge`, `math_proof_judgement`, `frontierscience_judge`, `equivalence_llm_judge`, `bunsenbench_chemistry_mcq`, `critpt`, `ether0`, `inverse_if`, `abstention`, `openenv`, `verifif`, `genrm_compare`, `tales`, `grl_tetris` - Benchmarks card added to footer **`evaluation/aggregate-metrics.mdx`** - Filled empty `description` frontmatter field - Added Benchmarks card to footer (return path from the benchmarks→aggregate-metrics link) **`fern/versions/main.yml`** - Renamed Evaluation section to "Evaluate" — consistent with the action-oriented nav pattern used by "Build Verifiers", "Configure Agents", etc. ## Addresses #1761 acceptance criteria - [x] `evaluation/benchmarks.mdx` created covering benchmark vs environment distinction, selection guidance, category overview, and result interpretation - [x] CLI discovery commands documented on both the Benchmarks and Browse Environments pages - [x] Interpreting Results section with cards for Build Verifiers, Aggregate Metrics, Diagnose Results, and Reward Profiling - [x] Navigation card in evaluation index and cross-page footer links complete the nav loop - [ ] `fern check` — pending CI (no Fern org provisioning locally) ## Test plan - [ ] Fern preview auto-generated on this PR — check the preview comment for rendered output - [ ] Review `benchmarks.mdx` structure and content accuracy - [ ] Confirm 14 added rows in `environment-list.mdx` match actual resources server directories - [ ] Confirm `index.mdx` evaluation loop reads correctly end to end - [ ] `fern check` passes in CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Seph Mard <smard@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
This was referenced Jun 30, 2026
sephmard
added a commit
that referenced
this pull request
Jun 30, 2026
Supersedes #1836 (closed due to merge conflict with #1791 and #1826). Fixes 11 tracked issues across documentation and tutorials. DOC-23e863df is dropped — the `example-resources-servers.mdx` file it targeted was removed by #1791, which also added the needed redirects for `/resources-server`. ## P1 Issues | ID | File | Fix | |---|---|---| | DOC-525c894a | `build-verifiers/index.mdx:17` | `GymnasiumServer` incorrectly implied to be importable from `nemo_gym` core. Added note: lives in `resources_servers/gymnasium/` and must be imported with `from resources_servers.gymnasium import GymnasiumServer` | | LINK-f3437a67 | `resources_servers/proof_judge/README.md:54` | Hard-coded local filesystem path `/Users/<redacted>/Desktop/ls/Gym/...` replaced with relative link `configs/proof_judge.yaml` | | TUT-0e5ca040 | `get-started/quickstart.mdx:95` | `gym list benchmarks` sample output was 3-column and stale. Updated to 4-column format (Domain column added) with current count of 76. Domain values verified against `Domain` enum in `nemo_gym/config_types.py` | | TUT-2ef29637 | `training-tutorials/offline-training-w-rollouts.mdx:111` | `rollout['output']` and `rollout.get('output', [])` do not exist — model output lives at `rollout['response']['output']`. Fixed in all three places: filter, SFT, and DPO snippets | | TUT-85a2f46b | `evaluation-tutorials/evalplus.mdx:52` | Dangling trailing backslash on last argument of `gym env start` block causes shell to hang. Removed | ## P2 Issues | ID | File | Fix | |---|---|---| | DOC-9f8467fd | `contribute/agent-skills.mdx:29` | `nemo-gym-reward-profiling` skill description listed deprecated `ng_run`, `ng_collect_rollouts`, `ng_reward_profile`. Updated to `gym env start`, `gym eval run`, `gym eval profile` (verified against SKILL.md) | | DOC-ebccd396 | `get-started/installation.mdx:27` | Version pin `nemo-gym==0.3.1` references a phantom release. Updated to `nemo-gym==0.4.0` | | TUT-bbc10555 | `get-started/installation.mdx:58` | `gym --version` sample showed `NeMo Gym v0.3.0`. Updated to `v0.4.0` | | LINK-58668ec0 | `environments/workplace_assistant/README.md:30` | Broken link `notebooks/synthetic-data-generation/` (relative to wrong directory). Repointed to `../../resources_servers/workplace_assistant/notebooks/synthetic-data-generation/` where the notebook actually lives | | LINK-c38be399 | `responses_api_agents/stirrup_agent/README.md:14` | Stale TOC entry `[Full GDPVal Evaluation](#full-gdpval-evaluation)` has no matching heading. Removed | | LINK-e8de6b7e | `fern/versions/v0.2.1/.../about-workplace-assistant.mdx:28` | Bare same-page anchor `#training-nemo-rl-grpo-index` has no matching target. Changed to cross-page link `/v0.2/training-tutorials/nemo-rl-grpo` | ## Dropped | ID | Reason | |---|---| | DOC-23e863df | `example-resources-servers.mdx` was deleted by #1791, which also added proper `/resources-server` redirects to docs.yml | ## Test plan - [ ] Fern preview auto-generated — verify the Fern page changes render correctly - [ ] `fern check` passes in CI - [ ] `proof_judge/configs/proof_judge.yaml` exists at relative path (confirmed) - [ ] `resources_servers/workplace_assistant/notebooks/synthetic-data-generation/` exists (confirmed) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Seph Mard <smard@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1761. Part of #1436.
Summary
New page —
evaluation/benchmarks.mdxDedicated Benchmarks reference page covering: what a benchmark is (evaluation-configured overlay on top of an environment's resources server); how benchmarks relate to environments with a real config example (
aime24); CLI discovery (gym list benchmarks,gym search); benchmark selection criteria table; overview of all six benchmark categories (80+); and an Interpreting Results section with cards linking to Build Verifiers, Aggregate Metrics, Diagnose Results, and Reward Profiling.evaluation/index.mdx/agent-server/agent-skills)--resumenow correctly attributed togym eval run, notgym eval aggregateevaluation/environment-list.mdxarena_judge,math_proof_judgement,frontierscience_judge,equivalence_llm_judge,bunsenbench_chemistry_mcq,critpt,ether0,inverse_if,abstention,openenv,verifif,genrm_compare,tales,grl_tetrisevaluation/aggregate-metrics.mdxdescriptionfrontmatter fieldfern/versions/main.ymlAddresses #1761 acceptance criteria
evaluation/benchmarks.mdxcreated covering benchmark vs environment distinction, selection guidance, category overview, and result interpretationfern check— pending CI (no Fern org provisioning locally)Test plan
benchmarks.mdxstructure and content accuracyenvironment-list.mdxmatch actual resources server directoriesindex.mdxevaluation loop reads correctly end to endfern checkpasses in CI🤖 Generated with Claude Code