[mpatel31415]

What does this PR do?

Adds gym compare, a new CLI command that diffs a baseline eval run against a candidate run and writes a report.
Related epic: #1493 and issue addressed by this PR: #2531

Note

This to be fully completed needs statistical testing and rollout quality, this PR add a skeleton of the report and machine readible output file, report deltas and ci.

Later output of statistical tests and  rollout quality work can be just plugged in follow-up steps.

Output report example

compare_report.md

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-mpatelka-gym-compare-v0.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/reference/cli-commands

[giuliolovisotto]

@mpatel31415 a number of small issues found by agents in these threads. I suggest you just point your agent to the threads.

[giuliolovisotto]

For the record alongside the review: the happy path was validated end-to-end, not just statically. Two live GPQA eval runs were collected against a hosted inference endpoint (--limit 30, 32 repeats/task, 120 rollouts each, zero failures), and gym eval compare at 824fbf7 was run on the real artifacts. Everything worked: sibling metrics discovery, agent selection, binary-mode detection from real 0/1 rewards, correct flip accounting (4 common tasks, 0 flips — the runs agree within noise, mean/reward 0.983 vs 0.975 with overlapping 95% CIs), CI columns exactly where the footnote says they should be, and both report artifacts plus the terminal recap rendering correctly.

The findings in the review were each verified by executing this PR's compare pipeline (details in the inline threads); this comment is just to confirm the feature itself works well on real runs — nice work. Two review points were also visible live in the real output: the "Drop" column header over signed deltas, and the report-size scaling (156 KB JSON / 190-row details table for a 4-task comparison, via pass@k metric fan-out — see the thread on diff.py:272).

[giuliolovisotto]

Re-review at e53ace4: all 14 threads addressed and now resolved.

Verification: the six behavioral fixes were re-verified by re-running the original executable repros against this head — key-metric union (renamed wer row now present), output-dir resolution (stays under cwd with NEMO_GYM_EXTRA_ROOTS set), avg_sample_std_dev filtering, _looks_binary half-present min/max (now binary), the shared Δ (cand − base) header, and the flip reward-list cap (400-task × 8-repeat report shrank 267→124 KiB with counts intact). The remaining fixes (writer migration to aggregate_metrics_path_for, did_you_mean reuse, docs "1", explicit field=FLIP_FIELD, dead-filter cleanup, secret redaction in the provenance command) verified by reading the new diff — all good, and the compare test suite passes locally (166/166). The join-guard deferral to FEP-1461 and the comma-flag decision both make sense as scoped calls.

Formal approval to follow separately — holding it only until the currently red unit-test CI on this head is sorted.