[Kipok]

What does this PR do?

Use symbolic-only grading for APEX25, matching APEX Shortlist, and a dedicated GPT-5.6 Luna medium judge for ArXivMath May/June and BrokenArXiv May/June, including both default and system-prompt configurations. Add the June 2026 datasets alongside the existing May benchmarks.

• APEX25 uses the existing math_with_judge with should_use_judge: false: require a nonempty complete boxed answer and grade with math-verify. It has no judge service or judge credential dependency.
• Both ArXivMath months use math_with_judge: require a nonempty complete boxed answer, verify symbolically first, then require two positive judge verdicts with answer order swapped.
• Name their resource and agent instances apex25_math_with_judge_*, arxivmath_0526_math_with_judge_*, and arxivmath_0626_math_with_judge_*, including README commands. This intentionally replaces the previous *_math_with_autograder_* names; configuration overrides must also target resources_servers.math_with_judge.
• Both BrokenArXiv months retain its existing full-response 0/1/2 rubric and parser, while clearing the inherited Ultra output cap and sampling overrides.
• Share a configurable public Responses API setup, with medium reasoning, concurrency 16, and no explicit judge output cap or sampling overrides. Update run examples to 32/16/16 collection repeats and document credential setup.

Existing dataset pins and policy prompts, April benchmarks, and grader Python implementations are unchanged. The benchmark READMEs describe usage; the decision rationale is recorded below.

June additions

Benchmark	Public dataset	Problems	Example repeats
arxivmath_0626	MathArena/arxivmath-0626	48	16
brokenarxiv_0626	MathArena/brokenarxiv-0626	54	16

Dataset revisions are pinned to dca0d771ab20c1b8e8c50d0921bf693f7bfe663d and 73dd424784fbdeab599557fcba3d77559c89d1ee, respectively (CC BY-SA 4.0). June's official prompts are preserved, with the same boxed-backslash normalization as May. The system-prompt variants reuse the same prepared problem data and change only the prompt. August's tool-enabled setup is outside this PR.

All 102 source records were checked against the pinned datasets. Both prompt variants were rendered for every row and collated through Gym, producing 48/54 rows with no dataset-level repeat multiplication. No June policy or judge calls were made: these are locally validated additions, not new score baselines.

Validation informing the decision

Judge-only replays used saved responses from an internal model, without regenerating policy answers. Each replay checked complete coverage, unchanged policy responses, valid verdicts, and individual reward differences.

Benchmark	Saved responses	Comparison	Outcome
APEX25	384 (12 × 32)	math_with_autograder + Luna versus math_with_judge + Luna	Every final reward agreed; no judge failures
ArXivMath May	640 (40 × 16)	math_with_autograder + Ultra versus math_with_judge + Luna	Every final reward agreed, including 135 symbolic misses recovered by the judge; no judge failures
BrokenArXiv May	800 (50 × 16)	Same rubric with Luna, Ultra, Terra medium/high, Sol medium and Gemini 3.1 Pro medium	All 800 Luna verdicts valid without serving or parsing failures; approximately 96% exact-grade agreement with alternatives

APEX25 had no judge-only accepts in either internal-model run (384 and 768 saved responses); symbolic successes were 223 and 462 respectively. Disabling fallback therefore preserves every corrected reward. Its defaults now follow the symbolic-only approach used by APEX Shortlist and MathArena, although Gym and MathArena use different parsers. Earlier synthetic judge controls are not evidence that APEX25 needs a fallback. ArXivMath had 137 positive first judgments, of which 135 remained positive after swapping. One rejected first acceptance compared k=3 against 4. This supports retaining the second check; it does not isolate answer-order effects from sampling variability.

Why Luna?

On BrokenArXiv, Luna's mean score differed from Terra medium by −0.25 percentage points, Sol medium by −0.0625 points, and Gemini medium by +1.625 points. A paired bootstrap over the 50 problems, keeping each problem's 16 responses together, gave exploratory 95% intervals of [−1.31, +0.63], [−1.06, +0.81], and [+0.69, +2.75] points, respectively (100,000 resamples). These concern score shifts, not judge accuracy or equivalence.

Similar totals hide disagreements: among answers where either judge awarded credit, disagreements were 24–30%. Targeted, unblinded review found mistakes on both sides. Luna sometimes confused the false statement with the true reference; other judges sometimes applied a contradiction deduction despite the rubric forbidding it for reinterpreted definitions, where Luna followed the rule. Neither majority agreement nor these selected cases establishes an accuracy ranking.

Luna is a practical cost/reliability choice: recorded usage at published rates made it roughly 10–20 times cheaper than the tested Terra, Sol and Gemini settings, without evidence quantifying an accuracy benefit from that premium. This is a workload-specific estimate, not provider billing. Published pricing: Luna, Terra, Sol, Gemini.

These are contributor-reported checks on one internal model, not public gold labels. Independent blinded adjudication and additional models would be needed to establish relative judge accuracy. MathArena uses symbolic grading without an LLM fallback for APEX/ArXivMath and Gemini 3.1 Pro medium for BrokenArXiv, so the ArXivMath/BrokenArXiv defaults do not exactly reproduce its grading methodology. APEX25 is also symbolic-only, but its parser differs. Record the judge alongside results, keep it fixed across model comparisons, and retain historical grader attribution.

Local checks

• Downloaded and prepared both pinned June datasets; source-field, coverage, prompt, and collation checks passed for all four new variants. Preparation and collation functions were exercised directly: the local Python 3.14 environment has an existing Hydra CLI help-parser incompatibility.

• 54 existing tests passed across math_with_judge and false_statement_judge.

• Offline Gym inheritance resolution and actual model-adapter request construction passed for all ten benchmark variants, with both default and overridden provider settings (20 combinations). Rechecked with the June additions; checked judge routing, medium reasoning, request limits, and one-fold dataset materialization. After the APEX25 update, both APEX25 variants resolve without a judge service; the other four benchmarks retain Luna medium.

• pre-commit run --all-files passed.

• The prior live replays used a compatible provider; the public OpenAI endpoint itself was not exercised. No new policy generations or GPU jobs were submitted for this PR.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated edits are excluded.
• Existing relevant tests pass locally; configuration composition and request construction were checked offline.
• Pre-commit checks pass locally (pre-commit run --all-files).
• All commits have DCO sign-off (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[bxyu-nvidia]

/ok to test a05e2d4