[gwarmstrong]

Add imo_proofbench benchmark and imo_proofbench_judge resource server

Summary

Migrates the IMO ProofBench benchmark from
NeMo Skills into Gym. Verification is a strong LLM judge
plus sympy-based symbolic equivalence, mirroring
Skills' eval_type=math chained with prompt_config=judge/imo_proofbench.
The new imo_proofbench_judge resource server vendors Skills' boxed-
extraction (search_boxed), the 3-format judgement parser
(is_correct_judgement), the prefill_judgement literal-match
shortcut, and math_equal (sympy via math_verify +
latex2sympy2_extended) — so the headline any_correct = symbolic_correct OR judge_correct matches Skills.

Includes

• benchmarks/imo_proofbench/ — benchmark configuration, data
  preparation, and the model prompt template.

  • config.yaml — registers imo_proofbench_judge_simple_agent,
    points at the prepared JSONL, declares a Gemini judge model server
    via judge_base_url / judge_api_key / judge_model_name.
  • prepare.py — pulls imobench/proofbench.csv

• resources_servers/imo_proofbench_judge/ — Skills-parity verifier.

  • app.py — vendors:
    • search_boxed (rightmost \boxed{…} brace-balanced extraction,
      mirroring nemo_skills.evaluation.math_grader.search_boxed)
    • parse_judgement_verdict with Skills' 3-format priority
      (Judgement: Yes/No → \boxed{Correct/Incorrect} →
      <points>N out of 7</points> with points >= PASS_THRESHOLD)
    • parse_points for the score-band metrics
    • math_equal (MCQ short-circuit, latex2sympy2_extended
      normalization, math_verify symbolic fallback) so Gym's
      symbolic_correct is computed independently of the LLM judge,
      matching Skills' MathEvaluator.eval_single step
    • verify() — extracts boxed, runs math_equal first; if
      symbolic-correct, synthesises Reasoning: The two answers are identical.\nJudgement: Yes and skips the judge; on no
      extraction, synthesises Reasoning: No answer was provided.\n Judgement: No; otherwise calls Gemini with the boxed content
      filling {predicted_answer} and parses with the 3-format
      priority.
    • _score_fn exposes any_correct, symbolic_correct,
      judge_correct, and judge_score_{0,1,6,7} for
      compute_pass_majority_metrics; compute_metrics adds
      category / level subset breakdowns.

Skills-parity validation

End-to-end parity numbers

Final 16-seed run on nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
(960 rollouts per pipeline):

Metric	Skills	Gym	Δ	Threshold
pass@1 / any_correct	6.46 %	6.25 %†	-0.21 pp	<5 pp ✓
pass@1 / symbolic_correct	6.35 %	6.25 %	-0.10 pp	✓
pass@1 / judge_correct	1.25 %	1.46 %‡	+0.21 pp	✓
pass@16 / any_correct	16.67 % (10/60)	13.33 % (8/60)	-3.34 pp	✓

The model produces near-identical boxed answers in both pipelines —
once the verifier runs the same math_equal Skills does, the headline
metric is identical to within sampling noise.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.