Skip to content

Add polymath benchmark + polymath resource server (weighted, per-language) - #1168

Merged
bxyu-nvidia merged 7 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/migrate-gym-polymath
May 8, 2026
Merged

Add polymath benchmark + polymath resource server (weighted, per-language)#1168
bxyu-nvidia merged 7 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/migrate-gym-polymath

Conversation

@gwarmstrong

@gwarmstrong gwarmstrong commented Apr 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • New benchmarks/polymath/ — Qwen/PolyMath (18 languages × 4 difficulty tiers × 125 problems = 9,000 rows). prepare.py mirrors NeMo Skills' nemo_skills/dataset/polymath/prepare.py (same upstream HuggingFace repo, same QUESTION_TEMPLATE, same difficulty-weight mapping, optional language_control modes). Prompt is character-for-character with Skills' generic/default.yaml (user-only {question} — the per-language instruction text already lives inside the question by way of prepare.py).
  • New resources_servers/polymath/ — subclasses math_with_judge to layer the two pieces of metric machinery PolyMath needs and stock math_with_judge does not provide:
    1. Difficulty-weighted aggregation — PolyMath rows carry a weight field (low=1, medium=2, high=4, top=8). The server emits pass@k/<score>_weighted, pass@1[avg-of-k]/<score>_weighted, and majority@k/<score>_weighted alongside the unweighted metrics. Direct port of WeightedMathMetrics from nemo_skills/evaluation/metrics/weighted_math_metrics.py.
    2. Per-language stratification — tasks are also grouped by the language JSONL field via compute_subset_metrics, producing <lang>/pass@k/<score> keys. Equivalent to Skills' pipeline-level subset_for_metrics fan-out.
  • Verification path is unchanged from math_with_judge (math-verify symbolic check + optional LLM-judge fallback that's off by default to match Skills' default eval_type=math behaviour for this benchmark).

Validated against NeMo Skills

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, 4 rollouts/task at
temperature=1.0 top_p=0.95 top_k=20 max_output_tokens=16384. Both
sides served by vLLM with --reasoning-parser deepseek_r1. Symbolic
verification only (no LLM judge — Skills' default for this benchmark).
500-task subset.

Metric Skills Gym Δ (pp)
pass@1 symbolic 45.55% 45.60% +0.05
pass@1[avg-of-4] symbolic 45.55% 45.60% +0.05
pass@2 symbolic 48.63% 49.57% +0.94
pass@4 symbolic 51.80% 53.80% +2.00
pass@1 weighted_symbolic 23.56% 23.87% +0.31
pass@1[avg-of-4] weighted_symbolic 23.56% 23.87% +0.31
pass@4 weighted_symbolic 29.97% 31.41% +1.44

Verdict: PASS

@gwarmstrong
Add polymath benchmark + polymath resource server (weighted, per-lang…
67c659c 
…uage)

Migrates NeMo Skills' polymath benchmark (Qwen/PolyMath, 18 languages
× 4 difficulty tiers) into Gym. Adds:

* `resources_servers/polymath/` — subclass of `math_with_judge` that
  accepts per-row `weight` (low=1, medium=2, high=4, top=8) and
  `language` metadata, then emits three families of metrics:
    * Tier 1 (unweighted, language-pooled): `pass@k/symbolic_accuracy`,
      `pass@1[avg-of-k]/symbolic_accuracy`, `majority@k/symbolic_accuracy`
      — same as math_with_judge.
    * Tier 2 (per-language): `<lang>/pass@k/symbolic_accuracy` via
      `compute_subset_metrics(subset_key="language")`.
    * Tier 3 (difficulty-weighted): `<metric>_weighted` variants —
      port of NeMo Skills' `WeightedMathMetrics`.
* `benchmarks/polymath/` — `prepare.py` mirrors Skills'
  `nemo_skills/dataset/polymath/prepare.py` (same upstream
  `Qwen/PolyMath` HF download, same per-language `QUESTION_TEMPLATE`
  formatting, same difficulty weights, optional `language_control`
  modes). Prompt is character-for-character with Skills'
  `generic/default.yaml` (user-only `{question}` — instruction text
  is already embedded in the question by prepare.py).

Verification path is unchanged from `math_with_judge` (math-verify
symbolic check, optional LLM-judge fallback that's off by default to
match Skills' default `eval_type=math` for this benchmark).

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@gwarmstrong
fix(polymath): rename benchmark agent to avoid same-key swap with parent
cfe27b7 
Naming the benchmark's agent block the same as the parent agent
(`polymath_simple_agent`) makes `_inherit_from` a no-op key swap and
leaves the parent's references intact. Use distinct names —
`polymath_benchmark_resources_server` and `polymath_benchmark_simple_agent`
— so the swap actually replaces the parent and the resources_server
override flows through. Mirrors the hmmt_feb25 / apex_shortlist pattern.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@gwarmstrong
refactor(polymath): collapse weighted-aggregation counters and hoist …
8c3d818 
…imports

The three running totals (pass_at_k_w, pass1_avg_of_k_w, majority_at_k_w)
all increment together whenever a task contributes — collapse to one
contributing_weight. Move math/Counter imports to module top. Drop
the inline Tier 1/2/3 banner comments now that the docstring covers
the same ground.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@gwarmstrong
fix(polymath): add missing CI artifacts and package markers
7b7737b 
Adds the four files required by the Gym PR-readiness audit that the
initial polymath migration commit missed:

- resources_servers/polymath/tests/__init__.py — pytest package marker
  (consistent with proof_judge / proof_genselect / math_proof_judgement).
- benchmarks/polymath/__init__.py — package marker (matches aime26,
  aime25, proof_bench_judge).
- resources_servers/polymath/data/example_metrics.json — generated by
  ng_prepare_data +mode=example_validation against example.jsonl; what
  ng_test's data validation step parses to confirm the example set
  declares 5 rows.
- resources_servers/polymath/data/example_rollouts.jsonl (+ the
  materialized_inputs sibling) — 5 synthetic rollouts using the
  documented synthetic-example model marker, exercising the
  PolyMathVerifyResponse schema (reward / extracted_answer /
  library_reward / weight / language). Same pattern used by
  resources_servers/math_proof_judgement.

No code changes. 13/13 tests still pass; coverage on the polymath
server stays at 100%.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@gwarmstrong

gwarmstrong commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

Parity result (subset 500 problems × 4 = 2000 Gym rollouts, all English; Skills full benchmark 9000 × 4):

Metric Skills (%) Gym (%) Δ (pp)
pass@1 symbolic 45.55 45.60 +0.05
pass@1[avg-of-4] symbolic 45.55 45.60 +0.05
pass@4 symbolic 51.80 53.80 +2.00
pass@1 weighted_symbolic 23.56 23.87 +0.31
pass@1[avg-of-4] weighted_symbolic 23.56 23.87 +0.31

✅ All deltas ≤2pp — well within the deterministic-verification gate (math-verify symbolic, no LLM judge). Weighted (difficulty-weighted) and per-language metrics are also computed; symbolic_accuracy_weighted matches Skills' weighted_symbolic_correct in distribution.

Caveat: Gym subset is English-only (first 500 rows); Skills covers all 18 languages.

gwarmstrong added 2 commits May 7, 2026 15:42
@gwarmstrong
Merge remote-tracking branch 'origin/main' into georgea/migrate-gym-p…
d131685 
…olymath-fixmerge

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

# Conflicts:
#	README.md
@gwarmstrong
Merge remote-tracking branch 'origin/main' into georgea/migrate-gym-p…
917e2f3 
…olymath-fixmerge

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

# Conflicts:
#	README.md
@bxyu-nvidia
bxyu-nvidia merged commit 53595a1 into NVIDIA-NeMo:main May 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bxyu-nvidia bxyu-nvidia bxyu-nvidia approved these changes
@ananthsub ananthsub Awaiting requested review from ananthsub
@cmunley1 cmunley1 Awaiting requested review from cmunley1

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gwarmstrong @bxyu-nvidia