[gwarmstrong]

One-sentence summary

Migrates the longbench-v2 benchmark (THUDM/LongBench-v2, 503
long-context MCQ questions) from NeMo Skills into Gym on top of the
existing mcqa resource server. No new servers or agents, no
metric-infra changes. Scope is deliberately data-prep-only: the new
prepare.py mirrors the Skills prepare method and the final file is
validated against Skills' output for every row.

Includes

• benchmarks/longbench_v2/ — full LongBench-v2 test split
  • Data source: THUDM/LongBench-v2 (HuggingFace, single train split
    holding all 503 evaluation questions) — identical upstream dataset
    to Skills' nemo_skills/dataset/longbench-v2/prepare.py
  • LongBench-v2 covers 6 long-context domains (8k-2M words):
    single-doc QA, multi-doc QA, long in-context learning, long-dialogue
    history, code-repo understanding, long structured data
  • Reuses the existing mcqa resource server (no server changes);
    prepare.py emits the standard mcqa options list and
    grading_mode: strict_single_letter_boxed
• benchmarks/longbench_v2/prompts/default.yaml — character-for-character
  port of nemo_skills/prompt/config/eval/longbench/default.yaml
  (single user message, CoT-style, ends with Answer: \boxed{A/B/C/D})

Tiktoken <|endoftext|> handling

LongBench-v2 contexts contain raw <|endoftext|> strings as plain text
inside long-document corpora. tiktoken refuses to encode them by
default. Both Skills' upstream prepare.py (when run unpatched) and an
earlier draft of this Gym prepare.py aborted on these rows. The fix
(commit a4bd3e8e on this branch) passes disallowed_special=() so
the count is computed treating the literals as ordinary text — we only
need a length, not a faithful token-id sequence. The same patch is
applied on the Skills side in the parity-check recipe so both sides
are exercised symmetrically.

Validated against NeMo Skills

Scope is data-prep parity only. This PR does not include rollout or
metrics-parity numbers (no model runs were made). The new
prepare() output is compared row-by-row against Skills' prepare.py
output. Every Skills field is identical on every row:

==================================================================
Data-prep parity · Skills prepare.py vs Gym prepare()
==================================================================
Benchmark           Skills rows    Gym rows    Identical
------------------------------------------------------------------
longbench_v2             503            503    503 / 503

Per-field equality on the 13 Skills fields (index, context,
question, choice_A..D, expected_answer, domain, sub_domain,
difficulty, length, context_tokens): value_diff=0 on all 503
rows. Expected-answer agreement: 503 / 503. Normalized-question overlap:
100 % (15 questions repeat verbatim across different long contexts in
the upstream dataset, hence 488 unique normalized questions for 503
rows). Gym's options list matches choice_A..D on all 503 rows.

Grader-mode note

Skills' eval_mcq runs two extraction passes: try \boxed{} first,
fall back to (?i)Answer\s*:\s*([A-Z]). The mcqa resource server
exposes either pass independently (strict_single_letter_boxed or
lenient_answer_colon) but not the combined fallback. Since the
prompt explicitly instructs the model to emit Answer: \boxed{X}, the
boxed pass catches the dominant case and strict_single_letter_boxed
(the mcqa default and the convention used by every other mcqa-backed
Gym benchmark) is used here. This is a property of the existing mcqa
server and is out of scope for this data-only PR — if a future Skills-vs-Gym
end-to-end check shows a no_answer gap larger than ~2 pp, it would
be addressed inside mcqa (e.g. by adding a
lenient_boxed_then_answer_colon mode), not by re-doing this data port.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.