[gwarmstrong]

Migrates the birdbench (BIRD text-to-SQL) benchmark from NeMo Skills into
Gym. Uses execution-based verification against the per-db_id SQLite
databases, matching the Skills evaluator byte-for-byte (including the
"SELECT 1" no-answer filler and the --.*?$-with-DOTALL comment quirk).

Includes

• resources_servers/bird_sql/ — execution-based SQL text-to-SQL resource server
  • Skills BirdEvaluator._extract_answer ported byte-for-byte in
    extract_sql_from_response (CODEBLOCK mode, SELECT 1 filler,
    --.*?$-with-DOTALL comment stripping)
  • SQL execution via asyncio.to_thread under an asyncio.Semaphore
    (max_concurrency=32, sql_execution_timeout_s=30.0), so the server
    works under contention with a vLLM-spawned Ray cluster on multi-node DP
  • Result-set comparison uses the same unordered set(pred) == set(gold) rule
    as Skills
  • Auto-installs BIRD dev DBs (1.4 GB, 11 SQLite files) from
    bird-bench.oss-cn-beijing.aliyuncs.com on server startup via
    ensure_bird_sql(); cached under .bird_sql/
  • compute_metrics() exposes pass@k/accuracy, pass@1[avg-of-k]/accuracy,
    majority@k/accuracy, and per-difficulty breakdowns via
    compute_subset_metrics(field="difficulty") (simple / moderate /
    challenging)
• benchmarks/birdbench/ — benchmark config, prepare.py, prompt template
  • Data source: BIRD dev split (1534 tasks across 11 SQLite databases),
    downloaded from bird-bench.oss-cn-beijing.aliyuncs.com/dev.zip
  • Per-row schema {question, gt_sql, sql_context, difficulty, db_id, id} —
    identical to NeMo Skills dataset/birdbench/prepare.py::format_entries
  • sql_context is an iterdump()-style schema + truncated INSERT chain
    (≤10 INSERTs per run, VALUES (...), (...) chains collapsed after 10
    tuples) — byte-for-byte port of Skills' regex rules so the prompt context
    matches on both sides

Validated against NeMo Skills

Run on nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, BIRD dev (1534 tasks),
4 rollouts per task (6136 rollouts per side), sampling
T=1.0 top_p=0.95 top_k=20 max_output_tokens=65536, on draco-oci. Skills'
BirdMetrics doesn't compute pass@k / majority@k — the Skills column
below shows pooled per-rollout accuracy, which is equivalent to
pass@1[avg-of-4].

=================================================================================
BIRD exec-based eval | 4 rollouts/task | T=1.0 top_p=0.95 | Nemotron-3-Nano-30B
=================================================================================
Metric                          Skills         Gym        Delta
---------------------------------------------------------------
pass@1[avg-of-4]/accuracy        27.88%      28.67%       +0.8%
simple/pass@1[avg-of-4]          34.89%      35.22%       +0.3%
moderate/pass@1[avg-of-4]        17.62%      19.34%       +1.7%
challenging/pass@1[avg-of-4]     16.03%      16.72%       +0.7%

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

looks good, testing this on my end

[cmunley1]

ng_collect_rollouts     +agent_name=birdbench_bird_sql_simple_agent     +input_jsonl_fpath=resources_servers/bird_sql/data/example.jsonl   
  +output_jsonl_fpath=birdbench_rollouts.jsonl     +limit=1
Clearing output fpath since `resume_from_cache=False`!
Limiting the number of rows to 1
Using `birdbench_bird_sql_simple_agent` for rows that do not already have an agent ref
Repeating rows 1 times (in a pattern of abc to aabbcc)!
Reading rows: 0it [00:00, ?it/s]
INFO:     127.0.0.1:39692 - "GET /global_config_dict_yaml HTTP/1.1" 200 OK
Collecting rollouts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00,  6.67s/it]
Sorting results to ensure consistent ordering
Computing aggregate metrics
INFO:     127.0.0.1:32700 - "GET /global_config_dict_yaml HTTP/1.1" 200 OK

Key metrics for birdbench_bird_sql_simple_agent:
{
    "mean/input_tokens": 4443.0,
    "mean/output_tokens": 99.0,
    "pass@1[avg-of-1]/accuracy": 100.0,
    "pass@1/accuracy": 100.0,
    "simple/pass@1[avg-of-1]/accuracy": 100.0
}
Finished rollout collection! View results at:
Fully materialized inputs: birdbench_rollouts_materialized_inputs.jsonl
Rollouts: birdbench_rollouts.jsonl
Aggregate metrics: birdbench_rollouts_aggregate_metrics.json

Single rollout test with openai model works for me smoothly