[gwarmstrong]

Add speed-bench benchmark and resource server

Migrates the SPEED-Bench speculative-decoding throughput benchmark from NeMo Skills into NeMo Gym. Verification reads the model server's Prometheus /metrics counters before and after generation to derive acceptance length (AL) and acceptance rate (AR) — there is no notion of answer correctness; the metrics are server-cumulative properties of the run window.

Includes

• resources_servers/speed_bench/ — new resource server that scrapes vLLM vllm:spec_decode_* (or SGLang sglang:spec_accept_*) Prometheus counters across the benchmark window and returns running aggregates per task.
  • vLLM scrape ported from nemo_skills/inference/eval/specdec.py::fetch_vllm_spec_decode_metrics.
  • SGLang scrape ported via Prometheus weighted-delta on the running-average gauges (sglang:spec_accept_length, sglang:spec_accept_rate) and the request/generation-token counters — mirrors Skills' compute_sglang_spec_decode_delta.
• responses_api_agents/speed_bench_agent/ — new fixed-replay multi-turn agent.
  • simple_agent sends a list of messages in one model call; SPEED-Bench's qualitative config has 185/880 multi-turn rows where Skills' SpecdecGenerationTask plays each user turn separately so the model sees its own intermediate replies. The new agent reproduces that turn-by-turn replay.
  • Single-turn rows collapse to a single model call (same shape as simple_agent).
• benchmarks/speed-bench/ — qualitative + throughput_2k configs.
  • Each Gym config maps to its own per-config agent + resource server (so ng_prepare_benchmark's "exactly one dataset per agent" assertion is satisfied).
  • prepare.py and prepare_throughput_2k.py reuse Skills' _resolve_external_data helper (loaded by file path) so the 14-source interpolation logic isn't duplicated.
  • Each row's responses_create_params.input is a list of {"role": "user", "content": "<turn>"} messages with no interspersed assistants — speed_bench_agent plays them sequentially at rollout time.
• responses_api_models/local_vllm_model/configs/Qwen/Qwen3-30B-A3B-Instruct-2507-ngram-specdec.yaml — ready-to-use local_vllm_model config that bakes ngram speculative decoding into vllm_serve_kwargs.speculative_config. Drop it into +config_paths=[…] alongside the speed_bench config and you have a working spec-decode-enabled vLLM out of the box.
• nemo_gym/config_types.py — adds "NVIDIA Evaluation Dataset License Agreement" to the dataset license whitelist (SPEED-Bench is hosted under exactly that license; without the literal, ng_prepare_data fails Pydantic validation).

Validated against NeMo Skills

Two side-by-side runs on a single 8 × H100 node, Qwen3-30B-A3B target, vLLM ngram speculative decoding (num_speculative_tokens=3, prompt_lookup_max=5, prompt_lookup_min=2), temperature=1.0, top_p=0.95, max_output_tokens=1024, single seed each side. (Single seed because spec-decode metrics are server-cumulative on a single benchmark window — splitting into multiple seeds would change the semantic, not just produce variance estimates.)

=================================================================
qualitative · 880 prompts · 185 multi-turn · ~1125 output tokens
=================================================================
Metric                         Skills        Gym        Delta
-----------------------------------------------------------------
spec_acceptance_length         1.7661       1.7642     -0.11%
spec_acceptance_rate          25.54%       25.48%     -0.0656pp
avg_tokens                     1125         1125.30    +0.03%
num_entries                    880          880        0

=================================================================
throughput_2k · 1536 prompts · ~975 output tokens
=================================================================
Metric                         Skills        Gym        Delta
-----------------------------------------------------------------
spec_acceptance_length         1.6800       1.6806     +0.04%
spec_acceptance_rate          22.67%       22.69%     +0.0214pp
avg_tokens                     972          974.94     +0.30%
num_entries                    1536         1536       0

Both spec-decode quality metrics are within sampling noise on both configs; data-prep parity (row counts, src_id sequences, multi-turn row counts) is byte-equivalent. The natural drop from qualitative→throughput_2k (AL 1.77→1.68) reflects ngram lookup having a lower hit rate on long padded prompts; both pipelines see the drop identically.

Deferred

• The throughput_{1k,8k,16k,32k} configs: prepare.py knows how to prepare them (--config throughput_8k etc.) but only qualitative and throughput_2k were validated end-to-end.
• Skills' SGLang per-request metrics-file fallback (--export-metrics-to-file) is not ported — the Prometheus delta gives matching headline numbers and is sufficient for parity.
• Per-position acceptance rates are emitted per-row but excluded from get_key_metrics() headline.

Notes for reviewers

• The model server must be launched with speculative decoding enabled (e.g. --speculative-config '{"method": "ngram", ...}' or the vllm_serve_kwargs.speculative_config: block in the demo local_vllm_model config). Without it, every rollout records spec_decode_unavailable: true and AL/AR are null. Both READMEs document this prominently.
• The SGLang scrape path is wired but not validated against an actual SGLang server in this PR — set server_type_for_metrics: sglang in the resources server config and ensure the model server is SGLang. Unit tests cover the parser, weighted-delta arithmetic, and the verify-time code path; missing piece is end-to-end SGLang validation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.