🔥 Jadwal & Hasil: NVIDIA NeMo/Gym/pull/

gwarmstrong · 2026-04-28T23:43:26Z

Add file-path audio support to the vllm_model audio sidechannel

Audio rows in Gym today carry the entire audio payload as a base64
data-URI on metadata.audio_url — fine for the LibriSpeech-PC POC but
unworkable for the larger ASR / audio benchmarks we want to migrate
next (asr-leaderboard, MUSAN, audiobench, mmau-pro, numb3rs). For
those, the input JSONL would be tens to hundreds of GB before we
collect a single rollout, and every materialized output JSONL
duplicates the audio per seed — base64 inflation gets multiplied by
seed count for free.

This change lets a row reference audio by file path instead, while
keeping the existing inline-data form fully supported. The vllm_model
server reads the file and base64-encodes it at request time — same
strategy NeMo Skills already uses for its audio benchmarks, so the
ergonomics for migrated benchmarks line up.

What's new on the metadata sidechannel

Three coexisting, mutually exclusive ways to carry audio on a row:

metadata.audio_data — pre-built data:audio/...;base64,… URI
inlined into the JSONL. Self-contained; no audio root needed at
serve time. (Renamed from the previous audio_url since the value
is always inline data, never a real URL.)
metadata.audio_path — a single file path. Resolved against a new
audio_root config field on the model.
metadata.audio_paths — list of file paths for multi-clip rows,
mirroring NeMo Skills' audios schema.

Spliced content blocks remain {"type": "audio_url", ...} — that's
vLLM's wire format and unchanged.

Why this shape

File paths cut on-disk size by orders of magnitude. A rough
accounting on a synthetic LibriSpeech-PC-shaped corpus
(~2.4k clips × ~56 KB) shows the input JSONL going from ~172 MB
(data form) to ~328 KB (path form) — ~540×, with proportional
savings on each materialized rollout JSONL.
No new dependency on the vLLM server config. A file:// URL
passthrough mode was prototyped and dropped; it would have required
the vLLM server to be started with --allowed-local-media-path,
which isn't worth the operational complexity for the additional
savings on top of base64-at-request-time.
Skills parity for the migration path. The request-time pipeline
now functionally mirrors VLLMMultimodalModel.content_text_to_list
in Skills (read path → resolve → base64 → audio-block-before-text),
so benchmark migrations don't have to invent a different schema.
No breaking changes for librispeech_pc. The existing inline form
still works — the prepare script just emits the renamed
audio_data key; downstream behavior is byte-identical.

What's in the PR

responses_api_models/vllm_model/app.py — audio_root config
field, _resolve_audio_path_to_url helper, and the extended splice
block.
responses_api_models/vllm_model/tests/test_app.py — splits the
audio splice tests into TestAudioDataSplice (inline form,
unchanged behavior) and TestAudioPathSplice (path form, 8 cases:
absolute path encoding, relative + audio_root, missing
audio_root → raise, missing file → raise, unknown extension →
raise, multi-clip ordering, audio_paths-not-a-list, three-way mutex).
benchmarks/librispeech_pc/prepare.py — emits the renamed
audio_data key.

Test plan

TestAudioDataSplice (6 cases) — covers the existing inline
behavior librispeech_pc relies on, all passing.
TestAudioPathSplice (8 cases) — covers the new path form.
End-to-end run of librispeech_pc rollouts on dfw against a
real vLLM audio model with both audio_data and audio_path
JSONLs to confirm parity.

Lets a row reference audio by file path (single or list) on ``responses_create_params.metadata`` instead of inlining it as a base64 data-URI. Cuts the on-disk JSONL by ~540x for LibriSpeech-PC-shaped data and proportionally for multi-seed materialized rollouts — the larger ASR/audio benchmarks we're migrating next (asr-leaderboard, MUSAN, audiobench, mmau-pro, numb3rs) need this to be tractable. Three mutually exclusive metadata keys: * ``audio_data`` — pre-built ``data:audio/...;base64,`` URI inlined into the JSONL (renamed from ``audio_url``; the value is always inline data, not a real URL) * ``audio_path`` — single file path; resolved against a new ``audio_root`` config field and base64-encoded at request time * ``audio_paths`` — list of file paths, mirroring NeMo Skills' ``audios`` multi-clip schema The spliced vLLM content block stays ``{"type": "audio_url", ...}`` — that's vLLM's wire format. Existing librispeech_pc inline behavior is preserved (just under the renamed key). Tests: ``TestAudioDataSplice`` (6, regression for inline form), ``TestAudioPathSplice`` (8, new path form: abs/relative resolution, missing-root and missing-file errors, unknown-extension error, multi-clip ordering, list-type validation, three-way mutex). A synthetic size-comparison demo (``benchmarks/librispeech_pc/demo_size_compare.py``) lets reviewers see the JSONL delta without downloading OpenSLR. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

copy-pr-bot · 2026-04-28T23:43:30Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

These were useful as worked examples but aren't part of the framework change. The path-mode prepare and the synthetic size-comparison script belong with the librispeech_pc migration work, not the vllm_model audio sidechannel PR. Keeps this PR focused on the framework support for ``audio_path`` / ``audio_paths``. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

# Add file-path audio support to the vllm_model audio sidechannel Audio rows in Gym today carry the entire audio payload as a base64 data-URI on `metadata.audio_url` — fine for the LibriSpeech-PC POC but unworkable for the larger ASR / audio benchmarks we want to migrate next (asr-leaderboard, MUSAN, audiobench, mmau-pro, numb3rs). For those, the input JSONL would be tens to hundreds of GB before we collect a single rollout, and every materialized output JSONL duplicates the audio per seed — base64 inflation gets multiplied by seed count for free. This change lets a row reference audio by **file path** instead, while keeping the existing inline-data form fully supported. The vllm_model server reads the file and base64-encodes it at request time — same strategy NeMo Skills already uses for its audio benchmarks, so the ergonomics for migrated benchmarks line up. ## What's new on the metadata sidechannel Three coexisting, mutually exclusive ways to carry audio on a row: - `metadata.audio_data` — pre-built `data:audio/...;base64,…` URI inlined into the JSONL. Self-contained; no audio root needed at serve time. (Renamed from the previous `audio_url` since the value is always inline data, never a real URL.) - `metadata.audio_path` — a single file path. Resolved against a new `audio_root` config field on the model. - `metadata.audio_paths` — list of file paths for multi-clip rows, mirroring NeMo Skills' `audios` schema. Spliced content blocks remain `{"type": "audio_url", ...}` — that's vLLM's wire format and unchanged. ## Why this shape - **File paths cut on-disk size by orders of magnitude.** A rough accounting on a synthetic LibriSpeech-PC-shaped corpus (~2.4k clips × ~56 KB) shows the input JSONL going from ~172 MB (data form) to ~328 KB (path form) — ~540×, with proportional savings on each materialized rollout JSONL. - **No new dependency on the vLLM server config.** A `file://` URL passthrough mode was prototyped and dropped; it would have required the vLLM server to be started with `--allowed-local-media-path`, which isn't worth the operational complexity for the additional savings on top of base64-at-request-time. - **Skills parity for the migration path.** The request-time pipeline now functionally mirrors `VLLMMultimodalModel.content_text_to_list` in Skills (read path → resolve → base64 → audio-block-before-text), so benchmark migrations don't have to invent a different schema. - **No breaking changes for librispeech_pc.** The existing inline form still works — the prepare script just emits the renamed `audio_data` key; downstream behavior is byte-identical. ## What's in the PR - `responses_api_models/vllm_model/app.py` — `audio_root` config field, `_resolve_audio_path_to_url` helper, and the extended splice block. - `responses_api_models/vllm_model/tests/test_app.py` — splits the audio splice tests into `TestAudioDataSplice` (inline form, unchanged behavior) and `TestAudioPathSplice` (path form, 8 cases: absolute path encoding, relative + audio_root, missing audio_root → raise, missing file → raise, unknown extension → raise, multi-clip ordering, audio_paths-not-a-list, three-way mutex). - `benchmarks/librispeech_pc/prepare.py` — emits the renamed `audio_data` key. ## Test plan - [x] `TestAudioDataSplice` (6 cases) — covers the existing inline behavior librispeech_pc relies on, all passing. - [x] `TestAudioPathSplice` (8 cases) — covers the new path form. - [ ] End-to-end run of librispeech_pc rollouts on dfw against a real vLLM audio model with both `audio_data` and `audio_path` JSONLs to confirm parity. --------- Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Signed-off-by: cmunley1 <cmunley@nvidia.com>

## Summary - Adds the HuggingFace Open ASR Leaderboard (8 ESB subsets: librispeech-clean/other, voxpopuli, tedlium, gigaspeech, spgispeech, earnings22, ami) as a data-only benchmark. - Reuses the existing `asr_with_pc` resource server in `task_type=ASR` mode (Whisper-normalized WER, added in #1144) — no new server logic. - Audio is referenced via `responses_create_params.metadata.audio_path` (#1170) so the multi-GB FLAC corpus stays on disk instead of being inlined as base64 in every JSONL row. ## Why ASR-Leaderboard is the standard cross-domain WER benchmark suite for English ASR systems. Migrating it as data-only — no new server logic — extends Gym's audio coverage with zero verification risk. Both the prompt template (system + user messages) and the minimum-duration / empty-reference filtering match the upstream NeMo-Skills implementation exactly, so a data-only parity check (row count, `expected_answer`, audio path) is sufficient. ## Files - `benchmarks/asr_leaderboard/__init__.py` - `benchmarks/asr_leaderboard/config.yaml` — chains to `asr_with_pc.yaml` and pins `task_type: ASR`. - `benchmarks/asr_leaderboard/prepare.py` — downloads `hf-audio/esb-datasets-test-only-sorted`, saves FLACs to `<work_dir>/<dataset>/<id>.flac`, writes the combined `asr_leaderboard_benchmark.jsonl`. - `benchmarks/asr_leaderboard/prompts/default.yaml` — `"You are a helpful assistant. /no_think"` + `"Transcribe the following audio."` - `benchmarks/asr_leaderboard/README.md` - `benchmarks/asr_leaderboard/data/.gitignore` ## Test plan - [x] Smoke-tested `_format_row` and `_audio_filename` on synthetic dataset entries (empty-text skip, sub-0.1s skip, slash-in-id rewriting, AMI-style `id/name.suffix` stem extraction). - [ ] Skills-vs-Gym data-parity check (row count exact, `expected_answer` 100% agreement, audio paths 100% agreement) — runs in the migration recipe after merge once both prepare jobs have downloaded the corpus. --------- Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

Migrates the AudioBench benchmark from NeMo Skills, scoped to the judge half. Introduces: - resources_servers/audiobench_judge/ — LLM-as-a-judge resource server with a 0-5 rating prompt that is byte-equivalent to NeMo Skills' judge/audiobench.yaml. extract_judge_result mirrors Skills' AudioMetrics._extract_judge_result precedence (Rating > Judgement > yes/no fallback). 25 unit tests, 98 percent line coverage on app.py. - benchmarks/audiobench_alpaca_audio/ — first AudioBench benchmark on top of audiobench_judge. Uses the file-path audio sidechannel from PR NVIDIA-NeMo#1170; prepare.py writes absolute audio paths to keep the JSONL self-contained across per-server cwd boundaries. The remaining 33 judge-scored and 32 nonjudge AudioBench sub-datasets follow the same pattern; they will land as small follow-on PRs (one benchmarks/audiobench_<name>/ each, sharing audiobench_judge for the judged datasets and asr_with_pc for the deterministic ones). Signed-off-by: gwarmstrong <georgea@nvidia.com>

## Summary - Rename the bundled ASR smoke data metadata key from `audio_url` to `audio_data`, matching the `vllm_model` contract introduced in #1170. - Update the generator and all checked-in rollout fixtures. - Clarify that `--model-type vllm_model` selects the Gym adapter and still requires `--model-url` to point to an audio-capable vLLM endpoint. - Add a regression test that compares the committed dataset with generator output and validates each WAV data URI. ## Problem The production LibriSpeech-PC preparation script emits `metadata.audio_data`, but the five-row smoke dataset and its generator still emitted `metadata.audio_url`. The current `vllm_model` adapter does not consume that legacy metadata key. It forwarded the 42 KB data URI as ordinary request metadata, which OpenAI rejected because metadata strings are limited to 512 characters. The reported command also selected the `vllm_model` Gym adapter while leaving `policy_base_url` pointed at `https://api.openai.com/v1`. This adapter emits the vLLM `audio_url` content format. It requires a compatible audio-capable vLLM endpoint. ## Reproduction before this change ```bash gym env start \ --model-type vllm_model \ --resources-server asr_with_pc gym eval run --no-serve \ --agent asr_with_pc_simple_agent \ --input resources_servers/asr_with_pc/data/example.jsonl \ --output results/asr_with_pc_rollouts.jsonl \ --num-repeats 1 ``` When `policy_base_url` is the OpenAI API, the request fails with: ```text Invalid metadata.audio_url: string too long. Expected a string with maximum length 512, but got a string with length 42750 instead. ``` ## Correct server command ```bash gym env start \ --model-type vllm_model \ --model-url http://<vllm-host>:<port>/v1 \ --model <audio-capable-model> \ --resources-server asr_with_pc ``` ## Testing - `RAY_TMPDIR=/tmp uv run gym env test --resources-server asr_with_pc` (`47 passed`) - `uv run pytest responses_api_models/vllm_model/tests/test_app.py -k AudioDataSplice -x` (`6 passed`) - `pre-commit run --all-files` in a clean checkout - Parsed all changed JSONL files with `jq` - Confirmed each JSONL differs only by the metadata key rename A live rollout was not run because no audio-capable vLLM endpoint is available in this workspace. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

## Summary - Rename the bundled ASR smoke data metadata key from `audio_url` to `audio_data`, matching the `vllm_model` contract introduced in #1170. - Update the generator and all checked-in rollout fixtures. - Clarify that `--model-type vllm_model` selects the Gym adapter and still requires `--model-url` to point to an audio-capable vLLM endpoint. - Add a regression test that compares the committed dataset with generator output and validates each WAV data URI. ## Problem The production LibriSpeech-PC preparation script emits `metadata.audio_data`, but the five-row smoke dataset and its generator still emitted `metadata.audio_url`. The current `vllm_model` adapter does not consume that legacy metadata key. It forwarded the 42 KB data URI as ordinary request metadata, which OpenAI rejected because metadata strings are limited to 512 characters. The reported command also selected the `vllm_model` Gym adapter while leaving `policy_base_url` pointed at `https://api.openai.com/v1`. This adapter emits the vLLM `audio_url` content format. It requires a compatible audio-capable vLLM endpoint. ## Reproduction before this change ```bash gym env start \ --model-type vllm_model \ --resources-server asr_with_pc gym eval run --no-serve \ --agent asr_with_pc_simple_agent \ --input resources_servers/asr_with_pc/data/example.jsonl \ --output results/asr_with_pc_rollouts.jsonl \ --num-repeats 1 ``` When `policy_base_url` is the OpenAI API, the request fails with: ```text Invalid metadata.audio_url: string too long. Expected a string with maximum length 512, but got a string with length 42750 instead. ``` ## Correct server command ```bash gym env start \ --model-type vllm_model \ --model-url http://<vllm-host>:<port>/v1 \ --model <audio-capable-model> \ --resources-server asr_with_pc ``` ## Testing - `RAY_TMPDIR=/tmp uv run gym env test --resources-server asr_with_pc` (`47 passed`) - `uv run pytest responses_api_models/vllm_model/tests/test_app.py -k AudioDataSplice -x` (`6 passed`) - `pre-commit run --all-files` in a clean checkout - Parsed all changed JSONL files with `jq` - Confirmed each JSONL differs only by the metadata key rename A live rollout was not run because no audio-capable vLLM endpoint is available in this workspace. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>

gwarmstrong and others added 3 commits April 28, 2026 16:55

Merge branch 'main' into georgea/audio-read-from-file

018563d

Merge branch 'main' into georgea/audio-read-from-file

18815ff

gwarmstrong requested review from bxyu-nvidia and cmunley1 April 29, 2026 01:38

cwing-nvidia requested a review from ananthsub April 29, 2026 09:16

bxyu-nvidia approved these changes Apr 29, 2026

View reviewed changes

Merge branch 'main' into georgea/audio-read-from-file

acd04f6

gwarmstrong enabled auto-merge (squash) April 29, 2026 16:16

gwarmstrong merged commit 75ecbba into NVIDIA-NeMo:main Apr 29, 2026
6 checks passed

This was referenced Apr 29, 2026

feat(benchmarks): add asr_leaderboard benchmark (data-only) #1176

Merged

Add audiobench benchmark and resource server gwarmstrong/Gym#4

Closed

gwarmstrong mentioned this pull request May 8, 2026

Add audiobench_alpaca_audio benchmark + audiobench_judge resource server #1274

Open

gwarmstrong mentioned this pull request Sep 4, 2026

[6667815] Sync LibriSpeech-PC smoke audio metadata #3098

Merged

Navigation Menu

Repository navigation

Add file-path audio support to the vllm_model audio sidechannel - #1170

Add file-path audio support to the vllm_model audio sidechannel#1170
gwarmstrong merged 5 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/audio-read-from-file

gwarmstrong commented Apr 28, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gwarmstrong commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add file-path audio support to the vllm_model audio sidechannel

What's new on the metadata sidechannel

Why this shape

What's in the PR

Test plan

Uh oh!

copy-pr-bot Bot commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gwarmstrong commented Apr 28, 2026 •

edited

Loading