[muskank7]

What does this PR do?

Adds an Aegis v4 safety-evaluation resources server backed by nvidia/Nemotron-3-Content-Safety and registers it as the manifest-backed aegis_v4_safety benchmark.

The integration supports two workflows:

• generate responses from a target model and classify the prompt/response pair with Aegis v4;
• reverify datasets that already contain target-model responses without generating them again.

It accepts text prompts with an optional single image, records structured user/response safety labels and categories, supports final-answer-only or reasoning-plus-final evaluation, and reports unresolved judgments separately from safe/unsafe results. It also includes a dataset converter, example fixtures, documentation, and focused tests.

There is no linked issue; this is a new benchmark integration.

Benchmark metadata and data

• benchmarks/aegis_v4_safety/manifest.yaml declares the safety-domain benchmark and mirrors its Gym composition.
• The included five project-authored prompts are an Apache-2.0 smoke dataset for checking the integration, not a comprehensive safety benchmark.
• The resources server exports a service-free verifier fixture covering full reward, zero reward, and malformed input.
• A safety-only reward can be maximized by refusing every request. It measures response safety, not helpfulness or task correctness, and should not be used by itself as an RL reward.

Validation

• venv/bin/pytest -q resources_servers/aegis_v4_safety/tests — 23 passed
• venv/bin/ruff check resources_servers/aegis_v4_safety benchmarks/aegis_v4_safety — passed
• venv/bin/ruff format --check resources_servers/aegis_v4_safety benchmarks/aegis_v4_safety — passed
• pre-commit run --all-files — passed
• gym env test +entrypoint=resources_servers/aegis_v4_safety +should_validate_data=true — 23 tests passed and the five-row example data, metrics, and rollouts validated
• gym env validate aegis_v4_safety --kind benchmark --sync --json — passed with 5 benchmark rows, no warnings, and no manifest synchronization drift
• Publication contract validation — benchmark discovered as experimental and all 3 verifier-fixture cases passed

Rollouts

Ran a live five-example end-to-end rollout on Slurm (job 7362340) through:

dataset -> Nemotron 3.5 Super VL EA target endpoint -> Gym -> Aegis v4 endpoint

• 5/5 rollout records were produced with no transport failures.
• Both unsafe prompts were classified as unsafe user requests, while the target model's refusals were classified as safe responses.
• Two completed benign responses were classified safe.
• One benign prompt exhausted the target model's 1,024-token limit in reasoning without producing a final answer; the environment correctly recorded it as unresolved rather than sending empty text to Aegis or treating it as safe.

The five real rollout records are included in resources_servers/aegis_v4_safety/data/example_rollouts.jsonl.

The benchmark-manifest follow-up is metadata and service-free fixture coverage, so it does not require another model rollout.

Compatibility and benchmark impact

This is additive and does not change existing environments or public APIs. The target model and Aegis model use separate endpoint configurations, so users can replace the target model without modifying the verifier. Existing-response scoring avoids a second target-model inference pass.

The default response_text_mode is final_only; reasoning_plus_final is opt-in. Unresolved results receive the configured unresolved reward and are reported separately from resolved safety rates.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Documentation and benchmark metadata added.
• Pre-commit checks pass locally (pre-commit run --all-files).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/ok to test 64fbf25

[cmunley1]

lgtm! do you want to add an entry in environments/ or benchmarks/? If it is eval only, you should put it in benchmarks/.

If for training, it seems a model may be able to reward hack by refusing everything to receive full score? Just a thought!

[muskank7]

Thanks! Yes, this is evaluation-only at this stage, so I’ll add an entry under benchmarks/ and keep Aegis v4 as the reusable resources server.
Agreed on the reward-hacking risk: the current safety-only reward could encourage blanket refusals, so it should not be used as a standalone training objective. Supporting training would require combining safety with task/helpfulness rewards and appropriate refusal handling.

[muskank7]

Added the requested benchmark entry in commit 0e0533c. It includes the benchmark manifest/config, project-authored five-row smoke data and preparation metadata, documentation, and a service-free verifier fixture. I also documented that the safety score is evaluation-only and can be reward-hacked by refusing everything, so it should not be used alone as an RL reward. Local validation passed: 23 Aegis tests, manifest validation with five rows and no warnings, all three verifier-fixture cases, legacy data validation, Ruff, and the full pre-commit suite.

[cmunley1]

/ok to test 0e0533c

[cmunley1]

is this intentional? 1k output seems small, if reasoning model.

[cmunley1]

/ok to test 005c848