Repository navigation
feat(aalcr): add versioned v1.1 protocol - #3152
Merged
Merged
Conversation
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Signed-off-by: Andrea Santilli <asantilli@nvidia.com>
Collaborator
Author
|
/ok to test 7cb16ba |
Contributor
|
/ok to test 7cb16ba |
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
## Summary - preserve the existing `aalcr` selector as the pinned v1.0 dataset and legacy judge protocol - add `aalcr/config_v1_1` for dataset revision `9a77ef56b717057ade24ceab4d273712a0b4f19e`, including its 16 corrected answers - configure the official GPT-5.6 Luna judge with medium reasoning for v1.1; read its credential from `JUDGE_API_KEY` with no fallback to another judge - implement the published v1.1 system prompt and strict JSON verdict protocol - attach dataset/protocol provenance to prepared rows and reject mismatched grading configurations - preserve the historical v1.0 behavior for valid, malformed, and empty judge responses ## Compatibility evidence - all 100 pinned v1.0 prepared rows match Gym `e446e4f` exactly after excluding the three additive provenance fields - v1.0 sends the exact historical user-only judge prompt and produces identical rewards for `CORRECT`, `INCORRECT`, malformed, and empty verdicts - legacy prepared rows without provenance fields load with v1.0 defaults - v1.1 changes exactly the 16 published answer keys; questions, documents, policy prompts, and other task inputs are unchanged - v1.1 sends the published system/user prompts, uses Luna-medium, accepts strict JSON `CORRECT`/`INCORRECT`, and rejects malformed or cross-version inputs ## Testing - `uv run pytest -q benchmarks/aalcr/tests resources_servers/aalcr/tests/test_app.py` (32 passed) - `uv run pytest -q tests/unit_tests/test_benchmarks.py` (49 passed) - pre-commit and secret-detector checks - `gym eval prepare --benchmark aalcr` (100 rows) - keyless `gym eval prepare --benchmark aalcr/config_v1_1` (100 rows) - config tests verify missing-key startup validation and forwarding `reasoning_effort=medium` with model `openai/openai/gpt-5.6-luna` - direct data and verifier A/B against Gym `e446e4f415b9cde0e95bb813c85e9e3e23f5d893` --------- Signed-off-by: Andrea Santilli <asantilli@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
katherineh123
pushed a commit
to katherineh123/Gym
that referenced
this pull request
Sep 18, 2026
…eMo#3510) ## Expected An empty or whitespace-only judge reply should produce a retryable `judge_failed` result, excluded from scores by default—not count as an incorrect policy answer. ## Actual AA-LCR's legacy protocol returns `reward=0` and `invalid_judge_response=True` for these replies, including the ungraded sample in the score. ## Fix Move the existing empty-reply check before protocol selection. Nonempty verdicts, empty policy answers, and v1.1 behavior are unchanged. This intentionally revises the legacy compatibility behavior preserved in NVIDIA-NeMo#3152; scores may change for runs containing empty judge replies. Validation: 231 tests and pre-commit passed. Still Draft: real-model rollout and isolated server/data validation remain incomplete. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
aalcrselector as the pinned v1.0 dataset and legacy judge protocolaalcr/config_v1_1for dataset revision9a77ef56b717057ade24ceab4d273712a0b4f19e, including its 16 corrected answersJUDGE_API_KEYwith no fallback to another judgeCompatibility evidence
e446e4fexactly after excluding the three additive provenance fieldsCORRECT,INCORRECT, malformed, and empty verdictsCORRECT/INCORRECT, and rejects malformed or cross-version inputsTesting
uv run pytest -q benchmarks/aalcr/tests resources_servers/aalcr/tests/test_app.py(32 passed)uv run pytest -q tests/unit_tests/test_benchmarks.py(49 passed)gym eval prepare --benchmark aalcr(100 rows)gym eval prepare --benchmark aalcr/config_v1_1(100 rows)reasoning_effort=mediumwith modelopenai/openai/gpt-5.6-lunae446e4f415b9cde0e95bb813c85e9e3e23f5d893