Repository navigation
[GDPVal] Add opt-in persistence of raw judge responses - #1225
Merged
Merged
Conversation
1 of 2 tasks
agronskiy
approved these changes
May 4, 2026
Kh4L
marked this pull request as ready for review
May 4, 2026 20:54
agronskiy
force-pushed
the
spanev-gdpval-persist-raw-judge-responses
branch
from
May 4, 2026 21:29
07a42ae to
dcfa377
Compare
Signed-off-by: Serge Panev <spanev@nvidia.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
agronskiy
force-pushed
the
spanev-gdpval-persist-raw-judge-responses
branch
from
May 4, 2026 21:32
dcfa377 to
3ecf805
Compare
agronskiy
enabled auto-merge (squash)
May 4, 2026 21:32
This was referenced May 5, 2026
marta-sd
added a commit
that referenced
this pull request
Jul 10, 2026
…1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post #1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces #1215 Supersedes #1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with #1225 (raw_responses kwarg on `run_trials`), #1228 (preconvert + `clean_up_list`), and #1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Jul 15, 2026
…VIDIA-NeMo#1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post NVIDIA-NeMo#1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces NVIDIA-NeMo#1215 Supersedes NVIDIA-NeMo#1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with NVIDIA-NeMo#1225 (raw_responses kwarg on `run_trials`), NVIDIA-NeMo#1228 (preconvert + `clean_up_list`), and NVIDIA-NeMo#1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com> Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a config flag
persist_raw_judge_responses: bool = FalseonGDPValResourcesServerConfig. When True, the raw judge completion text from every trial is preserved onverify_response.judge_response:judge_response.per_ref_repeat[i].raw_responses: list[str](one per trial, ordered by trial index; trialiwas swapped iffi % 2 != 0).judge_response.raw_responses: list[str].Off by default. Raw responses are 10–50 KB ×
num_trials×num_ref_repeats×num_tasks, so always-on would meaningfully bloat rollout JSONLs and W&B uploads. Turn on for debug runs to post-mortem judge verdicts (e.g. understanding why a comparison-mode run lost 87% of pairwise comparisons).Why
When a comparison-mode or rubric run produces unexpected scores, the only persisted signal today is parsed counts/numbers — the judge's actual rationale, criterion-by-criterion grades, and verdict text are fetched and discarded. With this flag, those raw strings survive into the rollout JSONL so post-mortem analysis is possible without re-running.
Dependency
The structured-rubric scorer modified here is added by #1222. Until #1222 lands, this PR's diff includes its commit; after #1222 merges, a rebase against main auto-drops the duplicate (patch-id match).
Test plan
test_persist_raw_judge_responses_comparison,test_persist_raw_judge_responses_rubric— verify both modes pass the flag through to the underlying scorers and that returned raw text appears in the response.raw_responsesfield shape (in flight on HSG).Replaces #1224
Supersedes draft #1224 which carried 9 commits / 826 lines from a stale Kh4L:dev stack (5 already-merged-to-main with different SHAs). Closing that one.