Repository navigation
fix(rolemrc): correct BLEU/METEOR + judge wiring, report published roll-ups; add gym-native RAGTruth/RoleMRC benchmarks - #2822
Merged
bxyu-nvidia merged 10 commits intoSep 15, 2026
Conversation
- BLEU: sacrebleu.sentence_bleu is smoothed and 0-100, so it never emits a hard 0 and ran several times above upstream. Reimplement evaluate.load("bleu") (13a tok, max_order=4, unsmoothed).
- METEOR: concurrent NLTK WordNet lookups corrupted a shared file handle and _safe_call turned the failures into silent 0s. Serialize behind a lock and warm wordnet/punkt at startup.
- compute_metrics: emit auto/<metric>/mean corpus means and the judge roll-ups avg_simple / avg_weighted / avg_simple_no_mt (the headline). These aggr
per aspect, so they are not recoverable from a mean of per-row rewards.
42 -refused rows; avg_simple* are count-independent and match.
- Judge ran on policy_model, i.e. graded its own output. Wire a dedicated judge_model via ${judge_*}, and add rolemrc_judge_serve.yaml for harnesses
that own generation (rolemrc_judge.yaml fails validation there on its a policy_model reference).
- Send only judge params set in config, dropping nulls, so reasoning mode that reject temperature/top_p can omit them. An empty judge reply is now a judge failure with a reason, not a parsed 0.
- prepare_rolemrc.py asserts the upstream task histogram (ROLEMRC_ALLOW_DATASET_DRIFT=1 overrides) and prints per-aspect call co
- ragtruth: strip trailing space from pinned _PUBLIC_COMMIT (malformed URL).
Tests: TestJudgeFailureReporting, TestJudgeSamplingParams.
Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com>
… wiring only; each chains to the resources server's config and inherits its *_simple_agent. Scoring and aggregation stay in the resources server. - ragtruth: all three slices in one file, so the headline is the upstream all-slice accuracy while compute_metrics still splits by task_type. - rolemrc: config.yaml (reference mode, no model server needed for scoring) and config_judge.yaml (5-aspect judge, requires a separate judge endpoint via +judge_base_url / +judge_api_key / +judge_model_name) prompt_config: null throughout — rows already carry a built responses_create_params.input. Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com>
github-actions
Bot
added
the
sla:triage-overdue
- Carry the conversation in `verifier_metadata`. A caller that only POSTs /verify may not deliver `responses_create_params`, which dropped the system instruction and earlier turns from the judge prompt. - Add `judge_api` (`chat_completions` default, `responses` opt-in) with a matching `judge_chat_completion_create_params`. Not cosmetic: on identical prompts the same judge scored `judge/avg_simple_no_mt` 0.55 through /v1/chat/completions vs 0.31 through /v1/responses. - Reasoning traces are stripped by splitting on `</think>` / `<channel|>`; nested-instruction lead-ins are removed anywhere, not just as a prefix; a verdict must be a bare integer, anything else is a bad aspect scored 0. - Judge budget to `max_tokens: 2048` — a reasoning judge spends it before emitting the score. Also a comment pass over rolemrc, ragtruth, spartqa: comments explain the code instead of comparing it to the original benchmark, and no longer name the external runner that drives the Gym servers. Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com>
…ragthuth-conf-fix
…ragthuth-conf-fix # Conflicts: # resources_servers/ragtruth/prepare_ragtruth.py # resources_servers/rolemrc/configs/rolemrc_judge.yaml
Collaborator
Author
|
@cmunley1 could you please take a look? The changes are minor but important to run correctly with gym. |
bxyu-nvidia
previously approved these changes
Sep 4, 2026
Contributor
|
/ok to test 04b72fb |
github-actions
Bot
removed
the
sla:triage-overdue
… in verifier_metadata Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com>
Contributor
|
/ok to test ed61ca0 |
bxyu-nvidia
approved these changes
Sep 15, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RoleMRC's numbers didn't reproduce the published report, and the judge config
graded the model under test with itself. Also adds
benchmarks/entries soRAGTruth and RoleMRC run via
gym env start --benchmark …, not only through anexternal harness hitting
*_serve.yaml.Fixes
evaluate.load("bleu")reimplemented — 13a tok, max_order=4, unsmoothedmean_reward(= ROUGE-L) + per-dimension meansauto/*/mean+judge/avg_simple{,_no_mt}/avg_weightedpolicy_model— self-gradingjudge_modelPlus:
rolemrc_judge_serve.yaml(judge/verify, no agent —rolemrc_judge.yamlfails validation there on its agent's
policy_modelref); null judge params aredropped so reasoning models can omit
temperature/top_p;prepare_rolemrc.pyasserts the upstream task histogram; RAGTruth's pinned commit had a trailing
space that malformed the download URL.
Judge fixes (
resources_servers/rolemrc/)Conversation —
prepare_rolemrc.pywritesverifier_metadata: {"conversation": messages}on judge rows (reference rowsunchanged);
app.pyadds_normalize_turns()and_conversation_messages(),which prefers that and falls back to
responses_create_params.input.data/example_judge.jsonlregenerated.API surface — new
judge_api+judge_chat_completion_create_paramsconfigfields (
judge_responses_create_paramskept forjudge_api: responses);model_post_initvalidates the field the surface needs;_judge_payload()and_call_judge_aspect()build and unwrap either dialect. Both judge configsswitched to the chat body (
messages: [],n: 1,temperature: 0.0,max_tokens: 2048).Text handling —
_strip_reasoning_for_judge()replaces_strip_think()onthe policy response;
_extract_nested_content()replaces lead-ins anywhere andfalls back to the full string without a
". ";_parse_judge_score()returns(score, is_bad), requires a bare integer, does not clamp;_NESTED_INSTRUCTIONkeeps a significant trailing space as
\x20.requirements.txt:+regex.Testing
gym env test --resources-server rolemrc.Fill in the Testing line — I couldn't verify you ran real rollouts.## Checklist
pre-commit run --all-files) (so CI lint/format/copyright pass).git commit -s) (so the DCO check passes).