[pachmu]

AgentIF — agentic instruction-following benchmark

Adds a resources server for AgentIF, ported from the THU-KEG
AgentIF benchmark (707 real-world
agentic scenarios). Each scenario carries a list of natural-language constraints
that the model's response must satisfy; constraints are scored by a pipeline of
typed checkers declared in the data.

What it does

Each constraint runs through a checker pipeline whose step types are:

Step type	Mechanism
llm	Calls the judge model; rewrites the working text to the judge's output
llm_conditional_check	Calls the judge to test if the constraint applies; skips (→ None) if not
code	Executes a dataset-provided check_following(response) in a sandboxed globals dict

The pipeline resolves to True / False / None (None = unscored: conditional
guard unmet, judge failure, or code error). Two headline metrics follow upstream:

Metric	Definition
CSR (Constraint Success Rate)	n_true / (n_true + n_false) across all scored constraints
ISR (Instruction Success Rate)	fraction of rows where every scored constraint passed

Metrics

• csr — corpus-level constraint success rate (primary reward signal)
• isr — instruction success rate (all-or-nothing per row)
• mean_reward — mean per-row CSR
• by_dimension/{vanilla,condition,example}/accuracy — per-dimension breakdown
• by_type/{formatting,semantic,tool}/accuracy — per-type breakdown
• n_null_total — total unscored constraints (diagnostics)

Contents

• app.py — verify() (constraint pipeline + CSR reward) + compute_metrics() /
  get_key_metrics(). Strips <think> blocks before scoring. Code checkers run
  in an executor; judge concurrency is bounded by asyncio.Semaphore.
• prepare_agentif.py — converts the upstream THU-KEG/AgentIF eval.json into
  Gym JSONL format; strips pre-existing score keys so the verifier re-derives
  verdicts from the model's generation.
• configs/agentif.yaml — simple-agent + policy + judge model wiring for
  eval/training.
• configs/agentif_serve.yaml — serve-only config (resources server + judge, no
  policy) for nemo-evaluator gym://...protocol=native flow.
• data/example.jsonl — committed synthetic samples covering constraint types and
  dimensions.
• tests/test_app.py — checker pipeline (all step types), conditional branching,
  upstream parity (code-fail → llm skipped), compute_metrics, dimension/type
  breakdowns, key-metric selection. 39 tests.

Testing

ng_test +entrypoint=resources_servers/agentif      # 39 passed
ruff check resources_servers/agentif/              # clean

Notes

• verified: false — not yet baselined. Reward profiling across an
  open-instruct / open-thinking / closed-source model suite is a follow-up
  before flipping to verified: true.
• Recommended judge: gpt-4o-mini (upstream used gpt-4o-2024-11-20). Judge
  credentials are passed at launch via +judge_base_url=... / +judge_api_key=...
  / +judge_model_name=....
• code checkers use exec in a fresh per-call globals dict (never the process
  globals()), so concurrent rollouts cannot interfere. The AgentIF dataset is
  trusted (curated by THU-KEG); checkers only inspect the model's response string.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/claude review

[claude]

BLOCKER — code checkers that reference top-level helpers/constants silently score None, corrupting CSR/ISR.

exec(exec_src, shared_globals, local_vars) with two distinct dicts uses Python's module-vs-class-body semantics: every top-level def/assignment binds into local_vars, but each defined function captures shared_globals as its __globals__. So when check_following calls a sibling helper or reads a module-level constant defined in the same source, the lookup goes to shared_globals + builtins and never local_vars → NameError. _run_code_check catches it, returns (None, err), and _score_constraint demotes the constraint to None (dropped from ISR/CSR).

Not hypothetical — data/example.jsonl row 2 defines top-level helpers (get_company_info, get_sub_company_info, get_company_register, …) called from within the checker. Those constraints all silently drop out of scoring. The separate import-only loop above rescues import statements but not helper defs or constants, so the workaround is incomplete.

BLAST RADIUS: every AgentIF row whose code checker uses a helper or module-level constant scores None instead of True/False. CSR/ISR are computed only over scored constraints, so reported numbers are silently biased toward trivial single-function checkers — wrong eval numbers in model reports, wrong reward signal if used for training.

FIX: exec into a single namespace so top-level names and function __globals__ coincide:

ns: dict = {"response": working}
exec(exec_src, ns)                       # one dict → helpers/constants visible to check_following
if "check_following" not in ns:
    return None, "check_following not defined"
return ns["check_following"](working), None

This also makes the separate _IMPORT_RE pre-exec loop unnecessary. Add a test with a helper-referencing checker — current tests only cover self-contained check_following bodies, which is why this passed CI.

[claude]

HOLD — one correctness BLOCKER in the code-checker exec path; everything else is solid.

BLOCKER (inline on resources_servers/agentif/app.py:146)

_run_code_check runs exec(exec_src, shared_globals, local_vars) with two distinct dicts. Under Python's module-vs-class-body semantics, top-level defs/constants bind into local_vars while each defined function captures shared_globals as __globals__ — so any check_following that calls a sibling helper or reads a module-level constant raises NameError, gets caught, and the constraint silently scores None (dropped from CSR/ISR). data/example.jsonl row 2 already contains such checkers (get_company_info, get_sub_company_info, …). Result: silently biased CSR/ISR in eval reports / reward signal. Fix: exec into a single namespace (exec(exec_src, ns) with ns = {"response": working}), which also removes the need for the import-only pre-exec loop. Add a helper-referencing checker test — current tests only exercise self-contained bodies, so this slipped past CI.

Reviewed and clear

• Async correctness: judge calls go through server_client.post (aiohttp) with an asyncio.Semaphore; code checkers run via run_in_executor off the event loop; no httpx, no ray.get(), no missing awaits. Good.
• Judge failure handling: _call_judge swallows exceptions → None → constraint scored None, no hang/crash. Reasonable.
• Metrics contract: compute_metrics/get_key_metrics signatures match the mixin; reward = ISR per-row, CSR corpus-wide — internally consistent with the tests.
• Config conventions: null judge_* defaults at file end, verified: false, serve-vs-agent split documented; TypedDict defaults not abused.
• Ancillary changes (ragtruth trailing-space commit-hash fix, rolemrc license CC→MIT) are low-risk metadata edits.

Once the exec namespace fix lands with a regression test, this is a ship.

[cmunley1]

would be better to use a sandbox if possible

[pachmu]

Not sandboxing check_following — it's curated upstream data (THU-KEG), not untrusted input, same trust level as the benchmark's own code. Happy to revisit if we ever take community-submitted checkers.

[cmunley1]

some minor comments on verifier robustness, good to merge if you want

[cmunley1]

This check seems a little weak, if it contains both words it might be misclassified? e.g. NO, ... not YES receives reward 1.0

[pachmu]

added _parse_yes_no() function to address this issue

[cmunley1]

Should judge/verifier failures count as evaluation failures rather than being dropped as null? one passing constraint plus one verifier exception can return reward 1.0.

[pachmu]

Fixed, errors now count as evaluation failures for the row: isr_counted covers scored-or-errored constraints and isr_pass requires n_error == 0.

[pachmu]

@cmunley1 the comments has been addressed

[cmunley1]

/ok to test 0f8f4fa