[gchlebus]

What does this PR do?

NeMoGymAsyncOpenAI._request_with_retry treated every 429 as a transient rate limit. When the upstream key was actually spent (budget_exceeded / insufficient_quota), each request burned all its retries against a dead endpoint, and every later judge or model call on the same client did the same. In long eval jobs this meant hours of rollouts silently scored against a key that would never answer.

This PR adds a per-client fail-fast latch to the shared OpenAI client:

• A 429 whose parsed error.code / error.type is budget_exceeded or insufficient_quota trips the client. Generic "quota exceeded" wording (used by per-minute limits that recover) still retries as before.
• A 401/403 whose body names an invalid key (invalid_api_key, invalid_api_token, authentication_error, or "invalid/incorrect api key" text) also trips the client. Other 401/403 responses are returned once, unchanged from today.
• Once tripped, every subsequent _request / _request_with_retry call on that client raises PermanentEndpointError without touching the wire. In-flight retry loops re-check the latch before their next attempt, so sibling tasks stop as soon as one trips.
• PermanentEndpointError subclasses aiohttp.ClientResponseError, so existing except ClientResponseError handlers (for example the judge paths that read .status) keep working. Each raise builds a fresh exception instead of re-raising a cached one, so rejected requests are not retained.
• server_utils.raise_for_status accepts pre-read content, because the retry loop now reads the body before deciding whether to retry. This lets the final attempt raise with the real error body instead of an already-consumed stream.
• The max_http_attempts row in the OpenAI model-server docs now describes the retryable set and the fail-fast cases accurately.

The latch is per NeMoGymAsyncOpenAI instance. Session pinning / failover across multiple model-server instances is out of scope here.

No GitHub issue: this came out of an internal eval run where a spent inference key was retried for the remainder of the job. Reviewed and approved internally before being moved here.

Validation

Run against the final diff in this branch:

uv run --extra dev pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_server_utils.py -q
# 318 passed

uv run pre-commit run --files fern/versions/latest/pages/model-server/openai.mdx nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.py tests/unit_tests/test_server_utils.py
# all hooks pass (ruff-format made one blank-line fix, folded into the second commit)

New tests cover: the quota and auth body classifiers (parametrized over real OpenAI / inference-gateway error shapes), fail-fast on spent-key 429 and invalid-key 401/403, generic-quota and transient 429s still retrying, exhausted transient 429 raising without tripping, auth errors without a key marker being returned once, requests skipping the wire after a trip, an in-flight retry stopping after a sibling trips, create_chat_completion surfacing the permanent error's .status, and raise_for_status accepting prefetched content.

Rollouts

N/A. This changes HTTP retry behavior in the shared client, not environment or agent logic. The behavior is exercised end to end in unit tests with mocked upstream responses. Not exercised against a live spent key.

Compatibility and benchmark impact

• No config changes. max_http_attempts semantics are unchanged for transient errors.
• Behavior change: after a spent-key or invalid-key response, later calls on that client raise PermanentEndpointError (a ClientResponseError with the original .status) immediately instead of retrying. Callers that previously saw the 429/401/403 after exhausting retries now see it on the first request and without the retry delay.
• No benchmark-result impact for healthy endpoints.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header. (No new files.)
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-gchlebus-judge-quota-fail-fast.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/model-server/openai

[gchlebus]

/ok to test

[copy-pr-bot]

/ok to test

@gchlebus, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[gchlebus]

/ok to test 590cd97

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35636030089