Skip to content

fix(opensandbox): surface OOM status after exec 502 - #2604

Merged
bxyu-nvidia merged 4 commits into
mainfrom
hemild/fix-opensandbox-oom-status
Aug 19, 2026
Merged

bxyu-nvidia merged 4 commits into
mainfrom
hemild/fix-opensandbox-oom-status

Conversation

@hemildesai

@hemildesai hemildesai commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • inspect OpenSandbox lifecycle status after an exec backend returns HTTP 502
  • surface a typed OOM error with lifecycle context while preserving the original behavior for non-OOM failures
  • add race-aware unit coverage for status reconciliation and non-OOM behavior

Example OOM message

SandboxBackendUnreachableError: Sandbox was OOM-killed. OpenSandbox status: state='Failed', reason='FAILED', message='1/1 observed pods failed; primary reason=OOMKilled; sample pod=<pod-name>; container sandbox terminated with OOMKilled (exit code 137)'; sandbox_id='<sandbox-id>'

Test plan

  • pytest -q tests/unit_tests/test_opensandbox_provider.py (45 passed)
  • scoped pre-commit hooks on both changed files
  • local live E2E: in-flight exec returned 502, SDK reconciled to Failed / OOMKilled, Gym surfaced the OOM diagnosis, and cleanup succeeded

πŸ€– Generated with Claude Code

OpenSandbox already reports lifecycle details after a container OOMs, but
Gym returned only the proxy 502. Surface an actionable OOM error after a
bounded status reconciliation window while preserving non-OOM failures.

Add focused unit coverage and a low-resource end-to-end reproducer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The deployment-specific E2E script is useful for local validation but does
not belong in the shipped Gym repository.

Signed-off-by: Hemil Desai <hemild@nvidia.com>
Keep the OOM diagnosis and lifecycle status without prescribing a memory
configuration change to callers.

Signed-off-by: Hemil Desai <hemild@nvidia.com>
@hemildesai
hemildesai marked this pull request as ready for review August 18, 2026 23:01
@hemildesai

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

Copy link
Copy Markdown
Contributor Author

/ok to test 9e20c69

@hemildesai
hemildesai marked this pull request as draft August 18, 2026 23:03
@hemildesai
hemildesai marked this pull request as ready for review August 18, 2026 23:03
Raise the enriched OOM error as soon as reconciliation finds it instead of
storing and re-reading the same lifecycle fields.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@hemildesai

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

Copy link
Copy Markdown
Contributor Author

/ok to test 42f732d

@hemildesai
hemildesai marked this pull request as draft August 18, 2026 23:08
@hemildesai
hemildesai marked this pull request as ready for review August 18, 2026 23:09
@claude

claude Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

SHIP β€” LGTM, no reliability concerns.

Focused, defensive change to OpenSandbox exec error classification. On a submission/polling 502 (or an existing SandboxBackendUnreachableError), it polls read-only get_info for up to 5s and, if the status reports an OOM-kill, raises a more specific SandboxBackendUnreachableError; otherwise it re-raises the original error unchanged.

Verified:

  • No exec double-run risk. The added path only calls get_info (read-only status), never re-submits the command, so retry semantics are untouched.
  • Error routing is correct. Non-502 / non-typed exceptions re-raise immediately (provider.py:1244-1245); the hard-cap TimeoutError (no status code) propagates as before. get_info absent or raising both fall through to re-raise the original error.
  • Async is clean. Loop uses asyncio.get_running_loop().time(), await self._await_sdk_call(...), bounded per-call timeout_s=min(2.0, remaining_s), and await asyncio.sleep(...). No httpx, no ray.get(), no missing await.
  • Tests assert real behavior β€” OOM classification, message truncation (<800 / [:500]), __cause__ chaining, poll count, and the non-OOM re-raise path.

Worst case is a bounded ~5s extra latency on a 502 when the status stays non-terminal β€” acceptable for the improved diagnosability.

@bxyu-nvidia
bxyu-nvidia merged commit 1f972cb into main Aug 19, 2026
32 checks passed
@bxyu-nvidia
bxyu-nvidia deleted the hemild/fix-opensandbox-oom-status branch August 19, 2026 03:20

This branch was previously deployed

1 inactive deployment
public β€” 42f732d7 Deployed Aug 18, 2026 by copy-pr-bot[bot] via release / finalize / notify #1523
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants