You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
resources_servers/terminal_bench_4/tests/test_verifier_boundary.py failed once in CI on a wall-clock assertion and passed when the job was re-run on the same commit.
Where it failed: line 406, in the check that a slow verifier preparation stays within its budget:
The test's fake sandbox sleeps between 0.12 s and 0.5 s per step and then asserts a tight elapsed-time window, so on a loaded runner the measured time can exceed the upper bound without any change in behaviour.
Run: "Unit tests / Server suite (shard 6)" on commit 445e407 (PR #5343). The first attempt failed; the re-run of the failed jobs passed. No commit in that PR touches terminal_bench_4.
Possible fix: assert the ordering and the budget enforcement rather than an absolute wall-clock window (for example, that the verifier stopped waiting at the budget and returned the expected reward), or widen the upper bound to tolerate scheduler delays.
Activity