Skip to content

Flaky test: terminal_bench_4 test_verifier_boundary wall-clock bound #5434

Open

Description

@AdamRajfer

resources_servers/terminal_bench_4/tests/test_verifier_boundary.py failed once in CI on a wall-clock assertion and passed when the job was re-run on the same commit.

Where it failed: line 406, in the check that a slow verifier preparation stays within its budget:

assert 0.18 <= elapsed < 0.28
E       assert 0.8193796259984083 < 0.28

The test's fake sandbox sleeps between 0.12 s and 0.5 s per step and then asserts a tight elapsed-time window, so on a loaded runner the measured time can exceed the upper bound without any change in behaviour.

Run: "Unit tests / Server suite (shard 6)" on commit 445e407 (PR #5343). The first attempt failed; the re-run of the failed jobs passed. No commit in that PR touches terminal_bench_4.

Possible fix: assert the ordering and the budget enforcement rather than an absolute wall-clock window (for example, that the verifier stopped waiting at the budget and returned the expected reward), or widen the upper bound to tolerate scheduler delays.

Similar to #5429 and #5433.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions