[wprazuch]

What is wrong

swebench_pro retries an inconclusive verification on a fresh sandbox
(inconclusive_verification_retries, default 2 → 3 attempts at app.py:350). The one timeout that
exists does not cover most of an attempt, and nothing bounds the attempts in aggregate:

• evaluation_timeout (shipped as 3600) is passed to run_verification(timeout_s=...), so it
  bounds only the test command inside the sandbox.
• self._create_sandbox(...) is not covered by it.
• eval_sandbox.stop() in the finally block is not covered by it.
• 3 × 3600 s is therefore a legal budget for one rollout — more than what remains of a 4 h job.

Because rollout collection ends only when the last rollout does, one instance can hold an entire run
open. Every other rollout can be finished and every GPU idle, and the job still runs to its wall
clock and is killed.

Evidence

A 2193-rollout SWE-bench Pro run on 20 nodes, killed at a 4 h wall with 2192 rollouts complete.
2185 of 2193 were done at 2.22 h; the 2192nd landed at 2.93 h; the last one never did. From ~2:03
elapsed the serving cluster was idle — decode reporting Running: 3-7 reqs.

The one unrecorded rollout had finished its agent phase at +0.94 h. Reconstructed from the on-disk
verification artifacts (logs/run_evaluation/<session>/<instance>/ mtimes):

event	duration
verification attempt 1	3630 s
verification attempt 2	3641 s
attempt 3	started, ~53 min elapsed when Slurm killed the job

Both completed attempts ran to the full evaluation_timeout and returned; the third was inside
its budget when the wall hit. The rollout consumed 2 h 55 m of verification and was on track for
~3 h 01 m. Nothing hung — the budget is simply larger than the job.

For scale, per-rollout verification totals derived from the same artifacts are p50 66 s, p99 461 s,
and the healthy maximum is 735 s. (patch_verification_time_taken in the rollout record is
last-attempt-only — app.py overwrites it each attempt and zeroes it on exception — so it reports
66/431/735 and is structurally unable to show this failure.)

The instance is not systematically bad: across four runs it has 13 verification sessions, and 12
of them took 115–644 s. The trigger was this rollout's own patch — it changes a hook's return
signature across 9 files, the patch applied cleanly, and the jest run started and never terminated.
A model patch that hangs a test suite is normal and expected. Spending three hours on it is not.
That is also why quarantining the instance is not the fix.

What this changes

• verification_attempt_timeout (default 1200 s) bounds one attempt, sandbox creation included.
  That is ~2.6× the observed p99 of 461 s and above the healthy max of 735 s.
• verification_total_timeout (default 2700 s) bounds all attempts for one rollout — the
  load-bearing cap. Each attempt's ceiling is clamped to whatever is left of it.
• verification_stop_timeout (default 120 s) bounds the teardown in finally, which would
  otherwise defeat the attempt timeout it runs after.
• A timed-out attempt takes the existing failure path, so the rollout is scored and recorded.
• Setting the ceilings to null restores the previous behaviour exactly.

Tests cover a hung attempt, the aggregate budget cutting the retry sequence short, the unset-ceiling
path still using all three attempts, and the clamping arithmetic.
resources_servers/swebench_pro/tests/ is green (48 passed).

Independent confirmation: a replay of the shipped certificate fails the same way

The configuration behind this benchmark's published certificate was re-run unchanged — same
container, same Gym pin, same 4P/6D shape on 10 nodes, composed from the published release tag. The
original run of that configuration completed in 3:47:35. The replay TIMED OUT at 04:00:17 with
2192 of 2193 rollouts.

The single missing rollout was in verification. Its instance has a Retrying verification ... on a new sandbox line in the resources-server log with no successor — no "still inconclusive after 3
attempt(s)", no completion — and the log stopped writing 22 minutes before the wall.

That is the same signature as the 20-node run described above, five days apart and at a different
node count. Two independent runs now end one rollout short with that rollout inside the retry
budget. It also rules out the alternative explanation we were carrying, that a recent serving-stack
change had caused a throughput regression: the pre-change configuration does not reliably finish
either. Aggregate throughput is not what decides these runs; the verification tail is.

Scope: this does not fix every SWE-bench Pro timeout

Across four TIMEOUT runs of this benchmark, 86 of 89 unrecorded rollouts were sitting in
verification and 3 were still in the agent — so verification dominates. But the proximate causes
differ, and only one of the four is the defect fixed here:

• 20-node run: one rollout, 2 h 55 m of verification. Without it the job finishes with 59 minutes
  to spare. Fully attributable.
• Two 10-node runs: missed the wall by 4.0 and 4.2 minutes with 4 and 3 rollouts outstanding.
  Fixing verification perfectly still leaves them short. These are makespan misses.
• A fourth run: 81 rollouts short, with 56 verification attempts in flight and incomplete at
  the wall, ages 768–1950 s against a p50 of 60 s — all within their 3600 s budget. That looks like
  verification-tier contention, not one unbounded rollout. The 1200 s per-attempt default would have
  cut 55 of those 56; the aggregate cap alone would have cut none.

For the benchmark PIC — a separate scoring question

On one run, 558 of 2189 rollouts (25.5%) across 225 of 731 instances exhausted all three
attempts and were still inconclusive (360 "N of M graded tests never reported an outcome", 198
"parser reported no tests at all"), and are scored 0.

These look correct rather than buggy: test_stderr.log shows the graded suites failing to compile
against the model's patch (unknown field clientFunc, not enough arguments in call to sess.dial,
Cannot find module ..., [build failed]), 67 of the 225 affected instances have at least one
conclusive rollout elsewhere in the same job and 61 have a reward-1.0 rollout, and retries recover
only 4 of 562 (0.7%) — deterministic, not flaky infrastructure.

So this is a scoring-policy question, not a verifier bug: should a rollout whose patch prevents the
graded suite from reporting be scored 0, or excluded and reported as a coverage number? Worth a
decision either way, since it moves a quarter of the sample.