unit-tests: DictionaryMultiPageTest drain reports success with the page not on disk (#6406 item 4) #6415

Split from #6406 item 4 ("a dictionary page is missing from the file after a drain that reported success"), which named this as possibly the one item in that batch that isn't a test bug.

The failure

DictionaryMultiPageTest.aDictionaryPageThatIsClaimedButMissingFailsLoudly fails on main's unit-tests lane with:

[the premise: the file really holds page 0 and nothing after it]
expected: 1L
 but was: 0L

right after the test asserts waitAllPagesOfDatabaseAreFlushed(other) returned true. So the drain reported success while PaginatedComponentFile.getTotalPages() (channel.size() / pageSize, not a cached value) still says the page never reached the file.

Current rate (checked 2026-08-18, last 16 completed mvn-test.yml runs on main): 3/16 (32099192174, 32116404436, 32150657917), always the identical assertion, always this exact test. Consistent with #6406's original "4 of the last 20" count.

What I ruled out (issue #6406's PR, l.garulli)

#6406 item 4 suggested this lead: every per-database map in PageManagerFlushThread is keyed by LocalDatabase.equals() (compares databasePath, not instance identity), and the test's other database could be aliasing a stale entry. That hazard is real in general but does not apply here:

• It was already fixed for flushedPagesPerDatabase and slotsInUse by Follow-ups from #6281: path-keyed flush-thread maps, an unchecked worker pause, and async-dispatched DDL #6303/fix(#6303): a stale batch stops answering for the database at its path, an index build waits for the workers to actually park, and async DDL runs off them #6309 (both now carry their counter on the batch itself, resolved once at enqueue time, never re-looked-up by path).
• pageIndex's own per-database counter (FlushPageIndex.pending) is guarded by removeAllOfDatabase running under the database's own write lock, which excludes a concurrent putAll for the same database (see the class's own javadoc). It isn't guarded against a different, path-aliased database, but this test's other database is created fresh (TestHelper.createDatabase drops any existing DB at that path first) and nothing else in the suite uses that exact path, so there's no other instance to alias with.
• I traced every possible interleaving of FlushPageIndex.putAll/removeIfSame for the shard's known two-commit sequence (schema init writes page 0's header, then getIdByName's addItemToPage modifies the same page again) by hand. The counter's arithmetic is self-correcting under every ordering I could construct - it never crosses zero while a newer, unflushed copy of the page is still the one indexed. I could not find a hole in the counting mechanism itself.
• flushPage's "file dropped" branch (fileManager.existsFile(fileId) false, releases the WAL ack without writing) is the only path I found that decrements the pending count without writing bytes - but it requires the file to have been dropped, which never happens to the dictionary file in this test.

What I tried and couldn't reproduce

• 3300+ iterations of the test's exact logic in a single JVM, with 4 concurrent noise databases hammering the shared flush thread to maximize scheduling jitter: zero repros.
• 15 fresh-JVM runs of just this test method (mvn test -Dtest=DictionaryMultiPageTest#aDictionaryPageThatIsClaimedButMissingFailsLoudly): zero repros.
• 2 full engine module suite runs (~12750 tests each, matching CI's actual invocation) locally: zero repros, including of this specific test.

So this reproduces on CI's runners but not on a dev machine even under deliberately adversarial local conditions - suggests either something specific to the CI runners' I/O/scheduling characteristics, or a window narrower than what I could force locally.

A data point for whoever picks this up

A fork run gathering fresh CI data for this investigation noticed a broader shape in the same time window: three other, unrelated tests also failed on "expected a positive/non-zero value, got 0" assertions in this period - Issue6111StaleSnapshotReadFloorTest.retryFiresAndResolvesTheFloorWhenNothingIsRunning (retry counter), Issue5596MergeCoverageTest.aCleanAppendConflictIsStillMerged (merge counter), MaterializedViewRefreshAtomicityTest (index size on disk, slow-unit-tests lane). Whether these share a root cause with this one or are coincidentally-shaped unrelated flakes is unknown - flagging it because "the same shape of assertion failing across several unrelated flush/counter-adjacent tests in the same window" is the kind of thing worth checking before assuming they're independent.

Suggested next steps

• Add temporary diagnostic logging around waitAllPagesOfDatabaseAreFlushed / flushPage (e.g. log the exact branch taken in flushPage - written vs. file-dropped vs. exception - for every page of this test's database) and let it ride in CI a few times to catch the mechanism in the act, since local reproduction has not worked.
• Check whether the CI runners' filesystem (overlay/network FS on GitHub Actions) has different FileChannel.write/size() visibility semantics than a local disk - PaginatedComponentFile.write() and getTotalPages() both assume a write() that returns has extended channel.size() for a same-process, same-channel read, which is POSIX-guaranteed but worth confirming holds on the actual runner image.
• If the broader "0L" pattern above recurs, treat it as the same investigation rather than as separate flakes.

CC @lvca