Repository navigation
HA: Ratis SegmentedRaftLog.close() can deadlock with an in-flight cache eviction #9556
Description
Summary
Found while writing the regression test for #9549 (PR #9552). In Apache Ratis 3.3.1, SegmentedRaftLog.close() takes the log's write lock and then calls cacheEviction.close(), which interrupts and joins the cache-eviction thread. If that thread was woken just before (by a cache miss or a segment roll) and shouldEvict() was true, it is parked in writeLock() in checkAndEvictCache(). ReentrantReadWriteLock.lock() ignores the interrupt, so close() waits forever.
main: SegmentedRaftLog.close(SegmentedRaftLog.java:536) -> AwaitToRun.close -> Thread.join
n...-cacheEviction-AwaitToRun: SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:334) -> RaftLogBase.writeLock -> park
Reproduced locally with a bare SegmentedRaftLog (open, read entries that miss the cache, close right away). The test hung for 30 minutes until it was killed. Issue9549RaftLogCacheBoundTest now waits for the eviction thread to go idle before closing.
Why it matters for ArcadeDB
A server shutdown or in-place restart (RaftHAServer division close) that races with a catch-up read or a segment roll could hang in the Raft log close. The window is small, but it gets wider when evictions are frequent. #9552 makes them more frequent on small heaps, because fewer segments are cached.
To do
- Check whether this is fixed in a newer Ratis release, or file it upstream.
- Check whether the ArcadeDB division close / in-place restart path has a timeout that would turn this hang into a recoverable failure.
Activity
address review: warn once per JVM, guard logSegmentSize parsing, docu…
fix(#9556): stop the Raft log cache-eviction thread before closing th…
robfrank commentedon Oct 9, 2026
Worked around by #9565. Ratis 3.3.1's log close holds the log's write lock while it joins a cache-eviction thread that is parked on that same lock. Upstream fixed it as RATIS-2719, but no Ratis release ships it yet (3.3.1 is still the latest). A new class stops each division log's eviction thread before the close takes the lock, which is the same reordering as the upstream fix. It is called before all three Ratis server closes in
RaftHAServer(stop, in-place restart, aborted restart) and waits at most 10 s.Status of the points in the issue: (1) the deadlock: worked around for closes ArcadeDB performs; fixed upstream but unreleased. (2) Neither stop nor the in-place restart puts a time limit on its own Ratis close: not fixed, tracked in #9561.
Still open:
Issue8900ClosedServerInPlaceRestartITfails in teardown (DatabaseAreNotIdentical, types 8 <> 7); it fails onmaintoo, so it is not caused by this change.❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.
fix(#9556): stop the Raft log cache-eviction thread before closing th…