Repository navigation
FixedGrpcLogAppender.forceMatchIndex rewinds Ratis FollowerInfoImpl.matchIndex via reflection with no synchronization, violating monotonicity #4833
Description
Affected version: 26.6.1
Component: FixedGrpcLogAppender.getNextIndexForInconsistency / forceMatchIndex
Severity: MEDIUM
Summary
On the heartbeat-inconsistency path the override reflectively reads the private
matchIndex field of FollowerInfoImpl and calls `setUnconditionally(replyNextIndex
- 1, …)
, deliberately rewindingmatchIndexdownward. Ratis keepsmatchIndexmonotonic (max) precisely because commit-index computation and the SUCCESS handler assume it never moves backward. The read-check-write is not atomic against concurrentupdateMatchIndex` / commit computation.
Code
ratis_FixedGrpcLogAppender.java:63-104 (reflective setUnconditionally; silent
no-op WARNING fallback at :92-103 if the Ratis field layout changes).
Impact
Rare race on the leader's match-index bookkeeping during follower-restart-on-idle
recovery; correctness depends on Ratis-internal semantics the override violates. If
a future Ratis version renames the field, the workaround silently no-ops and
re-exposes RATIS-2523.
Suggested fix
Prefer the public API or upstream the fix; if reflection stays, rewind under the
same lock Ratis uses for FollowerInfoImpl, assert newMatchIndex >= 0, and add a
startup self-check that fails loud (not silent no-op) on a field-layout change.
Activity
lvca commentedon Jun 30, 2026
Confirmed and fixed in main.
The HA leader could, in a rare timing window, push a follower's progress marker backwards while a normal reply was raising it at the same time. The reset is now done as a single atomic step that only applies if nothing changed in the meantime, so a legitimate update always wins and the marker never moves backward.
I also hardened the internal workaround so that if a future Apache Ratis upgrade changes its internals, the node fails fast at startup with a clear error instead of silently skipping the fix. Added regression tests covering the concurrent case and the startup self-check.
Note: this is an internal robustness fix; clusters were not losing data because of it.
fix(#4833): make RATIS-2523 matchIndex rewind atomic and fail-loud
fix(#4833): make RATIS-2523 matchIndex rewind atomic and fail-loud