Repository navigation
[ha-raft] Define the user-visible contract for post-quorum local commit failure (committed-but-reported-failed)Â #5064
Description
Found while fixing #4940 (PR #5061). In RaftReplicatedDatabase.applyPhase2 (~415-445), when commit2ndPhase fails locally AFTER the Raft entry is quorum-committed, reconcileLeaderPagesAfterPhase2Failure persists the pages from the payload WAL bytes and the exception is rethrown: the user is told the commit FAILED while the data is durably committed cluster-wide.
With #5061's fix, a pre-local-WAL failure in that flow additionally resets user-held new-record RIDs to provisional, so an application-level retry of the same objects would INSERT DUPLICATES rather than update the already-reconciled records.
Pre-existing anomaly inherent to post-quorum local failure (the #4740 design steps the leader down). Proposal: surface a distinct 'committed-but-local-apply-failed' error type instead of a generic commit failure, so applications can distinguish 'retry' from 'already durable'.
Discovered during the 2026-07 audit follow-up (#4962).
Activity
Merge pull request #5075 from ArcadeData/fix/5064-committed-remotely
lvca commentedon Jul 7, 2026
Resolved in #5075 (merged, option A from the design discussion).
The contract: when the replication quorum durably commits a transaction but the leader's local phase-2 apply fails, the application now receives
TransactionCommittedRemotelyException(engine exception package, catchable without an ha-raft dependency): the transaction IS committed cluster-wide, the message states whether local pages were reconciled, and it must NOT be retried.The duplicate-insert fix: a
remotelyCommitteddurability regime inTransactionContext, set by the HA layer after quorum commit - a local failure past that point releases resources without rolling back user-held record identities (they are the identities the cluster committed) and without fencing (pre-append only; post-append still takes the intentional #5053 fence). Application retries no longer insert duplicates of already-committed records.The wire contract, all three layers: raw over embedded API; HTTP maps it to a non-retryable 409 (never a retry-inviting 5xx), including when wrapped by script/auto-commit wrappers; follower-forwarded writes reconstruct the exact type via the leader-exception factory instead of collapsing it to a generic TransactionException.
Verified red-first at every layer: 3-node IT (embedded + over-the-wire 409), HA unit for the ALL-quorum recovery path, engine regime test on the reused-context sticky-flag hazard, server handler tests incl. wrapped variants. Eight review rounds, the last four endorsements.
fix(ArcadeData#5064): distinct contract for post-quorum local commit …
address review: restore orphaned javadoc, cover the ALL-quorum exit
fix(#5064): distinct contract for post-quorum local commit failure
fix(#6201, #6202): a retryable conflict is retried and reported as on…