Skip to content

BootstrapElection.decideAndAct abandons remaining databases on the first leadership transfer, with nondeterministic source selectionΒ #4807

Closed

Description

@ruispereira

Affected version: 26.6.1
Component: BootstrapElection.decideAndAct
Severity: MEDIUM

Summary

The method returns TRANSFERRED on the first database whose source is a remote
peer, leaving other databases (where the local node was the source)
uncommitted/unbootstrapped. The per-DB map iteration order is nondeterministic
(HashMap), and the "prefer self" tie-break can make the new leader pick itself
for a DB where another node had the higher lastTxId.

Code

BootstrapElection.java:314-396 (:391 early return TRANSFERRED; :343-349 prefer-self; :225 HashMap).

Impact

Bootstrap baseline can be derived from a node that is not the most-recent for some
databases; nondeterministic source selection undermines the "highest lastTxId
wins" guarantee.

Suggested fix

Decide sources for all databases first (deterministic order, e.g. sorted by name);
commit all local-source DBs before transferring; choose a single deterministic
transfer target.

Activity

  1. added this to the 26.7.1 milestone on Jun 29, 2026
  2. self-assigned this
    on Jun 29, 2026
  3. added theissue type on Jun 29, 2026
  4. added a commit that references this issue on Jun 29, 2026
  5. lvca commented on Jun 29, 2026

    @lvca
    Member

    Fixed

  6. lvca commented on Jun 29, 2026

    @lvca
    Member

    ArcadeDB runs one Raft group with a single leader for all databases (RaftHAServer.java:186). Every write flows through that one leader, so the last leader necessarily holds the freshest copy of all databases at shutdown. A follower can lag, but never lead on any database. So "different database freshest on a different node" cannot happen via normal replication - only via an operator manually staging mismatched backups outside Raft, which the existing localLastTxId > baseline "refusing to overwrite" SEVERE guard already catches.

    The fix: Replaced the per-database transfer loop with a single deterministic cluster-wide source election:

    • electSourceNode() - each database votes for the peer(s) holding its freshest copy; the winner is most-databases-won, then highest aggregate lastTxId, then prefer-self, then lex-lowest peer id. Order-independent, so every peer computes the same winner.
    • Remote winner β†’ transfer leadership once, committing nothing (commit index stays 0, so the existing gate stays valid); the new leader elects itself and commits everything in one pass.
    • Local winner β†’ commit one baseline per database in sorted order. Nothing abandoned.
  7. added a commit that references this issue on Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions