Repository navigation
HA: no alert or handling when every node has quarantined the same database (no resync source left) #9553
Description
Follow-up of #9547 (split out of that issue's second "Expected" point).
In the #9547 soak, a file-id collision made ArcadeStateMachine.handleUnexpectedApplyError quarantine database chaos on all three nodes within the same second. Each quarantine triggers a targeted snapshot resync from the leader and hands leadership off, but when every node is quarantined there is no healthy copy to resync from: node 1 answered HTTP 503 "being replaced from the leader's snapshot", and the leader answered HTTP 404 "Database 'chaos' is not available" because it was quarantined too.
The fix for #9547 removes the cause seen in that run (a stale proposer's schema entry reaching the log). It does not handle the general state "every voter has quarantined the same database", whatever the cause.
Expected
- Detect that no voter holds a non-quarantined copy of the database (the cluster view already carries each peer's diverged databases).
- Raise a dedicated alert that says so and names the recovery: pick the copy to keep and force-accept it (A quarantine that already stands on a sole voter (restored from disk, or raised before the cluster shrank) still can never lift: add an operator override #9449 accept-diverged path), or restore from backup. Today the operator sees only per-node SEVERE lines and a stalled resync loop.
- Optionally: refuse to keep retrying a resync whose only possible source is itself quarantined.
Entry point: ArcadeStateMachine.handleUnexpectedApplyError -> quarantineDatabase / triggerDatabaseResync, plus ClusterAlerts.
Activity
feat(#9553): alert and recovery when every voter has quarantined the …
robfrank commentedon Oct 9, 2026
Implemented by #9567. Each node now reports the databases it holds quarantined in the capability poll it already runs against every peer, so any node can tell when every voter holds the same database quarantined.
GET /api/v1/clusterthen raises a critical alert,no-healthy-copy-on-any-voter, that names the way out: accept one copy, make that node leader, or restore from a backup. The accept-diverged override from #9449 used to refuse any node that was not the sole voter, so it is now also allowed when every voter reports the database quarantined; before lifting, the node asks every other voter live whether it serves the database and refuses if one does or cannot be asked. A resync is no longer retried from a leader that holds the same database quarantined. A peer that is unknown, stale or on an older build is never counted as quarantined, so a down node keeps the alert off.Still open:
❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.
feat(#9553): alert and recovery when every voter has quarantined the …