HA: leader floods log when a follower restarts, and the restarting replica logs no resync progress #4810

Summary

In a running Raft cluster under heavy write load, restarting a node produces two poor logging experiences: the leader floods the log, and the restarting replica is completely silent about its resync.

1. Master node floods the log

When a follower restarts and is briefly unreachable, the leader emits thousands of WARN lines, errorCount climbing into the thousands:

WARNI [GrpcLogAppender] arcadesplit-0_2434@group-E6A6271798FD->arcadesplit-2_2434-FixedGrpcLogAppender:
  Follower failed (request=null, errorCount=4442); keep nextIndex (105325) unchanged and retry.
  (Repeated 2 times in the last 5.001s)
WARNI [GrpcLogAppender] arcadesplit-0_2434@group-E6A6271798FD->arcadesplit-2_2434-FixedGrpcLogAppender:
  Follower failed (request=null, errorCount=4444); keep nextIndex (105326) unchanged and retry.
  (Repeated 2 times in the last 5.000s)

The line itself is harmless: Apache Ratis is correctly holding the follower's nextIndex and retrying, and replication resumes automatically once the follower returns. But the volume drowns the log during exactly the window an operator most needs to read it. The message originates in Apache Ratis 3.2.2's own org.apache.ratis.grpc.server.GrpcLogAppender (parent of our FixedGrpcLogAppender), routed SLF4J -> slf4j-jdk14 -> JUL -> ArcadeDB's logger.

2. Replica node is silent during resync

When the restarted replica comes back and resyncs, it logs nothing about catching up. An operator cannot tell whether resync is progressing, stuck, or finished. The snapshot-install path already logs start/end, but the common restart-under-load case is Raft log catch-up (AppendEntries replay), which is entirely silent. Long snapshot downloads also log only start/end with no in-flight progress.

Proposed direction (log hygiene, not membership change)

The flood is a symptom of a transient, self-healing event. The intended fix is to say it once, then summarize - not to mutate Raft membership (auto-deregistering an unreachable peer was considered and rejected: it shifts quorum math under load, can deadlock on the membership change itself, and causes remove/add flapping on rolling restarts).

Master side:

• Suppress the repeated Ratis WARN via a JUL Filter on the GrpcLogAppender logger (pass the first failure per follower, suppress repeats).
• Emit an ArcadeDB-authored periodic summary instead: follower-2 unreachable for 45s (errorCount~4442, holding nextIndex=105325) every ~30s, plus a one-shot recovery line on reconnect.
• No Raft membership change.

Replica side:

• Log catch-up visibility: HA resync started (mode=catch-up): N entries behind, periodic progress, HA resync finished.
• Add per-database progress to long snapshot downloads (bytes/throughput/ETA).
• Unified, grep-able lifecycle bookends shared by both catch-up and snapshot paths.

All cadences configurable, new logging default-on, reusing the existing HealthMonitor timer and per-follower FixedGrpcLogAppender hooks (no new threads).

Full design: docs/superpowers/specs/2026-06-29-ha-resync-logging-design.md.