Skip to content

Reclaim unreferenced files: the engine has no GC for files an abandoned instalment sequence leaves on a follower #6189

Closed

Description

@lvca

Split out of #6168 (item 2) so it is tracked on its own, which is what that item asked for: the other five follow-ups
shipped in #6173, this one is a deletion and wants its own change and its own risk assessment.

The state today

#6143 shipped a diagnostic, not a fix. CHECK DATABASE reports unreferencedFiles, every node publishes an
arcadedb.ha.schema.unreferenced_files gauge (memoized behind a modification/version gate since #6173), and
com.arcadedb.engine.UnreferencedFiles will prove three shapes and deliberately nothing else. What it never does is
delete. An operator who reads the SEVERE line, or watches the gauge climb, still has to stop the node and remove
files by hand.

That was the right call for a diagnostic - a file whose reference the walk simply does not know how to follow would
be data - but it leaves the actual problem unsolved: this database has no unreferenced-file garbage collection.

Where the files come from

A schema session that ships its WAL in instalments announces the files it creates before the change is published, and
retires them itself if it never gets there. It cannot do that once it has lost leadership mid-session - a node that
is no longer leader cannot make the cluster do anything - so the files stay on the OTHER nodes, referenced by
nothing. The same shape is left on any node by a DDL that throws after creating a file without publishing it.

The design that was worked out and deliberately NOT taken

Recorded so it is not rediscovered from scratch:

A follower knows exactly which files an unpublished instalment sequence created (it created them itself, from
filesToAdd on entries marked moreChunksFollow). The next entry that PUBLISHES - moreChunksFollow false - is
the signal that the sequence is over. At that point it could drop any file from the tracked set that the newly
loaded schema still does not reference. The legitimate ones ARE referenced by then, so they survive by
construction; only the abandoned ones fall out.

It is bounded, targeted, and needs no reachability analysis.

Why it was not taken, stated plainly

It is a file deletion driven by inference, on a replica, on a code path whose history (#4083, #4743, #5443,
#5492) is silent divergence rather than exceptions - the failure mode is not a stack trace, it is a follower that
quietly no longer matches the leader. The impact being bought is only wasted disk. That is a bad ratio unless the
deletion is provably safe.

Two things any implementation has to answer:

  1. Restart durability. The tracked set has to survive a restart, or the reclaim silently stops covering the case
    where the node holding the files is restarted before the next DDL - which is the likeliest case in practice, since
    an abandoned instalment usually accompanies the leadership change that a restart caused.
  2. What proves a file is dead. "The newly loaded schema does not reference it" is the same question
    UnreferencedFiles answers, and that class is explicit that it can prove only three shapes and treats everything
    else as claimed because a false positive here means deleting live data. A reclaim needs the converse guarantee
    (a proof of absence), which is strictly stronger than what the diagnostic provides.

Worth considering instead

An operator-triggered reclaim rather than an inferred one - e.g. CHECK DATABASE in FIX mode reclaiming what
UnreferencedFiles.scan already proves, on a node the operator chose, with the finding printed first. Deletion stays
a decision somebody took, on a node they picked, rather than one a replica inferred while running. It gives up
automatic cleanup, and it buys an audit trail and a much smaller blast radius. Whichever way it goes, the choice
should be explicit in the change.

Related: #6143 (the diagnostic), #6152 (the instalment work), #6173 (the other five follow-ups).

Activity

  1. added this to the 26.9.1 milestone on Aug 15, 2026
  2. self-assigned this
    on Aug 17, 2026
  3. added a commit that references this issue on Aug 18, 2026
  4. removed
    in progressThe team is actively working on this issue
    on Aug 19, 2026
  5. added
    enhancementNew feature or request
    haHigh availability / replication (Raft)
    on Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requesthaHigh availability / replication (Raft)

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions