LSM_VECTOR index in-memory bloat: VectorLocation objects never removed from ConcurrentHashMap (OOM after weeks of operation) #5516

Activity

1. KyaniteSolutions commented on Jul 28, 2026

   

   KyaniteSolutions

   on Jul 28, 2026

   Author

   More actions

   Java reproduction script:

   import com.arcadedb.database.Database;
   import com.arcadedb.database.DatabaseFactory;
   import com.arcadedb.schema.Type;
   import com.arcadedb.schema.Schema;
   
   public class VectorIndexBloatRepro {
       public static void main(String[] args) throws Exception {
           String dbPath = "/tmp/arcadedb-vector-bloat-repro";
           new java.io.File(dbPath).deleteRecursively();
   
           DatabaseFactory factory = new DatabaseFactory(dbPath);
           Database db = factory.create();
   
           // 1. Create schema with embedding property + LSM_VECTOR index
           db.transaction(() -> {
               var type = db.getSchema().createVertexType("TestEntity");
               type.createProperty("id", Type.STRING);
               type.createProperty("embedding", Type.ARRAY_OF_FLOATS);
               type.createTypeIndex(Schema.INDEX_TYPE.LSM_TREE, true, "id");
           });
   
           db.command("sql", "CREATE INDEX TestEntity_embedding_idx ON TestEntity (embedding) " +
               "LSM_VECTOR METADATA { dimensions: 384, similarity: 'COSINE', quantization: 'INT8' }");
           System.out.println("Created LSM_VECTOR index");
   
           // 2. Store 1000 embeddings
           float[] fakeEmbedding = new float[384];
           for (int i = 0; i < 384; i++) fakeEmbedding[i] = i * 0.001f;
           for (int i = 1; i <= 1000; i++) {
               db.transaction(() -> {
                   var vertex = db.newVertex("TestEntity");
                   vertex.set("id", "entity_" + i);
                   vertex.set("embedding", fakeEmbedding);
                   vertex.save();
               });
           }
           System.out.println("Stored 1000 embeddings");
   
           // 3. Re-store same 1000 embeddings 10 times (simulating periodic sync cycles)
           for (int cycle = 1; cycle <= 10; cycle++) {
               for (int i = 1; i <= 1000; i++) {
                   db.transaction(() -> {
                       var rs = db.query("sql", "SELECT FROM TestEntity WHERE id = ?", "entity_" + i);
                       if (rs.hasNext()) {
                           var vertex = rs.next().getVertex().get().modify();
                           vertex.set("embedding", fakeEmbedding);
                           vertex.save();
                       }
                   });
               }
           }
           System.out.println("Re-stored 1000 embeddings x 10 cycles = 10,000 put() calls");
   
           // 4. Heap histogram — should show ~10,000 VectorLocation objects
           printHeapHistogram("After 10K put() calls");
   
           // 5. DROP INDEX + CREATE INDEX
           db.command("sql", "DROP INDEX TestEntity_embedding_idx");
           db.command("sql", "CREATE INDEX TestEntity_embedding_idx ON TestEntity (embedding) " +
               "LSM_VECTOR METADATA { dimensions: 384, similarity: 'COSINE', quantization: 'INT8' }");
           System.out.println("\nDropped + recreated index");
   
           // 6. Heap histogram — stale entries should still be in memory
           printHeapHistogram("After DROP + CREATE INDEX");
   
           // 7. Close + reopen
           db.close();
           db = factory.open();
           System.out.println("\nClosed + reopened database");
   
           // 8. Heap histogram — should now be clean
           printHeapHistogram("After db.close() + db.open()");
   
           // Verify data is intact
           var rs = db.query("sql", "SELECT count(*) AS cnt FROM TestEntity");
           System.out.println("\nVertex count after reopen: " + rs.next().getProperty("cnt"));
   
           db.close();
           new java.io.File(dbPath).deleteRecursively();
       }
   
       static void printHeapHistogram(String label) throws Exception {
           System.out.println("\n--- Heap histogram: " + label + " ---");
           System.gc();
           Thread.sleep(500);
           long pid = ProcessHandle.current().pid();
           Process p = new ProcessBuilder("jcmd", String.valueOf(pid), "GC.class_histogram")
               .redirectOutput(ProcessBuilder.Redirect.PIPE)
               .redirectError(ProcessBuilder.Redirect.DISCARD)
               .start();
           String output = new String(p.getInputStream().readAllBytes());
           p.waitFor();
           output.lines()
               .filter(l -> l.contains("VectorLocation") || l.contains("ConcurrentHashMap$Node"))
               .limit(3)
               .forEach(l -> System.out.println(l.trim()));
       }
   }

   To run: javac -cp arcadedb-engine-26.5.1.jar VectorIndexBloatRepro.java && java -cp .:arcadedb-engine-26.5.1.jar VectorIndexBloatRepro
2. lvca

   self-assigned this

   on Jul 29, 2026
3. lvca

   added the

   Bug

   issue type on Jul 29, 2026
4. lvca

   added this to the 26.8.1 milestone on Jul 29, 2026
5. lvca commented on Jul 29, 2026

   

   lvca

   on Jul 29, 2026

   Member

   More actions

   Fixed in #5521 - thanks for the heap histogram and the reproduction script, they made this quick to pin down.

   What was actually wrong. Your diagnosis was right about the symptom and there were two causes behind it:

   1. A tombstoned vector id kept its whole VectorLocation resident - the location, its RID, the map node and a slot in the reverse index, ~90 bytes - purely to carry a deleted boolean. Nothing ever reads a dead id's offset again, so that memory bought nothing. A superseded id is now one bit in a compact set: ~1.2MB for the 9.3M ids that were costing you ~970MB. It also means opening the database no longer pulls every tombstoned page entry into memory.

   2. Each of your updates was being indexed twice - once while the commit serialized the record, once replaying the operations queued on the transaction. A normal LSM index collapses that onto the same key, but a vector put mints a new vector id every time, so every re-embedding cycle burned two ids and wrote two entries plus two tombstones per vertex. That is why your bloat ratio was roughly double what the arithmetic suggested. Updates now mint exactly one.

   Measured on 100 vertices re-embedded 10 times: 2100 resident locations before, 100 after, and a reopen loads 100 instead of 2100.

   Your workaround is no longer needed. DROP INDEX did already clear the in-memory map (there is a regression test for it now), but the close-and-reopen dance was doing the real work of dropping the accumulated tombstones. That accumulation is gone.

   On the disk side, which the issue did not mention but your workload will hit. The index file grows with every write and nothing reclaimed it, so after weeks of 10-minute sync cycles you would have had a multi-GB index for ~4K live vectors. COMPACT INDEX <name> is the reclaim path - and while verifying it for this fix we found it was broken in four ways (it resurrected deleted vectors, duplicated them on a second run, never dropped the file it replaced, and left the index unreadable after a restart). All four are fixed in the same PR. On the test workload it takes 9 pages / 2304KB / 10500 entries down to 1 page / 256KB / 500 entries.

   Compaction is not automatic - it stays an explicit operation for now - so for a workload like yours the recommendation is to run COMPACT INDEX on the vector index periodically, e.g. after a sync cycle, instead of dropping and recreating the index at startup.
6. lvca

   added a commit that references this issue on Jul 29, 2026

   fix(engine) #5516: LSM_VECTOR in-memory bloat, and a compaction that …

   ...

   e6fd2cc
7. lvca

   closed this as completedin #5521on Jul 29, 2026
8. lvca

   added 4 commits that reference this issue on Jul 30, 2026

   fix(engine) #5568: stop the location index from evicting live vectors…

   ...

   51a3fd2

   fix(engine) #5568: report the location index's retained heap, log the…

   ...

   20e0369

   fix(engine) #5568: publish the vector index location map atomically i…

   ...

   de1df64

   feat(engine): an LSM_VECTOR index compacts itself once its file is mo…

   ...

   c60958f
9. lvca

   added 2 commits that reference this issue on Aug 7, 2026

   fix(engine) #5588: restore the delta-scan tombstone guard the comment…

   ...

   3a9826b

   perf(engine) #5588: lay the vector location index out in primitive ar…

   ...

   6042d44
10. robfrank

    added 3 commits that reference this issue on Aug 14, 2026

    fix(engine) #5516: LSM_VECTOR in-memory bloat, and a compaction that …

    ...

    b68aaba

    fix(engine) #5568: publish the vector index location map atomically i…

    ...

    ac6b931

    feat(engine): an LSM_VECTOR index compacts itself once its file is mo…

    ...

    c1c5967