Skip to content

Index warnings from cluster FOLLOWER #4063

New issue
New issue
Closed
Bug
Closed
Index warnings from cluster FOLLOWER#4063
Bug
Assignees
lvca
Labels
waiting for user
Milestone
26.5.1

Description

@danieljuhl
danieljuhl
opened on May 3, 2026
ArcadeDB Server v26.4.2 (build 96a306bed54479737b22243d9fabbae68bae01cf/1776998135905/UNKNOWN) is starting up...
Running on Linux 6.8.0-90-generic - OpenJDK 64-Bit Server VM 21.0.10

We have a cluster with 3 nodes, and during initial data loading using GraphBatch endpoint, we get a lot of warnings on the FOLLOWER(s), all of the pattern:

[LocalSchema] Cannot find index 'Something_0_965489876545678' defined in type 'SomeType'. Ignoring it

It's a clear (new) database that we are loading data to, and before we call the /batch endpoint, we create the types, properties and indexes using SQL statements.

There is no warnings or errors on LEADER but both of our FOLLOWER(s) has many of the above errors.

We have a type Entity and multiple sub-types eg. EntityA and EntityB which extends Entity

May 03 12:11:10 arcadedb-02 java[155733]: 2026-05-03 12:11:03.038 WARNI [LocalSchema] Cannot find index 'EntityA_0_793160134276612' defined in type 'Entity'. Ignoring it
May 03 12:11:18 arcadedb-02 java[155733]: 2026-05-03 12:11:10.829 WARNI [LocalSchema] Cannot find index 'EntityA_0_793167928772833' defined in type 'Entity'. Ignoring it
May 03 12:11:26 arcadedb-02 java[155733]: 2026-05-03 12:11:18.630 WARNI [LocalSchema] Cannot find index 'EntityA_0_793175727767536' defined in type 'Entity'. Ignoring it
May 03 12:11:34 arcadedb-02 java[155733]: 2026-05-03 12:11:26.728 WARNI [LocalSchema] Cannot find index 'EntityA_0_793183833278173' defined in type 'Entity'. Ignoring it
May 03 12:11:42 arcadedb-02 java[155733]: 2026-05-03 12:11:34.846 WARNI [LocalSchema] Cannot find index 'EntityA_0_793191946663059' defined in type 'Entity'. Ignoring it
May 03 12:11:51 arcadedb-02 java[155733]: 2026-05-03 12:11:42.984 WARNI [LocalSchema] Cannot find index 'EntityA_0_793200087621888' defined in type 'Entity'. Ignoring it
May 03 12:12:00 arcadedb-02 java[155733]: 2026-05-03 12:11:51.505 WARNI [LocalSchema] Cannot find index 'EntityA_0_793208608505679' defined in type 'Entity'. Ignoring it
May 03 12:12:09 arcadedb-02 java[155733]: 2026-05-03 12:12:00.171 WARNI [LocalSchema] Cannot find index 'EntityA_0_793217269381666' defined in type 'Entity'. Ignoring it
May 03 12:12:16 arcadedb-02 java[155733]: 2026-05-03 12:12:09.313 WARNI [LocalSchema] Cannot find index 'EntityA_0_793226415257694' defined in type 'Entity'. Ignoring it
May 03 12:12:25 arcadedb-02 java[155733]: 2026-05-03 12:12:16.521 WARNI [LocalSchema] Cannot find index 'EntityA_0_793233622259273' defined in type 'Entity'. Ignoring it
May 03 12:12:32 arcadedb-02 java[155733]: 2026-05-03 12:12:25.485 WARNI [LocalSchema] Cannot find index 'EntityA_0_793242580851786' defined in type 'Entity'. Ignoring it
May 03 12:12:42 arcadedb-02 java[155733]: 2026-05-03 12:12:32.784 WARNI [LocalSchema] Cannot find index 'EntityA_0_793249889058572' defined in type 'Entity'. Ignoring it
May 03 12:12:49 arcadedb-02 java[155733]: 2026-05-03 12:12:42.167 WARNI [LocalSchema] Cannot find index 'EntityA_0_793259264731271' defined in type 'Entity'. Ignoring it
May 03 12:12:59 arcadedb-02 java[155733]: 2026-05-03 12:12:49.752 WARNI [LocalSchema] Cannot find index 'EntityA_0_793266853486772' defined in type 'Entity'. Ignoring it
May 03 12:13:06 arcadedb-02 java[155733]: 2026-05-03 12:12:59.277 WARNI [LocalSchema] Cannot find index 'EntityA_0_793276379419539' defined in type 'Entity'. Ignoring it
May 03 12:13:16 arcadedb-02 java[155733]: 2026-05-03 12:13:06.923 WARNI [LocalSchema] Cannot find index 'EntityA_0_793284023733098' defined in type 'Entity'. Ignoring it
May 03 12:13:26 arcadedb-02 java[155733]: 2026-05-03 12:13:16.770 WARNI [LocalSchema] Cannot find index 'EntityA_0_793293872562000' defined in type 'Entity'. Ignoring it
May 03 12:13:35 arcadedb-02 java[155733]: 2026-05-03 12:13:26.809 WARNI [LocalSchema] Cannot find index 'EntityA_0_793303913657196' defined in type 'Entity'. Ignoring it
May 03 12:13:39 arcadedb-02 java[155733]: 2026-05-03 12:13:35.578 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:39 arcadedb-02 java[155733]: 2026-05-03 12:13:39.350 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:43 arcadedb-02 java[155733]: 2026-05-03 12:13:39.350 WARNI [LocalSchema] Cannot find index 'EntityB_0_793316343733384' defined in type 'Entity'. Ignoring it
May 03 12:13:43 arcadedb-02 java[155733]: 2026-05-03 12:13:43.598 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:47 arcadedb-02 java[155733]: 2026-05-03 12:13:43.598 WARNI [LocalSchema] Cannot find index 'EntityB_0_793320699361734' defined in type 'Entity'. Ignoring it
May 03 12:13:47 arcadedb-02 java[155733]: 2026-05-03 12:13:47.700 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:51 arcadedb-02 java[155733]: 2026-05-03 12:13:47.700 WARNI [LocalSchema] Cannot find index 'EntityB_0_793324801200591' defined in type 'Entity'. Ignoring it
May 03 12:13:51 arcadedb-02 java[155733]: 2026-05-03 12:13:51.782 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:55 arcadedb-02 java[155733]: 2026-05-03 12:13:51.782 WARNI [LocalSchema] Cannot find index 'EntityB_0_793328884655385' defined in type 'Entity'. Ignoring it
May 03 12:13:55 arcadedb-02 java[155733]: 2026-05-03 12:13:55.919 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:13:59 arcadedb-02 java[155733]: 2026-05-03 12:13:55.920 WARNI [LocalSchema] Cannot find index 'EntityB_0_793333021413279' defined in type 'Entity'. Ignoring it
May 03 12:13:59 arcadedb-02 java[155733]: 2026-05-03 12:13:59.403 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:03 arcadedb-02 java[155733]: 2026-05-03 12:13:59.403 WARNI [LocalSchema] Cannot find index 'EntityB_0_793336502165967' defined in type 'Entity'. Ignoring it
May 03 12:14:03 arcadedb-02 java[155733]: 2026-05-03 12:14:03.803 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:08 arcadedb-02 java[155733]: 2026-05-03 12:14:03.803 WARNI [LocalSchema] Cannot find index 'EntityB_0_793340906409171' defined in type 'Entity'. Ignoring it
May 03 12:14:08 arcadedb-02 java[155733]: 2026-05-03 12:14:08.333 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:12 arcadedb-02 java[155733]: 2026-05-03 12:14:08.333 WARNI [LocalSchema] Cannot find index 'EntityB_0_793345435582676' defined in type 'Entity'. Ignoring it
May 03 12:14:12 arcadedb-02 java[155733]: 2026-05-03 12:14:12.912 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:16 arcadedb-02 java[155733]: 2026-05-03 12:14:12.912 WARNI [LocalSchema] Cannot find index 'EntityB_0_793350015146719' defined in type 'Entity'. Ignoring it
May 03 12:14:16 arcadedb-02 java[155733]: 2026-05-03 12:14:16.637 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:21 arcadedb-02 java[155733]: 2026-05-03 12:14:16.637 WARNI [LocalSchema] Cannot find index 'EntityB_0_793353737058257' defined in type 'Entity'. Ignoring it
May 03 12:14:21 arcadedb-02 java[155733]: 2026-05-03 12:14:21.441 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:26 arcadedb-02 java[155733]: 2026-05-03 12:14:21.442 WARNI [LocalSchema] Cannot find index 'EntityB_0_793358544667454' defined in type 'Entity'. Ignoring it
May 03 12:14:26 arcadedb-02 java[155733]: 2026-05-03 12:14:26.355 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:30 arcadedb-02 java[155733]: 2026-05-03 12:14:26.355 WARNI [LocalSchema] Cannot find index 'EntityB_0_793363455823458' defined in type 'Entity'. Ignoring it
May 03 12:14:30 arcadedb-02 java[155733]: 2026-05-03 12:14:30.745 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:35 arcadedb-02 java[155733]: 2026-05-03 12:14:30.745 WARNI [LocalSchema] Cannot find index 'EntityB_0_793367850708232' defined in type 'Entity'. Ignoring it
May 03 12:14:35 arcadedb-02 java[155733]: 2026-05-03 12:14:35.927 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:41 arcadedb-02 java[155733]: 2026-05-03 12:14:35.928 WARNI [LocalSchema] Cannot find index 'EntityB_0_793373026933178' defined in type 'Entity'. Ignoring it
May 03 12:14:41 arcadedb-02 java[155733]: 2026-05-03 12:14:41.104 WARNI [LocalSchema] Cannot find index 'EntityA_0_793312680393078' defined in type 'Entity'. Ignoring it
May 03 12:14:46 arcadedb-02 java[155733]: 2026-05-03 12:14:41.104 WARNI [LocalSchema] Cannot find index 'EntityB_0_793378208000681' defined in type 'Entity'. Ignoring it

Activity

  1. lvca
    self-assigned this
    on May 3, 2026

  2. lvca commented on May 3, 2026

    @lvca
    lvca
    on May 3, 2026
    Member

    @danieljuhl do you remember if you created the database and full schema in cluster or on a single server and then started the cluster?

  3. lvca
    added this to the 26.5.1 milestone on May 3, 2026
  4. lvca
    added the
    Bug
    issue type on May 3, 2026
  5. lvca
    added a commit that references this issue on May 3, 2026

    fix: HA replication index mismatch

    2b81e04

  6. lvca commented on May 3, 2026

    @lvca
    lvca
    on May 3, 2026
    Member

    @danieljuhl I push a fix in main branch to solve this problem. Can you please check again? If the issue is still open, please send me the full logs from leader and follower(s).

  7. lvca
    added a commit that references this issue on May 3, 2026

    fix: skip compaction during replicated transaction

    c374693

  8. danieljuhl commented on May 3, 2026

    @danieljuhl
    danieljuhl
    on May 3, 2026
    Author

    @lvca thanks for your quick response, and yes, we have a running cluster, and then the database and full schema is creating using SQL statements against the LEADER. I've tried several times, resetting and also deleting the database.

    I see you have a fix in main, and I'll see if I can get main up and running on our cluster and test it.

    Is it just a warning, as stated, or is the indexes actually missing on the FOLLOWER(s) ?

  9. danieljuhl commented on May 3, 2026

    @danieljuhl
    danieljuhl
    on May 3, 2026
    Author

    Quick question - what is the easiest way to test the main branch? We have installed on our Ubuntu instances using the assets attached to the releases.

  10. lvca commented on May 3, 2026

    @lvca
    lvca
    on May 3, 2026
    Member

    I believed the issue was in a concurrent compaction of LSM Tree index, but without all the logs, I can't be sure. This fix circumvent 1) compaction inside a replicated tx and 2) reattach the new compacted indexes to be findable inside a distributed transaction

    The easiest ways to have the latest build from main is:

    1. cook it by yourself:
    • clone arcadedb form github
    • assure you have JDK 21+ and Maven installed on your machine
    • run mvn clean install -DskipTests -Pdocker (-Pdocker install the latest snapshot on your local docker desktop)
    • find your new cooked version under package/target
    1. Or alternatively, wait about 15 minutes Github Action cooks the build for you. We have an action on github that is triggered on each commit, it takes about 15 minutes (if no critical errors) to deploy the latest snapshot to Maven Central.

  11. lvca commented on May 11, 2026

    @lvca
    lvca
    on May 11, 2026
    Member

    Closing this issue, in case the problem is still here, comment here, please.

  12. lvca
    closed this as completedon May 11, 2026
  13. lvca
    added
    waiting for user
    on May 11, 2026
  14. robfrank
    added 2 commits that reference this issue on May 12, 2026

    fix: HA replication index mismatch

    866e37b

    fix: skip compaction during replicated transaction

    f933e9b
  15. tae898
    added 3 commits that reference this issue on Jun 28, 2026

    chore(deps): bump com.google.api.grpc:proto-google-common-protos from…

    a92d212

    fix: HA replication index mismatch

    b90ea11

    fix: skip compaction during replicated transaction

    5704a00
  16. mergify
    added 2 commits that reference this issue on Jul 1, 2026

    chore(deps): bump org.postgresql:postgresql from 42.7.11 to 42.7.12 […

    3a1b987

    chore(deps): bump org.postgresql:postgresql from 42.7.12 to 42.7.13 […

    9b28abe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

waiting for user

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions