Repository navigation
Index creation hinders async record creation #310
Description
ArcadeDB Version: 22.1.3
JDK Version: Corretto 11
OS: Windows 10
Expected behavior
ArcadeDb should internally balance the creation of indexes with the record creation so that the create record calls are not lost. Alternatively, an exception should be thrown to alert the user that such operations are not supported.
Actual behavior
The index and vertex creation are executed without errors, but the edge creation fails since a subset of "Software" vertices are not created.
Steps to reproduce
To create the scenario and trigger an exception using edge creation:
for (i in 0..250000) {
val vertex = database.newVertex("Tester")
vertex.set("id", i.toString())
database.async().createRecord(vertex) {}
}
database.schema.createTypeIndex(Schema.INDEX_TYPE.LSM_TREE, true, "Tester", "id")
for (i in 0..250000) {
val vertex = database.newVertex("Software")
vertex.set("id", i.toString())
database.async().createRecord(vertex) {}
}
database.schema.createTypeIndex(Schema.INDEX_TYPE.LSM_TREE, true, "Software", "id")
database.async().waitCompletion()
for (i in 0..250000) {
database.async().newEdgeByKeys(
"Tester",
arrayOf("id"),
arrayOf(i),
"Software",
arrayOf("id"),
arrayOf(i),
false,
"testerOf",
false,
true,
null
)
}
After running the above, running a SELECT query will show that the required "Software" vertex is indeed not present.
Temporary workaround: add database.async().waitCompletion() before index creation.
Activity
lvca commentedon Feb 2, 2022
This is interesting. Before reproducing it, it seems that the creation of the index (synchronous operation) doesn't index the concurrent insertion of records in the queue for asynchronous operations. I guess the problem is that the event trigger for indexing documents, vertices, and edges is installed when the index completes the creation, but it's not taking care of concurrent operations.
Working on this asap.
lvca commentedon Feb 3, 2022
I was able to reproduce it. The issue is trickier than I thought. The index is created in its own transaction, so new vertices created in parallel and in the different transaction(s) don't see the index until the transaction is completed.
4 remaining items
fix: index creation now requires no async tasks are running
lvca commentedon Jun 13, 2023
Fixing this issue with an exception in case there are running asynchronous tasks.
sblommers commentedon May 28, 2024
I have this now when importing a OrientDB database with version 24.5.1 when it finishes and wants to create the indexes.
ERROR:
com.arcadedb.exception.CommandExecutionException: Error on importing database
at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:73)
at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:91)
at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:52)
at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:65)
at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:116)
at com.arcadedb.database.LocalDatabase.command(LocalDatabase.java:1327)
at com.arcadedb.console.Console.executeSQL(Console.java:602)
at com.arcadedb.console.Console.execute(Console.java:273)
at com.arcadedb.console.Console.parse(Console.java:742)
at com.arcadedb.console.Console.interactiveMode(Console.java:131)
at com.arcadedb.console.Console.execute(Console.java:185)
at com.arcadedb.console.Console.main(Console.java:148)
Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'file:///tmp/orientdb3.2.30export.gz (compressed=true size=2454688091)'
at com.arcadedb.integration.importer.Importer.load(Importer.java:69)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:62)
... 11 more
Caused by: com.arcadedb.integration.importer.ImportException: Error on importing OrientDB database
at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:51)
at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:107)
at com.arcadedb.integration.importer.Importer.load(Importer.java:54)
... 14 more
Caused by: com.arcadedb.exception.NeedRetryException: Cannot create a new index while asynchronous tasks are running
at com.arcadedb.schema.TypeIndexBuilder.create(TypeIndexBuilder.java:57)
at com.arcadedb.schema.TypeIndexBuilder.create(TypeIndexBuilder.java:42)
at com.arcadedb.schema.LocalSchema.getOrCreateTypeIndex(LocalSchema.java:567)
at com.arcadedb.integration.importer.OrientDBImporter.createIndexes(OrientDBImporter.java:1037)
at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:231)
at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
... 16 more
chore(deps): bump https://github.com/macisamuele/language-formatters-…
fix: index creation now requires no async tasks are running
chore(deps): bump https://github.com/macisamuele/language-formatters-…
chore(deps): bump the github-actions group across 1 directory with 8 …