Repository navigation
When restarting ArcadeDB embedded server: Error on loading schema. The schema will be reset #1412
Description
ArcadeDB Version:
ArcadeDB Server v23.11.1
OS and JDK Version:
Running on Linux 6.5.6-76060506-generic - OpenJDK 64-Bit Server VM 21.0.1
Expected behavior
All Schema changes should persist if server is restarted
Actual behavior
When restarting server: Error on loading schema. The schema will be reset
Steps to reproduce
I am starting an embedded server and then running a script to create tables (all with IF NOT EXISTS) and then loading data into the tables. All is well until I stop the server and start up again because I get this message:
2024-01-06 01:15:12.452 INFO [ArcadeDBServer] <ArcadeDB_0> Starting ArcadeDB Server in development mode with plugins [] ...
2024-01-06 01:15:12.453 INFO [ArcadeDBServer] <ArcadeDB_0> - Metrics Collection Started...<ArcadeDB_0> Error on loading schema. The schema will be reset
com.arcadedb.exception.SchemaException: Type 'menuItem' not defined
at com.arcadedb.schema.LocalProperty.setOfType(LocalProperty.java:67)
at com.arcadedb.schema.LocalDocumentType.createProperty(LocalDocumentType.java:279)
at com.arcadedb.schema.LocalSchema.readConfiguration(LocalSchema.java:999)
at com.arcadedb.schema.LocalSchema.load(LocalSchema.java:193)
at com.arcadedb.database.LocalDatabase.openInternal(LocalDatabase.java:1771)
at com.arcadedb.database.LocalDatabase.open(LocalDatabase.java:195)
at com.arcadedb.database.DatabaseFactory.open(DatabaseFactory.java:78)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:464)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:318)
at com.arcadedb.server.ArcadeDBServer.loadDatabases(ArcadeDBServer.java:489)
at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:145)
at permeagility.util.Database.startMeUp(Database.java:97)
at permeagility.util.Database.(Database.java:60)
at permeagility.web.Server.initializeServer(Server.java:1270)
at permeagility.web.Server.main(Server.java:227)
Luckily, my table structure is re-initialized at startup but if I create other tables, during a session of the web server and load data into them, I can stop and restart the server and sometimes the table is just missing in the schema, if I recreate the table with the same name, the data appears in it so it appears that the data is solid but somehow the schema is not persisting all the changes. I tried setting autoTransaction(true) but I get the same result.
Is there something I should be doing to validate or flush schema changes to ensure they are durable? I tried CHECK DATABASE FIX but I could not see any difference.
Thanks.
Activity
lvca commentedon Jan 6, 2024
How do you stop the server? Are you running it into a container?
Also it's possible you have a database instance with an open transaction that lasts forever, so the schema changes are in a nested transaction, waiting the 1st level tx to finish but maybe it never does?
glennirwin commentedon Jan 6, 2024
I have a shutdown hook calling server.stop(); When called, it shows:
2024-01-06 12:36:19.218 SEVER [ArcadeDBServer] Received shutdown signal.
I thought it could be the transaction thing too which is why I changed it to use AutoTransaction and removed all my begins/commits/rollbacks, but I got the same results.
Is anyone else having problems like this?
lvca commentedon Jan 6, 2024
I should provide a method to return the nesting level of TX, probably for the next release. For now, if you're in debug mode, can you check this before changing the schema?
How many TXs are in the stack?
lvca commentedon Jan 6, 2024
Just created a new method
Database.getNestedTransactions()that returns an integer, it will be in the next release.glennirwin commentedon Jan 7, 2024
I added that line of code in a few places to display the size() of the transactions object and it always returns 1.
I tried adding some logic to commit() that one transaction but I just get:
DatabaseOperationException: Transaction not begun
I have autoTransaction set to true, and no begins commits or rollbacks anywhere in my code. Below is what I get when I restart up after shutting down. I am ignoring the servermonitor exception (I have been able to turn it off with a runtime option but it doesn't seem to make a difference - please correct me if I am wrong). I also get 'Reusing found bucket' messages afterwards on many tables (not sure if that is normal).
2024-01-06 18:58:01.848 INFO [ArcadeDBServer] <ArcadeDB_0> ArcadeDB Server v23.11.1 (build 798b2b6/1701763071895/main) is starting up...
2024-01-06 18:58:01.854 INFO [ArcadeDBServer] <ArcadeDB_0> Running on Linux 6.5.6-76060506-generic - OpenJDK 64-Bit Server VM 21.0.1
2024-01-06 18:58:01.858 INFO [ArcadeDBServer] <ArcadeDB_0> Starting ArcadeDB Server in development mode with plugins [] ...
2024-01-06 18:58:01.859 INFO [ArcadeDBServer] <ArcadeDB_0> - Metrics Collection Started...<ArcadeDB_0> Error on loading schema. The schema will be reset
com.arcadedb.exception.SchemaException: Type 'menuItem' not defined
at com.arcadedb.schema.LocalProperty.setOfType(LocalProperty.java:67)
at com.arcadedb.schema.LocalDocumentType.createProperty(LocalDocumentType.java:279)
at com.arcadedb.schema.LocalSchema.readConfiguration(LocalSchema.java:999)
at com.arcadedb.schema.LocalSchema.load(LocalSchema.java:193)
at com.arcadedb.database.LocalDatabase.openInternal(LocalDatabase.java:1771)
at com.arcadedb.database.LocalDatabase.open(LocalDatabase.java:195)
at com.arcadedb.database.DatabaseFactory.open(DatabaseFactory.java:78)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:464)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:318)
at com.arcadedb.server.ArcadeDBServer.loadDatabases(ArcadeDBServer.java:489)
at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:145)
at permeagility.util.Database.startMeUp(Database.java:98)
at permeagility.util.Database.(Database.java:61)
at permeagility.web.Server.initializeServer(Server.java:1272)
at permeagility.web.Server.main(Server.java:228)
2024-01-06 18:58:02.084 INFO [HttpServer] <ArcadeDB_0> - Starting HTTP Server (host=0.0.0.0 port=2480-2489 httpsPort=2490-2499)...
2024-01-06 18:58:02.127 INFO [HttpServer] <ArcadeDB_0> - HTTP Server started (host=0.0.0.0 port=2480 httpsPort=2490)
2024-01-06 18:58:02.127 INFO [ArcadeDBServer] <ArcadeDB_0> Available query languages: [sqlscript, java, js, sql]
2024-01-06 18:58:02.128 INFO [ArcadeDBServer] <ArcadeDB_0> ArcadeDB Server started in 'development' mode (CPUs=12 MAXRAM=7.82GB)Exception in thread "Thread-6" Connected to database name=db version=23.11.1 (build 798b2b6/1701763071895/main)
java.lang.IllegalAccessError: class com.arcadedb.server.monitor.ServerMonitor (in unnamed module @0x7bec725a) cannot access class sun.management.ManagementFactoryHelper (in module java.management) because module java.management does not export sun.management to unnamed module @0x7bec725a
at com.arcadedb.server.monitor.ServerMonitor.checkJVMHotSpot(ServerMonitor.java:107)
at com.arcadedb.server.monitor.ServerMonitor.monitor(ServerMonitor.java:61)
at com.arcadedb.server.monitor.ServerMonitor.lambda$start$0(ServerMonitor.java:52)
at java.base/java.lang.Thread.run(Thread.java:1583)
DatabaseConnection.getConnection() Transactions=1
DatabaseSetup.checkInstallation
2024-01-06 18:58:02.198 WARNI [TypeBuilder] <ArcadeDB_0> Reusing found bucket 'restricted_0' for type 'restricted'
2024-01-06 18:58:02.199 WARNI [TypeBuilder] <ArcadeDB_0> Reusing found bucket 'restricted_1' for type 'restricted'
2024-01-06 18:58:02.200 WARNI [TypeBuilder] <ArcadeDB_0> Reusing found bucket 'restricted_2' for type 'restricted'
2024-01-06 18:58:02.201 WARNI [TypeBuilder] <ArcadeDB_0> Reusing found bucket 'restricted_3' for type 'restricted'
-- end of insert --
If I stop and restart again, it gives the same error but on a different table. Which could make sense as I am re-running the table/property create script every time I startup. Further restarts show the schema error on the same second table every time after that.
Is there any way to force a flush or checkpoint or something on the schema to lock it down? or what am I doing wrong?
are schema changes even supposed to be transactional? how would that work? I would prefer everything autocommit until a transaction is explicitly requested in the code but that's just my opinion and arcadedb seems like it should support both. I started out using the transactions in my code as documented but had problems and found my way to autoTransaction=true but it doesn't seem to make a difference.
So far I am happy with ArcadeDB, it took a little while to migrate my code from Orientdb but I am pretty happy with the API and the performance, now if we could just get the schema to stick...
Also: If I try to pull up the studio on my embedded server with a browser it just says: Not found
Shouldn't that work? I can see that it grabs the port but doesn't seem to work.
glennirwin commentedon Jan 7, 2024
I just tried out the new release 23.12.2, same result.
lvca commentedon Jan 7, 2024
About studio, @robfrank moved it to a separate jar, be sure to have it in the class path. I'll try to reproduce the issue with auto TX true
glennirwin commentedon Jan 8, 2024
I added arcadedb-studio to my maven dependencies and it works now, thanks.
glennirwin commentedon Jan 8, 2024
I have created a crude example test code which demonstrates the problem isolated from everything else. It will run a begin/commit if run without options and if given the option 'auto' it will do it with autoTransaction=true.
Each time it is run, it will look for the existence of some types (menuItem and then identity are the ones that go missing) and if not found, it will run a script to create all the tables (using IF NOT EXISTS on everything).
See attached code
SetupTest.zip
lvca commentedon Jan 8, 2024
Awesome, thanks! I was able to reproduce it.
lvca commentedon Jan 8, 2024
I found the issue: you declared some properties with OF, like this:
The issue is in the fact the properties are loaded while some type is still not defined.
OFis a quite recent feature, and this scenario wasn't tested. The solution is easy: create a 2 steps load where all the types are created first, then all the properties.lvca commentedon Jan 8, 2024
I'm going also to improve the message in the exception to understand exactly what kind of issue this is.
fix: reload of the schema with linked properties (using LIST OF state…
lvca commentedon Jan 8, 2024
Fixed. Your test has been transformed to be part of our test suite. Thanks @glennirwin!
fix: reload of the schema with linked properties (using LIST OF state…
chore(deps): bump the github-actions group with 2 updates [skip ci]