ec2608d is the important commit. I can revert a976b30 if it's not wanted; I was just cleaning up the areas I touched with the other commit since they were still using raw types everywhere.

This also might mean one.microstream.use-current-thread-class-loader workaround can be removed, since all classes are loaded using the "main" ApplicationContext class loader. I can continue this PR to remove that workaround if you want, I don't think it would take very long.

Hi @magneticflux-

This PR doesn't solve the issue with the DevTools completely in my test application. With the changes, the application starts (and does not complain about the missing bean) but when making a change in the application and with the restart, I still get the Class cast exception because of the different classloaders that are in use.

The solution is to put this

restart.include.microstream=/microstream-.*.jar

within spring-devtools.properties file, just as you need to do with version 7.1 of MicroStream.

What am I missing?

@rdebusscher I am able to reproduce that issue, and it occurs because the StorageManagerProvider retains a static map that caches already-created storage managers. We don't want to recreate the EmbeddedStorageManager each reload though, so instead of injecting the current ApplicationContext and retaining it, we can inject a ObjectFactory<ApplicationContext> that queries the current context whenever a ClassLoader is needed. Does 5f31c0d fix the issue you were seeing?

I also ran into another issue, unrelated to Spring Boot. When redefining classes using DCEVM (only Java 8, use TravaOpenJDK for Java 11, or JetBrainsRuntime for Java 11 or 17), a BinaryStorer.Item with an object and corresponding PersistenceTypeHandler uses out-of-date memoryOffsets and storers. This can cause data corruption or crashes in development after adjusting fields in a live object, but doesn't cause issues if you aren't storing objects you redefine.

This doesn't seem easily fixable though. If you want to keep caching reflective accesses (like BinaryHandlerGenericType does with persistableFields), you'd need to check the private field classRedefinedCount on each Class and recompute if it changes.

Incremented by the VM on each call to JVM TI RedefineClasses() that redefines this class or a superclass.

private transient volatile int classRedefinedCount;

Hi @magneticflux-

The usage of ObjectFactory<ApplicationContext> doesn't solve it since the classloader is not the main problem. As you mentioned, some static map holds the old classes and causes the ClassCastException.

And the usage of the spring-devtools.properties file solves this by making sure that the MicroStream classes are part of the restart classloader. But the usage of the restart classloader is not causing the recreation of the EmbeddedStorageManager. Reload in Spring Boot drops the Spring Bean container and recreates it. The MicroStream Spring Boot integration defines the EmbeddedStorageManager as bean and thus recreation of the Storage Manager is part of the expected behaviour of Reload functionality.

I disagree with adding MicroStream's classes to the restart classloader. The restart classloader is only supposed to contain classes that are possible to change - i.e. the end-user's classes. MicroStream internal classes do not change, and thus should not be included in the restart classloader.

As it stands now, in the current release of MicroStream, reloading the Spring application does destroy the EmbeddedStorageManager by calling the shutdown() method. However, reloading the application does not reload the StorageManagerProvider class, so its static field is not cleared and the old, shutdown, EmbeddedStorageManagers are able to be retrieved by StorageManagerFactory after the restart.

Since StorageManagerProvider is already a singleton (by default), I suggest we make the static map field an instance field. This would allow it to forget the EmbeddedStorageManager beans and create new ones after restarting.

Additionally, I think I have a workaround for my class redefinition issue using Spring's LoadTimeWeaverAware. We might be able to define a no-op ClassFileTransformer that watches for classes persisted by MicroStream and throws UnsupportedOperationException if the class is being modified. IntelliJ's default behavior when redefinition fails is to trigger a restart, which would then correctly load and persist the new classes.

I will fixup+rebase my initial commit to make StorageManagerProvider's static field an instance field, and add a new commit attempting to prevent stored classes from being redefined.

My technique to prevent storage classes from being redefined did not work because Java Agent self-Instrumentation is not supported, and that's the only way to receive notification of class redefinition events. If you want resilience against class redefinition, I think the only way is to check the classRedefinedCount field for changes upon (de)serialization.

From within IDE, the Spring Boot app is working, with reload. But when running through 'java -jar', I get an null Root (when dev-tools as optional as in documentation, without this dependency, it also works). So I need more time to test out what is going on.

@rdebusscher I've tried to reproduce the issue you mentioned while running the Spring Boot jar directly, but I couldn't using the Gradle starter project. I've shared my reproduction here: https://github.com/magneticflux-/microstream-issue517-demo, it requires the snapshot to be installed to your local Maven repo.

Using that reproduction:

• I can successfully run...
  • Inside IntelliJ IDEA
  • Using Gradle ./gradlew bootRun
  • Directly using ./gradlew bootJar && java -jar build/libs/microstream-issue517-demo-0.0.1-SNAPSHOT.jar

Details:

When running inside IntelliJ, changing application.properties, clicking "Build Project" (to copy resources), and then clicking "Update" causes the application to perform a quick restart (~0.3 seconds instead of ~5 seconds for a full restart) which properly shuts down and restarts the MicroStream storage system.

While in debug mode inside IntelliJ, changing a class and clicking "Build Project" causes the class to be reloaded near-instantly without restarting the application. Like I mentioned before, this can cause issues with serialization when the reloaded class has already had a PersistenceTypeHandler that caches the list of field types and offsets.

I found the problem, asynchronous initialisation which means you can operate on a null Root. Not related to this PR. #530

We have a look what is needed (regarding intellectual Property flow) to accept your PR as we are moving the code to the Eclipse Foundation.