ArcadeDB Version:

ArcadeDB Server v23.9.1 (build //)

OS and JDK Version:

Running on Linux 6.2.16-3-pve - OpenJDK 64-Bit Server VM 21

Expected behavior

Application read the newest record which is submitted by previous commits without concurrent commits under async flush mode.

Actual behavior

Application may read the old record without concurrent commits under async flush mode.

Steps to reproduce

Continuously insert records with large size (property values long string for example) with serveral commits using async flush mode and read them immediately

Additional description

This bug is due to the way modified pages are flushed into files under async flush mode. When asyncFlush transaction is committed, the modified pages are passed to the FlushThread and then recorded in readCache of PageManager, then FlushThread flush the pages into files. This usually works, until large amount of pages are modified continuously. In this case, the readCache may exceed the limit and drop the pages which not yet writen to files. If function loadPage tries to read the dropped page at the same time, it will not find the page in readCache, and will read the old page from the files.

This bug may produce several Exceptions, such as RecordNotFoundException if a page with new records not flushed yet. Moreover, the bug may cause ConcurrentModificationException even if two transactions are submitted by same thread, which is quite confusing. This happens when the latter transaction read a old page of former one to modify, and the new page has been flushed into files when the latter transaction commits. The database found that the version of modified page is different from the one of page in file, so it throws a ConcurrentModificationException.

This is the log when the ConcurrentModificationException is thrown:

2023-10-25 15:00:03.938 SEVER [PageManager] Reached max RAM for page cache. Freeing pages from cache (target=2152628224 current=4305256448 max=4294967296 threadId=41)
2023-10-25 15:00:31.163 SEVER [PageManager] Reached max RAM for page cache. Freeing pages from cache (target=2147483648 current=4294967296 max=4294967296 threadId=41)
2023-10-25 15:00:37.886 SEVER [PageManager] Reached max RAM for page cache. Freeing pages from cache (target=2147483648 current=4294967296 max=4294967296 threadId=41)
2023-10-25 15:00:58.637 SEVER [PageManager] Reached max RAM for page cache. Freeing pages from cache (target=2147483648 current=4294967296 max=4294967296 threadId=41)
2023-10-25 15:00:59.967 SEVER [PageManager] Concurrent modification on page PageId(78/416) in file '_stat_16_camera_number_0.78.65536.v0.bucket' (current v.66 <> database v.67). Please retry the operation (threadId=41)
2023-10-25 15:01:00.489 SEVER [PageManager] Reached max RAM for page cache. Freeing pages from cache (target=2147483648 current=4294967296 max=4294967296 threadId=41)

As for the fix of this bug, maybe check whether the cached page to remove from readCache is flushed into files is an option, or there may be some better choices.