Skip to content

Error with index rebuilding #3867

@ExtReMLapin

Description

@ExtReMLapin

Hello,

In production on data I cannot share we're metting difficulties with stability of arcadedb.

The database is quite heavy, but again that's just our opinion, we adde inside it a lot of data. (Folder is 9.8 Gb)

While some of the issues are starting to be a little mitigated by performances patches, there are others issues like crash , OOM crashes (ubunting killing it).

One of those issues are for example this error during graph rebuilding :

2026-04-16 00:15:06.657 INFO  [LSMVectorIndex] Graph build building: 35833/36080 (vector accesses=35841, heap=18371,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:11.662 INFO  [LSMVectorIndex] Graph build building: 35996/36080 (vector accesses=36004, heap=25043,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:14.065 INFO  [LSMVectorIndex] Graph build building: 36080/36080 (vector accesses=36088, heap=28259,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:25.091 INFO  [LSMVectorIndex] JVector graph index built successfully
2026-04-16 00:15:25.093 INFO  [LSMVectorIndex] Graph build persisting: 0/744824 (vector accesses=0, heap=29799,1/32768,0MB, offheap=0,6MB, files=187,8MB [idx=6,0, graph=181,8, pq=0,0, compacted=0,0])
2026-04-16 00:15:25.094 INFO  [LSMVectorIndexGraphFile] Starting graph write (sequential) with chunking: 744824 nodes, 50MB chunk size
2026-04-16 00:15:25.094 INFO  [LSMVectorIndexGraphFile] Writing graph WITHOUT inline vectors - topology only (vectors fetched from documents on-demand)
2026-04-16 00:15:26.546 INFO  [LSMVectorIndex] Graph persistence chunk complete: 50,0MB written
2026-04-16 00:15:27.817 INFO  [LSMVectorIndex] Graph persistence chunk complete: 50,0MB written
2026-04-16 00:15:27.818 SEVER [LSMVectorIndexGraphFile] Error writing graph to pages: Transaction error on commitcom.arcadedb.exception.TransactionException: Transaction error on commit
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:718)
        at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:133)
        at com.arcadedb.database.LocalDatabase.lambda$commit$5(LocalDatabase.java:499)
        at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1602)
        at com.arcadedb.database.LocalDatabase.commit(LocalDatabase.java:493)
        at com.arcadedb.index.vector.LSMVectorIndex.lambda$buildGraphFromScratchWithRetry$11(LSMVectorIndex.java:1569)
        at com.arcadedb.index.vector.ContiguousPageWriter.write(ContiguousPageWriter.java:211)
        at com.arcadedb.index.vector.ContiguousPageWriter.writeInt(ContiguousPageWriter.java:133)
        at io.github.jbellis.jvector.graph.disk.OnDiskSequentialGraphIndexWriter.write(OnDiskSequentialGraphIndexWriter.java:145)
        at com.arcadedb.index.vector.LSMVectorIndexGraphFile.writeGraph(LSMVectorIndexGraphFile.java:232)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratchWithRetry(LSMVectorIndex.java:1577)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratch(LSMVectorIndex.java:1027)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratch(LSMVectorIndex.java:1016)
        at com.arcadedb.index.vector.LSMVectorIndex.lambda$startAsyncGraphRebuild$13(LSMVectorIndex.java:1956)
        at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.IllegalArgumentException: Cannot resize the buffer (autoResizable=false)
        at com.arcadedb.database.Binary.checkForAllocation(Binary.java:677)
        at com.arcadedb.database.Binary.size(Binary.java:609)
        at com.arcadedb.engine.CachedPage.loadMetadata(CachedPage.java:63)
        at com.arcadedb.engine.PageManager.loadPage(PageManager.java:426)
        at com.arcadedb.engine.PageManager.getMostRecentVersionOfPage(PageManager.java:180)
        at com.arcadedb.engine.PageManager.checkPageVersion(PageManager.java:217)
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:696)
        ... 14 more
2026-04-16 00:15:27.819 SEVER [LSMVectorIndex] PERSIST: Failed to persist graph for CHUNK_EMBEDDING_0_593681324616448 (nodes=744824, storeVectorsInGraph=false, txStatus=INACTIVE): IndexException - Error writing graph to pages
2026-04-16 00:15:27.819 INFO  [LSMVectorIndex] Built graph for index: CHUNK_EMBEDDING_0_593681324616448
2026-04-16 00:15:27.819 INFO  [LSMVectorIndex] Async graph rebuild completed for index: CHUNK_EMBEDDING_0_593681324616448
2026-04-16 00:15:54.107 INFO  [LSMVectorIndex] JVector graph index built successfully

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions