I am periodically seeing errors like this with my call to merge_insert:
Caused by:
0: LanceError(IO): Execution error: The input to a take operation specified fragment id 0 but this fragment does not exist in the dataset, /Users/ogchen/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.38.2/src/dataset/write/merge_insert.rs:851:46
1: Execution error: The input to a take operation specified fragment id 0 but this fragment does not exist in the dataset
2: Execution error: The input to a take operation specified fragment id 0 but this fragment does not exist in the dataset
My app is currently batching up rows every 10 seconds, calling merge_insert based on an ID column, and then calling optimize afterwards.
I have noticed that setting use_index to false on merge_insert fixes this problem, so I am suspecting that the index on the ID column can sometimes be corrupted?
Equally I have found that the order of compaction vs optimizing index after the call to merge_insert matters. I only see the issue if I do compaction -> index optimize, but do not see the issue with index optimize -> compaction.
Note: This is with lancedb version 0.22.2
I am periodically seeing errors like this with my call to merge_insert:
My app is currently batching up rows every 10 seconds, calling merge_insert based on an ID column, and then calling optimize afterwards.
I have noticed that setting
use_indexto false onmerge_insertfixes this problem, so I am suspecting that the index on the ID column can sometimes be corrupted?Equally I have found that the order of compaction vs optimizing index after the call to merge_insert matters. I only see the issue if I do compaction -> index optimize, but do not see the issue with index optimize -> compaction.
Note: This is with lancedb version 0.22.2