Repository navigation
feat(gc): optimize chunk tables and delete superseded versions - #50
Merged
Merged
Conversation
Every LanceDB write leaves a table version behind and nothing removed them: one 170k-row cache held 16,772 versions, 8.0 GiB of manifests beside 2.0 GiB of data. `lgrep gc` gains an optimize_caches pass that runs LanceDB optimize on each chunks table (compaction, deletion of versions superseded more than an hour ago, index updates). optimize also adds new rows to the vector index. The index is trained once on the first rows indexed (340 of 169,733 on that cache), so the pass retrains it when most rows lie outside it, and vector and hybrid search now re-rank candidates by exact distance (refine_factor 10). On that cache top-10 agreement with exact search went from 0.60 to 0.98, at about 22 ms per query against 270 ms for today's scan. Connections now recheck the latest table version every 5 s. Without it a long-lived server handle keeps a version the pass deleted and fails every later search and write until restart. Refs LGREP-36.
The coverage rule fired only on a cache's first run: optimize then folds new rows into the stale partitions and adds one index segment per run. Retraining whenever an index exists keeps it trained on the current rows and one segment deep. Tests now prove the retrain and the hybrid re-rank, and the docs state LanceDB's age rule (time a version was written) correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every LanceDB write leaves a table version, and nothing removes them. The semantic cache on the maintainer host reached 97 GiB. One 170k-row cache held 16,772 versions: 8.0 GiB of manifests beside 2.0 GiB of data. Opening a 47k-version table took 6.6 s.
Change
lgrep gcgains a fourth pass,optimize_caches. It runs LanceDBoptimizeon each chunks table: compaction, deletion of every version but the latest written more than an hour ago, and index updates. The dry run (default) reports each cache's version count and size.optimizeadds new rows to the vector index's existing partitions, as one more segment per run; the index is trained once on the first rows indexed (340 of 169,733 on the cache above). The pass retrains the vector index first whenever the table has one.refine_factor(10)).read_consistency_interval). Without it, a long-lived server handle keeps a version the pass deleted and fails every later search and write until restart.Evidence (copy of the 170k-row cache)
refine_factor(10)The run took 275 s and peaked at 2.3 GiB. In a two-process test, a writer that committed during
optimizealways won andoptimizeraised a retryable conflict, so the pass reports that cache and the next run retries.Tests
tests/test_table_maintenance.py: version deletion keeps rows; dry run changes nothing; retrain keeps the index one segment deep across runs (fails without it); an open store reads and writes after another process optimizes (fails without the consistency interval); exact match ranks first through the PQ index, vector and hybrid (21 of 40 without re-ranking);optimize_cachesreporting, reclaim, and per-cache failure. Full suite: 1249 passed, 1 xfailed. Reviewed by an independent reviewer; its test and doc fixes are included.