[ENH] Garbage collect usearch index files by Sicheng-Pan · Pull Request #6416 · chroma-core/chroma

Sicheng-Pan · 2026-02-12T06:11:16Z

s## Description of changes

Summarize the changes made by this PR.

Improvements & Bug fixes
- N/A
New functionality
- Wire up GC to clean up usearch binary files

Test plan

How are these changes tested?

Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

github-actions · 2026-02-12T06:11:24Z

Sicheng-Pan · 2026-02-12T06:11:34Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

propel-code-bot · 2026-02-12T06:12:44Z

Garbage collector now cleans up USearch index artifacts

Extends GC operators so USearch/Spann index binaries are discovered, preserved when needed, and deleted when unused. Adds USearchIndex::format_storage_key usage and new QUANTIZED_SPANN_* constants in file discovery paths plus enables the usearch feature on the chroma-index dependency.

Key Changes

• Updated ListFilesAtVersionsOperator to recognize QUANTIZED_SPANN_* file types and translate segment paths into USearch binary storage keys via USearchIndex::format_storage_key.
• Modified FetchSparseIndexFilesOperator to treat USearch files like HNSW indices, skipping fetch but collecting prefixes for deferred deletion unless the version is preserved.
• Enhanced ComputeUnusedFilesOperator to emit USearch .bin keys into unused_hnsw_prefixes, ensuring GC deletes orphaned quantized and raw centroids.
• Enabled the usearch feature on chroma-index within rust/garbage_collector/Cargo.toml so the crate exposes USearchIndex helpers.

Possible Issues

• No automated tests cover the USearch GC paths, so regressions (e.g., changing key formatting) would go unnoticed.
• If segment paths ever omit the USearch shard suffix, Segment::extract_prefix_and_id may fail and prevent GC cleanup for those files.

This summary was automatically generated by @propel-code-bot

propel-code-bot · 2026-02-12T18:56:06Z

rust/garbage_collector/src/operators/compute_unused_files.rs

+                // For usearch index files, add the single .bin file directly.
+                if file_type == QUANTIZED_SPANN_RAW_CENTROID
+                    || file_type == QUANTIZED_SPANN_QUANTIZED_CENTROID
+                {
+                    let quantized = file_type == QUANTIZED_SPANN_QUANTIZED_CENTROID;
+                    for file_path in file_paths.paths.iter() {
+                        let (prefix, id) =
+                            Segment::extract_prefix_and_id(file_path).map_err(|e| {
+                                tracing::error!(error = %e, "Failed to extract prefix and ID");
+                                ComputeUnusedFilesError::InvalidUuid(e, file_path.to_string())
+                            })?;
+                        let s3_key =
+                            USearchIndex::format_storage_key(prefix, IndexUuid(id), quantized);
+                        unused_hnsw_prefixes.push(s3_key);
+                    }
+                    continue;
+                }


[Testing] The new SPANN handling in compute_unused_between_successive_versions isn’t covered by any tests—the existing suites only exercise sparse index blocks and HNSW prefixes. Without a regression test we can’t detect if future refactors stop emitting the formatted .bin key, which would silently leak storage. Please extend the tests in this file to create a CollectionSegmentInfo containing a QUANTIZED_SPANN_* entry and assert that compute_unused_between_successive_versions returns the expected USearchIndex::format_storage_key(...) value in unused_hnsw_prefixes.

Context for Agents

The new SPANN handling in `compute_unused_between_successive_versions` isn’t covered by any tests—the existing suites only exercise sparse index blocks and HNSW prefixes. Without a regression test we can’t detect if future refactors stop emitting the formatted `.bin` key, which would silently leak storage. Please extend the tests in this file to create a `CollectionSegmentInfo` containing a `QUANTIZED_SPANN_*` entry and assert that `compute_unused_between_successive_versions` returns the expected `USearchIndex::format_storage_key(...)` value in `unused_hnsw_prefixes`. File: rust/garbage_collector/src/operators/compute_unused_files.rs Line: 123

Sicheng-Pan · 2026-02-14T03:39:07Z

Merge activity

Feb 14, 3:39 AM UTC: A user started a stack merge that includes this pull request via Graphite.
Feb 14, 4:16 AM UTC: @Sicheng-Pan merged this pull request with Graphite.

- **[ENH]: Cache rust git submodules in mounted volume (#6424)** - **[CHORE](k8s) increase dev CPU limits from 100m to 200-300m (#6435)** - **[ENH] replace live cloud tests with k8s integration tests (#6434)** - **[ENH] Make dirty_log_collections metric mcmr-aware. (#6353)** - **[ENH] Quantized Spann Segment Writer (#6397)** - **[ENH] Wire up quantized writer in compaction (#6399)** - **[ENH] Quantized Spann Segment Reader (#6405)** - **[ENH] Wire up quantized reader in new orchestrator (#6409)** - **[ENH] Garbage collect usearch index files (#6416)** - **[ENH] Trace quantized spann implementation (#6425)** - **[ENH]: Precompute data chunk len() (#6442)** - **[BUG]: Compaction version file flush was incomplete on MCMR (#6423)** - **[DOC]: Fixed broken links in Readme (#6440)** - **[DOC] Fix link to Rust documentation (#6443)** - **[ENH]: Allow users to disable FTS in schema (#6214)** --------- Co-authored-by: Robert Escriva <[email protected]> Co-authored-by: Macronova <[email protected]> Co-authored-by: Nilpotent <[email protected]> Co-authored-by: anderk222 <[email protected]> Co-authored-by: Sanket Kedia <[email protected]>

This was referenced Feb 12, 2026

[ENH] Quantized Spann Segment Writer #6397

Merged

[ENH] Wire up quantized writer in compaction #6399

Merged

This was referenced Feb 12, 2026

[ENH] Quantized Spann Segment Reader #6405

Merged

[ENH] Wire up quantized reader in new orchestrator #6409

Merged

Sicheng-Pan marked this pull request as ready for review February 12, 2026 06:11

This comment has been minimized.

Sign in to view

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from 34250b1 to da75d4e Compare February 12, 2026 18:48

propel-code-bot bot reviewed Feb 12, 2026

View reviewed changes

Sicheng-Pan mentioned this pull request Feb 12, 2026

[ENH] Trace quantized spann implementation #6425

Merged

1 task

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from da75d4e to dc0d345 Compare February 12, 2026 21:27

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from d067206 to 2acf8f0 Compare February 12, 2026 21:27

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from dc0d345 to a6a9ce3 Compare February 12, 2026 21:33

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from 2acf8f0 to 588f737 Compare February 12, 2026 21:33

This comment has been minimized.

Sign in to view

Sicheng-Pan added 13 commits February 12, 2026 16:39

[ENH] Quantized Spann Segment

37f4e42

Schema helper

a6d39cf

Writer and provider

8d81b20

VectorWriter wire up

a72c285

Simple persistence test

212499d

Provider config

2885f53

Use old provider

16f267f

Leave provider change for upstream

b9befb0

Prefetch metadata

d58781f

[ENH] Wire up quantized writer in compaction

31c4639

Update provider

166ddd7

Update quantization schema config

3fa3361

Regen

9c45337

Extend test

833b8cc

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from 588f737 to b64d5d2 Compare February 13, 2026 00:43

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from a6a9ce3 to e43785c Compare February 13, 2026 00:43

Address comments

80ce108

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from b64d5d2 to 9debc15 Compare February 13, 2026 01:39

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch 2 times, most recently from e3c0a2b to eb342bb Compare February 13, 2026 05:39

Arc matrix

c2e95b6

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from eb342bb to e1fb908 Compare February 13, 2026 19:19

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from 96c6e0e to 1a1b6e8 Compare February 13, 2026 19:19

This comment has been minimized.

Sign in to view

HammadB approved these changes Feb 13, 2026

View reviewed changes

Sicheng-Pan added 5 commits February 13, 2026 15:32

Fix test

6875112

[ENH] Wire up quantized reader in new orchestrator

f14d16d

Update merge impl

e23a296

Address comments

300f968

Pipelining

b49d43a

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from 1a1b6e8 to 7cefa67 Compare February 14, 2026 00:05

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from e1fb908 to 5769085 Compare February 14, 2026 00:05

Sicheng-Pan added 2 commits February 13, 2026 16:09

Avoid data loading in CPU opeartor

8f4b1ec

[ENH] Garbage collect usearch index files

def34fc

Sicheng-Pan force-pushed the 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator branch from 7cefa67 to 8f4b1ec Compare February 14, 2026 00:09

Sicheng-Pan force-pushed the 02-11-_enh_garbage_collect_usearch_index_files branch from 5769085 to def34fc Compare February 14, 2026 00:09

Sicheng-Pan changed the base branch from 02-11-_enh_wire_up_quantized_reader_in_new_orchestrator to graphite-base/6416 February 14, 2026 03:40

Sicheng-Pan changed the base branch from graphite-base/6416 to main February 14, 2026 04:15

Sicheng-Pan merged commit 188d6fe into main Feb 14, 2026
69 of 70 checks passed

tanujnay112 mentioned this pull request Feb 18, 2026

[CHORE]: fast forward rc/2026-02-13 to 3cb097601ddd379ab109456727a926e593e43a5c #6456

Merged

Sicheng-Pan deleted the 02-11-_enh_garbage_collect_usearch_index_files branch February 25, 2026 23:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Garbage collect usearch index files#6416

[ENH] Garbage collect usearch index files#6416
Sicheng-Pan merged 27 commits intomainfrom
02-11-_enh_garbage_collect_usearch_index_files

Sicheng-Pan commented Feb 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 12, 2026

Uh oh!

Sicheng-Pan commented Feb 12, 2026 •

edited

Loading

Uh oh!

propel-code-bot bot commented Feb 12, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

propel-code-bot bot Feb 12, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

Sicheng-Pan commented Feb 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Sicheng-Pan commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Migration plan

Observability plan

Documentation Changes

Uh oh!

github-actions bot commented Feb 12, 2026

Reviewer Checklist

Testing, Bugs, Errors, Logs, Documentation

System Compatibility

Quality

Uh oh!

Sicheng-Pan commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

propel-code-bot bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

propel-code-bot bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

Sicheng-Pan commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sicheng-Pan commented Feb 12, 2026 •

edited

Loading

Sicheng-Pan commented Feb 12, 2026 •

edited

Loading

propel-code-bot bot commented Feb 12, 2026 •

edited

Loading

Sicheng-Pan commented Feb 14, 2026 •

edited

Loading