Skip to content

feat(vector): replace hnsw_rs with TurboQuant 4-bit backend#30

Merged
epicsagas merged 2 commits into
mainfrom
orbit-turboquant-backend
Jun 9, 2026
Merged

feat(vector): replace hnsw_rs with TurboQuant 4-bit backend#30
epicsagas merged 2 commits into
mainfrom
orbit-turboquant-backend

Conversation

@epicsagas

Copy link
Copy Markdown
Owner

Summary

Replaces the hnsw_rs HNSW backend with TurboQuant 4-bit compressed vector index for 8x memory reduction and faster search.

Problem

Current VectorStore uses hnsw_rs for in-memory search with f32 raw vectors. For large vaults:

  • HNSW index build is slow (ef_build=200)
  • Memory usage is high (10K vectors × 1024-dim ≈ 400MB)
  • Cache eviction causes frequent full rebuilds from SQLite

Solution

Before: SQLite (raw f32 BLOBs) + hnsw_rs HNSW (in-memory, f32)
After:  SQLite (raw f32 BLOBs) + TurboQuant 4-bit (in-memory, 8x compressed)

TurboQuant benefits:

  • 8x memory reduction (4-bit quantization)
  • SIMD-accelerated ANN search
  • Built-in search_filtered and remove (native, no SQLite workarounds)
  • Index persistence via save/load — no rebuild on restart

Changes

File Change
Cargo.toml Add llm-kernel-vector-index dep, turboquant feature, update presets
src/vector.rs TurboQuant backend (conditional compilation)
src/lib.rs Feature-gate re-exports for both backends
src/main.rs Feature-gate mod vector for turboquant

Feature Gates

turboquant = llm-kernel-vector-index + rusqlite + llm-kernel/embedding  (new default)
vector    = hnsw_rs  (fallback, no TurboQuant deps)

Verification

790 tests passed (turboquant feature)
clippy: 0 warnings
fmt: clean
release build: OK

Notes

  • [patch.crates-io] redirects llm-kernel to git — temporary until llm-kernel-vector-index is published on crates.io
  • Test dimensions changed 3→8 (TurboQuant requires dim % 8 == 0)
  • hnsw_rs remains available as fallback via vector feature

epicsagas added 2 commits June 9, 2026 09:40
- Add llm-kernel-vector-index dependency (TurboQuant) behind turboquant feature
- TurboQuant provides 8x memory compression, SIMD-accelerated search, and
  built-in filtered search/remove
- hnsw_rs kept as fallback behind vector feature (without turboquant)
- SQLite remains for metadata; TurbovecIndex handles in-memory search
- Index persistence via save/load to .alcove/vectors.tvim
- full-macos and full-cross presets now use turboquant
- Patch llm-kernel to git to ensure trait coherence with vector-index crate

Closes #25 (P3 TurboQuant adoption)
TurboQuant depends on faer/nalgebra which links against OpenBLAS.
Add apt-get install step to check, cross-check-linux, and test-linux jobs.
@epicsagas epicsagas merged commit fc8a45a into main Jun 9, 2026
20 checks passed
@epicsagas epicsagas deleted the orbit-turboquant-backend branch June 9, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant