Skip to content

Add SpillStore trait with local-disk implementation #7300

Description

@wjones127

Index builds need a uniform place to put temporary, reclaimable intermediate state (posting-list segments, shuffle runs, btree pages) that can tier beyond memory when a machine is starved.

Add a SpillStore trait, available on Session, providing temporary scratch with an RAII lifecycle (handles delete on drop) and trafficking in opaque byte streams.

Provide a local-disk implementation that writes to temp dirs and returns a clean error when a configured size cap is exceeded (similar to DataFusion's DiskManager).

This is the foundation for memory-budgeted index builds: builders react to memory pressure (see the MemoryPool threading work) by spilling intermediate state through SpillStore. The trait should be a clean seam so alternative implementations can be injected via Session.

Acceptance

  • SpillStore trait + local-disk impl
  • RAII cleanup of scratch handles
  • Clean, typed error on disk-cap exhaustion (no panic)

Metadata

Metadata

Assignees

Labels

A-indexVector index, linalg, tokenizerenhancementNew feature or requestrustRust related tasks

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions