Index builds need a uniform place to put temporary, reclaimable intermediate state (posting-list segments, shuffle runs, btree pages) that can tier beyond memory when a machine is starved.
Add a SpillStore trait, available on Session, providing temporary scratch with an RAII lifecycle (handles delete on drop) and trafficking in opaque byte streams.
Provide a local-disk implementation that writes to temp dirs and returns a clean error when a configured size cap is exceeded (similar to DataFusion's DiskManager).
This is the foundation for memory-budgeted index builds: builders react to memory pressure (see the MemoryPool threading work) by spilling intermediate state through SpillStore. The trait should be a clean seam so alternative implementations can be injected via Session.
Acceptance
SpillStore trait + local-disk impl
- RAII cleanup of scratch handles
- Clean, typed error on disk-cap exhaustion (no panic)
Index builds need a uniform place to put temporary, reclaimable intermediate state (posting-list segments, shuffle runs, btree pages) that can tier beyond memory when a machine is starved.
Add a
SpillStoretrait, available onSession, providing temporary scratch with an RAII lifecycle (handles delete on drop) and trafficking in opaque byte streams.Provide a local-disk implementation that writes to temp dirs and returns a clean error when a configured size cap is exceeded (similar to DataFusion's
DiskManager).This is the foundation for memory-budgeted index builds: builders react to memory pressure (see the MemoryPool threading work) by spilling intermediate state through
SpillStore. The trait should be a clean seam so alternative implementations can be injected viaSession.Acceptance
SpillStoretrait + local-disk impl