Reduced write amplification with faster data visibility #112
Description
Summary
Dramatically reduce write amplification and improve data visibility by enabling append-only writes with immediate query access to uncommitted data. The query engine reads WAL files directly before compaction, making new data visible in milliseconds rather than waiting for merge operations.
Current Limitation
[Write Path] [Query]
│ │
Sort on write ──────────────────► Wait for merge
│ │
High write amplification Delayed visibility
With This Change
[Write Path] [Query]
│ │
Append only (fast) ──────────────► Read WAL directly
│ │
Minimal amplification Instant visibility
│
Background compaction (async)
Benefits
| Metric | Improvement |
|---|---|
| Write amplification | Significantly reduced — append-only on hot path |
| Data visibility | Milliseconds instead of seconds — query reads WAL directly |
| Write throughput | Higher — no sorting on critical path |
| Resource efficiency | Compaction happens in background, doesn't block writes or reads |
How It Works
- Query engine handles unordered data via predicate pushdown
- Writes append to WAL without immediate sorting
- Queries read both compacted partitions AND uncommitted WAL segments
- Background compaction merges WAL into sorted partitions when convenient
Use Cases
- High-frequency ingestion requiring immediate queryability
- Real-time dashboards showing latest data
- Trading systems where milliseconds of latency matter
- Any workload where write speed and data freshness are critical