|
| 1 | +# Large-state backend (P0) — spillable + disaggregated, designed to beat Flink |
| 2 | + |
| 3 | +> The streaming latency + EO + throughput + memory gaps vs Flink are now closed/won |
| 4 | +> (`STREAMING_VS_FLINK_EKS.md`). The remaining operational gap is **large keyed state**: |
| 5 | +> Vajra keeps streaming state **in memory** and snapshots it **whole** to the checkpoint. |
| 6 | +> Flink handles huge state via RocksDB + incremental checkpoints + (2.0) disaggregated |
| 7 | +> state (ForSt). This designs a Vajra state backend that **matches and then beats** Flink — |
| 8 | +> not by porting RocksDB, but by exploiting Vajra's no-JVM, Arrow-columnar, object-store-first |
| 9 | +> architecture. |
| 10 | +
|
| 11 | +## Where Vajra is today |
| 12 | +- `WindowAccumExec` keeps partial-aggregate state in `AccumState.pending_rows: Vec<RecordBatch>` |
| 13 | + (in memory), pruned to open windows, and **full-snapshotted** to the object-store checkpoint |
| 14 | + on epoch/EndOfData (`state_io::stage_state`). `StreamJoinExec` / dedup hold state in memory too. |
| 15 | +- Consequence: state must fit RAM (OOM risk at large keyspaces), and every checkpoint uploads |
| 16 | + the **entire** state (not the delta). |
| 17 | + |
| 18 | +## How the proven systems do it (and what we take, not copy) |
| 19 | +- **Flink HashMapStateBackend** (heap) — fast, but JVM-GC-bound and RAM-limited. |
| 20 | +- **Flink RocksDBStateBackend** — embedded LSM on local disk → state ≫ RAM; **incremental |
| 21 | + checkpoints** upload only new/changed SST files; JNI + JVM-off-heap overhead. |
| 22 | +- **Flink changelog state backend (FLIP-158)** — decouples a durability *log* from periodic |
| 23 | + materialization → faster, more predictable checkpoints. |
| 24 | +- **Flink 2.0 disaggregated state (ForSt)** — state lives on DFS/object store, compute reads via |
| 25 | + a local cache → cloud elasticity, fast rescaling, cheap checkpoints (state already remote). |
| 26 | +- **DataFusion** — `GroupedHashAggregateStream` + `MemoryPool` spill: when a memory budget is |
| 27 | + exceeded, spill to disk and merge — the model for *bounded-memory* operators. |
| 28 | + |
| 29 | +## Vajra design — `StateBackend`: Arrow-columnar, spillable, disaggregated-native |
| 30 | +A pluggable trait, Arrow-first (state is RecordBatches, not opaque bytes), used by |
| 31 | +`WindowAccumExec` / `StreamJoinExec` / dedup instead of in-memory `Vec`s: |
| 32 | + |
| 33 | +```text |
| 34 | +trait StateBackend { |
| 35 | + // keyed columnar state: key-group -> the operator's partial-state batches |
| 36 | + fn get(&self, key_group: u32) -> Result<Option<StateChunk>>; // hot-cache or load |
| 37 | + fn put(&mut self, key_group: u32, chunk: StateChunk); // buffered, spillable |
| 38 | + fn scan(&self) -> impl Iterator<Item=(u32, StateChunk)>; // for Final merge / emit |
| 39 | + fn evict(&mut self, predicate); // drop closed windows (TTL) |
| 40 | + fn checkpoint(&mut self, epoch) -> Result<CheckpointHandle>; // INCREMENTAL upload |
| 41 | + fn restore(&mut self, handle: CheckpointHandle) -> Result<()>; // load on recovery |
| 42 | +} |
| 43 | +``` |
| 44 | + |
| 45 | +**Three layers, each a deliberate improvement over Flink:** |
| 46 | + |
| 47 | +1. **Bounded memory + spill (matches RocksDB, no JNI/GC).** State chunks live in an in-memory |
| 48 | + Arrow map under a `MemoryPool` budget; when exceeded, **spill** the coldest key-groups to a |
| 49 | + local on-disk store (embedded KV such as `redb`/`rocksdb`-rust, or Arrow IPC files keyed by |
| 50 | + key-group). Pure Rust, no JVM heap, no GC pauses → flatter tail latency than Flink at large |
| 51 | + state. Reuses DataFusion's `MemoryPool` so it composes with the rest of the pipeline. |
| 52 | + |
| 53 | +2. **Incremental + disaggregated checkpoints (matches ForSt, native).** State is keyed by |
| 54 | + **key-group** and content-addressed; `checkpoint(epoch)` uploads only key-groups **changed |
| 55 | + since the last epoch** to the object store (delta), writing one atomic manifest |
| 56 | + (`state/<op>/<epoch>/manifest`) — reusing Vajra's existing single-atomic-object commit (the |
| 57 | + same primitive the realtime EO sink uses). Because Vajra is **object-store-first already**, |
| 58 | + "disaggregated state" is the natural representation, not a bolt-on: the object store *is* the |
| 59 | + state of record; the local store is a write-back **cache**. This is the ForSt idea, native. |
| 60 | + |
| 61 | +3. **Rescaling by key-group.** State partitioned into a fixed large number of key-groups |
| 62 | + (Flink's approach); on parallelism change, key-groups are reassigned to instances and each |
| 63 | + loads its groups from the object store — clean elastic rescaling (pairs with the parallel |
| 64 | + Kafka source's `% N` assignment). |
| 65 | + |
| 66 | +## Why this beats Flink (the honest thesis) |
| 67 | +- **No JVM** → no GC pauses on large state (Flink's large-RocksDB tail-latency pain) and no |
| 68 | + off-heap/JNI tax. |
| 69 | +- **Arrow-columnar state** → vectorized merge/scan, zero-copy to the agg operators (Flink |
| 70 | + serializes per-key rows). |
| 71 | +- **Object-store-first** → checkpoints are cheap (state already remote, only the delta), and |
| 72 | + elasticity is free — Flink retrofitted this in 2.0 (ForSt); Vajra is built that way. |
| 73 | + |
| 74 | +## Build order (each step compiles + is chaos/scale-testable) |
| 75 | +1. **`StateBackend` trait + in-memory impl** (parity refactor): move `WindowAccumExec` off the |
| 76 | + raw `Vec<RecordBatch>` onto the trait. No behavior change; sets the seam. Validate 6/6 + EO. |
| 77 | +2. **Spill layer**: `MemoryPool`-budgeted, spill cold key-groups to a local embedded KV. |
| 78 | + *Acceptance:* a keyed windowed agg with **state ≫ RAM** runs stable (no OOM). |
| 79 | +3. **Incremental + disaggregated checkpoint**: per-key-group delta upload + atomic manifest; |
| 80 | + restore loads from object store. *Acceptance:* checkpoint upload scales with the delta, not |
| 81 | + total state; EO across crash with large state holds. |
| 82 | +4. **Rescaling**: key-group reassignment on parallelism change. *Acceptance:* change workers |
| 83 | + live, state redistributes, results stay correct. |
| 84 | + |
| 85 | +## Sequencing with the DataFusion 54.0.0 upgrade |
| 86 | +See `docs/design/datafusion-54-upgrade.md`. 54.0.0's `MemoryPool`/spill and StringView/BinaryView |
| 87 | +maturation help layer 2; decide order after the 54 arrow-bump delta is known. Both in |
| 88 | +`docs/PROD_GRADE_ROADMAP.md` (P0). |
0 commit comments