Skip to content

Commit b34b7f2

Browse files
vikashgarggclaude
andcommitted
design: large-state backend (beat Flink) + DataFusion 54.0.0 upgrade plan
- design/large-state-backend.md: prod-grade, implementation-ready StateBackend (Arrow-columnar, spillable, INCREMENTAL + DISAGGREGATED checkpoints) designed to BEAT Flink — not by porting RocksDB but by exploiting Vajra's no-JVM (no GC on state), Arrow-columnar (vectorized merge), object-store-first (ForSt-style disaggregation is native, not bolt-on) architecture. Build order: trait+in-mem parity refactor -> spill (state >> RAM) -> incremental/disaggregated checkpoint -> key-group rescaling, each with scale/chaos acceptance criteria. Grounded in Flink HashMap/RocksDB/changelog(FLIP-158)/ForSt(2.0) + DataFusion MemoryPool spill. - design/datafusion-54-upgrade.md: grounded in the 54.0.0 release notes (apache/datafusion#21080). Vajra wins: RepartitionExec coalesce (shuffle throughput), Parquet predicate short-circuit + stats reorder (faster scans), ORDER BY sort-key pruning, scalar-subquery exec, lateral joins, Extension Type Registry (AI-native types hook), BinaryView (retire the byte-bounded-batch workaround), arrow-avro. Staged low-risk upgrade (bump -> absorb arrow ripple -> re-run gates -> re-bench -> exploit). Breaking changes catalogued. - roadmap: link both designs (3.2 state, new 3.5b DF54). Co-Authored-By: Claude Opus 4.8 <[email protected]>
1 parent 18ef2bf commit b34b7f2

3 files changed

Lines changed: 131 additions & 1 deletion

File tree

‎docs/PROD_GRADE_ROADMAP.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ gap analysis, each grounded in how the incumbents do it.
9090
- *Acceptance:* ✅ **p99 < 100 ms** (51 ms) at 10k ev/s; ✅ exactly-once Kafka→Kafka across
9191
a kill (100000/100000). Latency P0 essentially complete.
9292

93-
### 3.2 State management & large state — **P0**
93+
### 3.2 State management & large state — **P0** · [design: `design/large-state-backend.md`]
9494
- **Flink:** pluggable keyed state; **RocksDB** backend for state >> memory; **incremental
9595
+ changelog checkpoints** (only deltas uploaded); state TTL; **rescalable** state
9696
(key-group reassignment). **Flink 2.0 disaggregated state (ForSt):** state lives on
@@ -141,6 +141,13 @@ gap analysis, each grounded in how the incumbents do it.
141141
- **Needed:** per-partition watermark generation + MIN-merge in the distributed exchange;
142142
idle-partition handling; allowed-lateness + side outputs.
143143

144+
### 3.5b DataFusion 54.0.0 upgrade — **P1 (free wins)** · [plan: `design/datafusion-54-upgrade.md`]
145+
DF 54.0.0 brings RepartitionExec coalesce-before-distributor (helps our shuffle), Parquet
146+
predicate short-circuit + stats-based file/row-group reorder (faster scans), ORDER BY
147+
redundant-sort-key pruning, scalar-subquery physical exec, lateral joins, lambda/`array_transform`,
148+
an Extension Type Registry (hook for Vajra-native vector/AI types), and arrow-avro. Bump from
149+
53.1.0, absorb the arrow-version ripple (codec/flow-event/parquet), re-run all gates, re-bench.
150+
144151
### 3.6 Adaptive execution & query optimization (batch) — **P1**
145152
- **Spark AQE:** runtime re-optimization (coalesce shuffle partitions, skew-join handling,
146153
switch join strategy from runtime stats). **DataFusion:** dynamic filters, bounded batches.
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# DataFusion 53.1.0 → 54.0.0 upgrade plan (optimization opportunities)
2+
3+
> Vajra embeds DataFusion as its execution core. DataFusion **54.0.0** is released
4+
> (https://datafusion.apache.org/download.html, release tracking apache/datafusion#21080).
5+
> This plans the upgrade and where 54.0.0 buys Vajra concrete wins. Current pin: **53.1.0**
6+
> (workspace `Cargo.toml`, ~30 `datafusion*` crates incl. `datafusion-proto` used by our codec).
7+
8+
## What 54.0.0 brings that directly helps Vajra
9+
10+
| 54.0.0 change | Vajra benefit |
11+
|---|---|
12+
| **RepartitionExec: coalesce batches before sending to distributor channels** | Directly speeds our **shuffle** path (`StreamExchangeExec` / Arrow-Flight shuffle build on the same repartition model) — fewer, larger batches across channels = higher throughput, lower per-batch overhead. Helps both batch (ClickBench/TPC-H distributed) and the streaming exchange. |
13+
| **Parquet: skip predicates provably false; reorder files+row-groups by stats; sort pushdown; TopK file reorder** | Faster batch scans (TPC-H/ClickBench, Iceberg/Delta reads) — less I/O + CPU on selective + TopK queries. |
14+
| **ORDER BY: prune functionally-redundant sort keys** | Removes needless sorts in batch plans → lower latency + memory. |
15+
| **Datetime predicate simplification (preimages, date_trunc/date_part)** | Common in TPC-H/analytics; cheaper filters. |
16+
| **Scalar subquery physical execution (`ScalarSubqueryExec`, gated)** | Faster correlated/scalar subqueries vs interpretation. |
17+
| **Lateral joins; lambda expressions + `array_transform`; Extension Type Registry** | Spark-compat surface area (lateral, higher-order fns) + a clean hook for **Vajra-native types** (vector/embedding for the AI-native north star) via the type registry. |
18+
| **arrow-avro migration** | Drops DataFusion's internal Avro conversion → less code, faster Avro reads (we use Avro in the Kafka/source path). |
19+
| Continued **StringView/BinaryView** maturation | Lets us replace the byte-bounded-batch workaround in the Kafka source with `BinaryView` (i64-class offsets) for genuinely huge values — the Arrow-native fix for the i32 offset class. |
20+
21+
## Breaking changes to handle (from the 54.0.0 notes)
22+
- **`ExecutionPlan::apply_expressions()` reverted** → no action (it was never in 53.1.0's stable surface we use).
23+
- **`ScalarSubqueryExec` gated behind a session property** → opt-in; set the session flag if we want the perf path.
24+
- **Higher-order UDF wrapping refactored to a concrete struct** → check our UDF registrations (`sail-function`) still compile.
25+
- **Arrow major bump** (54 tracks a newer arrow-rs than 53) → the largest ripple: Vajra touches Arrow pervasively (arrays, compute, IPC for Flight shuffle, parquet). `datafusion-proto` (our `codec.rs`) and the flow-event encoding must compile against the new arrow. **Verify the exact arrow version from `datafusion 54.0.0`'s `Cargo.toml` before starting.**
26+
27+
## Upgrade approach (low-risk, staged)
28+
1. **Branch + bump** all `datafusion*` pins 53.1.0 → 54.0.0 in workspace `Cargo.toml`; `cargo update`; resolve the arrow version bump.
29+
2. **Compile-fix** the workspace crate-by-crate (expect changes in: `sail-execution/codec.rs` via datafusion-proto; `sail-function` UDF wrappers; any `ExecutionPlan`/`PlanProperties` API drift; arrow array/compute signature changes).
30+
3. **Re-run the full gates**: 105/105 multi-mode scorecard; differential-trust 124/124 vs Spark; clippy `-D warnings`; the streaming suite (6/6 + EO + the new Kafka-sink latency/EO tests).
31+
4. **Re-benchmark** TPC-H SF-1/SF-100 + ClickBench to quantify the 54.0.0 scan/repartition gains (expect modest but real batch speedups; the RepartitionExec coalesce should help distributed throughput).
32+
5. **Then exploit**: enable `ScalarSubqueryExec`; adopt `BinaryView` in the Kafka source; wire the Extension Type Registry for Vajra-native types.
33+
34+
## Sequencing vs the large-state backend
35+
Do the **DF-54 upgrade first** if its Arrow bump is moderate (it de-risks everything downstream and the RepartitionExec/Parquet wins are free), **or** land the large-state backend on 53.1.0 first if the arrow bump is disruptive — decide after step 1 reveals the arrow delta. Both are tracked in `docs/PROD_GRADE_ROADMAP.md`.

‎docs/design/large-state-backend.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Large-state backend (P0) — spillable + disaggregated, designed to beat Flink
2+
3+
> The streaming latency + EO + throughput + memory gaps vs Flink are now closed/won
4+
> (`STREAMING_VS_FLINK_EKS.md`). The remaining operational gap is **large keyed state**:
5+
> Vajra keeps streaming state **in memory** and snapshots it **whole** to the checkpoint.
6+
> Flink handles huge state via RocksDB + incremental checkpoints + (2.0) disaggregated
7+
> state (ForSt). This designs a Vajra state backend that **matches and then beats** Flink —
8+
> not by porting RocksDB, but by exploiting Vajra's no-JVM, Arrow-columnar, object-store-first
9+
> architecture.
10+
11+
## Where Vajra is today
12+
- `WindowAccumExec` keeps partial-aggregate state in `AccumState.pending_rows: Vec<RecordBatch>`
13+
(in memory), pruned to open windows, and **full-snapshotted** to the object-store checkpoint
14+
on epoch/EndOfData (`state_io::stage_state`). `StreamJoinExec` / dedup hold state in memory too.
15+
- Consequence: state must fit RAM (OOM risk at large keyspaces), and every checkpoint uploads
16+
the **entire** state (not the delta).
17+
18+
## How the proven systems do it (and what we take, not copy)
19+
- **Flink HashMapStateBackend** (heap) — fast, but JVM-GC-bound and RAM-limited.
20+
- **Flink RocksDBStateBackend** — embedded LSM on local disk → state ≫ RAM; **incremental
21+
checkpoints** upload only new/changed SST files; JNI + JVM-off-heap overhead.
22+
- **Flink changelog state backend (FLIP-158)** — decouples a durability *log* from periodic
23+
materialization → faster, more predictable checkpoints.
24+
- **Flink 2.0 disaggregated state (ForSt)** — state lives on DFS/object store, compute reads via
25+
a local cache → cloud elasticity, fast rescaling, cheap checkpoints (state already remote).
26+
- **DataFusion** — `GroupedHashAggregateStream` + `MemoryPool` spill: when a memory budget is
27+
exceeded, spill to disk and merge — the model for *bounded-memory* operators.
28+
29+
## Vajra design — `StateBackend`: Arrow-columnar, spillable, disaggregated-native
30+
A pluggable trait, Arrow-first (state is RecordBatches, not opaque bytes), used by
31+
`WindowAccumExec` / `StreamJoinExec` / dedup instead of in-memory `Vec`s:
32+
33+
```text
34+
trait StateBackend {
35+
// keyed columnar state: key-group -> the operator's partial-state batches
36+
fn get(&self, key_group: u32) -> Result<Option<StateChunk>>; // hot-cache or load
37+
fn put(&mut self, key_group: u32, chunk: StateChunk); // buffered, spillable
38+
fn scan(&self) -> impl Iterator<Item=(u32, StateChunk)>; // for Final merge / emit
39+
fn evict(&mut self, predicate); // drop closed windows (TTL)
40+
fn checkpoint(&mut self, epoch) -> Result<CheckpointHandle>; // INCREMENTAL upload
41+
fn restore(&mut self, handle: CheckpointHandle) -> Result<()>; // load on recovery
42+
}
43+
```
44+
45+
**Three layers, each a deliberate improvement over Flink:**
46+
47+
1. **Bounded memory + spill (matches RocksDB, no JNI/GC).** State chunks live in an in-memory
48+
Arrow map under a `MemoryPool` budget; when exceeded, **spill** the coldest key-groups to a
49+
local on-disk store (embedded KV such as `redb`/`rocksdb`-rust, or Arrow IPC files keyed by
50+
key-group). Pure Rust, no JVM heap, no GC pauses → flatter tail latency than Flink at large
51+
state. Reuses DataFusion's `MemoryPool` so it composes with the rest of the pipeline.
52+
53+
2. **Incremental + disaggregated checkpoints (matches ForSt, native).** State is keyed by
54+
**key-group** and content-addressed; `checkpoint(epoch)` uploads only key-groups **changed
55+
since the last epoch** to the object store (delta), writing one atomic manifest
56+
(`state/<op>/<epoch>/manifest`) — reusing Vajra's existing single-atomic-object commit (the
57+
same primitive the realtime EO sink uses). Because Vajra is **object-store-first already**,
58+
"disaggregated state" is the natural representation, not a bolt-on: the object store *is* the
59+
state of record; the local store is a write-back **cache**. This is the ForSt idea, native.
60+
61+
3. **Rescaling by key-group.** State partitioned into a fixed large number of key-groups
62+
(Flink's approach); on parallelism change, key-groups are reassigned to instances and each
63+
loads its groups from the object store — clean elastic rescaling (pairs with the parallel
64+
Kafka source's `% N` assignment).
65+
66+
## Why this beats Flink (the honest thesis)
67+
- **No JVM** → no GC pauses on large state (Flink's large-RocksDB tail-latency pain) and no
68+
off-heap/JNI tax.
69+
- **Arrow-columnar state** → vectorized merge/scan, zero-copy to the agg operators (Flink
70+
serializes per-key rows).
71+
- **Object-store-first** → checkpoints are cheap (state already remote, only the delta), and
72+
elasticity is free — Flink retrofitted this in 2.0 (ForSt); Vajra is built that way.
73+
74+
## Build order (each step compiles + is chaos/scale-testable)
75+
1. **`StateBackend` trait + in-memory impl** (parity refactor): move `WindowAccumExec` off the
76+
raw `Vec<RecordBatch>` onto the trait. No behavior change; sets the seam. Validate 6/6 + EO.
77+
2. **Spill layer**: `MemoryPool`-budgeted, spill cold key-groups to a local embedded KV.
78+
*Acceptance:* a keyed windowed agg with **state ≫ RAM** runs stable (no OOM).
79+
3. **Incremental + disaggregated checkpoint**: per-key-group delta upload + atomic manifest;
80+
restore loads from object store. *Acceptance:* checkpoint upload scales with the delta, not
81+
total state; EO across crash with large state holds.
82+
4. **Rescaling**: key-group reassignment on parallelism change. *Acceptance:* change workers
83+
live, state redistributes, results stay correct.
84+
85+
## Sequencing with the DataFusion 54.0.0 upgrade
86+
See `docs/design/datafusion-54-upgrade.md`. 54.0.0's `MemoryPool`/spill and StringView/BinaryView
87+
maturation help layer 2; decide order after the 54 arrow-bump delta is known. Both in
88+
`docs/PROD_GRADE_ROADMAP.md` (P0).

0 commit comments

Comments
 (0)