Skip to content

Commit 0e93720

Browse files
schenksjclaude
andcommitted
docs(delta): audit + correct all contrib/delta design docs for drift (apache#85)
The design docs had drifted from the kernel-read refactor (apache#76/apache#77/apache#80/apache#81/apache#82/ apache#84/#2/apache#78/apache#86). Audited all 13 docs against current code and corrected: - Removed the deleted ParquetSource + DV-sweep + DeltaSyntheticColumnsExec read stack as the "current" path everywhere; it is now kernel-read only (apache#50/apache#82), with DeltaKernelScanExec doing in-worker synthesis. The old stack is kept only as clearly-labeled history / rejected alternatives. - delta_scan.rs is a ~72-line shim delegating to comet_contrib_delta::planner (apache#77); column-mapping physicalisation dropped, kernel ships the schemas (apache#76). - CDF (readChangeFeed) is kernel-native via TableChanges -> CometDeltaCdfScanExec, split multi-partition (apache#84/#2) -- corrected docs that called it unsupported, declined, or a synthetic-columns fallback. - 08-known-limitations.md: removed all of Part B (B1-B9 were development-time regressions, all now fixed + guarded) and A3 (path-based CDF now engages native, apache#84); kept only genuine current limitations (A1 DPP residual, A2e credential residual, A4 VARIANT, A5 decline gates, A6 INT96 kernel gap, A7 CM-id repoint). 466 -> 230 lines. - Fixed config keys, build/module layout, JNI symbols, file paths, CI workflow references, and supported-feature lists (added CDF, _metadata, INT96) across the build / README / user-guide docs. Every claim verified against code; markdown passes prettier. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent a88ec8b commit 0e93720

13 files changed

Lines changed: 776 additions & 848 deletions

‎contrib/delta/docs/01-overview.md‎

Lines changed: 42 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -19,12 +19,13 @@
1919

2020
# Comet Delta Contrib — Design Overview
2121

22-
> **Note:** the executor-side **read path** described below (ParquetSource +
23-
> field-id resolution + DV-sweep + synthetic columns + rename ProjectionExec) has
24-
> been **replaced** by the "kernel reads" path: `DeltaKernelScanExec` reads each
25-
> file through delta-kernel-rs (0.24 / arrow-58) directly. The driver-side planning,
26-
> deployment modes, and "what this does NOT touch" sections below are still accurate;
27-
> for the current execution design see
22+
> **Read path:** Delta tables are read **natively through delta-kernel-rs**
23+
> ("kernel-read"), enabled by default. `DeltaKernelScanExec` reads each file
24+
> through delta-kernel-rs (0.24 / arrow-58), applying kernel's physical→logical
25+
> transform (column mapping incl. nested, partition injection, row-tracking) and
26+
> the deletion vector itself, and synthesises Delta's virtual columns in-worker.
27+
> The legacy ParquetSource + DV-sweep + stacked-synthetic-columns read path has
28+
> been **removed**; kernel-read is the only read path. For execution detail see
2829
> [10-iceberg-style-kernel-read.md](10-iceberg-style-kernel-read.md).
2930
3031
## Who this is for
@@ -46,11 +47,14 @@ that Spark's `FileSourceScanExec` doesn't expose cleanly enough for Comet to
4647
slot in underneath.
4748

4849
This contrib bypasses that wrapping. It plans Delta scans with
49-
`delta-kernel-rs` (the official Rust kernel maintained by the Delta team),
50-
hands the resolved file list to DataFusion's parquet reader, and synthesises
51-
Delta's "virtual" columns (`row_id`, `__delta_internal_is_row_deleted`, etc.)
52-
in native code. The result is end-to-end native execution for Delta reads,
53-
with no Spark-side parquet decoding on the hot path.
50+
`delta-kernel-rs` (the official Rust kernel maintained by the Delta team), and
51+
reads each resolved data file through kernel itself — kernel does the parquet
52+
read, applies the physical→logical transform (column mapping, partition
53+
injection, row-tracking materialisation), and applies the deletion vector. Any
54+
remaining Delta "virtual" columns (`__delta_internal_is_row_deleted`,
55+
`row_commit_version`, `_metadata.*`) are synthesised in-worker by
56+
`DeltaKernelScanExec`. The result is end-to-end native execution for Delta
57+
reads, with no Spark-side parquet decoding on the hot path.
5458

5559
## Mental model: a DSv2 substitute scan that fires before DSv2 binding
5660

@@ -86,46 +90,47 @@ produced by DataFusion in Rust and shipped to the JVM as Arrow record batches.
8690
│ │
8791
│ Catalyst logical plan │
8892
│ │ │
89-
│ │ DeltaScanRule.transformV1IfDelta (extension) │
93+
│ │ DeltaScanRule.transformV1IfDelta (CometScanRule arm) │
9094
│ ▼ │
91-
│ Plan with CometDeltaNativeScanExec in place of FileSourceScanExec │
95+
│ CometDeltaScanMarker wrapping the original FileSourceScanExec │
9296
│ │ │
97+
│ │ CometExecRule routes the marker to the Delta serde │
9398
│ │ CometDeltaNativeScan.convert (proto serde) │
94-
│ │ 1. delta-kernel-rs resolves snapshot │
95-
│ │ 2. Returns AddFile list + DV info + base row IDs │
96-
│ │ 3. Encode into DeltaScan proto (common + per-task) │
99+
│ │ 1. JNI → planDeltaScan: kernel enumerates the snapshot │
100+
│ │ and ships per file a transform_json + DV descriptor │
101+
│ │ + kernel-built physical/logical Arrow schemas │
102+
│ │ (DML rewrites: kernel-enumerate + path-filter) │
103+
│ │ 2. Encode into DeltaScan proto (common + per-task) │
97104
│ ▼ │
98-
│ Per-partition byte arrays (DeltaScanTask) via PlanDataInjector │
105+
│ CometDeltaNativeScanExec; per-partition DeltaScanTask byte arrays │
106+
│ injected via DeltaPlanDataInjector │
99107
└───────────────────┬─────────────────────────────────────────────────┘
100108
│ shipped to executors via Spark task serialisation
101109
▼
102110
┌─────────────────────────────────────────────────────────────────────┐
103111
│ SPARK EXECUTOR (JVM + Rust via JNI) │
104112
│ │
105-
│ CometNativeScanExec.compute │
113+
│ Comet native exec framework │
106114
│ │ │
107-
│ │ JNI → planDeltaScan(proto bytes) │
115+
│ │ delta_scan.rs shim → comet_contrib_delta::planner │
108116
│ ▼ │
109-
│ contrib_delta_scan::build_plan (Rust) │
117+
│ DeltaKernelScanExec (contrib/delta/native/src/kernel_scan.rs) │
110118
│ │ │
111-
│ │ Builds DataFusion ExecutionPlan tree: │
112-
│ │ ParquetSource (with field_id resolution) │
119+
│ │ For each file, read parquet via delta-kernel-rs, apply │
120+
│ │ kernel's transform (CM/partition/row-tracking) + DV, and │
121+
│ │ produce ALL output columns BY NAME — in-worker synthesis │
122+
│ │ (data + partitions + row_index/row_id/is_row_deleted/ │
123+
│ │ row_commit_version/_metadata.*) │
113124
│ │ ↓ │
114-
│ │ ProjectionExec (CM rename, if physical != logical names) │
115-
│ │ ↓ │
116-
│ │ DeltaSyntheticColumnsExec if any emit_* flag │
117-
│ │ OR │
118-
│ │ DeltaDvFilterExec else if any task has DV │
119-
│ │ ↓ │
120-
│ │ ProjectionExec (reorder if synthetics not a suffix) │
125+
│ │ ProjectionExec (reorder via final_output_indices, if any) │
121126
│ ▼ │
122127
│ Arrow RecordBatch stream → Comet's existing Arrow→JVM bridge │
123128
└─────────────────────────────────────────────────────────────────────┘
124129
```
125130

126131
The two non-obvious pieces are **the proto split** (a single "common" block
127-
plus per-partition task arrays) and the **native wrapping stack** (DV /
128-
rename / synthetics / reorder). Both are covered in
132+
plus per-partition task arrays) and the **in-worker synthesis** done by
133+
`DeltaKernelScanExec`. Both are covered in
129134
[02-planning.md](02-planning.md) and
130135
[03-native-execution.md](03-native-execution.md).
131136

@@ -134,9 +139,10 @@ rename / synthetics / reorder). Both are covered in
134139
Default builds (no `-Pcontrib-delta` Maven profile, no `contrib-delta` Cargo
135140
feature) ship with zero Delta surface area:
136141

137-
- The reflection bridge in `DeltaIntegration.scala` returns `None` at the
138-
first classpath lookup and stays that way for the JVM lifetime
139-
- The `contrib_delta_scan` arm in `native/core/src/execution/planner/` is
142+
- The reflection bridge in `spark/.../rules/DeltaIntegration.scala` returns the
143+
"not handled" sentinel at the first classpath lookup and stays that way for
144+
the JVM lifetime
145+
- The `delta_scan` arm in `native/core/src/execution/planner/delta_scan.rs` is
140146
`#[cfg(feature = "contrib-delta")]`-gated and compiles out of the dylib
141147
- The proto variant `delta_scan = 117` is present in the schema but never
142148
emitted
@@ -145,8 +151,8 @@ Delta-enabled builds (`-Pcontrib-delta` + `contrib-delta` Cargo feature):
145151

146152
- `contrib/delta/src/main/scala/...` lands on the classpath, including the
147153
Spark extension that registers `DeltaScanRule`
148-
- `contrib/delta/native/` is linked into `libcomet` as a static lib via
149-
re-export, contributing the `Java_…_planDeltaScan` JNI symbol
154+
- `contrib/delta/native/` (the `comet_contrib_delta` crate) is linked into
155+
`libcomet`, contributing the `Java_…_planDeltaScan` JNI symbol
150156
- The reflection bridge resolves on first call and caches the result
151157

152158
This is the same shape as the Iceberg contrib in this repo. The motivation is

0 commit comments

Comments
 (0)