|
| 1 | +# Plan: move delta-kernel-rs integration into a Comet "contrib" extension |
| 2 | + |
| 3 | +Driver: PR review feedback on `delta-kernel-phase-1` asked the Delta integration to live in a |
| 4 | +contrib section rather than core so the core stays format-agnostic and the heavyweight |
| 5 | +delta-kernel-rs dependency (with its own `object_store_kernel` pin at 0.12 vs core's |
| 6 | +`object_store` at 0.13) is opt-in. |
| 7 | + |
| 8 | +User-confirmed decisions: |
| 9 | +1. **Single PR.** No series of incremental PRs. All structural changes ship together. |
| 10 | +2. **Separate extension JAR with its own cdylib.** Delta contrib ships as a fully independent |
| 11 | + artifact whose native library is loaded at runtime when the contrib JAR is on the classpath. |
| 12 | +3. **Iceberg stays in core.** It is treated as a first-class Comet feature; no parallel move. |
| 13 | +4. **Proto stays as-is.** Keep the existing `DeltaScan` message verbatim; only its |
| 14 | + reader/writer ownership relocates from core to contrib. (See "Proto handling" below.) |
| 15 | + |
| 16 | +This plan executes AFTER P1 (rich credentials for log replay) is signed off — P1 lives in the |
| 17 | +files being moved and merging the contrib split before P1 would conflict messily. |
| 18 | + |
| 19 | +## 1. Target layout |
| 20 | + |
| 21 | +``` |
| 22 | +contrib/ |
| 23 | + delta/ |
| 24 | + pom.xml ← new Maven module (groupId=org.apache.datafusion, |
| 25 | + artifactId=comet-contrib-delta-spark${spark.version.short}_${scala.binary.version}) |
| 26 | + native/ |
| 27 | + Cargo.toml ← new Cargo crate (package=comet-contrib-delta) |
| 28 | + src/ |
| 29 | + lib.rs ← cdylib entry, JNI symbol exports, registration |
| 30 | + engine.rs ← from native/core/src/delta/engine.rs |
| 31 | + error.rs ← from native/core/src/delta/error.rs |
| 32 | + jni.rs ← from native/core/src/delta/jni.rs |
| 33 | + predicate.rs ← from native/core/src/delta/predicate.rs |
| 34 | + scan.rs ← from native/core/src/delta/scan.rs |
| 35 | + integration_tests.rs ← from native/core/src/delta/integration_tests.rs |
| 36 | + dv_filter.rs ← from native/core/src/execution/operators/delta_dv_filter.rs |
| 37 | + planner_ext.rs ← (NEW) implements ContribOperatorPlanner for OpStruct::DeltaScan |
| 38 | + scan_rule_ext.rs ← (NEW) glue called by Scala-side CometScanRuleExtension |
| 39 | + spark/ |
| 40 | + src/main/scala/org/apache/comet/contrib/delta/ |
| 41 | + DeltaScanExtension.scala ← (NEW) implements CometScanRuleExtension |
| 42 | + package.scala ← (NEW) ServiceLoader registration helpers |
| 43 | + src/main/scala/org/apache/comet/delta/ |
| 44 | + DeltaReflection.scala ← moved from spark/.../comet/delta/ |
| 45 | + src/main/scala/org/apache/comet/serde/operator/ |
| 46 | + CometDeltaNativeScan.scala ← moved |
| 47 | + src/main/scala/org/apache/spark/sql/comet/ |
| 48 | + CometDeltaNativeScanExec.scala ← moved |
| 49 | + src/main/resources/META-INF/services/ |
| 50 | + org.apache.comet.spi.CometScanRuleExtension ← lists DeltaScanExtension |
| 51 | + dev/ |
| 52 | + diffs/3.3.2.diff ← moved from dev/diffs/delta/3.3.2.diff |
| 53 | + run-regression.sh ← moved from dev/run-delta-regression.sh |
| 54 | +``` |
| 55 | + |
| 56 | +`contrib/delta/native/` produces `libcomet_contrib_delta_<arch>.dylib` packaged in the contrib |
| 57 | +JAR's `META-INF/native/`. The contrib JAR's class init (via ServiceLoader) extracts and loads |
| 58 | +this library; core's JNI is unaware. |
| 59 | + |
| 60 | +## 2. Files that move wholesale |
| 61 | + |
| 62 | +These are 100% Delta and have no incoming references from non-Delta callers outside the hook |
| 63 | +points listed in §4: |
| 64 | + |
| 65 | +### Rust → `contrib/delta/native/src/` |
| 66 | +- `native/core/src/delta/{engine,error,jni,mod,predicate,scan,integration_tests}.rs` |
| 67 | +- `native/core/src/execution/operators/delta_dv_filter.rs` → `contrib/delta/native/src/dv_filter.rs` |
| 68 | + |
| 69 | +### Scala |
| 70 | +- `spark/.../comet/delta/DeltaReflection.scala` |
| 71 | +- `spark/.../serde/operator/CometDeltaNativeScan.scala` |
| 72 | +- `spark/.../sql/comet/CometDeltaNativeScanExec.scala` |
| 73 | + |
| 74 | +### Test driver |
| 75 | +- `dev/diffs/delta/3.3.2.diff` and `dev/run-delta-regression.sh` |
| 76 | + |
| 77 | +## 3. Code that stays in core unchanged |
| 78 | + |
| 79 | +- `common/.../objectstore/NativeConfig.scala` — generic, used by core + Iceberg + contribs. |
| 80 | +- `native/core/src/parquet/objectstore/s3.rs::resolve_static_credentials` — generic (P1 |
| 81 | + placed it here intentionally; the contrib delta JNI module consumes it via core's public crate path). |
| 82 | +- The Iceberg integration in its entirety. |
| 83 | +- The `DeltaScan` / `DeltaScanCommon` / `DeltaScanTask` / `DeltaScanTaskList` proto messages |
| 84 | + in `native/proto/src/proto/operator.proto`. They stay where they are; only their reader and |
| 85 | + writer move to contrib. The proto crate is generic infrastructure (already a top-level Cargo |
| 86 | + member with no Delta logic), so no module change there. |
| 87 | + |
| 88 | +## 4. Code that needs extension points |
| 89 | + |
| 90 | +These are the touch-points outside the `delta/` directories that today reference Delta directly. |
| 91 | +Each gets replaced by a registration hook so core no longer mentions Delta. |
| 92 | + |
| 93 | +### Rust core |
| 94 | + |
| 95 | +| File | Current Delta refs | After move | |
| 96 | +|---|---|---| |
| 97 | +| `native/core/src/lib.rs` | `pub mod delta;` (line 70) | removed | |
| 98 | +| `native/core/src/execution/jni_api.rs` | 1 reference | removed (Delta JNI moves to contrib's cdylib) | |
| 99 | +| `native/core/src/execution/operators/mod.rs` | `mod delta_dv_filter` + re-export | removed | |
| 100 | +| `native/core/src/execution/operators/operator_registry.rs` (line 154) | `OpStruct::DeltaScan(_) => None` stub | replaced by generic ContribScan registry lookup | |
| 101 | +| `native/core/src/execution/planner.rs` (~31 refs) | full `OpStruct::DeltaScan(...)` arm at ~1320–1572, plus `ColumnMappingFilterRewriter` at 3115–3145 | the arm calls into `ContribOperatorPlanner::plan(payload, children, ctx)` looked up by `OpStruct` discriminator; the rewriter moves to contrib | |
| 102 | + |
| 103 | +**New SPI in core (`native/core/src/execution/planner/contrib.rs`):** |
| 104 | + |
| 105 | +```rust |
| 106 | +pub trait ContribOperatorPlanner: Send + Sync { |
| 107 | + fn op_kind(&self) -> ContribOpKind; |
| 108 | + fn plan( |
| 109 | + &self, |
| 110 | + proto: &OpStruct, |
| 111 | + children: Vec<Arc<SparkPlan>>, |
| 112 | + ctx: &PlanCtx<'_>, |
| 113 | + ) -> Result<Arc<dyn ExecutionPlan>, ExecutionError>; |
| 114 | +} |
| 115 | + |
| 116 | +pub fn register_contrib_planner(p: Arc<dyn ContribOperatorPlanner>); |
| 117 | +pub fn lookup_contrib_planner(op_kind: ContribOpKind) -> Option<Arc<dyn ContribOperatorPlanner>>; |
| 118 | +``` |
| 119 | + |
| 120 | +`ContribOpKind` is an enum that lists `DeltaScan` (and stays open for future kinds). The |
| 121 | +planner.rs match arm becomes a single lookup that dispatches by kind — the *body* of the |
| 122 | +match arm moves to contrib's `planner_ext.rs`. |
| 123 | + |
| 124 | +The contrib cdylib's `lib.rs` registers its planner in a `#[ctor]` (the `ctor` crate) or |
| 125 | +inside the contrib JNI's library-init JNI call. Either way, registration happens before any |
| 126 | +plan executes. |
| 127 | + |
| 128 | +### JVM / Scala core |
| 129 | + |
| 130 | +| File | Current Delta refs | After move | |
| 131 | +|---|---|---| |
| 132 | +| `spark/.../CometSparkSessionExtensions.scala` | 20 (mostly comments + one or two conf entries) | comments scrubbed; any Delta-specific conf moves into contrib's own `CometConf` extension | |
| 133 | +| `spark/.../rules/CometExecRule.scala` | 2 (case for `SCAN_NATIVE_DELTA_COMPAT` + import) | replaced by SPI scan dispatcher that calls registered extensions' serdes | |
| 134 | +| `spark/.../rules/CometScanRule.scala` | **149** (the densest Delta-aware site) | the Delta branch of `transformV1Scan` becomes a call into `CometScanRuleExtension`. The `withDeltaColumnMappingMetadata`, `applyRowTrackingRewrite`, input_file_name handling (#75 design A), nested-CM data_schema overlay (#79), DV materialisation for pre-mat FileIndex (#77), and `extractObjectStoreOptions` wiring all move into contrib's `DeltaScanExtension` | |
| 135 | + |
| 136 | +**New SPI in core (`spark/.../comet/spi/`):** |
| 137 | + |
| 138 | +```scala |
| 139 | +trait CometScanRuleExtension { |
| 140 | + def name: String |
| 141 | + def matches(relation: HadoopFsRelation): Boolean |
| 142 | + def transform( |
| 143 | + plan: SparkPlan, |
| 144 | + scanExec: FileSourceScanExec, |
| 145 | + session: SparkSession): Option[SparkPlan] |
| 146 | +} |
| 147 | + |
| 148 | +trait CometOperatorSerdeExtension { |
| 149 | + def serdes: Seq[(Class[_ <: SparkPlan], CometOperatorSerde[_])] |
| 150 | +} |
| 151 | + |
| 152 | +object CometExtensionRegistry { |
| 153 | + // ServiceLoader-based discovery from META-INF/services/. |
| 154 | + // Idempotent; safe to call multiple times. |
| 155 | + def load(): Unit |
| 156 | + def scanExtensions: Seq[CometScanRuleExtension] |
| 157 | + def serdeExtensions: Seq[CometOperatorSerdeExtension] |
| 158 | +} |
| 159 | +``` |
| 160 | + |
| 161 | +`CometSparkSessionExtensions` calls `CometExtensionRegistry.load()` during its extension |
| 162 | +registration, which lets the contrib JAR's `META-INF/services/...CometScanRuleExtension` entry |
| 163 | +register `DeltaScanExtension`. No code changes in core after that — discovery is by classpath |
| 164 | +presence. |
| 165 | + |
| 166 | +## 5. Build wiring |
| 167 | + |
| 168 | +### Maven |
| 169 | +- New module entry in root `pom.xml`: `<module>contrib/delta</module>`. |
| 170 | +- `contrib/delta/pom.xml`: |
| 171 | + - artifactId `comet-contrib-delta-spark${spark.version.short}_${scala.binary.version}` |
| 172 | + - Depends on `comet-spark`, `comet-common`. |
| 173 | + - Depends on `delta-spark` (provided/test scope — kept consistent with the regression suite). |
| 174 | + - Has its own native-library packaging step that places the contrib cdylib under |
| 175 | + `META-INF/native/` in the JAR. |
| 176 | +- `spark/pom.xml` — drop the Delta provided/test dep that's now in contrib. |
| 177 | + |
| 178 | +### Cargo |
| 179 | +- New crate `contrib/delta/native/` with `crate-type = ["cdylib"]`. Package name |
| 180 | + `comet-contrib-delta`. |
| 181 | +- Add to root `native/Cargo.toml` `members` and `default-members`. |
| 182 | +- Cargo dependency on `datafusion-comet` (the core crate) for the public SPI types |
| 183 | + (`ContribOperatorPlanner`, `register_contrib_planner`, etc). |
| 184 | +- `delta_kernel`, `object_store_kernel` dependencies move from `native/core/Cargo.toml` to |
| 185 | + `contrib/delta/native/Cargo.toml`. Core no longer pulls them in. |
| 186 | + |
| 187 | +### JNI symbol packaging and loading |
| 188 | +- The contrib JAR's resources contain `META-INF/native/<os>/<arch>/libcomet_contrib_delta.so|.dylib|.dll`. |
| 189 | +- A small helper in contrib's `package.scala` (called by `CometExtensionRegistry.load()` |
| 190 | + before the contrib's `DeltaScanExtension` is registered) extracts the platform-specific |
| 191 | + library to a temp dir and calls `System.load()`. Same pattern Comet core already uses for |
| 192 | + its own native lib. |
| 193 | +- Contrib's cdylib has a JNI `OnLoad` (or equivalent) that registers its |
| 194 | + `ContribOperatorPlanner` with the core crate's registry. The core crate's registry is a |
| 195 | + `OnceLock<Mutex<Vec<...>>>`, so this works across cdylib boundaries because the contrib |
| 196 | + cdylib links against `datafusion-comet` and shares the static. |
| 197 | + |
| 198 | + Caveat: Rust statics are per-DSO. The contrib cdylib must link to the core crate *as a |
| 199 | + shared library* (via the same `libcomet.dylib` already loaded by JVM core) — not as a |
| 200 | + Cargo dependency that statically links a second copy of `datafusion-comet`. We achieve |
| 201 | + this with `crate-type = ["cdylib"]` on contrib AND making core's crate available as a |
| 202 | + dylib for the contrib build profile. This is the trickiest item in the plan; if Rust's |
| 203 | + shared-static crossing turns out fiddly, we fall back to passing registration through a |
| 204 | + JNI call: contrib's library-init JNI calls a `register_contrib_delta_planner` Java method |
| 205 | + that uses JNI again to call into core's registry. |
| 206 | + |
| 207 | +### Spotless / scalafmt |
| 208 | +- Add `contrib/delta/` patterns to spotless config so the same formatting applies. |
| 209 | + |
| 210 | +## 6. Single-PR ordering inside the same branch |
| 211 | + |
| 212 | +All changes land in one branch, one PR. Inside the branch the work proceeds in this order so |
| 213 | +the regression stays green at each intermediate commit (helpful for reviewer bisect and for |
| 214 | +our own validation): |
| 215 | + |
| 216 | +1. **Land P1** on `delta-kernel-phase-1` (separate commit, separate validation; already |
| 217 | + pending). This is the precondition for the contrib work. |
| 218 | +2. **Add the SPI traits** (`CometScanRuleExtension`, `ContribOperatorPlanner`, etc.) in core |
| 219 | + with NO callers yet — pure additions. Regression unchanged. |
| 220 | +3. **Wire `CometScanRule` and `planner.rs` to call the registry**, with an in-tree shim |
| 221 | + `DeltaScanExtension` registered from core. Logic unchanged; only structurally pluggable |
| 222 | + now. Regression unchanged. |
| 223 | +4. **Create the new contrib/delta/ module** with the new build wiring (Maven module + |
| 224 | + Cargo crate). Initially empty — just the build skeleton. |
| 225 | +5. **Physically move the Delta files** out of core into `contrib/delta/`. Update imports. |
| 226 | + Wire contrib's ServiceLoader file. Drop the in-tree shim from step 3 once the |
| 227 | + contrib module's extension is discovered at load time. |
| 228 | +6. **Move the regression test driver** and validate the full Delta regression still passes |
| 229 | + with the contrib jar on the test classpath. |
| 230 | + |
| 231 | +All six steps are in one branch. The user reviews the single PR. |
| 232 | + |
| 233 | +## 7. Proto handling (single-PR shape) |
| 234 | + |
| 235 | +Per user decision: keep the existing `DeltaScan` / `DeltaScanCommon` / `DeltaScanTask` / |
| 236 | +`DeltaScanTaskList` proto messages exactly as they are. |
| 237 | + |
| 238 | +- The `.proto` files stay in `native/proto/src/proto/operator.proto`. |
| 239 | +- The generated Rust types stay in the `datafusion-comet-proto` crate (already a top-level |
| 240 | + Cargo member, format-agnostic). |
| 241 | +- The generated Java/Scala types stay in `spark`'s build output. |
| 242 | +- Only the consumers move: contrib's Rust crate imports the generated types from the proto |
| 243 | + crate; contrib's Scala code does likewise. |
| 244 | + |
| 245 | +No deprecation window needed; no backwards-compat shims. If we later want to generalize to |
| 246 | +a `ContribScan` envelope for future Hudi/DeltaSharing/etc. contribs we can do it as a |
| 247 | +focused follow-up — the SPI is already shaped for that. |
| 248 | + |
| 249 | +## 8. Risks / open questions to confirm with the user before starting |
| 250 | + |
| 251 | +- **Shared-static across DSOs (§5, JNI loading).** Rust statics live per-dylib. Cross-dylib |
| 252 | + registration depends on contrib linking against core as a `dylib` (not a static copy). |
| 253 | + If that turns out to require deeper toolchain work than expected, the fallback is a |
| 254 | + JNI-mediated registration (contrib calls back through Java which calls forward into |
| 255 | + core's JNI). Confirm we're OK with this risk profile before kickoff. If not, we ship the |
| 256 | + JNI-mediated path from day one. |
| 257 | +- **Spark Delta version skew.** The contrib JAR currently targets Delta 3.3.2 (Spark 3.5). |
| 258 | + Confirm whether contrib should also produce Spark 3.4 / Delta 2.4 and Spark 4.0 / |
| 259 | + Delta 4.0 variants in this PR or whether those land in follow-ups. (`dev/run-delta-regression.sh` |
| 260 | + already conditionally drives all three.) |
| 261 | +- **`META-INF/native/` library size.** Adding a second cdylib roughly doubles the native |
| 262 | + size of the contrib JAR vs core. Confirm acceptable, or whether contrib should ship one |
| 263 | + cdylib per supported OS/arch combo separately (Linux x86_64, Linux aarch64, macOS x86_64, |
| 264 | + macOS aarch64, Windows x86_64). |
0 commit comments