|
| 1 | +# feat(contrib): introduce contrib extension SPI |
| 2 | + |
| 3 | +Branch: `comet-contrib-spi` |
| 4 | +Base: `main` |
| 5 | +Commits: 18 (one architectural cluster + four review-fix rounds + a doc-completeness pass + the symmetric-distribution refactor) |
| 6 | + |
| 7 | +## Summary |
| 8 | + |
| 9 | +Adds the infrastructure for *contrib extensions* — self-contained modules that ship as |
| 10 | +part of Comet's release but plug in via a stable SPI rather than being hard-wired into |
| 11 | +core. Core gains no functional behaviour change; with no contrib feature/profile |
| 12 | +enabled, the dispatch hooks are no-ops and `registered_contrib_kinds()` is empty. |
| 13 | + |
| 14 | +The first concrete contrib (`contrib/example/`) is a worked reference: Scala extension |
| 15 | +classes, a `CometScanRuleExtension` implementation, a Rust rlib registering a |
| 16 | +`ContribOperatorPlanner`, a `META-INF/services/` entry, its own `.proto` schema with |
| 17 | +prost-build wiring, and unit tests covering registration, proto decode round-trip, and |
| 18 | +the error path. New contrib authors copy this directory layout. |
| 19 | + |
| 20 | +This is the first of a two-PR sequence. **PR2** (a follow-up off `main` after this |
| 21 | +lands) will port Comet's existing delta-kernel-rs integration onto this SPI and into |
| 22 | +`contrib/delta/`. The SPI shape has been validated end-to-end against a real Delta port |
| 23 | +on a separate branch — see the testing section below. |
| 24 | + |
| 25 | +## Distribution model |
| 26 | + |
| 27 | +Both halves of a contrib are bundled into Comet's released artifacts at build time when |
| 28 | +their matching flags are enabled. Contribs are not independently distributable — they |
| 29 | +ship inside Comet's release. |
| 30 | + |
| 31 | +- **JVM half** — Scala/Java sources under `contrib/<name>/src/main/scala/`, compiled |
| 32 | + INTO `comet-spark.jar` by activating `-Pcontrib-<name>` on `spark/pom.xml`. The |
| 33 | + contrib's `META-INF/services/` entries go along for the ride; ServiceLoader at |
| 34 | + runtime discovers them from inside `comet-spark.jar` itself. The contrib has a tiny |
| 35 | + `<packaging>pom</packaging>` Maven pom that exists solely to enumerate external |
| 36 | + deps (e.g., a Delta contrib's pom would carry `<dependency>io.delta:delta-spark</dependency>`). |
| 37 | +- **Native half** — a Rust `rlib` crate (NOT `cdylib`) linked INTO `libcomet` via the |
| 38 | + matching `--features contrib-<name>` Cargo flag on `native/core`. The contrib's |
| 39 | + `#[ctor]` registers its operator planners during library load. |
| 40 | + |
| 41 | +`mvn install -Pcontrib-example && cargo build --features contrib-example` produces a |
| 42 | +Comet build that includes the example contrib in both `comet-spark.jar` and `libcomet`. |
| 43 | +A vanilla `mvn install && cargo build` produces a build with zero contrib surface. |
| 44 | + |
| 45 | +The wire format between JVM and native uses a single generic envelope on the operator |
| 46 | +proto, `ContribOp { kind, payload }`. Core's planner dispatches by `kind`; the contrib's |
| 47 | +native crate registers planners against the same `kind` string the contrib's JVM code |
| 48 | +writes into the proto. |
| 49 | + |
| 50 | +## Architecture |
| 51 | + |
| 52 | +``` |
| 53 | + JVM Native (single libcomet) |
| 54 | +
|
| 55 | + CometSparkSessionExtensions.apply() #[cfg(feature = "contrib-example")] |
| 56 | + │ extern crate comet_contrib_example; |
| 57 | + ├─ injectQueryStagePrepRule( │ |
| 58 | + │ CometScanRule._apply ▼ |
| 59 | + │ ├─ CometExtensionRegistry.load() contrib's #[ctor] runs at lib load |
| 60 | + │ │ (lazy, first invocation) │ |
| 61 | + │ │ discovers META-INF/services ▼ |
| 62 | + │ │ inside comet-spark.jar comet_contrib_spi::registry (ArcSwap) |
| 63 | + │ │ { "delta-scan" → DeltaPlanner, |
| 64 | + │ ├─ preTransform fold pass "example-no-op" → NoOpPlanner, |
| 65 | + │ │ (V1 only, gated on "example-constant-scan"→ ConstantScanPlanner } |
| 66 | + │ │ COMET_NATIVE_SCAN_ENABLED) │ |
| 67 | + │ │ │ |
| 68 | + │ └─ per-scan dispatch: core's `OpStruct::ContribOp` arm: |
| 69 | + │ iterate registered exts lookup_contrib_planner_by_kind(co.kind) |
| 70 | + │ first match wins .plan(ctx, co.payload, children) |
| 71 | + │ │ |
| 72 | + └─ injectQueryStagePrepRule( ctx: ContribPlannerContext, gives the |
| 73 | + CometExecRule._apply contrib core's planner services: |
| 74 | + ├─ CometExtensionRegistry.load() - session_ctx |
| 75 | + │ (lazy mirror of above) - build_physical_expr |
| 76 | + │ - convert_spark_schema |
| 77 | + └─ (allExecs ∪ - prepare_object_store |
| 78 | + CometExtensionRegistry - build_parquet_datasource_exec |
| 79 | + .mergedSerdes) |
| 80 | + .get(op.getClass) |
| 81 | + ) |
| 82 | + ───── wire format ───── |
| 83 | + OpStruct.contrib_op { |
| 84 | + kind: "delta-scan", |
| 85 | + payload: <contrib-private bytes, |
| 86 | + decoded by the contrib's own prost::Message>, |
| 87 | + reserved 3 to 9; |
| 88 | + } |
| 89 | +``` |
| 90 | + |
| 91 | +## What's in this PR (18 commits) |
| 92 | + |
| 93 | +### Foundational SPI (5 commits) |
| 94 | + |
| 95 | +| Commit | What | |
| 96 | +|---|---| |
| 97 | +| `51eb0fff` | `ContribOp { kind, payload }` proto envelope; Rust planner registry SPI | |
| 98 | +| `f448693b` | Native `OpStruct::ContribOp` dispatcher arm in `planner.rs` | |
| 99 | +| `f23500df` | Scala SPI: `CometScanRuleExtension`, `CometOperatorSerdeExtension`, `CometExtensionRegistry` | |
| 100 | +| `42234b96` | `CometScanRule` (V1 + V2) and `CometExecRule` consult the registry | |
| 101 | +| `8b694715` | `CometSparkSessionExtensions.apply` hooks the registry | |
| 102 | + |
| 103 | +### Worked-reference contrib (3 commits) |
| 104 | + |
| 105 | +| Commit | What | |
| 106 | +|---|---| |
| 107 | +| `d1553b55` | Rust half of `contrib/example/`: rlib crate, `#[ctor]` registration; introduced `native/contrib-spi/` leaf crate to break a cyclic dep | |
| 108 | +| `5cb7099a` | JVM half of `contrib/example/`: Scala extension, ServiceLoader entry, integration test | |
| 109 | +| `8508ec50` | First version of the contributor guide | |
| 110 | + |
| 111 | +### SPI shape refinements (2 commits) |
| 112 | + |
| 113 | +| Commit | What | |
| 114 | +|---|---| |
| 115 | +| `e018076d` | Refinements from a Delta-port confidence check: `preTransform` tree-level hook on `CometScanRuleExtension`, proto layer in `contrib/example/`, class-keyed dispatch convention documented | |
| 116 | +| `14e49448` | `ContribPlannerContext` trait + `ParquetDatasourceParams` argument bundle (SPI gap #4 — see "Delta-port confidence check" below) | |
| 117 | + |
| 118 | +### Review-fix rounds (4 commits) |
| 119 | + |
| 120 | +| Commit | What | |
| 121 | +|---|---| |
| 122 | +| `8930b698` | First review pass: test isolation, V2-asymmetry doc, `#[non_exhaustive]` markers, `preTransform` corruption guard, `#[ctor]` panic safety, drop `contrib-example` from default features, gate registry load lazily, cache mergedSerdes, multi-extension dispatch semantics, `prepare_object_store` returns `Path`, gate preTransform on `COMET_NATIVE_SCAN_ENABLED`, 16 MiB payload cap, "none discovered" diagnostic | |
| 123 | +| `68fff43f` | Second pass: `CometExecRule` self-loads, stale docstring, dead doc refs, `Display` wildcard fix, corruption-guard rewrite (identity check, class-changing replacements caught), `ContribOp` size guard ordering, encryption-asymmetry positional-arg test, owned `String` for session_timezone, production-canary `#[cfg]` test | |
| 124 | +| `e4e6e6c6` | Third pass: `IdentityHashMap` survivors set, `synchronized` load() publication order, empty/whitespace kind rejection, doc accuracy, scope notes | |
| 125 | +| `6652963c` | Fourth pass: identity survivors + cost comment, `resetForTesting` synchronized, doc trims, whitespace kind rejection, `#[cfg(not(any(...)))]` form, public `resetForTesting`, wildcard-arm comment, `Display` debug repr, `ContribOp` proto `reserved` block, payload cap doc | |
| 126 | + |
| 127 | +### Contributor guide completeness (2 commits) |
| 128 | + |
| 129 | +| Commit | What | |
| 130 | +|---|---| |
| 131 | +| `91c40e0a` | Comprehensive rewrite: prerequisites, JVM-side proto guidance, full `plan()` body walkthrough using every `ContribPlannerContext` method, `CometOperatorSerde[T]` contract, diagnostics story, multi-Spark-version, end-to-end testing recipe, Cargo-canary maintenance note | |
| 132 | +| `2c46552c` | Second-pass review fixes | |
| 133 | + |
| 134 | +### Architectural pivot to symmetric distribution (2 commits) |
| 135 | + |
| 136 | +| Commit | What | |
| 137 | +|---|---| |
| 138 | +| `cf5253ed` | Bundle JVM half INTO `comet-spark.jar` via Maven profile. Mirrors the native side's "Cargo feature pulls rlib into libcomet" model. No more separate contrib JARs. ~70 lines of protobuf-shading boilerplate deleted from the contributor guide (shading now handled automatically by `comet-spark`'s existing shade execution). | |
| 139 | +| `c7656fcc` | Deps-only pom per contrib so contribs like Delta can pull in external Maven deps (e.g. `delta-spark`) without recreating a Maven reactor cycle. Registry primitive: `RwLock<HashMap>` → `ArcSwap<HashMap>` for lock-free reads on the dispatch hot path. | |
| 140 | + |
| 141 | +## Notable design decisions |
| 142 | + |
| 143 | +### Why bundle into one artifact? |
| 144 | + |
| 145 | +Previously the contrib's JVM half shipped as a separate Maven JAR users would |
| 146 | +`--packages` or `--jars` onto their classpath. That asymmetry made no sense given the |
| 147 | +native side already requires a Comet rebuild (Cargo feature flag) for the contrib to |
| 148 | +work — pretending the JVM half was distributable independently was a fiction. The |
| 149 | +symmetric design has one artifact per side, both varying based on which contribs were |
| 150 | +enabled at Comet build time. This eliminated the protobuf-shading recipe that |
| 151 | +externally-published contrib JARs needed (which was the single biggest source of doc |
| 152 | +complexity). |
| 153 | + |
| 154 | +### Why a separate `comet-contrib-spi` Rust crate? |
| 155 | + |
| 156 | +Cycle break. Core would need to depend on contribs (to link them); contribs need core's |
| 157 | +trait types (`ContribOperatorPlanner`). Solution: a leaf crate both depend on, with |
| 158 | +nothing depending back on core from a contrib. |
| 159 | + |
| 160 | +### Why `ContribPlannerContext`? |
| 161 | + |
| 162 | +Surfaced by the Delta-port confidence check. A real file-scan contrib needs five |
| 163 | +core-side facilities: `init_datasource_exec`, `prepare_object_store_with_configs`, |
| 164 | +`convert_spark_types_to_arrow_schema`, expression-planning (`create_expr`), and a |
| 165 | +`SessionContext` handle. Exposing these as a trait core implements (and contribs use |
| 166 | +via `&dyn ContribPlannerContext`) avoids a back-dep on core while giving contribs |
| 167 | +everything they need to compose with Comet's tuned parquet path. |
| 168 | + |
| 169 | +### Why `ArcSwap` instead of `RwLock` for the registry? |
| 170 | + |
| 171 | +Reads are on the dispatch hot path; writes happen exclusively during library init from |
| 172 | +sequential `#[ctor]`s. The init-once / read-many access pattern is what `ArcSwap` is |
| 173 | +designed for. The original `RwLock<HashMap>` would have introduced reader-writer |
| 174 | +contention with no actual concurrent-write workload to justify it. |
| 175 | + |
| 176 | +### Why bundling-via-source-injection rather than bundling-via-shaded-JAR? |
| 177 | + |
| 178 | +A separate Maven module per contrib whose JAR gets shaded into `comet-spark.jar` |
| 179 | +would form a Maven reactor cycle (contrib's pom depends on `comet-spark` for SPI |
| 180 | +types; `comet-spark`'s contrib profile depends on contrib's JAR). Source-injection |
| 181 | +avoids the cycle: the contrib's Scala sources are compiled INSIDE `comet-spark`'s |
| 182 | +own compilation pass (via `build-helper-maven-plugin`'s `add-source` goal); no |
| 183 | +separate compile, no per-contrib JAR, no cycle. External Maven deps (`delta-spark`, |
| 184 | +etc.) flow through the contrib's separate `<packaging>pom</packaging>` artifact. |
| 185 | + |
| 186 | +## The Delta-port confidence check |
| 187 | + |
| 188 | +Before opening this PR, I ported Comet's existing Delta integration (~3,200 lines on |
| 189 | +the `delta-kernel-phase-1` branch) onto this SPI as a confidence check. The port |
| 190 | +itself is not committed here — its purpose was to surface SPI gaps before review. |
| 191 | + |
| 192 | +Four gaps were found, all addressed in this PR: |
| 193 | + |
| 194 | +1. **No tree-level pre-pass hook** → added `CometScanRuleExtension.preTransform`. |
| 195 | +2. **No reference for the proto layer in `contrib/example/`** → added a trivial |
| 196 | + `ExampleConstantScan` message, `build.rs`, prost-build wiring, and tests. |
| 197 | +3. **Class-keyed serde dispatch convention undocumented** → documented in the |
| 198 | + contributor guide. |
| 199 | +4. **`ContribOperatorPlanner::plan` lacked access to core's parquet / expression |
| 200 | + machinery** → introduced `ContribPlannerContext` trait + `ParquetDatasourceParams` |
| 201 | + bundle in commit `14e49448`. The full ~150-line Delta dispatcher body compiled |
| 202 | + clean against the new SPI surface; every trait method was exercised end-to-end. |
| 203 | + |
| 204 | +Full findings live in `PR1-delta-port-findings.md` (not committed; review-prep |
| 205 | +artifact). |
| 206 | + |
| 207 | +Net conclusion: the SPI is the right shape for a real consumer. PR2's Delta port can |
| 208 | +proceed mechanically with no further SPI surprises expected. |
| 209 | + |
| 210 | +## Review iterations |
| 211 | + |
| 212 | +The branch went through four independent clean-context code review passes (general-purpose |
| 213 | +subagent reviews launched fresh on each iteration's HEAD). Each pass surfaced a different |
| 214 | +class of issue: |
| 215 | + |
| 216 | +- **Pass 1** (review of `14e49448`, fixed by `8930b698`): 6 blockers + 10 important + |
| 217 | + nits. Test isolation, SemVer markers, V2 asymmetry, `#[ctor]` panic safety, default |
| 218 | + feature leakage, ServiceLoader gating, more. |
| 219 | +- **Pass 2** (review of `8930b698`, fixed by `68fff43f`): 3 regressions + 4 polish |
| 220 | + items + 10 new findings. Stale class docstring, dead doc refs, `Display` |
| 221 | + info-loss, corruption-guard class-change bypass. |
| 222 | +- **Pass 3** (review of `68fff43f`, fixed by `e4e6e6c6`): 2 regressions + 8 polish. |
| 223 | + Survivors-set false positive, `load()` publication race. |
| 224 | +- **Pass 4** (review of `e4e6e6c6`, fixed by `6652963c`): no blockers found; 6 polish. |
| 225 | + Cost-comment accuracy, identity-map upgrade, payload-guard message ordering. |
| 226 | + |
| 227 | +A separate completeness validation of the contributor guide (the doc-only audit pass) |
| 228 | +identified ~13 gaps a real contrib author would hit. Closed in `91c40e0a` and |
| 229 | +`2c46552c`. |
| 230 | + |
| 231 | +## Build verification |
| 232 | + |
| 233 | +- `cargo check` (default features): green. |
| 234 | +- `cargo check --no-default-features`: green; zero contrib surface in the resulting |
| 235 | + `libcomet`. |
| 236 | +- `cargo test -p comet-contrib-spi`: 7 tests pass (registry round-trip, scoped |
| 237 | + registration, kinds snapshot, params constructor/setters, `ContribError` Display |
| 238 | + preservation). |
| 239 | +- `cargo test -p datafusion-comet --lib -- execution::planner::contrib`: 5 tests pass, |
| 240 | + including the production-canary that asserts default cdylib has no registered |
| 241 | + contribs and the encryption-asymmetry test that catches positional-arg swaps in |
| 242 | + `init_datasource_exec`. |
| 243 | +- `cargo test -p comet-contrib-example`: 4 tests pass (ctor registration, decode + |
| 244 | + build, zero rows, bad payload). |
| 245 | +- Maven `-Pcontrib-example` activates the profile cleanly; `comet-contrib-example-deps` |
| 246 | + builds first, then `comet-spark` with the contrib's sources injected. No reactor |
| 247 | + cycle. |
| 248 | + |
| 249 | +## What this PR is NOT |
| 250 | + |
| 251 | +- It does NOT migrate Delta / kernel-rs to the new SPI. That's PR2. |
| 252 | +- It does NOT exercise `CometOperatorSerdeExtension` from the example contrib (the |
| 253 | + example only demonstrates `CometScanRuleExtension`; the trait surface is documented |
| 254 | + and validated against the Delta-port confidence check). |
| 255 | +- The Maven `BanDuplicateClasses` enforcer is no longer overridden per-contrib |
| 256 | + (contribs are no longer separate Maven modules); the rule applies to `comet-spark` |
| 257 | + itself as before. |
| 258 | + |
| 259 | +## How to review |
| 260 | + |
| 261 | +Suggested reading order: |
| 262 | + |
| 263 | +1. `docs/source/contributor-guide/contrib-extensions.md` — author-facing guide; doubles |
| 264 | + as the architectural overview. |
| 265 | +2. `native/proto/src/proto/operator.proto` — the `ContribOp` envelope (small, look for |
| 266 | + the new variant on `OpStruct`). |
| 267 | +3. `native/contrib-spi/src/lib.rs` — the leaf SPI crate (~370 lines incl. tests). |
| 268 | +4. `spark/src/main/scala/org/apache/comet/spi/` — three small files defining the JVM |
| 269 | + SPI. |
| 270 | +5. `native/core/src/execution/planner.rs` — the `OpStruct::ContribOp` dispatcher arm |
| 271 | + (~lines 1960–2020). |
| 272 | +6. `native/core/src/execution/planner/contrib.rs` — `CorePlannerContext` adapter that |
| 273 | + exposes core's parquet/expression infrastructure to contribs through the SPI trait. |
| 274 | +7. `spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala` and |
| 275 | + `CometExecRule.scala` — the integration hooks. Each is a small insertion at the |
| 276 | + top of `_apply`; the `preTransform` fold runs once per plan. |
| 277 | +8. `contrib/example/` — the worked reference (deps-pom + Scala source + Cargo crate). |
| 278 | +9. `spark/pom.xml`'s `contrib-example` profile — the template for wiring a new |
| 279 | + contrib into the build. |
| 280 | + |
| 281 | +## Risks / follow-ups (tracked for PR2) |
| 282 | + |
| 283 | +- **`CometOperatorSerdeExtension` not yet exercised by a contrib.** The example |
| 284 | + contrib only demonstrates `CometScanRuleExtension`. PR2's Delta port will exercise |
| 285 | + the operator serde path via `CometDeltaNativeScanExec`'s dedicated class. |
| 286 | +- **Native test runner needs `libjvm` on the dyld path.** Running |
| 287 | + `cargo test -p datafusion-comet --lib` on macOS requires |
| 288 | + `DYLD_LIBRARY_PATH=$JAVA_HOME/lib/server` (only relevant when the test binary |
| 289 | + transitively links against the JNI crate). Preexisting on `main` — not introduced |
| 290 | + by this PR — but worth documenting. |
| 291 | +- **CI matrix should add a `-Pcontrib-example,--features contrib-example` row** so the |
| 292 | + bundling path is exercised in CI on every PR. Today only the slim build is in CI. |
0 commit comments