Skip to content

Commit a2ac715

Browse files
schenksjclaude
andcommitted
docs(contrib): rewrite PR1-description.md to reflect current architecture
The PR description had drifted significantly from the actual branch: - Said "9 commits"; the branch is now 18. - Claimed JVM contribs ship as separate Maven JARs and users opt in via classpath; the symmetric-distribution refactor inverted that. - Said the registry uses RwLock; it now uses ArcSwap. - Missing all the post-`e018076d` work: ContribPlannerContext, four review-fix rounds, the contributor-guide completeness pass, the symmetric-distribution pivot, the deps-pom pattern. - The architecture diagram still showed ServiceLoader-from-separate-JAR. - The Delta-port confidence check section listed three SPI gaps closed; the fourth (Gap #4: ContribPlannerContext) was added later and was missing. The new description accurately enumerates the 18 commits, walks through the current distribution model, and explains the design decisions (single-artifact bundling, ArcSwap registry, source-injection over JAR-shading, deps-only pom per contrib) that emerged across the iterations. It also documents the four review passes and the doc-completeness validation as part of the audit trail. Not committed: this file is a working artifact, used when manually opening the PR. The previous version was a draft; the current is review-ready. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent c7656fc commit a2ac715

1 file changed

Lines changed: 292 additions & 0 deletions

File tree

‎PR1-description.md‎

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
# feat(contrib): introduce contrib extension SPI
2+
3+
Branch: `comet-contrib-spi`
4+
Base: `main`
5+
Commits: 18 (one architectural cluster + four review-fix rounds + a doc-completeness pass + the symmetric-distribution refactor)
6+
7+
## Summary
8+
9+
Adds the infrastructure for *contrib extensions* — self-contained modules that ship as
10+
part of Comet's release but plug in via a stable SPI rather than being hard-wired into
11+
core. Core gains no functional behaviour change; with no contrib feature/profile
12+
enabled, the dispatch hooks are no-ops and `registered_contrib_kinds()` is empty.
13+
14+
The first concrete contrib (`contrib/example/`) is a worked reference: Scala extension
15+
classes, a `CometScanRuleExtension` implementation, a Rust rlib registering a
16+
`ContribOperatorPlanner`, a `META-INF/services/` entry, its own `.proto` schema with
17+
prost-build wiring, and unit tests covering registration, proto decode round-trip, and
18+
the error path. New contrib authors copy this directory layout.
19+
20+
This is the first of a two-PR sequence. **PR2** (a follow-up off `main` after this
21+
lands) will port Comet's existing delta-kernel-rs integration onto this SPI and into
22+
`contrib/delta/`. The SPI shape has been validated end-to-end against a real Delta port
23+
on a separate branch — see the testing section below.
24+
25+
## Distribution model
26+
27+
Both halves of a contrib are bundled into Comet's released artifacts at build time when
28+
their matching flags are enabled. Contribs are not independently distributable — they
29+
ship inside Comet's release.
30+
31+
- **JVM half** — Scala/Java sources under `contrib/<name>/src/main/scala/`, compiled
32+
INTO `comet-spark.jar` by activating `-Pcontrib-<name>` on `spark/pom.xml`. The
33+
contrib's `META-INF/services/` entries go along for the ride; ServiceLoader at
34+
runtime discovers them from inside `comet-spark.jar` itself. The contrib has a tiny
35+
`<packaging>pom</packaging>` Maven pom that exists solely to enumerate external
36+
deps (e.g., a Delta contrib's pom would carry `<dependency>io.delta:delta-spark</dependency>`).
37+
- **Native half** — a Rust `rlib` crate (NOT `cdylib`) linked INTO `libcomet` via the
38+
matching `--features contrib-<name>` Cargo flag on `native/core`. The contrib's
39+
`#[ctor]` registers its operator planners during library load.
40+
41+
`mvn install -Pcontrib-example && cargo build --features contrib-example` produces a
42+
Comet build that includes the example contrib in both `comet-spark.jar` and `libcomet`.
43+
A vanilla `mvn install && cargo build` produces a build with zero contrib surface.
44+
45+
The wire format between JVM and native uses a single generic envelope on the operator
46+
proto, `ContribOp { kind, payload }`. Core's planner dispatches by `kind`; the contrib's
47+
native crate registers planners against the same `kind` string the contrib's JVM code
48+
writes into the proto.
49+
50+
## Architecture
51+
52+
```
53+
JVM Native (single libcomet)
54+
55+
CometSparkSessionExtensions.apply() #[cfg(feature = "contrib-example")]
56+
│ extern crate comet_contrib_example;
57+
├─ injectQueryStagePrepRule( │
58+
│ CometScanRule._apply ▼
59+
│ ├─ CometExtensionRegistry.load() contrib's #[ctor] runs at lib load
60+
│ │ (lazy, first invocation) │
61+
│ │ discovers META-INF/services ▼
62+
│ │ inside comet-spark.jar comet_contrib_spi::registry (ArcSwap)
63+
│ │ { "delta-scan" → DeltaPlanner,
64+
│ ├─ preTransform fold pass "example-no-op" → NoOpPlanner,
65+
│ │ (V1 only, gated on "example-constant-scan"→ ConstantScanPlanner }
66+
│ │ COMET_NATIVE_SCAN_ENABLED) │
67+
│ │ │
68+
│ └─ per-scan dispatch: core's `OpStruct::ContribOp` arm:
69+
│ iterate registered exts lookup_contrib_planner_by_kind(co.kind)
70+
│ first match wins .plan(ctx, co.payload, children)
71+
│ │
72+
└─ injectQueryStagePrepRule( ctx: ContribPlannerContext, gives the
73+
CometExecRule._apply contrib core's planner services:
74+
├─ CometExtensionRegistry.load() - session_ctx
75+
│ (lazy mirror of above) - build_physical_expr
76+
│ - convert_spark_schema
77+
└─ (allExecs ∪ - prepare_object_store
78+
CometExtensionRegistry - build_parquet_datasource_exec
79+
.mergedSerdes)
80+
.get(op.getClass)
81+
)
82+
───── wire format ─────
83+
OpStruct.contrib_op {
84+
kind: "delta-scan",
85+
payload: <contrib-private bytes,
86+
decoded by the contrib's own prost::Message>,
87+
reserved 3 to 9;
88+
}
89+
```
90+
91+
## What's in this PR (18 commits)
92+
93+
### Foundational SPI (5 commits)
94+
95+
| Commit | What |
96+
|---|---|
97+
| `51eb0fff` | `ContribOp { kind, payload }` proto envelope; Rust planner registry SPI |
98+
| `f448693b` | Native `OpStruct::ContribOp` dispatcher arm in `planner.rs` |
99+
| `f23500df` | Scala SPI: `CometScanRuleExtension`, `CometOperatorSerdeExtension`, `CometExtensionRegistry` |
100+
| `42234b96` | `CometScanRule` (V1 + V2) and `CometExecRule` consult the registry |
101+
| `8b694715` | `CometSparkSessionExtensions.apply` hooks the registry |
102+
103+
### Worked-reference contrib (3 commits)
104+
105+
| Commit | What |
106+
|---|---|
107+
| `d1553b55` | Rust half of `contrib/example/`: rlib crate, `#[ctor]` registration; introduced `native/contrib-spi/` leaf crate to break a cyclic dep |
108+
| `5cb7099a` | JVM half of `contrib/example/`: Scala extension, ServiceLoader entry, integration test |
109+
| `8508ec50` | First version of the contributor guide |
110+
111+
### SPI shape refinements (2 commits)
112+
113+
| Commit | What |
114+
|---|---|
115+
| `e018076d` | Refinements from a Delta-port confidence check: `preTransform` tree-level hook on `CometScanRuleExtension`, proto layer in `contrib/example/`, class-keyed dispatch convention documented |
116+
| `14e49448` | `ContribPlannerContext` trait + `ParquetDatasourceParams` argument bundle (SPI gap #4 — see "Delta-port confidence check" below) |
117+
118+
### Review-fix rounds (4 commits)
119+
120+
| Commit | What |
121+
|---|---|
122+
| `8930b698` | First review pass: test isolation, V2-asymmetry doc, `#[non_exhaustive]` markers, `preTransform` corruption guard, `#[ctor]` panic safety, drop `contrib-example` from default features, gate registry load lazily, cache mergedSerdes, multi-extension dispatch semantics, `prepare_object_store` returns `Path`, gate preTransform on `COMET_NATIVE_SCAN_ENABLED`, 16 MiB payload cap, "none discovered" diagnostic |
123+
| `68fff43f` | Second pass: `CometExecRule` self-loads, stale docstring, dead doc refs, `Display` wildcard fix, corruption-guard rewrite (identity check, class-changing replacements caught), `ContribOp` size guard ordering, encryption-asymmetry positional-arg test, owned `String` for session_timezone, production-canary `#[cfg]` test |
124+
| `e4e6e6c6` | Third pass: `IdentityHashMap` survivors set, `synchronized` load() publication order, empty/whitespace kind rejection, doc accuracy, scope notes |
125+
| `6652963c` | Fourth pass: identity survivors + cost comment, `resetForTesting` synchronized, doc trims, whitespace kind rejection, `#[cfg(not(any(...)))]` form, public `resetForTesting`, wildcard-arm comment, `Display` debug repr, `ContribOp` proto `reserved` block, payload cap doc |
126+
127+
### Contributor guide completeness (2 commits)
128+
129+
| Commit | What |
130+
|---|---|
131+
| `91c40e0a` | Comprehensive rewrite: prerequisites, JVM-side proto guidance, full `plan()` body walkthrough using every `ContribPlannerContext` method, `CometOperatorSerde[T]` contract, diagnostics story, multi-Spark-version, end-to-end testing recipe, Cargo-canary maintenance note |
132+
| `2c46552c` | Second-pass review fixes |
133+
134+
### Architectural pivot to symmetric distribution (2 commits)
135+
136+
| Commit | What |
137+
|---|---|
138+
| `cf5253ed` | Bundle JVM half INTO `comet-spark.jar` via Maven profile. Mirrors the native side's "Cargo feature pulls rlib into libcomet" model. No more separate contrib JARs. ~70 lines of protobuf-shading boilerplate deleted from the contributor guide (shading now handled automatically by `comet-spark`'s existing shade execution). |
139+
| `c7656fcc` | Deps-only pom per contrib so contribs like Delta can pull in external Maven deps (e.g. `delta-spark`) without recreating a Maven reactor cycle. Registry primitive: `RwLock<HashMap>` → `ArcSwap<HashMap>` for lock-free reads on the dispatch hot path. |
140+
141+
## Notable design decisions
142+
143+
### Why bundle into one artifact?
144+
145+
Previously the contrib's JVM half shipped as a separate Maven JAR users would
146+
`--packages` or `--jars` onto their classpath. That asymmetry made no sense given the
147+
native side already requires a Comet rebuild (Cargo feature flag) for the contrib to
148+
work — pretending the JVM half was distributable independently was a fiction. The
149+
symmetric design has one artifact per side, both varying based on which contribs were
150+
enabled at Comet build time. This eliminated the protobuf-shading recipe that
151+
externally-published contrib JARs needed (which was the single biggest source of doc
152+
complexity).
153+
154+
### Why a separate `comet-contrib-spi` Rust crate?
155+
156+
Cycle break. Core would need to depend on contribs (to link them); contribs need core's
157+
trait types (`ContribOperatorPlanner`). Solution: a leaf crate both depend on, with
158+
nothing depending back on core from a contrib.
159+
160+
### Why `ContribPlannerContext`?
161+
162+
Surfaced by the Delta-port confidence check. A real file-scan contrib needs five
163+
core-side facilities: `init_datasource_exec`, `prepare_object_store_with_configs`,
164+
`convert_spark_types_to_arrow_schema`, expression-planning (`create_expr`), and a
165+
`SessionContext` handle. Exposing these as a trait core implements (and contribs use
166+
via `&dyn ContribPlannerContext`) avoids a back-dep on core while giving contribs
167+
everything they need to compose with Comet's tuned parquet path.
168+
169+
### Why `ArcSwap` instead of `RwLock` for the registry?
170+
171+
Reads are on the dispatch hot path; writes happen exclusively during library init from
172+
sequential `#[ctor]`s. The init-once / read-many access pattern is what `ArcSwap` is
173+
designed for. The original `RwLock<HashMap>` would have introduced reader-writer
174+
contention with no actual concurrent-write workload to justify it.
175+
176+
### Why bundling-via-source-injection rather than bundling-via-shaded-JAR?
177+
178+
A separate Maven module per contrib whose JAR gets shaded into `comet-spark.jar`
179+
would form a Maven reactor cycle (contrib's pom depends on `comet-spark` for SPI
180+
types; `comet-spark`'s contrib profile depends on contrib's JAR). Source-injection
181+
avoids the cycle: the contrib's Scala sources are compiled INSIDE `comet-spark`'s
182+
own compilation pass (via `build-helper-maven-plugin`'s `add-source` goal); no
183+
separate compile, no per-contrib JAR, no cycle. External Maven deps (`delta-spark`,
184+
etc.) flow through the contrib's separate `<packaging>pom</packaging>` artifact.
185+
186+
## The Delta-port confidence check
187+
188+
Before opening this PR, I ported Comet's existing Delta integration (~3,200 lines on
189+
the `delta-kernel-phase-1` branch) onto this SPI as a confidence check. The port
190+
itself is not committed here — its purpose was to surface SPI gaps before review.
191+
192+
Four gaps were found, all addressed in this PR:
193+
194+
1. **No tree-level pre-pass hook** → added `CometScanRuleExtension.preTransform`.
195+
2. **No reference for the proto layer in `contrib/example/`** → added a trivial
196+
`ExampleConstantScan` message, `build.rs`, prost-build wiring, and tests.
197+
3. **Class-keyed serde dispatch convention undocumented** → documented in the
198+
contributor guide.
199+
4. **`ContribOperatorPlanner::plan` lacked access to core's parquet / expression
200+
machinery** → introduced `ContribPlannerContext` trait + `ParquetDatasourceParams`
201+
bundle in commit `14e49448`. The full ~150-line Delta dispatcher body compiled
202+
clean against the new SPI surface; every trait method was exercised end-to-end.
203+
204+
Full findings live in `PR1-delta-port-findings.md` (not committed; review-prep
205+
artifact).
206+
207+
Net conclusion: the SPI is the right shape for a real consumer. PR2's Delta port can
208+
proceed mechanically with no further SPI surprises expected.
209+
210+
## Review iterations
211+
212+
The branch went through four independent clean-context code review passes (general-purpose
213+
subagent reviews launched fresh on each iteration's HEAD). Each pass surfaced a different
214+
class of issue:
215+
216+
- **Pass 1** (review of `14e49448`, fixed by `8930b698`): 6 blockers + 10 important +
217+
nits. Test isolation, SemVer markers, V2 asymmetry, `#[ctor]` panic safety, default
218+
feature leakage, ServiceLoader gating, more.
219+
- **Pass 2** (review of `8930b698`, fixed by `68fff43f`): 3 regressions + 4 polish
220+
items + 10 new findings. Stale class docstring, dead doc refs, `Display`
221+
info-loss, corruption-guard class-change bypass.
222+
- **Pass 3** (review of `68fff43f`, fixed by `e4e6e6c6`): 2 regressions + 8 polish.
223+
Survivors-set false positive, `load()` publication race.
224+
- **Pass 4** (review of `e4e6e6c6`, fixed by `6652963c`): no blockers found; 6 polish.
225+
Cost-comment accuracy, identity-map upgrade, payload-guard message ordering.
226+
227+
A separate completeness validation of the contributor guide (the doc-only audit pass)
228+
identified ~13 gaps a real contrib author would hit. Closed in `91c40e0a` and
229+
`2c46552c`.
230+
231+
## Build verification
232+
233+
- `cargo check` (default features): green.
234+
- `cargo check --no-default-features`: green; zero contrib surface in the resulting
235+
`libcomet`.
236+
- `cargo test -p comet-contrib-spi`: 7 tests pass (registry round-trip, scoped
237+
registration, kinds snapshot, params constructor/setters, `ContribError` Display
238+
preservation).
239+
- `cargo test -p datafusion-comet --lib -- execution::planner::contrib`: 5 tests pass,
240+
including the production-canary that asserts default cdylib has no registered
241+
contribs and the encryption-asymmetry test that catches positional-arg swaps in
242+
`init_datasource_exec`.
243+
- `cargo test -p comet-contrib-example`: 4 tests pass (ctor registration, decode +
244+
build, zero rows, bad payload).
245+
- Maven `-Pcontrib-example` activates the profile cleanly; `comet-contrib-example-deps`
246+
builds first, then `comet-spark` with the contrib's sources injected. No reactor
247+
cycle.
248+
249+
## What this PR is NOT
250+
251+
- It does NOT migrate Delta / kernel-rs to the new SPI. That's PR2.
252+
- It does NOT exercise `CometOperatorSerdeExtension` from the example contrib (the
253+
example only demonstrates `CometScanRuleExtension`; the trait surface is documented
254+
and validated against the Delta-port confidence check).
255+
- The Maven `BanDuplicateClasses` enforcer is no longer overridden per-contrib
256+
(contribs are no longer separate Maven modules); the rule applies to `comet-spark`
257+
itself as before.
258+
259+
## How to review
260+
261+
Suggested reading order:
262+
263+
1. `docs/source/contributor-guide/contrib-extensions.md` — author-facing guide; doubles
264+
as the architectural overview.
265+
2. `native/proto/src/proto/operator.proto` — the `ContribOp` envelope (small, look for
266+
the new variant on `OpStruct`).
267+
3. `native/contrib-spi/src/lib.rs` — the leaf SPI crate (~370 lines incl. tests).
268+
4. `spark/src/main/scala/org/apache/comet/spi/` — three small files defining the JVM
269+
SPI.
270+
5. `native/core/src/execution/planner.rs` — the `OpStruct::ContribOp` dispatcher arm
271+
(~lines 1960–2020).
272+
6. `native/core/src/execution/planner/contrib.rs` — `CorePlannerContext` adapter that
273+
exposes core's parquet/expression infrastructure to contribs through the SPI trait.
274+
7. `spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala` and
275+
`CometExecRule.scala` — the integration hooks. Each is a small insertion at the
276+
top of `_apply`; the `preTransform` fold runs once per plan.
277+
8. `contrib/example/` — the worked reference (deps-pom + Scala source + Cargo crate).
278+
9. `spark/pom.xml`'s `contrib-example` profile — the template for wiring a new
279+
contrib into the build.
280+
281+
## Risks / follow-ups (tracked for PR2)
282+
283+
- **`CometOperatorSerdeExtension` not yet exercised by a contrib.** The example
284+
contrib only demonstrates `CometScanRuleExtension`. PR2's Delta port will exercise
285+
the operator serde path via `CometDeltaNativeScanExec`'s dedicated class.
286+
- **Native test runner needs `libjvm` on the dyld path.** Running
287+
`cargo test -p datafusion-comet --lib` on macOS requires
288+
`DYLD_LIBRARY_PATH=$JAVA_HOME/lib/server` (only relevant when the test binary
289+
transitively links against the JNI crate). Preexisting on `main` — not introduced
290+
by this PR — but worth documenting.
291+
- **CI matrix should add a `-Pcontrib-example,--features contrib-example` row** so the
292+
bundling path is exercised in CI on every PR. Today only the slim build is in CI.

0 commit comments

Comments
 (0)