Repository navigation
feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] - #4
Merged
Merged
Conversation
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
from
June 21, 2026 14:14
f2ad00c to
e020fe6
Compare
schenksj
added a commit
that referenced
this pull request
Jun 22, 2026
…flaky re-run) + apache#4366 carveout links
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
from
July 17, 2026 02:13
e020fe6 to
e16970e
Compare
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
3 times, most recently
from
August 18, 2026 15:28
9b323ea to
87d5cff
Compare
schenksj
marked this pull request as ready for review
August 18, 2026 15:28
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
3 times, most recently
from
August 19, 2026 03:12
9ab285f to
0c34ec2
Compare
…b split, part 2]
Part 2 of the Delta contrib split: the build gate and the inert core wiring an
out-of-tree scan contrib plugs into. Nothing here is reachable on a default
build -- no contrib is registered, no contrib class is compiled, and the native
library carries zero contrib symbols.
Core gains two format-agnostic extension points, both discovered at runtime so
core holds no compile-time reference to any contrib:
- `CometScanContrib`, a ServiceLoader-discovered hook (mirroring
`PlanDataInjector`) that lets a contrib claim a V1 or V2 scan before
Comet's built-in handling runs, plus `CometContribScanMarker` so
`CometExecRule` can route a contrib's scan node to the contrib's own serde
handler by a plain type test.
- `ContribScan contrib_scan = 200`, a single permanent `Any`-shaped proto
envelope (`type_url` + packed `value`) dispatched by `type_url` on the
native side. Core's oneof never grows per-format, so independent contrib
PRs cannot collide on a field number -- as `main` taking field 118 for
`Sample` has since demonstrated.
Plus the build machinery: the `contrib-delta` Maven profile and Cargo feature,
and `dev/verify-contrib-delta-gate.sh`, which asserts a default build compiles
no contrib classes, packages no contrib `META-INF/services` files, and links no
contrib symbols.
Where the hooks sit, and why. Both run *before* Comet's built-in guards for
their scan kind, because a contrib may support things the built-in scan does
not -- the Delta contrib synthesises `_metadata.*` in its own reader, and a
contrib's table name may end in `files`/`snapshots` like an Iceberg metadata
table. Applying those guards first would decline such a scan before the contrib
was ever offered it. So `transformV1Scan` consults the contrib ahead of the
metadata-column guard, and the Iceberg metadata-table check moves out of the
outer `transformScan` match into `transformV2Scan`, after its hook. Core's
per-path metadata handling is otherwise untouched: `main` serves
`fileConstantMetadataColumns` natively in V1 and the Iceberg metadata columns
in V2, and both keep doing so.
Ownership contract. An implementation MUST return `None` for a scan it does not
own: contribs are offered a scan one at a time and the first claim wins, so a
contrib claiming another format's scan hides it from the contrib that could
have read it, with the outcome depending on unspecified ServiceLoader ordering.
"Own but cannot handle" is a distinct, expressible case -- claim the scan and
terminate it with `withFallbackReason` rather than declining. Core cannot
arbitrate competing claims (a claim is opaque; the only way to know a second
contrib would also have claimed is to ask it, which is what claiming prevents),
so the contract carries it.
Tests. `CometScanContribSuite` covers the registry contract on a default build:
no contribs registered (asserted against raw ServiceLoader discovery, not just
the registry -- `contribs` swallows a ServiceConfigurationError, so "empty"
alone is ambiguous), a stub discovered through a URLClassLoader whose claim is
returned, decline-passes-through, first-claim-wins with later contribs not
consulted, throw-is-a-decline, and LinkageError still propagating.
`CometScanRuleSuite` gains a V1 case asserting the fallback *reason* for
`_metadata.row_index`; verified red with the guard removed.
Co-Authored-By: Claude Opus 5 <[email protected]>
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
from
August 19, 2026 10:59
0c34ec2 to
7f8dc65
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this part is
Build gate + inert wiring. Establishes the
contrib-deltaMaven profile / Cargo feature and the inert wiring so a gated build compiles end to end, while the DEFAULT build stays byte-for-byte behavior-unchanged and carries zero Delta surface. No real Delta read logic yet — a Delta read that reaches native returns a cleanNotImplementedand falls back to vanilla Spark.Build gate
contrib-deltaprofile with per-Sparkdelta.version(3.5→3.3.2, 4.0→4.0.0, 4.1→4.1.0) + add-source ofcontrib/delta/src. Defaultspark.versionstays 4.1.2 (the delta-spark 4.1.1 pin is a separate, deferred decision).contrib-deltafeature (optional path dep on the contrib crate);native/Cargo.tomlexcludes../contribfrom the workspace.dev/verify-contrib-delta-gate.shproves default cargo/mvn/dylib carry zero Delta surface and the gated build pulls the right deps; wired into a minimaldelta_build_gate.ymljob.Inert wiring
Delta*messages +delta_scan = 118(117 isBroadcastNestedLoopJoin).OpStruct::DeltaScandispatch arm (not-compiled-in error on default builds; feature-gated shim into the contrib), exhaustive-match arms,convert_spark_types_to_arrow_schema→pub(crate).contrib/delta/native):plan_delta_scanreturnsDataFusionError::NotImplemented— just enough to satisfy the core shim so--features contrib-deltalinks.DeltaIntegration(reflective, returnsNoneuntil the contrib classes exist), the CometExecRule Delta-marker hook (CDF hook deferred), the CometScanRule Delta delegation + metadata-col reorder, and the leafDeltaConf.Why it's safe on default builds
No
-Pcontrib-delta/ no--features contrib-delta: the dispatch arm is a not-compiled-in error, the contrib crate isn't linked, andDeltaIntegration's reflective lookups resolve to nothing. The gate-verify script asserts this (0 Delta symbols in the defaultlibcomet, zeroio.deltain the effective pom, only theDeltaIntegrationbridge class compiled).Verification
default + gated native build, clippy (both feature states), the gate-verify script, gated + default JVM compile, spotless/scalastyle, and
cargo fmt— all green.🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.