Skip to content

feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] - #4

Merged
schenksj merged 1 commit into
mainfrom
pr/delta-A2-buildgate
Aug 20, 2026
Merged

schenksj merged 1 commit into
mainfrom
pr/delta-A2-buildgate

Conversation

@schenksj

Copy link
Copy Markdown
Owner

Fork-local review draft (Delta-contrib PR split, part 2 / unit A.2). Base is pr/delta-A1-spi so the diff shows only A.2. Stacks on part 1 (apache#4700). Opens upstream once part 1 is approved/merged. Tracking umbrella: apache#4366.

What this part is

Build gate + inert wiring. Establishes the contrib-delta Maven profile / Cargo feature and the inert wiring so a gated build compiles end to end, while the DEFAULT build stays byte-for-byte behavior-unchanged and carries zero Delta surface. No real Delta read logic yet — a Delta read that reaches native returns a clean NotImplemented and falls back to vanilla Spark.

Build gate

  • Maven contrib-delta profile with per-Spark delta.version (3.5→3.3.2, 4.0→4.0.0, 4.1→4.1.0) + add-source of contrib/delta/src. Default spark.version stays 4.1.2 (the delta-spark 4.1.1 pin is a separate, deferred decision).
  • Cargo contrib-delta feature (optional path dep on the contrib crate); native/Cargo.toml excludes ../contrib from the workspace.
  • dev/verify-contrib-delta-gate.sh proves default cargo/mvn/dylib carry zero Delta surface and the gated build pulls the right deps; wired into a minimal delta_build_gate.yml job.

Inert wiring

  • Proto Delta* messages + delta_scan = 118 (117 is BroadcastNestedLoopJoin).
  • Native OpStruct::DeltaScan dispatch arm (not-compiled-in error on default builds; feature-gated shim into the contrib), exhaustive-match arms, convert_spark_types_to_arrow_schema → pub(crate).
  • Stub contrib crate (contrib/delta/native): plan_delta_scan returns DataFusionError::NotImplemented — just enough to satisfy the core shim so --features contrib-delta links.
  • JVM bridge DeltaIntegration (reflective, returns None until the contrib classes exist), the CometExecRule Delta-marker hook (CDF hook deferred), the CometScanRule Delta delegation + metadata-col reorder, and the leaf DeltaConf.

Why it's safe on default builds

No -Pcontrib-delta / no --features contrib-delta: the dispatch arm is a not-compiled-in error, the contrib crate isn't linked, and DeltaIntegration's reflective lookups resolve to nothing. The gate-verify script asserts this (0 Delta symbols in the default libcomet, zero io.delta in the effective pom, only the DeltaIntegration bridge class compiled).

Verification

default + gated native build, clippy (both feature states), the gate-verify script, gated + default JVM compile, spotless/scalastyle, and cargo fmt — all green.


🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.

@schenksj
schenksj force-pushed the pr/delta-A2-buildgate branch from f2ad00c to e020fe6 Compare June 21, 2026 14:14
schenksj added a commit that referenced this pull request Jun 21, 2026
schenksj added a commit that referenced this pull request Jun 22, 2026
@schenksj
schenksj force-pushed the pr/delta-A2-buildgate branch from e020fe6 to e16970e Compare July 17, 2026 02:13
@schenksj
schenksj force-pushed the pr/delta-A2-buildgate branch 3 times, most recently from 9b323ea to 87d5cff Compare August 18, 2026 15:28
@schenksj
schenksj changed the base branch from pr/delta-A1-spi to main August 18, 2026 15:28
@schenksj
schenksj marked this pull request as ready for review August 18, 2026 15:28
@schenksj
schenksj force-pushed the pr/delta-A2-buildgate branch 3 times, most recently from 9ab285f to 0c34ec2 Compare August 19, 2026 03:12
…b split, part 2]

Part 2 of the Delta contrib split: the build gate and the inert core wiring an
out-of-tree scan contrib plugs into. Nothing here is reachable on a default
build -- no contrib is registered, no contrib class is compiled, and the native
library carries zero contrib symbols.

Core gains two format-agnostic extension points, both discovered at runtime so
core holds no compile-time reference to any contrib:

  - `CometScanContrib`, a ServiceLoader-discovered hook (mirroring
    `PlanDataInjector`) that lets a contrib claim a V1 or V2 scan before
    Comet's built-in handling runs, plus `CometContribScanMarker` so
    `CometExecRule` can route a contrib's scan node to the contrib's own serde
    handler by a plain type test.

  - `ContribScan contrib_scan = 200`, a single permanent `Any`-shaped proto
    envelope (`type_url` + packed `value`) dispatched by `type_url` on the
    native side. Core's oneof never grows per-format, so independent contrib
    PRs cannot collide on a field number -- as `main` taking field 118 for
    `Sample` has since demonstrated.

Plus the build machinery: the `contrib-delta` Maven profile and Cargo feature,
and `dev/verify-contrib-delta-gate.sh`, which asserts a default build compiles
no contrib classes, packages no contrib `META-INF/services` files, and links no
contrib symbols.

Where the hooks sit, and why. Both run *before* Comet's built-in guards for
their scan kind, because a contrib may support things the built-in scan does
not -- the Delta contrib synthesises `_metadata.*` in its own reader, and a
contrib's table name may end in `files`/`snapshots` like an Iceberg metadata
table. Applying those guards first would decline such a scan before the contrib
was ever offered it. So `transformV1Scan` consults the contrib ahead of the
metadata-column guard, and the Iceberg metadata-table check moves out of the
outer `transformScan` match into `transformV2Scan`, after its hook. Core's
per-path metadata handling is otherwise untouched: `main` serves
`fileConstantMetadataColumns` natively in V1 and the Iceberg metadata columns
in V2, and both keep doing so.

Ownership contract. An implementation MUST return `None` for a scan it does not
own: contribs are offered a scan one at a time and the first claim wins, so a
contrib claiming another format's scan hides it from the contrib that could
have read it, with the outcome depending on unspecified ServiceLoader ordering.
"Own but cannot handle" is a distinct, expressible case -- claim the scan and
terminate it with `withFallbackReason` rather than declining. Core cannot
arbitrate competing claims (a claim is opaque; the only way to know a second
contrib would also have claimed is to ask it, which is what claiming prevents),
so the contract carries it.

Tests. `CometScanContribSuite` covers the registry contract on a default build:
no contribs registered (asserted against raw ServiceLoader discovery, not just
the registry -- `contribs` swallows a ServiceConfigurationError, so "empty"
alone is ambiguous), a stub discovered through a URLClassLoader whose claim is
returned, decline-passes-through, first-claim-wins with later contribs not
consulted, throw-is-a-decline, and LinkageError still propagating.
`CometScanRuleSuite` gains a V1 case asserting the fallback *reason* for
`_metadata.row_index`; verified red with the guard removed.

Co-Authored-By: Claude Opus 5 <[email protected]>
@schenksj
schenksj force-pushed the pr/delta-A2-buildgate branch from 0c34ec2 to 7f8dc65 Compare August 19, 2026 10:59
@schenksj
schenksj merged commit 56527a3 into main Aug 20, 2026
72 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant