Skip to content

Commit 461fa4f

Browse files
schenksjclaude
andcommitted
feat(delta): rich Hadoop credential chain for kernel log replay
Closes the credential-asymmetry recommendation from the apache#87 audit (P1). Previously the delta-kernel-rs log-replay path built its `DeltaStorageConfig` from only the four static-key entries on the storage_options map (access/secret/session/region/endpoint/path-style), so a user with SimpleAWSCredentialsProvider / TemporaryAWSCredentialsProvider / AssumedRoleCredentialProvider / IAMInstanceCredentialsProvider configured would see data-file reads authenticate fine (those go through the rich s3.rs::create_store path) but log replay fail. This commit makes log replay use the SAME Hadoop credential provider chain by snapshot-resolving credentials at planning time: * native/core/src/parquet/objectstore/s3.rs: new public `resolve_static_credentials(configs, bucket)` calls the same `build_credential_provider` chain that `create_store` uses, then pulls a single (access_key, secret_key, session_token) tuple out via the resolved provider's `get_credential()`. Returns None for anonymous; surfaces errors when resolution fails. * native/core/src/delta/jni.rs: when the storage_options map lacks explicit static keys for an s3/s3a URL, the JNI entry point now calls `resolve_static_credentials` and populates the `DeltaStorageConfig` static fields with the resolved values. The rest of the pipeline (engine.rs object-store construction) stays unchanged. * native/core/src/parquet/mod.rs: bump the parquet::objectstore submodule visibility from `mod` to `pub(crate) mod` so the delta crate can import its public functions. The resolution is a SNAPSHOT (not refresh-capable) because Delta log replay completes in seconds — well within any reasonable credential TTL. Long-running data reads continue to use `create_store`'s full refresh-capable provider. JMap iteration uses `env.cast_local::<JString>(...)` rather than the unsafe `JString::from_raw(env, into_raw())` shortcut used elsewhere in this file — safer downcast for the same JLS-implied invariant. Also adds docs/contrib-delta-migration-plan.md — the audit's other two recommendations (P2 iceberg per-bucket session-token typo, P3 GCS/ADLS Hadoop translation for iceberg) were intentionally NOT touched here because they're out of scope for the delta-kernel branch; they're captured in the audit task description for a future iceberg PR. Verified: - Targeted DeltaDropColumnSuite: 21/21 pass. - DeletionVectorsSuite + DeltaColumnMappingSuite: 333/333 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent f84c6ba commit 461fa4f

4 files changed

Lines changed: 380 additions & 2 deletions

File tree

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
# Plan: move delta-kernel-rs integration into a Comet "contrib" extension
2+
3+
Driver: PR review feedback on `delta-kernel-phase-1` asked the Delta integration to live in a
4+
contrib section rather than core so the core stays format-agnostic and the heavyweight
5+
delta-kernel-rs dependency (with its own `object_store_kernel` pin at 0.12 vs core's
6+
`object_store` at 0.13) is opt-in.
7+
8+
User-confirmed decisions:
9+
1. **Single PR.** No series of incremental PRs. All structural changes ship together.
10+
2. **Separate extension JAR with its own cdylib.** Delta contrib ships as a fully independent
11+
artifact whose native library is loaded at runtime when the contrib JAR is on the classpath.
12+
3. **Iceberg stays in core.** It is treated as a first-class Comet feature; no parallel move.
13+
4. **Proto stays as-is.** Keep the existing `DeltaScan` message verbatim; only its
14+
reader/writer ownership relocates from core to contrib. (See "Proto handling" below.)
15+
16+
This plan executes AFTER P1 (rich credentials for log replay) is signed off — P1 lives in the
17+
files being moved and merging the contrib split before P1 would conflict messily.
18+
19+
## 1. Target layout
20+
21+
```
22+
contrib/
23+
delta/
24+
pom.xml ← new Maven module (groupId=org.apache.datafusion,
25+
artifactId=comet-contrib-delta-spark${spark.version.short}_${scala.binary.version})
26+
native/
27+
Cargo.toml ← new Cargo crate (package=comet-contrib-delta)
28+
src/
29+
lib.rs ← cdylib entry, JNI symbol exports, registration
30+
engine.rs ← from native/core/src/delta/engine.rs
31+
error.rs ← from native/core/src/delta/error.rs
32+
jni.rs ← from native/core/src/delta/jni.rs
33+
predicate.rs ← from native/core/src/delta/predicate.rs
34+
scan.rs ← from native/core/src/delta/scan.rs
35+
integration_tests.rs ← from native/core/src/delta/integration_tests.rs
36+
dv_filter.rs ← from native/core/src/execution/operators/delta_dv_filter.rs
37+
planner_ext.rs ← (NEW) implements ContribOperatorPlanner for OpStruct::DeltaScan
38+
scan_rule_ext.rs ← (NEW) glue called by Scala-side CometScanRuleExtension
39+
spark/
40+
src/main/scala/org/apache/comet/contrib/delta/
41+
DeltaScanExtension.scala ← (NEW) implements CometScanRuleExtension
42+
package.scala ← (NEW) ServiceLoader registration helpers
43+
src/main/scala/org/apache/comet/delta/
44+
DeltaReflection.scala ← moved from spark/.../comet/delta/
45+
src/main/scala/org/apache/comet/serde/operator/
46+
CometDeltaNativeScan.scala ← moved
47+
src/main/scala/org/apache/spark/sql/comet/
48+
CometDeltaNativeScanExec.scala ← moved
49+
src/main/resources/META-INF/services/
50+
org.apache.comet.spi.CometScanRuleExtension ← lists DeltaScanExtension
51+
dev/
52+
diffs/3.3.2.diff ← moved from dev/diffs/delta/3.3.2.diff
53+
run-regression.sh ← moved from dev/run-delta-regression.sh
54+
```
55+
56+
`contrib/delta/native/` produces `libcomet_contrib_delta_<arch>.dylib` packaged in the contrib
57+
JAR's `META-INF/native/`. The contrib JAR's class init (via ServiceLoader) extracts and loads
58+
this library; core's JNI is unaware.
59+
60+
## 2. Files that move wholesale
61+
62+
These are 100% Delta and have no incoming references from non-Delta callers outside the hook
63+
points listed in §4:
64+
65+
### Rust → `contrib/delta/native/src/`
66+
- `native/core/src/delta/{engine,error,jni,mod,predicate,scan,integration_tests}.rs`
67+
- `native/core/src/execution/operators/delta_dv_filter.rs` → `contrib/delta/native/src/dv_filter.rs`
68+
69+
### Scala
70+
- `spark/.../comet/delta/DeltaReflection.scala`
71+
- `spark/.../serde/operator/CometDeltaNativeScan.scala`
72+
- `spark/.../sql/comet/CometDeltaNativeScanExec.scala`
73+
74+
### Test driver
75+
- `dev/diffs/delta/3.3.2.diff` and `dev/run-delta-regression.sh`
76+
77+
## 3. Code that stays in core unchanged
78+
79+
- `common/.../objectstore/NativeConfig.scala` — generic, used by core + Iceberg + contribs.
80+
- `native/core/src/parquet/objectstore/s3.rs::resolve_static_credentials` — generic (P1
81+
placed it here intentionally; the contrib delta JNI module consumes it via core's public crate path).
82+
- The Iceberg integration in its entirety.
83+
- The `DeltaScan` / `DeltaScanCommon` / `DeltaScanTask` / `DeltaScanTaskList` proto messages
84+
in `native/proto/src/proto/operator.proto`. They stay where they are; only their reader and
85+
writer move to contrib. The proto crate is generic infrastructure (already a top-level Cargo
86+
member with no Delta logic), so no module change there.
87+
88+
## 4. Code that needs extension points
89+
90+
These are the touch-points outside the `delta/` directories that today reference Delta directly.
91+
Each gets replaced by a registration hook so core no longer mentions Delta.
92+
93+
### Rust core
94+
95+
| File | Current Delta refs | After move |
96+
|---|---|---|
97+
| `native/core/src/lib.rs` | `pub mod delta;` (line 70) | removed |
98+
| `native/core/src/execution/jni_api.rs` | 1 reference | removed (Delta JNI moves to contrib's cdylib) |
99+
| `native/core/src/execution/operators/mod.rs` | `mod delta_dv_filter` + re-export | removed |
100+
| `native/core/src/execution/operators/operator_registry.rs` (line 154) | `OpStruct::DeltaScan(_) => None` stub | replaced by generic ContribScan registry lookup |
101+
| `native/core/src/execution/planner.rs` (~31 refs) | full `OpStruct::DeltaScan(...)` arm at ~1320–1572, plus `ColumnMappingFilterRewriter` at 3115–3145 | the arm calls into `ContribOperatorPlanner::plan(payload, children, ctx)` looked up by `OpStruct` discriminator; the rewriter moves to contrib |
102+
103+
**New SPI in core (`native/core/src/execution/planner/contrib.rs`):**
104+
105+
```rust
106+
pub trait ContribOperatorPlanner: Send + Sync {
107+
fn op_kind(&self) -> ContribOpKind;
108+
fn plan(
109+
&self,
110+
proto: &OpStruct,
111+
children: Vec<Arc<SparkPlan>>,
112+
ctx: &PlanCtx<'_>,
113+
) -> Result<Arc<dyn ExecutionPlan>, ExecutionError>;
114+
}
115+
116+
pub fn register_contrib_planner(p: Arc<dyn ContribOperatorPlanner>);
117+
pub fn lookup_contrib_planner(op_kind: ContribOpKind) -> Option<Arc<dyn ContribOperatorPlanner>>;
118+
```
119+
120+
`ContribOpKind` is an enum that lists `DeltaScan` (and stays open for future kinds). The
121+
planner.rs match arm becomes a single lookup that dispatches by kind — the *body* of the
122+
match arm moves to contrib's `planner_ext.rs`.
123+
124+
The contrib cdylib's `lib.rs` registers its planner in a `#[ctor]` (the `ctor` crate) or
125+
inside the contrib JNI's library-init JNI call. Either way, registration happens before any
126+
plan executes.
127+
128+
### JVM / Scala core
129+
130+
| File | Current Delta refs | After move |
131+
|---|---|---|
132+
| `spark/.../CometSparkSessionExtensions.scala` | 20 (mostly comments + one or two conf entries) | comments scrubbed; any Delta-specific conf moves into contrib's own `CometConf` extension |
133+
| `spark/.../rules/CometExecRule.scala` | 2 (case for `SCAN_NATIVE_DELTA_COMPAT` + import) | replaced by SPI scan dispatcher that calls registered extensions' serdes |
134+
| `spark/.../rules/CometScanRule.scala` | **149** (the densest Delta-aware site) | the Delta branch of `transformV1Scan` becomes a call into `CometScanRuleExtension`. The `withDeltaColumnMappingMetadata`, `applyRowTrackingRewrite`, input_file_name handling (#75 design A), nested-CM data_schema overlay (#79), DV materialisation for pre-mat FileIndex (#77), and `extractObjectStoreOptions` wiring all move into contrib's `DeltaScanExtension` |
135+
136+
**New SPI in core (`spark/.../comet/spi/`):**
137+
138+
```scala
139+
trait CometScanRuleExtension {
140+
def name: String
141+
def matches(relation: HadoopFsRelation): Boolean
142+
def transform(
143+
plan: SparkPlan,
144+
scanExec: FileSourceScanExec,
145+
session: SparkSession): Option[SparkPlan]
146+
}
147+
148+
trait CometOperatorSerdeExtension {
149+
def serdes: Seq[(Class[_ <: SparkPlan], CometOperatorSerde[_])]
150+
}
151+
152+
object CometExtensionRegistry {
153+
// ServiceLoader-based discovery from META-INF/services/.
154+
// Idempotent; safe to call multiple times.
155+
def load(): Unit
156+
def scanExtensions: Seq[CometScanRuleExtension]
157+
def serdeExtensions: Seq[CometOperatorSerdeExtension]
158+
}
159+
```
160+
161+
`CometSparkSessionExtensions` calls `CometExtensionRegistry.load()` during its extension
162+
registration, which lets the contrib JAR's `META-INF/services/...CometScanRuleExtension` entry
163+
register `DeltaScanExtension`. No code changes in core after that — discovery is by classpath
164+
presence.
165+
166+
## 5. Build wiring
167+
168+
### Maven
169+
- New module entry in root `pom.xml`: `<module>contrib/delta</module>`.
170+
- `contrib/delta/pom.xml`:
171+
- artifactId `comet-contrib-delta-spark${spark.version.short}_${scala.binary.version}`
172+
- Depends on `comet-spark`, `comet-common`.
173+
- Depends on `delta-spark` (provided/test scope — kept consistent with the regression suite).
174+
- Has its own native-library packaging step that places the contrib cdylib under
175+
`META-INF/native/` in the JAR.
176+
- `spark/pom.xml` — drop the Delta provided/test dep that's now in contrib.
177+
178+
### Cargo
179+
- New crate `contrib/delta/native/` with `crate-type = ["cdylib"]`. Package name
180+
`comet-contrib-delta`.
181+
- Add to root `native/Cargo.toml` `members` and `default-members`.
182+
- Cargo dependency on `datafusion-comet` (the core crate) for the public SPI types
183+
(`ContribOperatorPlanner`, `register_contrib_planner`, etc).
184+
- `delta_kernel`, `object_store_kernel` dependencies move from `native/core/Cargo.toml` to
185+
`contrib/delta/native/Cargo.toml`. Core no longer pulls them in.
186+
187+
### JNI symbol packaging and loading
188+
- The contrib JAR's resources contain `META-INF/native/<os>/<arch>/libcomet_contrib_delta.so|.dylib|.dll`.
189+
- A small helper in contrib's `package.scala` (called by `CometExtensionRegistry.load()`
190+
before the contrib's `DeltaScanExtension` is registered) extracts the platform-specific
191+
library to a temp dir and calls `System.load()`. Same pattern Comet core already uses for
192+
its own native lib.
193+
- Contrib's cdylib has a JNI `OnLoad` (or equivalent) that registers its
194+
`ContribOperatorPlanner` with the core crate's registry. The core crate's registry is a
195+
`OnceLock<Mutex<Vec<...>>>`, so this works across cdylib boundaries because the contrib
196+
cdylib links against `datafusion-comet` and shares the static.
197+
198+
Caveat: Rust statics are per-DSO. The contrib cdylib must link to the core crate *as a
199+
shared library* (via the same `libcomet.dylib` already loaded by JVM core) — not as a
200+
Cargo dependency that statically links a second copy of `datafusion-comet`. We achieve
201+
this with `crate-type = ["cdylib"]` on contrib AND making core's crate available as a
202+
dylib for the contrib build profile. This is the trickiest item in the plan; if Rust's
203+
shared-static crossing turns out fiddly, we fall back to passing registration through a
204+
JNI call: contrib's library-init JNI calls a `register_contrib_delta_planner` Java method
205+
that uses JNI again to call into core's registry.
206+
207+
### Spotless / scalafmt
208+
- Add `contrib/delta/` patterns to spotless config so the same formatting applies.
209+
210+
## 6. Single-PR ordering inside the same branch
211+
212+
All changes land in one branch, one PR. Inside the branch the work proceeds in this order so
213+
the regression stays green at each intermediate commit (helpful for reviewer bisect and for
214+
our own validation):
215+
216+
1. **Land P1** on `delta-kernel-phase-1` (separate commit, separate validation; already
217+
pending). This is the precondition for the contrib work.
218+
2. **Add the SPI traits** (`CometScanRuleExtension`, `ContribOperatorPlanner`, etc.) in core
219+
with NO callers yet — pure additions. Regression unchanged.
220+
3. **Wire `CometScanRule` and `planner.rs` to call the registry**, with an in-tree shim
221+
`DeltaScanExtension` registered from core. Logic unchanged; only structurally pluggable
222+
now. Regression unchanged.
223+
4. **Create the new contrib/delta/ module** with the new build wiring (Maven module +
224+
Cargo crate). Initially empty — just the build skeleton.
225+
5. **Physically move the Delta files** out of core into `contrib/delta/`. Update imports.
226+
Wire contrib's ServiceLoader file. Drop the in-tree shim from step 3 once the
227+
contrib module's extension is discovered at load time.
228+
6. **Move the regression test driver** and validate the full Delta regression still passes
229+
with the contrib jar on the test classpath.
230+
231+
All six steps are in one branch. The user reviews the single PR.
232+
233+
## 7. Proto handling (single-PR shape)
234+
235+
Per user decision: keep the existing `DeltaScan` / `DeltaScanCommon` / `DeltaScanTask` /
236+
`DeltaScanTaskList` proto messages exactly as they are.
237+
238+
- The `.proto` files stay in `native/proto/src/proto/operator.proto`.
239+
- The generated Rust types stay in the `datafusion-comet-proto` crate (already a top-level
240+
Cargo member, format-agnostic).
241+
- The generated Java/Scala types stay in `spark`'s build output.
242+
- Only the consumers move: contrib's Rust crate imports the generated types from the proto
243+
crate; contrib's Scala code does likewise.
244+
245+
No deprecation window needed; no backwards-compat shims. If we later want to generalize to
246+
a `ContribScan` envelope for future Hudi/DeltaSharing/etc. contribs we can do it as a
247+
focused follow-up — the SPI is already shaped for that.
248+
249+
## 8. Risks / open questions to confirm with the user before starting
250+
251+
- **Shared-static across DSOs (§5, JNI loading).** Rust statics live per-dylib. Cross-dylib
252+
registration depends on contrib linking against core as a `dylib` (not a static copy).
253+
If that turns out to require deeper toolchain work than expected, the fallback is a
254+
JNI-mediated registration (contrib calls back through Java which calls forward into
255+
core's JNI). Confirm we're OK with this risk profile before kickoff. If not, we ship the
256+
JNI-mediated path from day one.
257+
- **Spark Delta version skew.** The contrib JAR currently targets Delta 3.3.2 (Spark 3.5).
258+
Confirm whether contrib should also produce Spark 3.4 / Delta 2.4 and Spark 4.0 /
259+
Delta 4.0 variants in this PR or whether those land in follow-ups. (`dev/run-delta-regression.sh`
260+
already conditionally drives all three.)
261+
- **`META-INF/native/` library size.** Adding a second cdylib roughly doubles the native
262+
size of the contrib JAR vs core. Confirm acceptable, or whether contrib should ship one
263+
cdylib per supported OS/arch combo separately (Linux x86_64, Linux aarch64, macOS x86_64,
264+
macOS aarch64, Windows x86_64).

‎native/core/src/delta/jni.rs‎

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,51 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_planDeltaScan(
7373
DeltaStorageConfig::default()
7474
} else {
7575
let jmap: JMap<'_> = env.cast_local::<JMap>(storage_options)?;
76-
extract_storage_config(env, &jmap)?
76+
let mut cfg = extract_storage_config(env, &jmap)?;
77+
// P1: when explicit static keys aren't on the map, walk the same Hadoop
78+
// credential-provider chain that `s3.rs::create_store` uses for data-file reads.
79+
// This lets log replay authenticate under SimpleAWSCredentialsProvider /
80+
// TemporaryAWSCredentialsProvider / AssumedRoleCredentialProvider /
81+
// IAMInstanceCredentialsProvider just like the data path, instead of failing
82+
// outright with non-static creds.
83+
//
84+
// SNAPSHOT resolution: we resolve once at planning time and store the resulting
85+
// access/secret/token strings on DeltaStorageConfig. Log replay completes in
86+
// seconds typically, well within any reasonable credential TTL.
87+
if cfg.aws_access_key.is_none() || cfg.aws_secret_key.is_none() {
88+
if let Ok(parsed_url) = url::Url::parse(&url_str) {
89+
let scheme = parsed_url.scheme();
90+
if scheme == "s3" || scheme == "s3a" {
91+
if let Some(bucket) = parsed_url.host_str() {
92+
let raw_options = jmap_to_hashmap(env, &jmap)?;
93+
match crate::parquet::objectstore::s3::resolve_static_credentials(
94+
&raw_options,
95+
bucket,
96+
) {
97+
Ok(Some((ak, sk, token))) => {
98+
cfg.aws_access_key = Some(ak);
99+
cfg.aws_secret_key = Some(sk);
100+
if token.is_some() {
101+
cfg.aws_session_token = token;
102+
}
103+
}
104+
Ok(None) => {
105+
// Anonymous credential provider was configured -- leave
106+
// the static fields empty so engine.rs builds an anonymous
107+
// S3 client.
108+
}
109+
Err(e) => {
110+
log::warn!(
111+
"Delta log-replay credential resolution failed for bucket {bucket}: {e}; \
112+
falling back to static-only DeltaStorageConfig (will fail if no static keys present)"
113+
);
114+
}
115+
}
116+
}
117+
}
118+
}
119+
}
120+
cfg
77121
};
78122

79123
// Phase 2: read column names for BoundReference resolution.
@@ -292,6 +336,36 @@ fn read_string_array(env: &mut Env, arr: &jni::objects::JObjectArray) -> CometRe
292336
Ok(result)
293337
}
294338

339+
/// Iterate a `java.util.Map<String, String>` into a Rust `HashMap`. Used when we need to
340+
/// pass the full Hadoop config map to a downstream consumer (e.g.,
341+
/// `s3::resolve_static_credentials`) that walks its own provider chain.
342+
///
343+
/// Uses `env.cast_local::<JString>(...)` to safely downcast each key/value entry rather
344+
/// than the `unsafe { JString::from_raw(..., into_raw()) }` shortcut used elsewhere in
345+
/// this file -- the runtime cast performs the same JNI-side type check the JLS implies
346+
/// for `Map<String, String>` but without the unchecked transmute.
347+
fn jmap_to_hashmap(
348+
env: &mut Env,
349+
jmap: &JMap<'_>,
350+
) -> CometResult<std::collections::HashMap<String, String>> {
351+
let mut out = std::collections::HashMap::new();
352+
jmap.iter(env).and_then(|mut iter| {
353+
while let Some(entry) = iter.next(env)? {
354+
let k = entry.key(env)?;
355+
let v = entry.value(env)?;
356+
let kstr: JString = env.cast_local::<JString>(k)?;
357+
let key = kstr.try_to_string(env)?;
358+
if !v.is_null() {
359+
let vstr: JString = env.cast_local::<JString>(v)?;
360+
let value = vstr.try_to_string(env)?;
361+
out.insert(key, value);
362+
}
363+
}
364+
Ok(())
365+
})?;
366+
Ok(out)
367+
}
368+
295369
/// `map.get(key)` for a `java.util.Map<String, String>` surfaced as a
296370
/// `JMap`. Returns `None` if the key is absent or the value is `null`.
297371
fn map_get_string(env: &mut Env, jmap: &JMap<'_>, key: &str) -> CometResult<Option<String>> {

‎native/core/src/parquet/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ pub mod read;
3030
pub mod schema_adapter;
3131

3232
mod cast_column;
33-
mod objectstore;
33+
pub(crate) mod objectstore;
3434

3535
use std::collections::HashMap;
3636
use std::task::Poll;

0 commit comments

Comments
 (0)