Skip to content

Commit d1553b5

Browse files
schenksjclaude
andcommitted
feat(contrib): SPI crate split + worked example contrib (PR1.7 part 1)
Substantive piece of PR1.7. Two structural changes break what was otherwise going to be a cyclic dependency between core and contribs: 1. New crate native/contrib-spi/ (`comet-contrib-spi`): - Defines the `ContribOperatorPlanner` trait that contribs implement. - Owns the process-wide registry (`register_contrib_planner`, `lookup_contrib_planner_by_kind`, `registered_contrib_kinds`). - Light-weight `ContribError` enum for SPI errors. Core converts to its own `ExecutionError` at the dispatch site. - 1 unit test covering registration round-trip. - Only deps: `datafusion` + `log`. No deps on core, no deps from core back. The SPI is the leaf both core and contribs depend on. 2. New crate contrib/example/native/ (`comet-contrib-example`): - rlib (NOT cdylib) -- linked INTO core's libcomet via the `contrib-example` Cargo feature flag on core. - Registers a `NoOpPlanner` against kind `"example-no-op"` via `#[ctor::ctor]`. The planner returns a sentinel error so tests can verify the full JVM->JNI->native->contrib dispatch chain. - Depends on `comet-contrib-spi` (NOT on core). - Real contribs follow the same shape: rlib + #[ctor] + thin dependency on the SPI crate. Core rewiring: - native/core/Cargo.toml: `comet-contrib-spi` mandatory dep; `comet-contrib-example` optional dep gated by feature `contrib-example` (default-on so released builds ship the example registered). - native/core/src/lib.rs: `extern crate comet_contrib_example` gated by the feature so #[ctor] runs at libcomet load. - native/core/src/execution/planner/contrib.rs: now just re-exports the SPI surface for backwards-compatible imports within core. - native/core/src/execution/planner.rs: ContribOp dispatcher now recursively builds native children, calls the SPI's `plan(payload, children) -> Arc<dyn ExecutionPlan>`, wraps in `SparkPlan`. Maps `ContribError` to `ExecutionError::GeneralError` with a clear contrib-identified prefix. Workspace wiring: - native/Cargo.toml: adds `contrib-spi` to default-members so the SPI crate is built/checked with the rest of the workspace. - Adds `../contrib/example/native` to workspace members (NOT default-members) so it shares the workspace lockfile and dependency overrides but isn't compiled standalone. Build state: `cargo check` on all three crates (core, contrib-spi, contrib-example) is clean. SPI unit test passes. The Maven side of the example contrib (pom.xml, Scala extension, ServiceLoader entry, integration test) is NOT in this commit -- it lands in a follow-up on the same branch. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 8b69471 commit d1553b5

10 files changed

Lines changed: 446 additions & 136 deletions

File tree

‎contrib/example/native/Cargo.toml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[package]
19+
name = "comet-contrib-example"
20+
description = "Worked reference implementation of a Comet contrib extension. Registers a no-op ContribOperatorPlanner under kind=\"example-no-op\" so the SPI dispatch path can be exercised end-to-end in tests."
21+
# Contrib crates live OUTSIDE the workspace root directory (`native/`) but are listed as
22+
# workspace members in `native/Cargo.toml`. Cargo's auto-discovery walks up the directory
23+
# tree, so without the explicit pointer it can't find `native/Cargo.toml` from
24+
# `contrib/example/native/`.
25+
workspace = "../../../native"
26+
version = { workspace = true }
27+
homepage = { workspace = true }
28+
repository = { workspace = true }
29+
authors = { workspace = true }
30+
license = { workspace = true }
31+
edition = { workspace = true }
32+
33+
[lib]
34+
# rlib (not cdylib): linked INTO core's cdylib via the `contrib-example` Cargo feature
35+
# flag on the core crate. There is exactly one libcomet.{so,dylib,dll} at runtime; the
36+
# contrib's #[ctor] runs during that single library's init.
37+
crate-type = ["rlib"]
38+
39+
[dependencies]
40+
# Depend on the thin SPI crate, NOT on core. This is what breaks the cycle: core
41+
# depends on contribs (Cargo feature → rlib link); both depend on contrib-spi; nothing
42+
# depends back on core from a contrib.
43+
comet-contrib-spi = { path = "../../../native/contrib-spi" }
44+
datafusion = { workspace = true }
45+
ctor = "0.4"
46+
log = "0.4"

‎contrib/example/native/src/lib.rs‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Worked reference implementation of a Comet contrib extension.
19+
//!
20+
//! Registers a single `ContribOperatorPlanner` under `kind = "example-no-op"`. The
21+
//! planner is intentionally trivial: it returns a clear `ContribError::Plan` so tests can
22+
//! verify the full dispatch chain (JVM serde → ContribOp envelope → JNI → native planner
23+
//! → contrib registry → this planner) without needing to actually execute anything.
24+
//!
25+
//! Real contribs (Delta, Hudi, etc.) replace `NoOpPlanner::plan` with a function that
26+
//! decodes the contrib's own proto message from `payload` and constructs an
27+
//! `ExecutionPlan` for the contrib's native operator.
28+
//!
29+
//! The whole crate is gated by `native/core/Cargo.toml`'s `contrib-example` feature flag.
30+
//! Build core without that feature (`cargo build --no-default-features`) and zero bytes
31+
//! of this crate end up in `libcomet`.
32+
33+
use std::sync::Arc;
34+
35+
use comet_contrib_spi::{
36+
register_contrib_planner, ContribError, ContribOperatorPlanner,
37+
};
38+
use datafusion::physical_plan::ExecutionPlan;
39+
40+
/// Stable identifier the example registers under. The Scala side writes this same string
41+
/// into `ContribOp.kind` when building a payload for the example operator. Convention:
42+
/// `<contrib-short-name>-<operator-short-name>`.
43+
pub const EXAMPLE_NO_OP_KIND: &str = "example-no-op";
44+
45+
/// A planner that intentionally does no plan-building work. It exists only to prove the
46+
/// dispatch chain is wired up correctly: tests construct an Operator with this kind, ship
47+
/// it through JNI, and assert that the returned error mentions this string.
48+
struct NoOpPlanner;
49+
50+
impl ContribOperatorPlanner for NoOpPlanner {
51+
fn plan(
52+
&self,
53+
_payload: &[u8],
54+
_children: Vec<Arc<dyn ExecutionPlan>>,
55+
) -> Result<Arc<dyn ExecutionPlan>, ContribError> {
56+
Err(ContribError::Plan(format!(
57+
"comet-contrib-example: NoOpPlanner reached for kind={EXAMPLE_NO_OP_KIND:?}; \
58+
this is the expected sentinel for SPI dispatch tests"
59+
)))
60+
}
61+
}
62+
63+
/// Registers `NoOpPlanner` against `EXAMPLE_NO_OP_KIND` at library-init time. Called by
64+
/// the linker before `main`/`JNI_OnLoad` because of `#[ctor::ctor]`. Comet's main
65+
/// `libcomet` is what gets loaded by the JVM; this constructor runs during its init.
66+
#[ctor::ctor]
67+
fn register() {
68+
log::info!(
69+
"comet-contrib-example: registering ContribOperatorPlanner kind={EXAMPLE_NO_OP_KIND:?}"
70+
);
71+
register_contrib_planner(EXAMPLE_NO_OP_KIND, Arc::new(NoOpPlanner));
72+
}

‎native/Cargo.lock‎

Lines changed: 55 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎native/Cargo.toml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,14 @@
1616
# under the License.
1717

1818
[workspace]
19-
default-members = ["core", "spark-expr", "common", "proto", "jni-bridge", "shuffle"]
20-
members = ["core", "spark-expr", "common", "proto", "jni-bridge", "shuffle", "hdfs", "fs-hdfs"]
19+
default-members = ["core", "spark-expr", "common", "proto", "jni-bridge", "shuffle", "contrib-spi"]
20+
# `contrib-spi` is the thin SPI surface that BOTH core and contribs depend on -- breaking
21+
# what would otherwise be a cyclic dep between core (links contribs via Cargo features)
22+
# and contribs (need core types). Contrib crates themselves live under
23+
# `../contrib/<name>/native` and are workspace members so workspace lockfile + workspace
24+
# dependencies apply; they're NOT default-members because they're consumed via core's
25+
# optional Cargo feature flags rather than built standalone.
26+
members = ["core", "spark-expr", "common", "proto", "jni-bridge", "shuffle", "hdfs", "fs-hdfs", "contrib-spi", "../contrib/example/native"]
2127
resolver = "2"
2228

2329
[workspace.package]

‎native/contrib-spi/Cargo.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[package]
19+
name = "comet-contrib-spi"
20+
description = "Stable SPI surface that contrib crates and Comet's core both depend on. Defines the ContribOperatorPlanner trait, the process-wide registry, and the lightweight error type. Separating this from the core crate breaks what would otherwise be a cyclic dependency (core links contribs via Cargo feature flags; contribs need core types)."
21+
version = { workspace = true }
22+
homepage = { workspace = true }
23+
repository = { workspace = true }
24+
authors = { workspace = true }
25+
license = { workspace = true }
26+
edition = { workspace = true }
27+
28+
[dependencies]
29+
datafusion = { workspace = true }
30+
log = "0.4"

0 commit comments

Comments
 (0)