Skip to content

Commit 0eafd4d

Browse files
jprochazktimsaucer
andauthored
chore: move datafusion-ffi integration tests into their own crate (apache#26564)
## Which issue does this PR close? - Closes apache#26559. ## Rationale for this change `datafusion-ffi` is built as a `cdylib` only so that its own integration tests can `dlopen` it. Every downstream build pays for compiling and linking that unused shared library, and `sccache` can't cache crates with a `cdylib` crate type. ## What changes are included in this PR? - New unpublished crate `datafusion-ffi-integration-tests` in `datafusion/ffi/integration-tests/`, built as a `cdylib`, which re-exports the `#[no_mangle]` test entry points from `datafusion_ffi::tests`. Its `integration-tests` feature enables `datafusion-ffi/integration-tests`, so regular workspace builds still don't compile the test module. - `datafusion/ffi/tests/` moves there unchanged, and `find_library` looks for the new library name. - `datafusion-ffi` is now a plain `rlib`. - The `test ffi` xtask step and the `datafusion-ffi` agent skill include the new crate. The new crate needs both `cdylib` and `rlib`: with only `cdylib`, Cargo doesn't build the library before running the crate's integration tests. ## What is the testing strategy for this PR? Covered by the existing FFI integration tests, which now load the new crate's `cdylib`. ## Are there any user-facing changes? The public API and ABI of `datafusion-ffi` are unchanged. Crates that build their own `cdylib` on top of `datafusion-ffi` are unaffected. --- This PR was prepared with the help of a coding agent (Claude). Co-authored-by: Tim Saucer <[email protected]>
1 parent 659b475 commit 0eafd4d

20 files changed

Lines changed: 102 additions & 7 deletions

File tree

‎.ai/skills/datafusion-ffi/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -197,21 +197,21 @@ Templates: `test_ffi_table_provider_local_bypass` and `test_round_trip_ffi_table
197197

198198
What unit tests catch: Rust-level correctness (translation logic, lifetime bugs, leaks under valgrind/miri, Send/Sync, error propagation, codec round-trips). What they **cannot** catch: real ABI bugs. Both producer and consumer share `#[repr(C)]` layout because they are the exact same struct definition in memory.
199199

200-
#### b. Cross-library integration tests (`tests/ffi_*.rs`, gated by the `integration-tests` feature)
200+
#### b. Cross-library integration tests (`integration-tests/tests/`, gated by the `integration-tests` feature)
201201

202-
The crate is published as `crate-type = ["cdylib", "rlib"]`. The integration tests in `datafusion/ffi/tests/` use `libloading` to `dlopen` the crate's own `cdylib` and call `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. The test executable links against the rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
202+
The integration tests live in the unpublished `datafusion-ffi-integration-tests` crate in `datafusion/ffi/integration-tests/`, so that `datafusion-ffi` itself is a plain rlib. That crate is built as a `cdylib` that exports `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. Its tests use `libloading` to `dlopen` that `cdylib`. The test executable links against the `datafusion-ffi` rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
203203

204204
Run with:
205205

206206
```bash
207-
cargo test -p datafusion-ffi --features integration-tests
207+
cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests
208208
```
209209

210210
To add coverage for a new wrapper:
211211

212212
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
213213
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
214-
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
214+
3. **Add the test** in `integration-tests/tests/cases/<area>.rs`, registered in `cases/mod.rs`, under `mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
215215

216216
When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
217217

@@ -347,7 +347,7 @@ When reviewing a PR that touches `datafusion/ffi/`:
347347
7. **Stabby types** for strings/vecs; crate-local `FFI_Option`/`FFI_Result` for optional/fallible payloads?
348348
8. **Async** uses `FfiFuture` + `.into_ffi()`, never blocking?
349349
9. **Codec** present on any method that ships an `Expr` / plan?
350-
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `tests/ffi_*.rs` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi --features integration-tests` must pass before merging an ABI-affecting PR.
350+
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `integration-tests/tests/` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests` must pass before merging an ABI-affecting PR.
351351
11. **No `datafusion` runtime dep** crept into `Cargo.toml`?
352352
12. **`Arc::clone(&x)`** everywhere — no implicit `x.clone()` on `Arc` (the lint will reject it but worth pre-flagging).
353353
13. **`cargo clippy --all-targets --all-features -- -D warnings`** clean on the crate?

‎Cargo.lock‎

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ members = [
3232
"datafusion/expr-common",
3333
"datafusion/execution",
3434
"datafusion/ffi",
35+
"datafusion/ffi/integration-tests",
3536
"datafusion/functions",
3637
"datafusion/functions-aggregate",
3738
"datafusion/functions-aggregate-common",

‎datafusion/ffi/Cargo.toml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@ workspace = true
3838

3939
[lib]
4040
name = "datafusion_ffi"
41-
crate-type = ["cdylib", "rlib"]
4241

4342
# Note to developers: do *not* add `datafusion` as a dependency in this crate.
4443
# It increases build times and library binary size for users.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[package]
19+
name = "datafusion-ffi-integration-tests"
20+
description = "Cross-library integration tests for datafusion-ffi"
21+
version = "0.1.0"
22+
edition = { workspace = true }
23+
license = { workspace = true }
24+
publish = false
25+
26+
[lints]
27+
workspace = true
28+
29+
[lib]
30+
crate-type = ["cdylib", "rlib"]
31+
32+
[features]
33+
integration-tests = ["datafusion-ffi/integration-tests"]
34+
35+
[dependencies]
36+
datafusion-ffi = { workspace = true }
37+
38+
[dev-dependencies]
39+
arrow = { workspace = true }
40+
arrow-schema = { workspace = true }
41+
datafusion = { workspace = true, default-features = false, features = ["sql"] }
42+
datafusion-catalog = { workspace = true }
43+
datafusion-common = { workspace = true }
44+
datafusion-execution = { workspace = true }
45+
datafusion-expr = { workspace = true }
46+
datafusion-physical-expr-common = { workspace = true }
47+
datafusion-physical-optimizer = { workspace = true }
48+
datafusion-physical-plan = { workspace = true }
49+
datafusion-proto = { workspace = true, default-features = false }
50+
datafusion-session = { workspace = true }
51+
tokio = { workspace = true }
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#[cfg(feature = "integration-tests")]
19+
pub use datafusion_ffi::tests::{
20+
datafusion_ffi_get_module, datafusion_ffi_test_create_exec_with_byte_metrics,
21+
};
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)