Skip to content

Commit b764b3d

Browse files
authored
Merge branch 'main' into filter-push-batch-with-filter
2 parents 23beb6a + 8332cfa commit b764b3d

906 files changed

Lines changed: 100940 additions & 27776 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/audit-datafusion-spark-expression/SKILL.md‎

Lines changed: 996 additions & 0 deletions
Large diffs are not rendered by default.

‎.ai/skills/datafusion-ffi/SKILL.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: datafusion-ffi
3-
description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `Foreign<X>` impls, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
3+
description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
44
---
55

66
# DataFusion FFI Skill
@@ -210,9 +210,11 @@ cargo test -p datafusion-ffi --features integration-tests
210210
To add coverage for a new wrapper:
211211

212212
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
213-
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract — adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
213+
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
214214
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
215215

216+
When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
217+
216218
What integration tests catch that unit tests cannot:
217219

218220
- **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.
@@ -235,7 +237,7 @@ What integration tests catch that unit tests cannot:
235237
- Primitives (`u8`/`u64`/`bool`/`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).
236238
- A `stabby::string::String` (`SString`) returned by value, with no other args or returns.
237239

238-
Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer — those exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
240+
Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer. These exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
239241

240242
If you skip the integration test for a layout change, you have effectively shipped untested ABI.
241243

@@ -313,6 +315,8 @@ If a method's body is non-trivial, the consumer-side default is non-trivial too.
313315
- **Logical `Expr` / `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.
314316
- **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From<Native> for FFI_X` and `From<&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.
315317
- **Errors**: every `FFI_X` method that can fail returns `FFI_Result<T>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src/util.rs` — do not roll your own.
318+
- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.
319+
- **Owned FFI returns**: consume an owned `FFI_X` with `From<FFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.
316320

317321
## Async, sessions, and task context
318322

‎.ai/skills/pr_review/SKILL.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
name: pr_review
3+
description: Review Apache DataFusion pull requests following the project's PR review guide. Use whenever asked to review a DataFusion PR or PR URL, and whenever creating a PR, to check the changes against the same criteria before submitting.
4+
---
5+
6+
# DataFusion PR Review
7+
8+
This skill describes the mechanics for doing PR reviews from the command line.
9+
10+
When creating a PR, skip the "Collect PR context" step and instead check the
11+
changes against each area of the
12+
[PR review guide](../../../docs/source/contributor-guide/pr_review.md) before
13+
submitting.
14+
15+
## Collect PR context
16+
17+
- Check out the PR locally: `gh pr checkout <PR number>` (ask first if the
18+
working tree has other work in progress).
19+
- Fetch the PR description, comments, and reviews:
20+
`gh pr view <PR number> --json title,body,comments,reviews`
21+
- Fetch CI status: `gh pr checks <PR number>`.
22+
23+
## Compute the diff
24+
25+
```bash
26+
# find the remote that points at apache/datafusion (e.g. `apache`, `upstream`, or `origin`)
27+
UPSTREAM=$(git remote -v | grep -m1 'apache/datafusion' | cut -f1)
28+
git fetch $UPSTREAM main
29+
git diff $(git merge-base HEAD $UPSTREAM/main)
30+
```
31+
32+
## Review checklist
33+
34+
Work through each area from the
35+
[PR review guide](../../../docs/source/contributor-guide/pr_review.md).

‎.asf.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ github:
6666
- "cargo check datafusion-proto features"
6767
- "cargo check datafusion features"
6868
- "cargo check datafusion-functions features"
69+
- "cargo check datafusion-spark features"
6970
- "cargo test (amd64)"
7071
- "cargo test datafusion-cli (amd64)"
7172
- "cargo examples (amd64)"
@@ -104,6 +105,9 @@ github:
104105
branch-54:
105106
required_pull_request_reviews:
106107
required_approving_review_count: 1
108+
branch-55:
109+
required_pull_request_reviews:
110+
required_approving_review_count: 1
107111
pull_requests:
108112
# enable updating head branches of pull requests
109113
allow_update_branch: true

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ body:
99
description: Please describe what you are trying to do.
1010
placeholder: >
1111
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
12-
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
12+
(This section helps DataFusion developers understand the context and *why* for this feature, in addition to the *what*)
1313
- type: textarea
1414
attributes:
1515
label: Describe the solution you'd like

‎.github/actions/setup-macos-aarch64-builder/action.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ runs:
4444
rustup default stable
4545
rustup component add rustfmt
4646
- name: Setup rust cache
47-
uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
47+
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
4848
with:
4949
save-if: ${{ github.ref_name == 'main' }}
5050
- name: Configure rust runtime env

‎.github/dependabot.yml‎

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,33 @@ updates:
6262
update-types:
6363
- "minor"
6464
- "patch"
65+
# dev/depcheck is a standalone crate excluded from the workspace, so it is
66+
# not covered by the workspace entry above
67+
- package-ecosystem: cargo
68+
directory: "/dev/depcheck"
69+
schedule:
70+
interval: weekly
71+
target-branch: main
72+
labels: [auto-dependencies]
73+
# the wasm test app has its own npm lockfile
74+
- package-ecosystem: npm
75+
directory: "/datafusion/wasmtest/datafusion-wasm-app"
76+
schedule:
77+
interval: weekly
78+
labels: [auto-dependencies]
79+
groups:
80+
# minor and patch bumps are grouped into a single PR
81+
all-npm-deps:
82+
applies-to: version-updates
83+
patterns:
84+
- "*"
85+
update-types:
86+
- "minor"
87+
- "patch"
6588
- package-ecosystem: "github-actions"
66-
directory: "/"
89+
# "/" covers workflows in .github/workflows; the glob covers composite
90+
# actions such as .github/actions/setup-builder
91+
directories: ["/", "/.github/actions/*"]
6792
schedule:
6893
interval: "weekly"
6994
open-pull-requests-limit: 10
@@ -72,8 +97,20 @@ updates:
7297
codeql-actions:
7398
patterns:
7499
- "github/codeql-action/*"
75-
- package-ecosystem: "pip"
76-
directory: "/docs"
100+
# Python dependencies are managed as a uv workspace rooted at the top-level
101+
# pyproject.toml with a shared uv.lock
102+
- package-ecosystem: "uv"
103+
directory: "/"
77104
schedule:
78105
interval: "weekly"
79106
labels: [auto-dependencies]
107+
groups:
108+
# all dependency bumps are grouped into a single PR
109+
all-uv-deps:
110+
applies-to: version-updates
111+
patterns:
112+
- "*"
113+
update-types:
114+
- "major"
115+
- "minor"
116+
- "patch"

‎.github/pull_request_template.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,19 @@ We generally require a GitHub issue to be filed for all bug fixes and enhancemen
1111
<!--
1212
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
1313
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
14+
15+
Please explain the problem you are trying to solve in terms of the user-visible
16+
behavior, rather than the implementation.
17+
18+
For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the
19+
implementation. "COUNT(DISTINCT) returns wrong results when the column contains
20+
nulls" is the user-visible problem.
1421
-->
1522

1623
## What changes are included in this PR?
1724

1825
<!--
19-
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
26+
There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR.
2027
-->
2128

2229
## Are these changes tested?
@@ -33,8 +40,6 @@ If tests are not included in your PR, please explain why (for example, are they
3340

3441
<!--
3542
If there are user-facing changes then we may require documentation to be updated before approving the PR.
36-
-->
3743
38-
<!--
3944
If there are any breaking changes to public APIs, please add the `api change` label.
4045
-->

‎.github/workflows/audit.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ jobs:
4545
steps:
4646
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4747
- name: Install cargo-audit
48-
uses: taiki-e/install-action@a6b2e2dcd845ddd7f509ce4f3ed3d922b80cc5d9 # v2.84.0
48+
uses: taiki-e/install-action@288e746965032cfcc232e09af2daf5f23c14d780 # v2.86.1
4949
with:
5050
tool: cargo-audit
5151
- name: Run audit check

‎.github/workflows/breaking_changes_detector.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ jobs:
8989
9090
- name: Install cargo-semver-checks
9191
if: steps.changed_crates.outputs.packages != ''
92-
uses: taiki-e/install-action@a6b2e2dcd845ddd7f509ce4f3ed3d922b80cc5d9 # v2.84.0
92+
uses: taiki-e/install-action@288e746965032cfcc232e09af2daf5f23c14d780 # v2.86.1
9393
with:
9494
tool: cargo-semver-checks
9595

0 commit comments

Comments
 (0)