Skip to content

Commit c466a17

Browse files
committed
Merge main and address spill review feedback
Preserve max-row spill statistics and intermediate sizing from main. Split output workspace before retaining the input grant, and retain construction workspace through intermediate spills while final aggregate replay releases it. Tighten the two spill assertions and align new aggregate memory-call snapshots with strict output construction accounting without changing pressure limits.
2 parents 850a3b2 + 350ca76 commit c466a17

1,091 files changed

Lines changed: 131103 additions & 32055 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/datafusion-ffi/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -197,21 +197,21 @@ Templates: `test_ffi_table_provider_local_bypass` and `test_round_trip_ffi_table
197197

198198
What unit tests catch: Rust-level correctness (translation logic, lifetime bugs, leaks under valgrind/miri, Send/Sync, error propagation, codec round-trips). What they **cannot** catch: real ABI bugs. Both producer and consumer share `#[repr(C)]` layout because they are the exact same struct definition in memory.
199199

200-
#### b. Cross-library integration tests (`tests/ffi_*.rs`, gated by the `integration-tests` feature)
200+
#### b. Cross-library integration tests (`integration-tests/tests/`, gated by the `integration-tests` feature)
201201

202-
The crate is published as `crate-type = ["cdylib", "rlib"]`. The integration tests in `datafusion/ffi/tests/` use `libloading` to `dlopen` the crate's own `cdylib` and call `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. The test executable links against the rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
202+
The integration tests live in the unpublished `datafusion-ffi-integration-tests` crate in `datafusion/ffi/integration-tests/`, so that `datafusion-ffi` itself is a plain rlib. That crate is built as a `cdylib` that exports `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. Its tests use `libloading` to `dlopen` that `cdylib`. The test executable links against the `datafusion-ffi` rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
203203

204204
Run with:
205205

206206
```bash
207-
cargo test -p datafusion-ffi --features integration-tests
207+
cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests
208208
```
209209

210210
To add coverage for a new wrapper:
211211

212212
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
213213
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
214-
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
214+
3. **Add the test** in `integration-tests/tests/cases/<area>.rs`, registered in `cases/mod.rs`, under `mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
215215

216216
When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
217217

@@ -347,7 +347,7 @@ When reviewing a PR that touches `datafusion/ffi/`:
347347
7. **Stabby types** for strings/vecs; crate-local `FFI_Option`/`FFI_Result` for optional/fallible payloads?
348348
8. **Async** uses `FfiFuture` + `.into_ffi()`, never blocking?
349349
9. **Codec** present on any method that ships an `Expr` / plan?
350-
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `tests/ffi_*.rs` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi --features integration-tests` must pass before merging an ABI-affecting PR.
350+
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `integration-tests/tests/` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests` must pass before merging an ABI-affecting PR.
351351
11. **No `datafusion` runtime dep** crept into `Cargo.toml`?
352352
12. **`Arc::clone(&x)`** everywhere — no implicit `x.clone()` on `Arc` (the lint will reject it but worth pre-flagging).
353353
13. **`cargo clippy --all-targets --all-features -- -D warnings`** clean on the crate?

‎.asf.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ github:
6969
- "cargo check datafusion-spark features"
7070
- "cargo test (amd64)"
7171
- "cargo test 'extended_tests' (amd64)"
72+
- "microbenchmark tests (amd64)"
7273
- "cargo test hash collisions (amd64)"
7374
- "cargo test datafusion-cli (amd64)"
7475
- "cargo examples (amd64)"
@@ -87,6 +88,7 @@ github:
8788
- "check Cargo.toml formatting"
8889
- "check configs.md and ***_functions.md is up-to-date"
8990
- "check example README is up-to-date"
91+
- "Check datafusion crates used in datafusion-examples"
9092
- "Verify MSRV (Min Supported Rust Version)"
9193
# needs to be updated as part of the release process
9294
# .asf.yaml doesn't support wildcard branch protection rules, only exact branch names
@@ -111,12 +113,56 @@ github:
111113
branch-55:
112114
required_pull_request_reviews:
113115
required_approving_review_count: 1
116+
117+
# GitHub merge queue settings for `main`.
118+
#
119+
# The merge queue is managed through rulesets
120+
# https://github.com/apache/infrastructure-asfyaml#rulesets
121+
#
122+
# Merge queue rules do not support wildcard ref patterns, so only the default
123+
# branch is covered; release branches keep plain branch protection above.
124+
rulesets:
125+
- name: Merge Queue
126+
target: branch
127+
enforcement: active
128+
conditions:
129+
ref_name:
130+
include:
131+
- "~DEFAULT_BRANCH"
132+
exclude: []
133+
rules:
134+
- type: merge_queue
135+
parameters:
136+
# Matches enabled_merge_buttons above.
137+
merge_method: SQUASH
138+
# Every entry's own build must be green, not only the head of the
139+
# group.
140+
grouping_strategy: ALLGREEN
141+
# How many queue entries are built and tested concurrently.
142+
max_entries_to_build: 5
143+
# How many green entries may land in one merge operation (note this
144+
# does not combine CI builds)
145+
max_entries_to_merge: 1
146+
min_entries_to_merge: 1
147+
min_entries_to_merge_wait_minutes: 5
148+
# How long the queue waits for a required check to report before
149+
# treating it as failed and evicting the entry. Set to 6h to handle
150+
# periods of high runner queue times. See
151+
# https://github.com/apache/datafusion/issues/26292
152+
check_response_timeout_minutes: 360
114153
pull_requests:
115154
# enable updating head branches of pull requests
116155
allow_update_branch: true
117156
allow_auto_merge: true
118157
# auto-delete head branches after being merged
119158
del_branch_on_merge: true
159+
# Limit open pull requests per user without write access.
160+
#
161+
# With coding agents, the number of open PRs now far exceeds our review capacity.
162+
# Setting a limit helps reviewers focus on fewer PRs at a time.
163+
creation_cap:
164+
enabled: true
165+
max_open_pull_requests: 3
120166

121167
# publishes the content of the `asf-site` branch to
122168
# https://datafusion.apache.org/

‎.envrc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
source_up_if_exists
2+
if command -v nix >/dev/null 2>&1; then
3+
use flake
4+
fi
5+
source_env_if_exists .envrc.local

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,12 @@ body:
2626
label: Additional context
2727
placeholder: >
2828
Add any other context about the problem here.
29+
- type: dropdown
30+
attributes:
31+
label: Willingness to contribute
32+
description: >
33+
The Apache DataFusion community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Apache DataFusion codebase?
34+
options:
35+
- I plan to contribute a fix for this bug independently
36+
- I would like to contribute a fix for this bug, but request guidance from the DataFusion community
37+
- I cannot contribute a fix for this bug at this time

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,12 @@ body:
2525
label: Additional context
2626
placeholder: >
2727
Add any other context or screenshots about the feature request here.
28+
- type: dropdown
29+
attributes:
30+
label: Willingness to contribute
31+
description: >
32+
The Apache DataFusion community encourages feature contributions. Would you or another member of your organization be willing to contribute to this feature for the Apache DataFusion codebase?
33+
options:
34+
- I plan to contribute to this feature independently
35+
- I would like to contribute to this feature, but request guidance from the DataFusion community
36+
- I cannot contribute to this feature at this time

‎.github/actions/setup-rust-runtime/action.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@ runs:
3030
# See for more details: https://github.com/rust-lang/rust/issues/119560
3131
run: |
3232
echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
33-
echo "RUSTFLAGS=-C debuginfo=line-tables-only -C incremental=false" >> $GITHUB_ENV
33+
echo "RUSTFLAGS=-C debuginfo=line-tables-only" >> $GITHUB_ENV
34+
# rustc's -C incremental takes a directory, not a boolean.
35+
echo "CARGO_INCREMENTAL=0" >> $GITHUB_ENV
3436
# Work around intermittent "[16] Error in the HTTP2 framing layer"
3537
# failures from curl when cargo fetches crates from crates.io.
3638
# Disabling HTTP/2 multiplexing forces cargo to serialize requests,
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Install rustup
19+
description: 'Install rustup and the toolchain from rust-toolchain.toml on a runner without Rust, such as ubuntu-slim'
20+
runs:
21+
using: "composite"
22+
steps:
23+
- name: Install rustup and the toolchain
24+
shell: bash
25+
run: |
26+
RETRY=("ci/scripts/retry" timeout 120)
27+
"${RETRY[@]}" curl -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs
28+
sh "$RUNNER_TEMP/rustup-init.sh" -y --no-modify-path --profile minimal --default-toolchain none
29+
CARGO_BIN="${CARGO_HOME:-$HOME/.cargo}/bin"
30+
export PATH="$CARGO_BIN:$PATH"
31+
echo "$CARGO_BIN" >> "$GITHUB_PATH"
32+
# With no toolchain argument, rustup installs the toolchain and the
33+
# components that rust-toolchain.toml names.
34+
"${RETRY[@]}" rustup toolchain install
35+
- name: Configure rust runtime env
36+
uses: ./.github/actions/setup-rust-runtime

‎.github/dependabot.yml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,6 @@ updates:
6262
update-types:
6363
- "minor"
6464
- "patch"
65-
# dev/depcheck is a standalone crate excluded from the workspace, so it is
66-
# not covered by the workspace entry above
67-
- package-ecosystem: cargo
68-
directory: "/dev/depcheck"
69-
schedule:
70-
interval: weekly
71-
target-branch: main
72-
labels: [auto-dependencies]
7365
# the wasm test app has its own npm lockfile
7466
- package-ecosystem: npm
7567
directory: "/datafusion/wasmtest/datafusion-wasm-app"

‎.github/pull_request_template.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
> [!IMPORTANT]
2+
>
3+
> **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push.
4+
> See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).
5+
16
## Which issue does this PR close?
27

38
<!--
@@ -10,7 +15,7 @@ We generally require a GitHub issue to be filed for all bug fixes and enhancemen
1015

1116
<!--
1217
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
13-
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
18+
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
1419
1520
Please explain the problem you are trying to solve in terms of the user-visible
1621
behavior, rather than the implementation.

‎.github/workflows/audit.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,13 @@ permissions:
4141

4242
jobs:
4343
security_audit:
44-
runs-on: ubuntu-latest
44+
runs-on: ubuntu-slim
4545
steps:
4646
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
47+
- name: Setup Rust toolchain
48+
uses: ./.github/actions/setup-rustup
4749
- name: Install cargo-audit
48-
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
50+
uses: taiki-e/install-action@8cca0918f8471c33515202b73f56ab27f2bed283 # v2.87.29
4951
with:
5052
tool: cargo-audit
5153
- name: Run audit check

0 commit comments

Comments
 (0)