Skip to content

Commit 702f761

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/26226-hash-join-build-order
# Conflicts: # datafusion/physical-plan/src/joins/hash_join/exec.rs
2 parents 5d2d39d + 86d7a49 commit 702f761

734 files changed

Lines changed: 64877 additions & 19168 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/datafusion-ffi/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -197,21 +197,21 @@ Templates: `test_ffi_table_provider_local_bypass` and `test_round_trip_ffi_table
197197

198198
What unit tests catch: Rust-level correctness (translation logic, lifetime bugs, leaks under valgrind/miri, Send/Sync, error propagation, codec round-trips). What they **cannot** catch: real ABI bugs. Both producer and consumer share `#[repr(C)]` layout because they are the exact same struct definition in memory.
199199

200-
#### b. Cross-library integration tests (`tests/ffi_*.rs`, gated by the `integration-tests` feature)
200+
#### b. Cross-library integration tests (`integration-tests/tests/`, gated by the `integration-tests` feature)
201201

202-
The crate is published as `crate-type = ["cdylib", "rlib"]`. The integration tests in `datafusion/ffi/tests/` use `libloading` to `dlopen` the crate's own `cdylib` and call `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. The test executable links against the rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
202+
The integration tests live in the unpublished `datafusion-ffi-integration-tests` crate in `datafusion/ffi/integration-tests/`, so that `datafusion-ffi` itself is a plain rlib. That crate is built as a `cdylib` that exports `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern "C"` entry point defined in `src/tests/mod.rs` and gated by `#[cfg(feature = "integration-tests")]`. Its tests use `libloading` to `dlopen` that `cdylib`. The test executable links against the `datafusion-ffi` rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.
203203

204204
Run with:
205205

206206
```bash
207-
cargo test -p datafusion-ffi --features integration-tests
207+
cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests
208208
```
209209

210210
To add coverage for a new wrapper:
211211

212212
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
213213
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
214-
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
214+
3. **Add the test** in `integration-tests/tests/cases/<area>.rs`, registered in `cases/mod.rs`, under `mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
215215

216216
When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
217217

@@ -347,7 +347,7 @@ When reviewing a PR that touches `datafusion/ffi/`:
347347
7. **Stabby types** for strings/vecs; crate-local `FFI_Option`/`FFI_Result` for optional/fallible payloads?
348348
8. **Async** uses `FfiFuture` + `.into_ffi()`, never blocking?
349349
9. **Codec** present on any method that ships an `Expr` / plan?
350-
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `tests/ffi_*.rs` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi --features integration-tests` must pass before merging an ABI-affecting PR.
350+
10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `integration-tests/tests/` exist for any wrapper that takes/returns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests` must pass before merging an ABI-affecting PR.
351351
11. **No `datafusion` runtime dep** crept into `Cargo.toml`?
352352
12. **`Arc::clone(&x)`** everywhere — no implicit `x.clone()` on `Arc` (the lint will reject it but worth pre-flagging).
353353
13. **`cargo clippy --all-targets --all-features -- -D warnings`** clean on the crate?

‎.asf.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ github:
6969
- "cargo check datafusion-spark features"
7070
- "cargo test (amd64)"
7171
- "cargo test 'extended_tests' (amd64)"
72+
- "microbenchmark tests (amd64)"
7273
- "cargo test hash collisions (amd64)"
7374
- "cargo test datafusion-cli (amd64)"
7475
- "cargo examples (amd64)"
@@ -112,6 +113,43 @@ github:
112113
branch-55:
113114
required_pull_request_reviews:
114115
required_approving_review_count: 1
116+
117+
# GitHub merge queue settings for `main`.
118+
#
119+
# The merge queue is managed through rulesets
120+
# https://github.com/apache/infrastructure-asfyaml#rulesets
121+
#
122+
# Merge queue rules do not support wildcard ref patterns, so only the default
123+
# branch is covered; release branches keep plain branch protection above.
124+
rulesets:
125+
- name: Merge Queue
126+
target: branch
127+
enforcement: active
128+
conditions:
129+
ref_name:
130+
include:
131+
- "~DEFAULT_BRANCH"
132+
exclude: []
133+
rules:
134+
- type: merge_queue
135+
parameters:
136+
# Matches enabled_merge_buttons above.
137+
merge_method: SQUASH
138+
# Every entry's own build must be green, not only the head of the
139+
# group.
140+
grouping_strategy: ALLGREEN
141+
# How many queue entries are built and tested concurrently.
142+
max_entries_to_build: 5
143+
# How many green entries may land in one merge operation (note this
144+
# does not combine CI builds)
145+
max_entries_to_merge: 1
146+
min_entries_to_merge: 1
147+
min_entries_to_merge_wait_minutes: 5
148+
# How long the queue waits for a required check to report before
149+
# treating it as failed and evicting the entry. Set to 6h to handle
150+
# periods of high runner queue times. See
151+
# https://github.com/apache/datafusion/issues/26292
152+
check_response_timeout_minutes: 360
115153
pull_requests:
116154
# enable updating head branches of pull requests
117155
allow_update_branch: true

‎.github/actions/setup-rust-runtime/action.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@ runs:
3030
# See for more details: https://github.com/rust-lang/rust/issues/119560
3131
run: |
3232
echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
33-
echo "RUSTFLAGS=-C debuginfo=line-tables-only -C incremental=false" >> $GITHUB_ENV
33+
echo "RUSTFLAGS=-C debuginfo=line-tables-only" >> $GITHUB_ENV
34+
# rustc's -C incremental takes a directory, not a boolean.
35+
echo "CARGO_INCREMENTAL=0" >> $GITHUB_ENV
3436
# Work around intermittent "[16] Error in the HTTP2 framing layer"
3537
# failures from curl when cargo fetches crates from crates.io.
3638
# Disabling HTTP/2 multiplexing forces cargo to serialize requests,
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Install rustup
19+
description: 'Install rustup and the toolchain from rust-toolchain.toml on a runner without Rust, such as ubuntu-slim'
20+
runs:
21+
using: "composite"
22+
steps:
23+
- name: Install rustup and the toolchain
24+
shell: bash
25+
run: |
26+
RETRY=("ci/scripts/retry" timeout 120)
27+
"${RETRY[@]}" curl -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs
28+
sh "$RUNNER_TEMP/rustup-init.sh" -y --no-modify-path --profile minimal --default-toolchain none
29+
CARGO_BIN="${CARGO_HOME:-$HOME/.cargo}/bin"
30+
export PATH="$CARGO_BIN:$PATH"
31+
echo "$CARGO_BIN" >> "$GITHUB_PATH"
32+
# With no toolchain argument, rustup installs the toolchain and the
33+
# components that rust-toolchain.toml names.
34+
"${RETRY[@]}" rustup toolchain install
35+
- name: Configure rust runtime env
36+
uses: ./.github/actions/setup-rust-runtime

‎.github/dependabot.yml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,6 @@ updates:
6262
update-types:
6363
- "minor"
6464
- "patch"
65-
# dev/depcheck is a standalone crate excluded from the workspace, so it is
66-
# not covered by the workspace entry above
67-
- package-ecosystem: cargo
68-
directory: "/dev/depcheck"
69-
schedule:
70-
interval: weekly
71-
target-branch: main
72-
labels: [auto-dependencies]
7365
# the wasm test app has its own npm lockfile
7466
- package-ecosystem: npm
7567
directory: "/datafusion/wasmtest/datafusion-wasm-app"

‎.github/pull_request_template.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
> [!IMPORTANT]
2+
>
3+
> **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push.
4+
> See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).
5+
16
## Which issue does this PR close?
27

38
<!--
@@ -10,7 +15,7 @@ We generally require a GitHub issue to be filed for all bug fixes and enhancemen
1015

1116
<!--
1217
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
13-
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
18+
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
1419
1520
Please explain the problem you are trying to solve in terms of the user-visible
1621
behavior, rather than the implementation.

‎.github/workflows/audit.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,13 @@ permissions:
4141

4242
jobs:
4343
security_audit:
44-
runs-on: ubuntu-latest
44+
runs-on: ubuntu-slim
4545
steps:
4646
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
47+
- name: Setup Rust toolchain
48+
uses: ./.github/actions/setup-rustup
4749
- name: Install cargo-audit
48-
uses: taiki-e/install-action@861a07ce7084f55488e375df125cdc99bba60eb7 # v2.87.23
50+
uses: taiki-e/install-action@8cca0918f8471c33515202b73f56ab27f2bed283 # v2.87.29
4951
with:
5052
tool: cargo-audit
5153
- name: Run audit check

‎.github/workflows/breaking_changes_detector.yml‎

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
# Detect semver-incompatible (breaking) API changes in crates modified by a PR.
1919
#
2020
# Only public workspace crates that have file changes are checked.
21-
# Internal crates (benchmarks, test-utils, sqllogictest, doc) are excluded.
21+
# Crates with `publish = false` (such as benchmarks and test-utils) and
22+
# datafusion-sqllogictest are excluded.
2223
#
2324
# This workflow only runs cargo-semver-checks and uploads the result as an
2425
# artifact. The actual PR comment is posted by a companion workflow
@@ -54,57 +55,58 @@ jobs:
5455
name: Check semver
5556
runs-on: ubuntu-latest
5657
steps:
58+
# The checkout is GitHub's test merge commit of the PR into the base
59+
# branch. Depth 2 fetches that commit and its two parents: the base
60+
# branch commit the PR was merged into, and the PR head.
5761
- name: Checkout
5862
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
5963
with:
60-
fetch-depth: 0
64+
fetch-depth: 2
6165

62-
# `origin` may point at a fork (when a contributor runs this locally) or
63-
# at a stale ref. Fetch the base branch from the PR's upstream repo into
64-
# a dedicated `apache/<base>` ref so the baseline is unambiguous and the
65-
# same ref name works locally (`git remote add apache ...`) and in CI.
66-
- name: Fetch base branch
67-
env:
68-
BASE_REF: ${{ github.base_ref }}
69-
REPO: ${{ github.repository }}
70-
run: git fetch "https://github.com/${REPO}.git" "${BASE_REF}:refs/remotes/apache/${BASE_REF}"
66+
# Compare against the base branch commit the PR was merged into, not the
67+
# base branch's current tip, which may include changes the PR is not
68+
# built on.
69+
- name: Determine base commit
70+
id: base
71+
run: |
72+
BASE_SHA=$(git rev-parse HEAD^1)
73+
echo "sha=$BASE_SHA" >> "$GITHUB_OUTPUT"
74+
echo "Base commit: $BASE_SHA"
7175
7276
- name: Determine changed crates
7377
id: changed_crates
7478
env:
75-
BASE_REF: ${{ github.base_ref }}
79+
BASE_SHA: ${{ steps.base.outputs.sha }}
7680
run: |
77-
PACKAGES=$(ci/scripts/changed_crates.sh changed-crates "apache/${BASE_REF}")
81+
PACKAGES=$(ci/scripts/changed_crates.sh changed-crates "$BASE_SHA")
7882
echo "packages=$PACKAGES" >> "$GITHUB_OUTPUT"
7983
echo "Changed crates: $PACKAGES"
8084
81-
# `datafusion-substrait` (and crates that depend on it via sqllogictest)
82-
# have a build script that calls protoc, which is not preinstalled on
83-
# ubuntu-latest runners.
84-
- name: Install Protobuf Compiler
85+
# The `substrait` crate's build script runs protoc, which is not
86+
# preinstalled on ubuntu-latest runners.
87+
- name: Install protoc
8588
if: steps.changed_crates.outputs.packages != ''
86-
run: |
87-
sudo apt-get update
88-
sudo apt-get install -y protobuf-compiler
89+
uses: taiki-e/install-action@8cca0918f8471c33515202b73f56ab27f2bed283 # v2.87.29
90+
with:
91+
8992

9093
- name: Install cargo-semver-checks
9194
if: steps.changed_crates.outputs.packages != ''
92-
uses: taiki-e/install-action@861a07ce7084f55488e375df125cdc99bba60eb7 # v2.87.23
95+
uses: taiki-e/install-action@8cca0918f8471c33515202b73f56ab27f2bed283 # v2.87.29
9396
with:
9497
tool: cargo-semver-checks
9598

9699
- name: Run cargo-semver-checks
97100
id: check_semver
98101
if: steps.changed_crates.outputs.packages != ''
99102
env:
100-
BASE_REF: ${{ github.base_ref }}
103+
BASE_SHA: ${{ steps.base.outputs.sha }}
101104
PACKAGES: ${{ steps.changed_crates.outputs.packages }}
102105
run: |
103106
set +e
104107
# `tee` lets cargo's output stream live into the Actions log
105108
# while we also keep a copy for the PR comment.
106-
# Using `apache` remote here to point to the repository the pull request is against
107-
ci/scripts/changed_crates.sh semver-check "apache/${BASE_REF}" $PACKAGES \
109+
ci/scripts/changed_crates.sh semver-check "$BASE_SHA" $PACKAGES \
108110
2>&1 | tee /tmp/semver-output.txt
109111
EXIT_CODE=${PIPESTATUS[0]}
110112
# Pass the result through an output instead of failing the job:
@@ -137,7 +139,7 @@ jobs:
137139
138140
- name: Upload artifact
139141
if: always()
140-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
142+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
141143
with:
142144
name: semver-check-result
143145
path: semver-artifact/

‎.github/workflows/breaking_changes_detector_comment.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,14 +61,14 @@ jobs:
6161
comment-on-pr:
6262
name: Comment on pull request
6363
if: github.event.workflow_run.event == 'pull_request'
64-
runs-on: ubuntu-latest
64+
runs-on: ubuntu-slim
6565
# Scoped to the minimum needed to upsert/delete the sticky comment.
6666
permissions:
6767
actions: read
6868
pull-requests: write
6969
steps:
7070
- name: Download semver-check artifact
71-
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
71+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
7272
with:
7373
name: semver-check-result
7474
run-id: ${{ github.event.workflow_run.id }}

‎.github/workflows/codeql.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,11 @@ jobs:
4545
persist-credentials: false
4646

4747
- name: Initialize CodeQL
48-
uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4
48+
uses: github/codeql-action/init@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4
4949
with:
5050
languages: actions
5151

5252
- name: Perform CodeQL Analysis
53-
uses: github/codeql-action/analyze@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4
53+
uses: github/codeql-action/analyze@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4
5454
with:
5555
category: "/language:actions"

0 commit comments

Comments
 (0)