Skip to content

Commit af8fc12

Browse files
authored
Merge branch 'main' into ji/len-fn-placement-move-down
2 parents 70e354a + 0576a0b commit af8fc12

905 files changed

Lines changed: 47134 additions & 8361 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/add-benchmarks/SKILL.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
name: add-benchmarks
3+
description: Guidelines for designing or extending DataFusion benchmarks.
4+
---
5+
6+
# Benchmark Design
7+
8+
Follow these guidelines when designing or adding a new benchmark.
9+
10+
## Design principles
11+
12+
### Use the highest-level interface
13+
14+
Use the highest-level interface possible, preferably SQL, while keeping the work
15+
around the operator being measured as cheap as possible. When benchmarking a
16+
function, exercise its evaluation path instead of benchmarking internal utility
17+
functions in isolation.
18+
19+
This makes benchmarks easier to maintain and helps assess how much an optimization
20+
matters to end-to-end runtime. It also helps avoid spending time optimizing code
21+
that accounts for only a small fraction of the total runtime.
22+
23+
Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion.
24+
25+
### Vary the key workload axes
26+
27+
First identify the key axes to vary. For example, for a join benchmark:
28+
29+
- Input size on each side.
30+
- Join-filter selectivity.
31+
- ...
32+
33+
Then choose benchmark cases that exercise representative variations. Full
34+
combinatorial coverage is unnecessary; focus on typical workloads that reflect
35+
real use cases.
36+
37+
When adding benchmark queries, simply tag each query with the decision made
38+
for every axis. For example, for a join benchmark with input sizes/filter selectivity
39+
to tune:
40+
41+
```sql
42+
-- Q1: Small left input, large right input; 0.1% of pairs match.
43+
SELECT *
44+
FROM generate_series(1, 100) AS l
45+
JOIN generate_series(1, 100000) AS r
46+
ON (l.value + r.value) % 1000 = 0;
47+
48+
-- Q2: Medium inputs on both sides; no filter.
49+
SELECT *
50+
FROM generate_series(1, 1000) AS l
51+
CROSS JOIN generate_series(1, 1000) AS r;
52+
```
53+
54+
## SQL benchmarks
55+
56+
For implementation details, see the
57+
[SQL benchmark README](../../../benchmarks/sql_benchmarks/README.md).
58+
59+
1. **Keep other operators cheap.**
60+
61+
When a SQL benchmark targets a specific operator, keep the work done by
62+
other operators as lightweight as possible. For example, use a data source
63+
such as `generate_series()` instead of a Parquet scan so scan overhead does
64+
not dominate the measurement. See the [`nlj` benchmark](../../../benchmarks/sql_benchmarks/nlj/) for examples.
65+
66+
2. **Integrate with the top-level benchmark script.**
67+
68+
Ensure the benchmark can be prepared and run through `bench.sh`:
69+
70+
```bash
71+
# Run from the benchmarks directory.
72+
73+
# Generate any required dataset.
74+
./bench.sh data new_bench
75+
76+
# Run the benchmark.
77+
./bench.sh run new_bench
78+
```
79+
80+
3. **Keep query runtimes practical.**
81+
82+
Tune the workload so each query takes roughly a few seconds per execution.
83+
This helps reduce the relative impact of timing noise while keeping the
84+
suite practical to run.

‎.asf.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,8 @@ github:
6868
- "cargo check datafusion-functions features"
6969
- "cargo check datafusion-spark features"
7070
- "cargo test (amd64)"
71+
- "cargo test 'extended_tests' (amd64)"
72+
- "cargo test hash collisions (amd64)"
7173
- "cargo test datafusion-cli (amd64)"
7274
- "cargo examples (amd64)"
7375
- "cargo test doc (amd64)"
@@ -76,6 +78,7 @@ github:
7678
- "verify benchmark results (amd64)"
7779
- "Run sqllogictest with Postgres runner"
7880
- "Run sqllogictest in Substrait round-trip mode"
81+
- "Run sqllogictests with the sqlite test suite"
7982
- "cargo test (macos-aarch64)"
8083
- "Verify Vendored Code"
8184
- "Check cargo fmt"

‎.github/workflows/audit.yml‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,8 @@ jobs:
4545
steps:
4646
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4747
- name: Install cargo-audit
48-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
48+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
4949
with:
5050
tool: cargo-audit
5151
- name: Run audit check
52-
# Note: you can ignore specific RUSTSEC issues using the `--ignore` flag ,for example:
53-
# run: cargo audit --ignore RUSTSEC-2026-0001
54-
# TODO: remove once object_store upgrades to quick-xml >= 0.41.0
55-
# https://github.com/apache/datafusion/issues/23297
56-
run: cargo audit --ignore RUSTSEC-2026-0194 --ignore RUSTSEC-2026-0195
52+
run: bash ci/scripts/security_audit.sh

‎.github/workflows/breaking_changes_detector.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ jobs:
8989
9090
- name: Install cargo-semver-checks
9191
if: steps.changed_crates.outputs.packages != ''
92-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
92+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
9393
with:
9494
tool: cargo-semver-checks
9595

‎.github/workflows/codeql.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,11 @@ jobs:
4545
persist-credentials: false
4646

4747
- name: Initialize CodeQL
48-
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
48+
uses: github/codeql-action/init@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
4949
with:
5050
languages: actions
5151

5252
- name: Perform CodeQL Analysis
53-
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
53+
uses: github/codeql-action/analyze@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
5454
with:
5555
category: "/language:actions"

‎.github/workflows/dependencies.yml‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ concurrency:
2424
on:
2525
push:
2626
branches-ignore:
27+
# Commits reach main through the merge queue, which runs this workflow at
28+
# the same SHA, so do not repeat the jobs after the merge. Release
29+
# branches and every other branch keep their post-push runs.
30+
- 'main'
2731
- 'gh-readonly-queue/**'
2832
- 'dependabot/**'
2933
pull_request:
@@ -51,9 +55,7 @@ jobs:
5155
with:
5256
rust-version: stable
5357
- name: Check dependencies
54-
working-directory: dev/depcheck
55-
run: |
56-
cargo run --locked
58+
run: bash ci/scripts/check_circular_dependencies.sh
5759

5860
detect-unused-dependencies:
5961
name: Detect Unused Dependencies
@@ -62,9 +64,13 @@ jobs:
6264
image: amd64/rust
6365
steps:
6466
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
67+
- name: Load tool versions
68+
run: |
69+
. ci/scripts/utils/tool_versions.sh
70+
echo "CARGO_MACHETE_VERSION=${CARGO_MACHETE_VERSION}" >> "$GITHUB_ENV"
6571
- name: Install cargo-machete
66-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
72+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
6773
with:
68-
tool: cargo-machete@0.9
74+
tool: cargo-machete@${{ env.CARGO_MACHETE_VERSION }}
6975
- name: Detect unused dependencies
70-
run: cargo machete --with-metadata
76+
run: bash ci/scripts/check_unused_dependencies.sh

‎.github/workflows/dev.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ name: Dev
1919
on:
2020
push:
2121
branches-ignore:
22+
# Commits reach main through the merge queue, which runs this workflow at
23+
# the same SHA, so do not repeat the jobs after the merge. Release
24+
# branches and every other branch keep their post-push runs.
25+
- 'main'
2226
- 'gh-readonly-queue/**'
2327
- 'dependabot/**'
2428
pull_request:
@@ -38,7 +42,7 @@ jobs:
3842
steps:
3943
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4044
- name: Install HawkEye
41-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
45+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
4246
with:
4347
4448
- name: Run license header check
@@ -66,7 +70,7 @@ jobs:
6670
source ci/scripts/utils/tool_versions.sh
6771
echo "LYCHEE_VERSION=${LYCHEE_VERSION}" >> "$GITHUB_ENV"
6872
- name: Install lychee
69-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
73+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
7074
with:
7175
tool: lychee@${{ env.LYCHEE_VERSION }}
7276
- name: Run markdown link check
@@ -91,7 +95,7 @@ jobs:
9195
# it may cause checks to fail more often.
9296
# We can upgrade it manually once a while.
9397
- name: Install typos
94-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
98+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
9599
with:
96100
97101
- name: Run typos check

‎.github/workflows/docs.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ jobs:
4343
path: asf-site
4444

4545
- name: Setup uv
46-
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
46+
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
4747

4848
- name: Install dependencies
4949
run: uv sync --package datafusion-docs
@@ -53,7 +53,7 @@ jobs:
5353
sudo apt-get update
5454
sudo apt-get install -y graphviz
5555
- name: Install cargo-depgraph
56-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
56+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
5757
with:
5858
5959

‎.github/workflows/docs_pr.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ jobs:
5050
submodules: true
5151
fetch-depth: 1
5252
- name: Setup uv
53-
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
53+
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
5454
- name: Install doc dependencies
5555
run: uv sync --package datafusion-docs
5656
- name: Install Graphviz
@@ -59,7 +59,7 @@ jobs:
5959
sudo apt-get update
6060
sudo apt-get install -y graphviz
6161
- name: Install cargo-depgraph
62-
uses: taiki-e/install-action@7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c # v2.87.6
62+
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
6363
with:
6464
6565
- name: Build docs html and check for warnings

‎.github/workflows/extended.yml‎

Lines changed: 28 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -15,37 +15,32 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18+
# Each test command below is implemented with `cargo xtask` so developers can
19+
# run the same suite locally. Append `--explain` to print the underlying
20+
# `cargo test` invocation without running it. Job setup, the clean working tree
21+
# check, and `cargo clean` remain in this workflow.
22+
#
23+
# See `xtask/README.md` for details about DataFusion's `xtask` tooling.
24+
# Track progress at https://github.com/apache/datafusion/issues/24487.
25+
1826
name: Datafusion extended tests
1927

2028
concurrency:
2129
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
2230
cancel-in-progress: true
2331

24-
# https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#running-your-pull_request-workflow-when-a-pull-request-merges
25-
#
26-
# These jobs are not run as part of PR checks as they are time-consuming
27-
# and should not fail often.
28-
#
29-
# We still run them as they provide important coverage to ensure correctness
30-
# in the (very rare) event of a hash failure or sqlite library query failure.
32+
# Run these required checks in the merge queue before merging into main.
33+
# On PR updates, skip the jobs so their required checks allow queue entry
34+
# without running the expensive suites. Do not add pull_request path filters:
35+
# a skipped workflow would leave required checks pending.
3136
on:
37+
pull_request:
38+
merge_group:
3239
push:
3340
branches:
34-
- main
35-
# support extended test suite for release candidate branches,
36-
# it is not expected to have many changes in these branches,
37-
# so running extended tests is not a burden
41+
# Release branches retain post-push coverage. main is covered by the
42+
# merge queue, so do not repeat the tests after merging.
3843
- 'branch-*'
39-
# Also run for changes to some critical areas that are most likely
40-
# to trigger errors in extended tests
41-
pull_request:
42-
branches: [ '**' ]
43-
paths:
44-
- 'datafusion/physical*/**/*.rs'
45-
- 'datafusion/expr*/**/*.rs'
46-
- 'datafusion/optimizer/**/*.rs'
47-
- 'datafusion/sql/**/*.rs'
48-
- 'datafusion-testing'
4944
workflow_dispatch:
5045
inputs:
5146
pr_head_sha:
@@ -60,13 +55,15 @@ jobs:
6055
# Run extended tests (with feature 'extended_tests')
6156
linux-test-extended:
6257
name: cargo test 'extended_tests' (amd64)
58+
# Long-running job: run in the merge queue, on release pushes, or manually.
59+
if: github.event_name != 'pull_request'
6360
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=32,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
6461
# note: do not use amd/rust container to preserve disk space
6562
steps:
6663
- uses: runs-on/action@efac073ea2507ec18797de3a81704201ade11d9d # v2.3.1
6764
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
6865
with:
69-
ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push
66+
ref: ${{ github.event.inputs.pr_head_sha }} # otherwise use the event's commit, including the merge group
7067
submodules: true
7168
fetch-depth: 1
7269
- name: Free Disk Space (Ubuntu)
@@ -86,22 +83,7 @@ jobs:
8683
run: |
8784
df -h
8885
- name: Run tests (excluding doctests)
89-
env:
90-
RUST_BACKTRACE: 1
91-
# Run more random scenarios of the spill pool fuzzer than the
92-
# default suite does (about a minute).
93-
DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000
94-
run: |
95-
cargo test \
96-
--profile ci \
97-
--exclude datafusion-examples \
98-
--exclude datafusion-benchmarks \
99-
--exclude datafusion-cli \
100-
--workspace \
101-
--lib \
102-
--tests \
103-
--bins \
104-
--features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
86+
run: cargo xtask ci step test extended
10587
- parallel:
10688
- name: Verify Working Directory Clean
10789
run: git diff --exit-code
@@ -111,14 +93,16 @@ jobs:
11193
# Check answers are correct when hash values collide
11294
hash-collisions:
11395
name: cargo test hash collisions (amd64)
96+
# Long-running job: run in the merge queue, on release pushes, or manually.
97+
if: github.event_name != 'pull_request'
11498
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
11599
container:
116100
image: amd64/rust
117101
steps:
118102
- uses: runs-on/action@efac073ea2507ec18797de3a81704201ade11d9d # v2.3.1
119103
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
120104
with:
121-
ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push
105+
ref: ${{ github.event.inputs.pr_head_sha }} # otherwise use the event's commit, including the merge group
122106
submodules: true
123107
fetch-depth: 1
124108
- name: Setup Rust toolchain
@@ -127,12 +111,14 @@ jobs:
127111
rust-version: stable
128112
- name: Run tests
129113
run: |
114+
cargo xtask ci step test hash-collisions
130115
cd datafusion
131-
cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --exclude datafusion-cli --workspace --lib --tests --features=force_hash_collisions,avro
132116
cargo clean
133117
134118
sqllogictest-sqlite:
135119
name: "Run sqllogictests with the sqlite test suite"
120+
# Long-running job: run in the merge queue, on release pushes, or manually.
121+
if: github.event_name != 'pull_request'
136122
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=32,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
137123
container:
138124
image: amd64/rust
@@ -141,13 +127,12 @@ jobs:
141127
- parallel:
142128
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
143129
with:
144-
ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push
130+
ref: ${{ github.event.inputs.pr_head_sha }} # otherwise use the event's commit, including the merge group
145131
submodules: true
146132
fetch-depth: 1
147133
# Don't use setup-builder to avoid configuring RUST_BACKTRACE which is expensive
148134
- name: Install protobuf compiler
149135
run: |
150136
apt-get update && apt-get install -y protobuf-compiler
151137
- name: Run sqllogictest
152-
run: |
153-
cargo test --features backtrace,parquet_encryption --profile ci-optimized --test sqllogictests -- --include-sqlite
138+
run: cargo xtask ci step test sqlite

0 commit comments

Comments
 (0)