Skip to content

Commit 9fffb0a

Browse files
committed
Merge upstream while retaining shared UNNEST null handling
Keep the current map lookup implementation and reuse the shared parent-null helper. The merged source passes workspace tests, strict Clippy, formatting and the real SQL regressions. Signed-off-by: 1fanwang <[email protected]>
2 parents d15680f + d477a2b commit 9fffb0a

280 files changed

Lines changed: 13833 additions & 2600 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/add-benchmarks/SKILL.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
name: add-benchmarks
3+
description: Guidelines for designing or extending DataFusion benchmarks.
4+
---
5+
6+
# Benchmark Design
7+
8+
Follow these guidelines when designing or adding a new benchmark.
9+
10+
## Design principles
11+
12+
### Use the highest-level interface
13+
14+
Use the highest-level interface possible, preferably SQL, while keeping the work
15+
around the operator being measured as cheap as possible. When benchmarking a
16+
function, exercise its evaluation path instead of benchmarking internal utility
17+
functions in isolation.
18+
19+
This makes benchmarks easier to maintain and helps assess how much an optimization
20+
matters to end-to-end runtime. It also helps avoid spending time optimizing code
21+
that accounts for only a small fraction of the total runtime.
22+
23+
Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion.
24+
25+
### Vary the key workload axes
26+
27+
First identify the key axes to vary. For example, for a join benchmark:
28+
29+
- Input size on each side.
30+
- Join-filter selectivity.
31+
- ...
32+
33+
Then choose benchmark cases that exercise representative variations. Full
34+
combinatorial coverage is unnecessary; focus on typical workloads that reflect
35+
real use cases.
36+
37+
When adding benchmark queries, simply tag each query with the decision made
38+
for every axis. For example, for a join benchmark with input sizes/filter selectivity
39+
to tune:
40+
41+
```sql
42+
-- Q1: Small left input, large right input; 0.1% of pairs match.
43+
SELECT *
44+
FROM generate_series(1, 100) AS l
45+
JOIN generate_series(1, 100000) AS r
46+
ON (l.value + r.value) % 1000 = 0;
47+
48+
-- Q2: Medium inputs on both sides; no filter.
49+
SELECT *
50+
FROM generate_series(1, 1000) AS l
51+
CROSS JOIN generate_series(1, 1000) AS r;
52+
```
53+
54+
## SQL benchmarks
55+
56+
For implementation details, see the
57+
[SQL benchmark README](../../../benchmarks/sql_benchmarks/README.md).
58+
59+
1. **Keep other operators cheap.**
60+
61+
When a SQL benchmark targets a specific operator, keep the work done by
62+
other operators as lightweight as possible. For example, use a data source
63+
such as `generate_series()` instead of a Parquet scan so scan overhead does
64+
not dominate the measurement. See the [`nlj` benchmark](../../../benchmarks/sql_benchmarks/nlj/) for examples.
65+
66+
2. **Integrate with the top-level benchmark script.**
67+
68+
Ensure the benchmark can be prepared and run through `bench.sh`:
69+
70+
```bash
71+
# Run from the benchmarks directory.
72+
73+
# Generate any required dataset.
74+
./bench.sh data new_bench
75+
76+
# Run the benchmark.
77+
./bench.sh run new_bench
78+
```
79+
80+
3. **Keep query runtimes practical.**
81+
82+
Tune the workload so each query takes roughly a few seconds per execution.
83+
This helps reduce the relative impact of timing noise while keeping the
84+
suite practical to run.

‎.github/workflows/dependencies.yml‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ concurrency:
2424
on:
2525
push:
2626
branches-ignore:
27+
# Commits reach main through the merge queue, which runs this workflow at
28+
# the same SHA, so do not repeat the jobs after the merge. Release
29+
# branches and every other branch keep their post-push runs.
30+
- 'main'
2731
- 'gh-readonly-queue/**'
2832
- 'dependabot/**'
2933
pull_request:
@@ -51,9 +55,7 @@ jobs:
5155
with:
5256
rust-version: stable
5357
- name: Check dependencies
54-
working-directory: dev/depcheck
55-
run: |
56-
cargo run --locked
58+
run: bash ci/scripts/check_circular_dependencies.sh
5759

5860
detect-unused-dependencies:
5961
name: Detect Unused Dependencies
@@ -62,9 +64,13 @@ jobs:
6264
image: amd64/rust
6365
steps:
6466
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
67+
- name: Load tool versions
68+
run: |
69+
. ci/scripts/utils/tool_versions.sh
70+
echo "CARGO_MACHETE_VERSION=${CARGO_MACHETE_VERSION}" >> "$GITHUB_ENV"
6571
- name: Install cargo-machete
6672
uses: taiki-e/install-action@3f74d7c16a4242f1c95561e98edc25d36adb4375 # v2.87.12
6773
with:
68-
tool: cargo-machete@0.9
74+
tool: cargo-machete@${{ env.CARGO_MACHETE_VERSION }}
6975
- name: Detect unused dependencies
70-
run: cargo machete --with-metadata
76+
run: bash ci/scripts/check_unused_dependencies.sh

‎.github/workflows/dev.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ name: Dev
1919
on:
2020
push:
2121
branches-ignore:
22+
# Commits reach main through the merge queue, which runs this workflow at
23+
# the same SHA, so do not repeat the jobs after the merge. Release
24+
# branches and every other branch keep their post-push runs.
25+
- 'main'
2226
- 'gh-readonly-queue/**'
2327
- 'dependabot/**'
2428
pull_request:

‎.github/workflows/rust.yml‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,17 @@
3030
# It is being rolled out incrementally across all workflows.
3131
# Track progress at https://github.com/apache/datafusion/issues/24487.
3232
#
33+
# # Pushes to `main`
34+
#
35+
# Commits reach upstream `main` through the merge queue, which runs this
36+
# workflow at the same SHA. A push to upstream `main` therefore runs only the
37+
# jobs that save caches, publish coverage, or run the non-required FFI check.
38+
# The other jobs carry an `if:` that skips them for that event alone. PRs, the
39+
# merge queue, `workflow_dispatch`, and every other branch run all jobs. A
40+
# direct push that bypasses the queue is not re-checked; dispatch the workflow
41+
# on `main` for a full run. `ci/scripts/check_asf_yaml_status_checks.py` keeps
42+
# the skipped set explicit.
43+
#
3344
name: Rust
3445

3546
concurrency:
@@ -80,6 +91,7 @@ jobs:
8091
run: cargo xtask ci step check workspace
8192
- name: Archive Cargo check artifacts
8293
id: archive-check
94+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
8395
continue-on-error: true
8496
shell: bash
8597
run: |
@@ -91,7 +103,7 @@ jobs:
91103
--exclude='registry/src/*/*/false' \
92104
-cf "$RUNNER_TEMP/cargo-check/check-registry.tar" -C "$CARGO_HOME" registry
93105
- name: Upload Cargo check artifacts
94-
if: steps.archive-check.outcome == 'success'
106+
if: ${{ steps.archive-check.outcome == 'success' && !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
95107
continue-on-error: true
96108
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
97109
with:
@@ -109,6 +121,7 @@ jobs:
109121
linux-datafusion-common-features:
110122
name: cargo check datafusion-common features
111123
needs: linux-build-lib
124+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
112125
runs-on: ubuntu-latest
113126
container:
114127
image: amd64/rust
@@ -139,6 +152,7 @@ jobs:
139152
linux-datafusion-substrait-features:
140153
name: cargo check datafusion-substrait features
141154
needs: linux-build-lib
155+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
142156
env:
143157
CARGO_INCREMENTAL: "0"
144158
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -185,6 +199,7 @@ jobs:
185199
linux-datafusion-proto-features:
186200
name: cargo check datafusion-proto features
187201
needs: linux-build-lib
202+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
188203
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
189204
container:
190205
image: amd64/rust
@@ -247,6 +262,7 @@ jobs:
247262
linux-cargo-check-datafusion:
248263
name: cargo check datafusion features
249264
needs: linux-build-lib
265+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
250266
env:
251267
CARGO_INCREMENTAL: "0"
252268
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -367,6 +383,7 @@ jobs:
367383
linux-cargo-check-datafusion-spark:
368384
name: cargo check datafusion-spark features
369385
needs: linux-build-lib
386+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
370387
runs-on: ubuntu-latest
371388
container:
372389
image: amd64/rust
@@ -448,6 +465,7 @@ jobs:
448465
linux-test-datafusion-cli:
449466
name: cargo test datafusion-cli (amd64)
450467
needs: linux-build-lib
468+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
451469
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
452470
steps:
453471
- uses: runs-on/action@efac073ea2507ec18797de3a81704201ade11d9d # v2.3.1
@@ -525,6 +543,7 @@ jobs:
525543
linux-test-doc:
526544
name: cargo test doc (amd64)
527545
needs: linux-build-lib
546+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
528547
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
529548
container:
530549
image: amd64/rust
@@ -547,6 +566,7 @@ jobs:
547566
linux-rustdoc:
548567
name: cargo doc
549568
needs: linux-build-lib
569+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
550570
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
551571
container:
552572
image: amd64/rust
@@ -562,6 +582,7 @@ jobs:
562582

563583
linux-wasm-pack:
564584
name: build and run with wasm-pack
585+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
565586
runs-on: ubuntu-24.04
566587
steps:
567588
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -589,6 +610,7 @@ jobs:
589610
verify-benchmark-results:
590611
name: verify benchmark results (amd64)
591612
needs: linux-build-lib
613+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
592614
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
593615
container:
594616
image: amd64/rust
@@ -621,6 +643,7 @@ jobs:
621643
sqllogictest-postgres:
622644
name: "Run sqllogictest with Postgres runner"
623645
needs: linux-build-lib
646+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
624647
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
625648
container:
626649
image: amd64/rust
@@ -658,6 +681,7 @@ jobs:
658681
sqllogictest-substrait:
659682
name: "Run sqllogictest in Substrait round-trip mode"
660683
needs: linux-build-lib
684+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
661685
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
662686
container:
663687
image: amd64/rust
@@ -718,6 +742,7 @@ jobs:
718742

719743
vendor:
720744
name: Verify Vendored Code
745+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
721746
runs-on: ubuntu-latest
722747
container:
723748
image: amd64/rust
@@ -735,6 +760,7 @@ jobs:
735760

736761
check-fmt:
737762
name: Check cargo fmt
763+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
738764
runs-on: ubuntu-latest
739765
container:
740766
image: amd64/rust
@@ -750,6 +776,7 @@ jobs:
750776
751777
check-workflow-tool-installs:
752778
name: Check GitHub Actions install tooling
779+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
753780
runs-on: ubuntu-latest
754781
steps:
755782
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -787,6 +814,7 @@ jobs:
787814
cargo-toml-formatting-checks:
788815
name: check Cargo.toml formatting
789816
needs: linux-build-lib
817+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
790818
runs-on: ubuntu-latest
791819
container:
792820
image: amd64/rust
@@ -810,6 +838,7 @@ jobs:
810838
config-docs-check:
811839
name: check configs.md and ***_functions.md is up-to-date
812840
needs: linux-build-lib
841+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
813842
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
814843
container:
815844
image: amd64/rust
@@ -845,6 +874,7 @@ jobs:
845874
examples-docs-check:
846875
name: check example README is up-to-date
847876
needs: linux-build-lib
877+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
848878
runs-on: ubuntu-latest
849879
container:
850880
image: amd64/rust
@@ -877,6 +907,7 @@ jobs:
877907
# - datafusion-cli
878908
msrv:
879909
name: Verify MSRV (Min Supported Rust Version)
910+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
880911
runs-on: ubuntu-latest
881912
container:
882913
image: amd64/rust

0 commit comments

Comments
 (0)