Skip to content

Commit 2904c4d

Browse files
committed
Merge apache/main into chore/track-datafusion-git-main; bump datafusion rev
Merged latest apache/main into the branch and bumped the datafusion* git dependencies from b79d3965 to efa84e83 (current apache/datafusion main HEAD). That bump forced a re-port of the physical-plan and proto-codec APIs: - statistics_with_args + StatisticsArgs::compute_child_statistics were refactored into StatisticsContext::compute + ExecutionPlan ::statistics_from_inputs(input_stats, args) + ExecutionPlan::child_stats_requests(partition). All Ballista operator overrides and call sites were migrated. Custom operators (ChaosExec, ShuffleWriterExec, SortShuffleWriterExec, BufferExec, RuntimeStatsExec, executor CollectExec) now declare a single ChildStats::At(partition) request and return their sole child's precomputed stats, restoring the correctness fix the original PR made when statistics_with_args replaced partition_statistics. - PhysicalExtensionCodec::try_decode/try_encode gained a PhysicalProtoConverterExtension argument; all test call sites now pass &DefaultPhysicalProtoConverter {}. - Partitioning::Range is a new enum variant; the shuffle-writer partition-mapping walker treats it as a K-space (like Hash), and the task-builder shuffle-reader rewrite falls back to Unknown for it. - ExecutionPlan::required_input_distribution is deprecated in favour of input_distribution_requirements(); task_builder was migrated. - apache/datafusion#23642 is fixed upstream, so the pin test that asserted the bug's presence was inverted into a regression test that the partition count survives round-trip. make_empty_exec_serde_safe itself is kept as belt-and-suspenders for legacy stage plans decoded from persisted state. - Plan-structure test expectations were updated for the new physical optimizer ordering (ProjectionExec now sits above SortExec in the aggregate/window fan-out) and the new RoundRobinBatch(2) repartition the optimizer inserts above the SortMergeJoin. - context_checks.rs's EXPLAIN ANALYZE snapshot merged the new ShuffleReaderExec fetch metrics from main with the peak_mem_used drop from the datafusion bump.
2 parents 50b1b9a + 9efb1f4 commit 2904c4d

294 files changed

Lines changed: 26150 additions & 5199 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/dependabot.yml‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,18 +23,43 @@ updates:
2323
interval: daily
2424
open-pull-requests-limit: 10
2525
target-branch: main
26-
labels: [auto-dependencies]
26+
labels: [ auto-dependencies ]
27+
commit-message:
28+
prefix: "chore(deps): "
2729
ignore:
2830
# arrow and datafusion are bumped manually
2931
- dependency-name: "arrow*"
30-
update-types: ["version-update:semver-major"]
32+
update-types: [ "version-update:semver-major" ]
3133
- dependency-name: "datafusion*"
32-
update-types: ["version-update:semver-major"]
34+
update-types: [ "version-update:semver-major" ]
3335
- dependency-name: "sqlparser"
34-
update-types: ["version-update:semver-major"]
36+
update-types: [ "version-update:semver-major" ]
37+
- package-ecosystem: cargo
38+
directory: "/python"
39+
schedule:
40+
interval: daily
41+
open-pull-requests-limit: 10
42+
target-branch: main
43+
labels: [ auto-dependencies ]
44+
commit-message:
45+
prefix: "chore(deps): "
46+
ignore:
47+
# arrow and datafusion are bumped manually
48+
- dependency-name: "arrow*"
49+
update-types: [ "version-update:semver-major" ]
50+
- dependency-name: "datafusion*"
51+
update-types: [ "version-update:semver-major" ]
52+
- dependency-name: "sqlparser"
53+
update-types: [ "version-update:semver-major" ]
3554
- package-ecosystem: "github-actions"
3655
directory: "/"
3756
schedule:
3857
interval: "daily"
3958
open-pull-requests-limit: 10
40-
labels: [auto-dependencies]
59+
labels: [ auto-dependencies ]
60+
commit-message:
61+
prefix: "chore(ci): "
62+
groups:
63+
github-codeql:
64+
patterns:
65+
- "github/codeql-action/*"
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
---
19+
name: "ASF Allowlist Check"
20+
21+
on:
22+
workflow_dispatch:
23+
pull_request:
24+
paths:
25+
- ".github/**"
26+
push:
27+
# Always run on every commit on the main branch in case of expired allows
28+
branches:
29+
- main
30+
31+
permissions:
32+
contents: read
33+
34+
jobs:
35+
asf-allowlist-check:
36+
runs-on: ubuntu-latest
37+
steps:
38+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
39+
with:
40+
persist-credentials: false
41+
- uses: apache/infrastructure-actions/allowlist-check@main

‎.github/workflows/build.yml‎

Lines changed: 43 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -43,22 +43,25 @@ jobs:
4343
- "3.12"
4444
- "3.13"
4545
steps:
46-
- uses: actions/[email protected].0
46+
- uses: actions/[email protected].1
4747

48-
- uses: actions/setup-python@v6
48+
- uses: actions/setup-python@v7
4949
with:
5050
python-version: ${{ matrix.python-version }}
5151

52-
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
53-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
52+
- name: Setup Rust toolchain
53+
run: |
54+
rustup toolchain install stable
55+
rustup default stable
56+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
5457

5558
- name: Install Protoc
5659
uses: arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b # v3.0.0
5760
with:
5861
version: "27.4"
5962
repo-token: ${{ secrets.GITHUB_TOKEN }}
6063

61-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
64+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
6265
with:
6366
enable-cache: true
6467

@@ -71,13 +74,13 @@ jobs:
7174
build:
7275
runs-on: ubuntu-latest
7376
steps:
74-
- uses: actions/[email protected].0
77+
- uses: actions/[email protected].1
7578
- name: Install Python
76-
uses: actions/setup-python@v6
79+
uses: actions/setup-python@v7
7780
with:
7881
python-version: "3.12"
7982

80-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
83+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
8184
with:
8285
enable-cache: true
8386

@@ -99,8 +102,8 @@ jobs:
99102
if: github.event_name != 'pull_request'
100103
runs-on: ubuntu-latest
101104
steps:
102-
- uses: actions/[email protected].0
103-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
105+
- uses: actions/[email protected].1
106+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
104107
with:
105108
enable-cache: true
106109
- name: Generate license file
@@ -121,16 +124,19 @@ jobs:
121124
python-version: [ "3.10" ]
122125
os: [ macos-latest ]
123126
steps:
124-
- uses: actions/[email protected].0
127+
- uses: actions/[email protected].1
125128

126-
- uses: actions/setup-python@v6
129+
- uses: actions/setup-python@v7
127130
with:
128131
python-version: ${{ matrix.python-version }}
129132

130-
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
131-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
133+
- name: Setup Rust toolchain
134+
run: |
135+
rustup toolchain install stable
136+
rustup default stable
137+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
132138
with:
133-
# FIXME: temporary workaround, see: https://github.com/Swatinem/rust-cache/issues/341
139+
# FIXME: temporary workaround, see: https://github.com/swatinem/rust-cache/issues/341
134140
cache-bin: ${{ matrix.os != 'macos-latest' }}
135141
workspaces: |
136142
python -> target
@@ -148,7 +154,7 @@ jobs:
148154
version: "27.4"
149155
repo-token: ${{ secrets.GITHUB_TOKEN }}
150156

151-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
157+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
152158
with:
153159
enable-cache: true
154160

@@ -174,10 +180,13 @@ jobs:
174180
name: Manylinux x86_64
175181
runs-on: ubuntu-latest
176182
steps:
177-
- uses: actions/[email protected].0
183+
- uses: actions/[email protected].1
178184

179-
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
180-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
185+
- name: Setup Rust toolchain
186+
run: |
187+
rustup toolchain install stable
188+
rustup default stable
189+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
181190

182191
- run: rm LICENSE.txt
183192
- name: Download LICENSE.txt
@@ -194,7 +203,7 @@ jobs:
194203
version: "27.4"
195204
repo-token: ${{ secrets.GITHUB_TOKEN }}
196205

197-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
206+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
198207
with:
199208
enable-cache: true
200209

@@ -216,10 +225,13 @@ jobs:
216225
name: Manylinux arm64
217226
runs-on: ubuntu-24.04-arm
218227
steps:
219-
- uses: actions/[email protected].0
228+
- uses: actions/[email protected].1
220229

221-
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
222-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
230+
- name: Setup Rust toolchain
231+
run: |
232+
rustup toolchain install stable
233+
rustup default stable
234+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
223235

224236
- run: rm LICENSE.txt
225237
- name: Download LICENSE.txt
@@ -236,7 +248,7 @@ jobs:
236248
version: "27.4"
237249
repo-token: ${{ secrets.GITHUB_TOKEN }}
238250

239-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
251+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
240252
with:
241253
enable-cache: true
242254

@@ -258,10 +270,13 @@ jobs:
258270
name: Source distribution
259271
runs-on: ubuntu-latest
260272
steps:
261-
- uses: actions/[email protected].0
273+
- uses: actions/[email protected].1
262274

263-
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
264-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
275+
- name: Setup Rust toolchain
276+
run: |
277+
rustup toolchain install stable
278+
rustup default stable
279+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
265280

266281
- run: rm LICENSE.txt
267282
- name: Download LICENSE.txt
@@ -278,7 +293,7 @@ jobs:
278293
version: "27.4"
279294
repo-token: ${{ secrets.GITHUB_TOKEN }}
280295

281-
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
296+
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
282297
with:
283298
enable-cache: true
284299

‎.github/workflows/codeql.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,16 +43,16 @@ jobs:
4343

4444
steps:
4545
- name: Checkout repository
46-
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
46+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4747
with:
4848
persist-credentials: false
4949

5050
- name: Initialize CodeQL
51-
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
51+
uses: github/codeql-action/init@7188fc363630916deb702c7fdcf4e481b751f97a # v4
5252
with:
5353
languages: actions
5454

5555
- name: Perform CodeQL Analysis
56-
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
56+
uses: github/codeql-action/analyze@7188fc363630916deb702c7fdcf4e481b751f97a # v4
5757
with:
5858
category: "/language:actions"

‎.github/workflows/dependencies.yml‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@
1717

1818
name: Dependencies
1919

20+
permissions:
21+
contents: read
22+
2023
concurrency:
2124
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
2225
cancel-in-progress: true
@@ -41,15 +44,35 @@ jobs:
4144
name: msrv dependency check
4245
runs-on: macos-latest
4346
steps:
44-
- uses: actions/[email protected].0
47+
- uses: actions/[email protected].1
4548
with:
4649
fetch-depth: 1
4750
- name: Setup Rust toolchain
4851
uses: ./.github/actions/setup-macos-builder
4952
with:
5053
rust-version: stable
51-
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
54+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
5255
- name: Check dependencies
5356
run: |
5457
cd dev/msrvcheck
5558
cargo run
59+
60+
python-lock:
61+
name: python Cargo.lock up to date
62+
runs-on: macos-latest
63+
steps:
64+
- uses: actions/[email protected]
65+
with:
66+
fetch-depth: 1
67+
- name: Setup Rust toolchain
68+
uses: ./.github/actions/setup-macos-builder
69+
with:
70+
rust-version: stable
71+
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
72+
- name: Verify python/Cargo.lock is in sync with manifests
73+
run: |
74+
cd python
75+
if ! cargo metadata --locked --format-version 1 >/dev/null; then
76+
echo "::error::python/Cargo.lock is out of date with the manifests. Run 'cd python && cargo update' and commit the updated Cargo.lock."
77+
exit 1
78+
fi

‎.github/workflows/dev.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@ jobs:
3131
runs-on: macos-latest
3232
steps:
3333
- name: Checkout
34-
uses: actions/[email protected].0
34+
uses: actions/[email protected].1
3535
- name: Setup Python
36-
uses: actions/setup-python@v6
36+
uses: actions/setup-python@v7
3737
with:
3838
python-version: "3.10"
3939
- name: Audit licenses
@@ -43,8 +43,8 @@ jobs:
4343
name: Use prettier to check formatting of documents
4444
runs-on: macos-latest
4545
steps:
46-
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v5.0.1
47-
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
46+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v5.0.1
47+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
4848
with:
4949
node-version: "20"
5050
- name: Prettier check

‎.github/workflows/dev_pr.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
github.event_name == 'pull_request_target' &&
3939
(github.event.action == 'opened' ||
4040
github.event.action == 'synchronize')
41-
uses: actions/labeler@v6.1.0
41+
uses: actions/labeler@v7.0.0
4242
with:
4343
repo-token: ${{ secrets.GITHUB_TOKEN }}
4444
configuration-path: .github/workflows/dev_pr/labeler.yml

0 commit comments

Comments
 (0)