Skip to content

Commit b9a0053

Browse files
authored
feat(CI): Introduce xtask to keep local test and CI commands in sync (#24083)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Part of #21048 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The end goal is easy local CI reproducer like ``` # run full ci tests cargo xtask ci full # run fast tests cargo xtask ci fast ``` This PR only try to encapsulate single CI step with `xtask`, and hope later build towards that goal. Running test commands locally mirroring CI become easier after this PR. For example, the default test run command is: https://github.com/apache/datafusion/blob/26fbf4d015d7071115c5fdadf1156291d8a2afb1/.github/workflows/rust.yml#L324-L335 Now both ci and local run can be executed with ``` # Use `cargo xtask help` for the full usage # syntax: cargo xtask ci step [step-name] [extra-args] cargo xtask ci step test workspace (...run tests) # Use 'explain' to show the underlying command cargo xtask ci step test workspace --explain (...print the full command) ``` The command is a bit verbose, since it's a common pattern (see top comments at `xtask/src/main.rs`) for all internal dev tools, so we might need additional domains like `cargo xtask benchmark [bench-name]` ### `xtask` implementation mechanism See `xtask/src/main.rs` top comments for details. TLDR: internal binary, no external dependencies. ### `xtask` v.s. shell script The existing CI jobs consist of lot of similar commands, that is parameterized with features, settings, so it would be easier to implement the runner that can handle args, to make the maintenance easier. We also want some extra functionalities, like the above `explain` feature, or more complex orchestration needed for the 'one-click local CI run' end goal. At this complexity, I find implementing it in rust is easier than shell scripts. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> In rust CI workflow (`rust.yml`), encapsulate `cargo test ...` and `cargo check ...` family into `xtask` runner To review the core changes, start from the top comments in `xtask/src/main.rs` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 090a045 commit b9a0053

8 files changed

Lines changed: 1092 additions & 73 deletions

File tree

‎.cargo/config.toml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[alias]
19+
xtask = "run --quiet --locked --package datafusion-xtask --"
20+
ci-step = "run --quiet --locked --package datafusion-xtask -- ci step"

‎.github/workflows/rust.yml‎

Lines changed: 66 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,21 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
# For some actions, we use Runs-On to run them on ASF infrastructure: https://datafusion.apache.org/contributor-guide/#ci-runners
19-
18+
# # CI runners
19+
#
20+
# Some actions use Runs-On to run on ASF infrastructure:
21+
# https://datafusion.apache.org/contributor-guide/#ci-runners
22+
#
23+
# # `cargo xtask`
24+
#
25+
# Individual CI commands are implemented with `cargo xtask` to make them easier
26+
# to reproduce locally.
27+
#
28+
# See `xtask/README.md` for details about DataFusion's `xtask` tooling.
29+
#
30+
# It is being rolled out incrementally across all workflows.
31+
# Track progress at https://github.com/apache/datafusion/issues/24487.
32+
#
2033
name: Rust
2134

2235
concurrency:
@@ -63,11 +76,8 @@ jobs:
6376
shared-key: "amd-ci-check" # this job uses it's own cache becase check has a separate cache and we need it to be fast as it blocks other jobs
6477
save-if: ${{ github.ref_name == 'main' }}
6578
- name: Prepare cargo build
66-
run: |
67-
# Adding `--locked` here to assert that the `Cargo.lock` file is up to
68-
# date with the manifest. When this fails, please make sure to commit
69-
# the changes to `Cargo.lock` after building with the updated manifest.
70-
cargo check --profile ci --workspace --all-targets --features integration-tests --locked
79+
# `--locked` asserts that Cargo.lock is up to date with the manifest.
80+
run: cargo xtask ci step check workspace
7181

7282
# Check datafusion-common features
7383
#
@@ -86,12 +96,12 @@ jobs:
8696
with:
8797
rust-version: stable
8898
- name: Check datafusion-common (default features)
89-
run: cargo check --profile ci --all-targets -p datafusion-common
99+
run: cargo xtask ci step check datafusion-common default
90100
#
91101
# Note: Only check libraries (not --all-targets) to cover end user APIs
92102
#
93103
- name: Check datafusion-common (no-default-features)
94-
run: cargo check --profile ci --no-default-features -p datafusion-common
104+
run: cargo xtask ci step check datafusion-common no-default
95105
# Note: don't check other feature flags as datafusion-common is not typically used standalone
96106

97107
# Check datafusion-substrait features
@@ -116,20 +126,20 @@ jobs:
116126
save-if: false # set in linux-test
117127
shared-key: "amd-ci"
118128
- name: Check datafusion-substrait (default features)
119-
run: cargo check --profile ci --all-targets -p datafusion-substrait
129+
run: cargo xtask ci step check datafusion-substrait default
120130
#
121131
# Note: Only check libraries (not --all-targets) to cover end user APIs
122132
#
123133
- name: Check datafusion-substrait (no-default-features)
124-
run: cargo check --profile ci --no-default-features -p datafusion-substrait
134+
run: cargo xtask ci step check datafusion-substrait no-default
125135
- name: Check datafusion-substrait (physical)
126-
run: cargo check --profile ci --no-default-features -p datafusion-substrait --features=physical
136+
run: cargo xtask ci step check datafusion-substrait physical
127137
- name: Install cmake
128138
run: |
129139
# note the builder setup runs apt-get update / installs protobuf compiler
130140
apt-get install -y cmake
131141
- name: Check datafusion-substrait (protoc)
132-
run: cargo check --profile ci --no-default-features -p datafusion-substrait --features=protoc
142+
run: cargo xtask ci step check datafusion-substrait protoc
133143

134144
# Check datafusion-proto features
135145
#
@@ -149,18 +159,18 @@ jobs:
149159
with:
150160
rust-version: stable
151161
- name: Check datafusion-proto (default features)
152-
run: cargo check --profile ci --all-targets -p datafusion-proto
162+
run: cargo xtask ci step check datafusion-proto default
153163
#
154164
# Note: Only check libraries (not --all-targets) to cover end user APIs
155165
#
156166
- name: Check datafusion-proto (no-default-features)
157-
run: cargo check --profile ci --no-default-features -p datafusion-proto
167+
run: cargo xtask ci step check datafusion-proto no-default
158168
- name: Check datafusion-proto (json)
159-
run: cargo check --profile ci --no-default-features -p datafusion-proto --features=json
169+
run: cargo xtask ci step check datafusion-proto json
160170
- name: Check datafusion-proto (parquet)
161-
run: cargo check --profile ci --no-default-features -p datafusion-proto --features=parquet
171+
run: cargo xtask ci step check datafusion-proto parquet
162172
- name: Check datafusion-proto (avro)
163-
run: cargo check --profile ci --no-default-features -p datafusion-proto --features=avro
173+
run: cargo xtask ci step check datafusion-proto avro
164174

165175
# Check datafusion-ffi features
166176
#
@@ -180,7 +190,7 @@ jobs:
180190
with:
181191
rust-version: stable
182192
- name: Check datafusion-ffi (no-default-features)
183-
run: cargo check --profile ci --no-default-features -p datafusion-ffi
193+
run: cargo xtask ci step check datafusion-ffi no-default
184194

185195

186196
# Check datafusion crate features
@@ -206,48 +216,48 @@ jobs:
206216
save-if: false # set in linux-test
207217
shared-key: "amd-ci"
208218
- name: Check datafusion (default features)
209-
run: cargo check --profile ci --all-targets -p datafusion
219+
run: cargo xtask ci step check datafusion default
210220
#
211221
# Note: Only check libraries (not --all-targets) to cover end user APIs
212222
#
213223
- name: Check datafusion (no-default-features)
214-
run: cargo check --profile ci --no-default-features -p datafusion
224+
run: cargo xtask ci step check datafusion no-default
215225
- name: Check datafusion (nested_expressions)
216-
run: cargo check --profile ci --no-default-features -p datafusion --features=nested_expressions
226+
run: cargo xtask ci step check datafusion nested_expressions
217227
- name: Check datafusion (array_expressions)
218-
run: cargo check --profile ci --no-default-features -p datafusion --features=array_expressions
228+
run: cargo xtask ci step check datafusion array_expressions
219229
- name: Check datafusion (avro)
220-
run: cargo check --profile ci --no-default-features -p datafusion --features=avro
230+
run: cargo xtask ci step check datafusion avro
221231
- name: Check datafusion (backtrace)
222-
run: cargo check --profile ci --no-default-features -p datafusion --features=backtrace
232+
run: cargo xtask ci step check datafusion backtrace
223233
- name: Check datafusion (compression)
224-
run: cargo check --profile ci --no-default-features -p datafusion --features=compression
234+
run: cargo xtask ci step check datafusion compression
225235
- name: Check datafusion (crypto_expressions)
226-
run: cargo check --profile ci --no-default-features -p datafusion --features=crypto_expressions
236+
run: cargo xtask ci step check datafusion crypto_expressions
227237
- name: Check datafusion (datetime_expressions)
228-
run: cargo check --profile ci --no-default-features -p datafusion --features=datetime_expressions
238+
run: cargo xtask ci step check datafusion datetime_expressions
229239
- name: Check datafusion (encoding_expressions)
230-
run: cargo check --profile ci --no-default-features -p datafusion --features=encoding_expressions
240+
run: cargo xtask ci step check datafusion encoding_expressions
231241
- name: Check datafusion (force_hash_collisions)
232-
run: cargo check --profile ci --no-default-features -p datafusion --features=force_hash_collisions
242+
run: cargo xtask ci step check datafusion force_hash_collisions
233243
- name: Check datafusion (math_expressions)
234-
run: cargo check --profile ci --no-default-features -p datafusion --features=math_expressions
244+
run: cargo xtask ci step check datafusion math_expressions
235245
- name: Check datafusion (parquet)
236-
run: cargo check --profile ci --no-default-features -p datafusion --features=parquet
246+
run: cargo xtask ci step check datafusion parquet
237247
- name: Check datafusion (regex_expressions)
238-
run: cargo check --profile ci --no-default-features -p datafusion --features=regex_expressions
248+
run: cargo xtask ci step check datafusion regex_expressions
239249
- name: Check datafusion (recursive_protection)
240-
run: cargo check --profile ci --no-default-features -p datafusion --features=recursive_protection
250+
run: cargo xtask ci step check datafusion recursive_protection
241251
- name: Check datafusion (serde)
242-
run: cargo check --profile ci --no-default-features -p datafusion --features=serde
252+
run: cargo xtask ci step check datafusion serde
243253
- name: Check datafusion (sql)
244-
run: cargo check --profile ci --no-default-features -p datafusion --features=sql
254+
run: cargo xtask ci step check datafusion sql
245255
- name: Check datafusion (string_expressions)
246-
run: cargo check --profile ci --no-default-features -p datafusion --features=string_expressions
256+
run: cargo xtask ci step check datafusion string_expressions
247257
- name: Check datafusion (unicode_expressions)
248-
run: cargo check --profile ci --no-default-features -p datafusion --features=unicode_expressions
258+
run: cargo xtask ci step check datafusion unicode_expressions
249259
- name: Check parquet encryption (parquet_encryption)
250-
run: cargo check --profile ci --no-default-features -p datafusion --features=parquet_encryption
260+
run: cargo xtask ci step check datafusion parquet_encryption
251261

252262
# Check datafusion-functions crate features
253263
#
@@ -266,26 +276,26 @@ jobs:
266276
with:
267277
rust-version: stable
268278
- name: Check datafusion-functions (default features)
269-
run: cargo check --profile ci --all-targets -p datafusion-functions
279+
run: cargo xtask ci step check datafusion-functions default
270280
#
271281
# Note: Only check libraries (not --all-targets) to cover end user APIs
272282
#
273283
- name: Check datafusion-functions (no-default-features)
274-
run: cargo check --profile ci --no-default-features -p datafusion-functions
284+
run: cargo xtask ci step check datafusion-functions no-default
275285
- name: Check datafusion-functions (crypto_expressions)
276-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=crypto_expressions
286+
run: cargo xtask ci step check datafusion-functions crypto_expressions
277287
- name: Check datafusion-functions (datetime_expressions)
278-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=datetime_expressions
288+
run: cargo xtask ci step check datafusion-functions datetime_expressions
279289
- name: Check datafusion-functions (encoding_expressions)
280-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=encoding_expressions
290+
run: cargo xtask ci step check datafusion-functions encoding_expressions
281291
- name: Check datafusion-functions (math_expressions)
282-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=math_expressions
292+
run: cargo xtask ci step check datafusion-functions math_expressions
283293
- name: Check datafusion-functions (regex_expressions)
284-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=regex_expressions
294+
run: cargo xtask ci step check datafusion-functions regex_expressions
285295
- name: Check datafusion-functions (string_expressions)
286-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=string_expressions
296+
run: cargo xtask ci step check datafusion-functions string_expressions
287297
- name: Check datafusion-functions (unicode_expressions)
288-
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=unicode_expressions
298+
run: cargo xtask ci step check datafusion-functions unicode_expressions
289299

290300
# Check datafusion-spark crate features
291301
#
@@ -347,19 +357,7 @@ jobs:
347357
- name: Run tests (excluding doctests and datafusion-cli)
348358
env:
349359
RUST_BACKTRACE: 1
350-
run: |
351-
cargo llvm-cov \
352-
--profile ci \
353-
--exclude datafusion-examples \
354-
--exclude ffi_example_table_provider \
355-
--exclude datafusion-cli \
356-
--workspace \
357-
--lib \
358-
--tests \
359-
--bins \
360-
--features serde,avro,json,backtrace,integration-tests,parquet_encryption,substrait \
361-
--codecov \
362-
--output-path target/codecov.json
360+
run: cargo xtask ci step test workspace
363361
- parallel:
364362
- name: Verify Working Directory Clean
365363
run: git diff --exit-code
@@ -405,7 +403,7 @@ jobs:
405403
AWS_SECRET_ACCESS_KEY: TEST-DataFusionPassword
406404
TEST_STORAGE_INTEGRATION: 1
407405
AWS_ALLOW_HTTP: true
408-
run: cargo test --features backtrace --profile ci -p datafusion-cli --lib --tests --bins
406+
run: cargo xtask ci step test cli
409407
- name: Verify Working Directory Clean
410408
run: git diff --exit-code
411409

@@ -458,7 +456,7 @@ jobs:
458456
with:
459457
rust-version: stable
460458
- name: Run doctests
461-
run: cargo test --profile ci --doc --features avro,json
459+
run: cargo xtask ci step test doctest
462460
- name: Verify Working Directory Clean
463461
run: git diff --exit-code
464462

@@ -532,11 +530,8 @@ jobs:
532530
mv *.tbl ../datafusion/sqllogictest/test_files/tpch/data
533531
- name: Verify that benchmark queries return expected results
534532
run: |
535-
# increase stack size to fix stack overflow
536-
export RUST_MIN_STACK=20971520
537-
export TPCH_DATA=`realpath datafusion/sqllogictest/test_files/tpch/data`
538-
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci -- --test-threads=1
539-
INCLUDE_TPCH=true cargo test --features backtrace,parquet_encryption,substrait --profile ci --package datafusion-sqllogictest --test sqllogictests
533+
cargo xtask ci step test benchmark-plan
534+
cargo xtask ci step test benchmark-sqllogic
540535
- name: Verify Working Directory Clean
541536
run: git diff --exit-code
542537

@@ -571,9 +566,7 @@ jobs:
571566
with:
572567
rust-version: stable
573568
- name: Run sqllogictest
574-
run: |
575-
cd datafusion/sqllogictest
576-
PG_COMPAT=true PG_URI="postgresql://postgres:postgres@$POSTGRES_HOST:$POSTGRES_PORT/db_test" cargo test --features backtrace --profile ci --features=postgres --test sqllogictests
569+
run: cargo xtask ci step test postgres
577570
env:
578571
# use postgres for the host here because we have specified a container for the job
579572
POSTGRES_HOST: postgres
@@ -600,7 +593,7 @@ jobs:
600593
# command cannot be run for all the .slt files. Run it for just one that works (limit.slt)
601594
# until most of the tickets in https://github.com/apache/datafusion/issues/16248 are addressed
602595
# and this command can be run without filters.
603-
run: cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip limit.slt
596+
run: cargo xtask ci step test substrait
604597

605598
# Temporarily commenting out the Windows flow, the reason is enormously slow running build
606599
# Waiting for new Windows 2025 github runner
@@ -638,7 +631,7 @@ jobs:
638631
uses: ./.github/actions/setup-macos-aarch64-builder
639632
- name: Run datafusion-ffi tests
640633
shell: bash
641-
run: cargo test --profile ci -p datafusion-ffi --lib --tests --features integration-tests
634+
run: cargo xtask ci step test ffi
642635

643636
vendor:
644637
name: Verify Vendored Code

‎Cargo.lock‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ members = [
6666
"benchmarks",
6767
"datafusion/macros",
6868
"datafusion/doc",
69+
"xtask",
6970
]
7071
exclude = ["dev/depcheck"]
7172
resolver = "2"

‎xtask/Cargo.toml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[package]
19+
name = "datafusion-xtask"
20+
version = "0.1.0"
21+
edition.workspace = true
22+
license.workspace = true
23+
publish = false
24+
25+
[dev-dependencies]
26+
insta = { workspace = true }
27+
28+
[lints]
29+
workspace = true

0 commit comments

Comments
 (0)