You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b9a0053
Browse filesBrowse the repository at this point in the historyBrowse files
feat(CI): Introduce xtask to keep local test and CI commands in sync (#24083)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes#123` indicates that this PR will close issue #123.
-->
Part of #21048
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
The end goal is easy local CI reproducer like
```
# run full ci tests
cargo xtask ci full
# run fast tests
cargo xtask ci fast
```
This PR only try to encapsulate single CI step with `xtask`, and hope
later build towards that goal. Running test commands locally mirroring
CI become easier after this PR.
For example, the default test run command is:
https://github.com/apache/datafusion/blob/26fbf4d015d7071115c5fdadf1156291d8a2afb1/.github/workflows/rust.yml#L324-L335
Now both ci and local run can be executed with
```
# Use `cargo xtask help` for the full usage
# syntax: cargo xtask ci step [step-name] [extra-args]
cargo xtask ci step test workspace
(...run tests)
# Use 'explain' to show the underlying command
cargo xtask ci step test workspace --explain
(...print the full command)
```
The command is a bit verbose, since it's a common pattern (see top
comments at `xtask/src/main.rs`) for all internal dev tools, so we might
need additional domains like `cargo xtask benchmark [bench-name]`
### `xtask` implementation mechanism
See `xtask/src/main.rs` top comments for details.
TLDR: internal binary, no external dependencies.
### `xtask` v.s. shell script
The existing CI jobs consist of lot of similar commands, that is
parameterized with features, settings, so it would be easier to
implement the runner that can handle args, to make the maintenance
easier.
We also want some extra functionalities, like the above `explain`
feature, or more complex orchestration needed for the 'one-click local
CI run' end goal.
At this complexity, I find implementing it in rust is easier than shell
scripts.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
In rust CI workflow (`rust.yml`), encapsulate `cargo test ...` and
`cargo check ...` family into `xtask` runner
To review the core changes, start from the top comments in
`xtask/src/main.rs`
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Copy file name to clipboardExpand all lines: .github/workflows/rust.yml
+66-73Lines changed: 66 additions & 73 deletions
Original file line number
Diff line number
Diff line change
@@ -15,8 +15,21 @@
15
15
# specific language governing permissions and limitations
16
16
# under the License.
17
17
18
-
# For some actions, we use Runs-On to run them on ASF infrastructure: https://datafusion.apache.org/contributor-guide/#ci-runners
19
-
18
+
# # CI runners
19
+
#
20
+
# Some actions use Runs-On to run on ASF infrastructure:
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci -- --test-threads=1
539
-
INCLUDE_TPCH=true cargo test --features backtrace,parquet_encryption,substrait --profile ci --package datafusion-sqllogictest --test sqllogictests
533
+
cargo xtask ci step test benchmark-plan
534
+
cargo xtask ci step test benchmark-sqllogic
540
535
- name: Verify Working Directory Clean
541
536
run: git diff --exit-code
542
537
@@ -571,9 +566,7 @@ jobs:
571
566
with:
572
567
rust-version: stable
573
568
- name: Run sqllogictest
574
-
run: |
575
-
cd datafusion/sqllogictest
576
-
PG_COMPAT=true PG_URI="postgresql://postgres:postgres@$POSTGRES_HOST:$POSTGRES_PORT/db_test" cargo test --features backtrace --profile ci --features=postgres --test sqllogictests
569
+
run: cargo xtask ci step test postgres
577
570
env:
578
571
# use postgres for the host here because we have specified a container for the job
579
572
POSTGRES_HOST: postgres
@@ -600,7 +593,7 @@ jobs:
600
593
# command cannot be run for all the .slt files. Run it for just one that works (limit.slt)
601
594
# until most of the tickets in https://github.com/apache/datafusion/issues/16248 are addressed
0 commit comments