Skip to content

Commit 1da3cf0

Browse files
chore(ci): Register the substrait check job with runs-on, route spark checks through xtask (apache#25195)
## Which issue does this PR close? - Part of apache#25148. Also finishes the `cargo check` half of apache#21048, which asks for the CI steps to be reproducible locally. ## Rationale for this change While measuring the `cargo check <crate> features` jobs for apache#25148 I found two inconsistencies in them. Neither is a policy question about which runner to use, so they seemed worth separating out. **1. The substrait check job never registers with `runs-on`.** Fifteen jobs in `rust.yml` select a `runs-on` runner with `extras=s3-cache`. Fourteen of them start with the `runs-on/action` step that registers the job. `cargo check datafusion-substrait features` is the one that does not, so on `push` and `merge_group` it lands on the `runs-on` runner but gets neither the external cache nor the job metrics the other fourteen report. It looks like an oversight rather than a choice; the job directly above and below it in the file both have the step. **2. The spark check job is the last one not going through `xtask`.** `cargo check datafusion-spark features` calls `cargo check` directly. Every other check step in the workflow goes through `cargo xtask ci step check`, which is what lets a contributor run the same thing locally. After this PR there is no raw `cargo check` left in `rust.yml`. ## What changes are included in this PR? Four lines. - Add the `runs-on/action` step to `linux-datafusion-substrait-features`, matching its siblings. - Replace the three `datafusion-spark` `cargo check` invocations with the `xtask` equivalents. The commands `xtask` emits are the same ones being replaced, which `--explain` confirms: | step | before | `cargo xtask ci step check datafusion-spark …` | | --- | --- | --- | | default features | `cargo check --profile ci --all-targets -p datafusion-spark` | `default` | | no-default-features | `cargo check --profile ci --no-default-features -p datafusion-spark` | `no-default` | | core | `cargo check --profile ci --no-default-features -p datafusion-spark --features=core` | `core` | I deliberately left out the larger changes those jobs invite, because they are the open question in apache#25148 rather than something to decide in a cleanup PR. For the record, what I measured there: five of the seven `cargo check <crate> features` jobs declare no `Rust Dependency Cache` step at all, only `datafusion` and `datafusion-substrait` do; three of the seven are pinned to `ubuntu-latest` rather than the `vars.USE_RUNS_ON` expression; and across those seven jobs about a third of the wall clock is setup and cache restore paid seven times over. Happy to follow up on any of that if it is wanted. ## What is the testing strategy for this PR? The three `xtask` commands were run locally against this branch and all pass: ``` cargo xtask ci step check datafusion-spark default cargo xtask ci step check datafusion-spark no-default cargo xtask ci step check datafusion-spark core ``` `python3 ci/scripts/check_asf_yaml_status_checks.py` reports `OK: All 32 required_status_checks match existing GitHub Actions jobs` — job names are unchanged, so the required checks are unaffected. `ci/scripts/check_no_cargo_install_in_workflows.sh` and `ci/scripts/typos_check.sh` are clean, and the file still parses as YAML with the same 26 jobs. The `runs-on/action` half cannot be verified outside the Apache repo, since the action is a no-op on standard GitHub runners. ## Are there any user-facing changes? No. CI configuration only. Co-authored-by: Andrew Lamb <[email protected]>
1 parent 396ea7c commit 1da3cf0

1 file changed

Lines changed: 4 additions & 3 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ jobs:
159159
container:
160160
image: amd64/rust
161161
steps:
162+
- uses: runs-on/action@efac073ea2507ec18797de3a81704201ade11d9d # v2.3.1
162163
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
163164
- name: Setup Rust toolchain
164165
uses: ./.github/actions/setup-builder
@@ -399,14 +400,14 @@ jobs:
399400
save-if: false # The workspace and functions checks save this cache on main.
400401
shared-key: "amd-ci-check"
401402
- name: Check datafusion-spark (default features)
402-
run: cargo check --profile ci --all-targets -p datafusion-spark
403+
run: cargo xtask ci step check datafusion-spark default
403404
#
404405
# Note: Only check libraries (not --all-targets) to cover end user APIs
405406
#
406407
- name: Check datafusion-spark (no-default-features)
407-
run: cargo check --profile ci --no-default-features -p datafusion-spark
408+
run: cargo xtask ci step check datafusion-spark no-default
408409
- name: Check datafusion-spark (core)
409-
run: cargo check --profile ci --no-default-features -p datafusion-spark --features=core
410+
run: cargo xtask ci step check datafusion-spark core
410411

411412
# Library and integration tests
412413
linux-test:

0 commit comments

Comments
 (0)