Repository navigation
Commit 1da3cf0
chore(ci): Register the substrait check job with runs-on, route spark checks through xtask (apache#25195)
## Which issue does this PR close?
- Part of apache#25148. Also finishes the `cargo check` half of apache#21048, which
asks for the CI steps to be reproducible locally.
## Rationale for this change
While measuring the `cargo check <crate> features` jobs for apache#25148 I
found two inconsistencies in them. Neither is a policy question about
which runner to use, so they seemed worth separating out.
**1. The substrait check job never registers with `runs-on`.**
Fifteen jobs in `rust.yml` select a `runs-on` runner with
`extras=s3-cache`. Fourteen of them start with the `runs-on/action` step
that registers the job. `cargo check datafusion-substrait features` is
the one that does not, so on `push` and `merge_group` it lands on the
`runs-on` runner but gets neither the external cache nor the job metrics
the other fourteen report. It looks like an oversight rather than a
choice; the job directly above and below it in the file both have the
step.
**2. The spark check job is the last one not going through `xtask`.**
`cargo check datafusion-spark features` calls `cargo check` directly.
Every other check step in the workflow goes through `cargo xtask ci step
check`, which is what lets a contributor run the same thing locally.
After this PR there is no raw `cargo check` left in `rust.yml`.
## What changes are included in this PR?
Four lines.
- Add the `runs-on/action` step to
`linux-datafusion-substrait-features`, matching its siblings.
- Replace the three `datafusion-spark` `cargo check` invocations with
the `xtask` equivalents.
The commands `xtask` emits are the same ones being replaced, which
`--explain` confirms:
| step | before | `cargo xtask ci step check datafusion-spark …` |
| --- | --- | --- |
| default features | `cargo check --profile ci --all-targets -p
datafusion-spark` | `default` |
| no-default-features | `cargo check --profile ci --no-default-features
-p datafusion-spark` | `no-default` |
| core | `cargo check --profile ci --no-default-features -p
datafusion-spark --features=core` | `core` |
I deliberately left out the larger changes those jobs invite, because
they are the open question in apache#25148 rather than something to decide in
a cleanup PR. For the record, what I measured there: five of the seven
`cargo check <crate> features` jobs declare no `Rust Dependency Cache`
step at all, only `datafusion` and `datafusion-substrait` do; three of
the seven are pinned to `ubuntu-latest` rather than the
`vars.USE_RUNS_ON` expression; and across those seven jobs about a third
of the wall clock is setup and cache restore paid seven times over.
Happy to follow up on any of that if it is wanted.
## What is the testing strategy for this PR?
The three `xtask` commands were run locally against this branch and all
pass:
```
cargo xtask ci step check datafusion-spark default
cargo xtask ci step check datafusion-spark no-default
cargo xtask ci step check datafusion-spark core
```
`python3 ci/scripts/check_asf_yaml_status_checks.py` reports `OK: All 32
required_status_checks match existing GitHub Actions jobs` — job names
are unchanged, so the required checks are unaffected.
`ci/scripts/check_no_cargo_install_in_workflows.sh` and
`ci/scripts/typos_check.sh` are clean, and the file still parses as YAML
with the same 26 jobs.
The `runs-on/action` half cannot be verified outside the Apache repo,
since the action is a no-op on standard GitHub runners.
## Are there any user-facing changes?
No. CI configuration only.
Co-authored-by: Andrew Lamb <[email protected]>1 parent 396ea7c commit 1da3cf0
1 file changed
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
| 162 | + | |
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
| |||
399 | 400 | | |
400 | 401 | | |
401 | 402 | | |
402 | | - | |
| 403 | + | |
403 | 404 | | |
404 | 405 | | |
405 | 406 | | |
406 | 407 | | |
407 | | - | |
| 408 | + | |
408 | 409 | | |
409 | | - | |
| 410 | + | |
410 | 411 | | |
411 | 412 | | |
412 | 413 | | |
| |||
0 commit comments