Repository navigation
Commit 8e0c668
ci: skip queue-verified Rust jobs on pushes to main (apache#25365)
## Which issue does this PR close?
- Part of apache#25148. This PR does not close the issue.
## Rationale for this change
Every commit reaches upstream `main` through the merge queue. The queue
runs the Rust workflow at that commit. The push to `main` then runs the
same 26 jobs at the same SHA.
Only seven of those jobs give new output after the merge:
- `linux-build-lib`, `linux-cargo-check-datafusion-functions`,
`linux-test`, `linux-test-example`, `clippy`, and `macos-aarch64` save
the caches that later PRs restore. The workflow saves caches only on
pushes to `main`.
- `linux-test` publishes coverage.
- `linux-datafusion-ffi-features` is not a required check in
`.asf.yaml`, so a PR can merge when it fails. The post-merge run shows
that failure on `main`.
From September 9 to September 16, 2026, the Rust workflow ran 116 times
for pushes to `main`. Each of those commits had a successful merge-queue
run at the same SHA before the push. The ten most recent pushes used a
mean of 71.7 runner-minutes each. The 19 skipped jobs account for 47.6
of those minutes: 19.5 on GitHub-hosted Linux runners and 28.0 on
RunsOn. At that merge rate, this change saves about 2,200 GitHub-hosted
Linux runner-minutes and 3,100 RunsOn runner-minutes per week. The
GitHub-hosted minutes count against the shared ASF limit that the issue
describes.
Tradeoff: a direct push to `main` that bypasses the merge queue also
skips the 19 jobs. A manual `workflow_dispatch` on `main` runs all 26
jobs. `extended.yml` already uses the same approach for `main`.
Post-merge runs also catch intermittent failures. In the sample week,
three post-merge runs failed. Each commit had passed the same jobs in
the queue. The logs show environment errors: a test memory-pool limit, a
ChromeDriver port bind, and a MinIO image pull. Two of the three failed
jobs are in the skipped set.
## What changes are included in this PR?
`.github/workflows/rust.yml`:
- 19 jobs get `if: ${{ !(github.event_name == 'push' && github.ref ==
'refs/heads/main' && github.repository == 'apache/datafusion') }}`. The
condition has no status function, so jobs that depend on
`linux-build-lib` still require that job to succeed.
- The Cargo check artifact archive and upload steps in `linux-build-lib`
get the same condition. Both consumers of that artifact are in the
skipped set.
- A header comment explains the behavior.
| Event | Rust jobs |
| ---------------------------------------- | --------: |
| Push to upstream `main` | 7 |
| PR or merge group | 26 |
| `workflow_dispatch` | 26 |
| Push to another branch, or a fork push | 26 |
`ci/scripts/check_asf_yaml_status_checks.py` gets a second check. It
requires that exactly the jobs in `POST_MERGE_SKIPPED_JOBS` carry the
condition, that the artifact steps carry it, and that every job that
downloads the `cargo-check` artifact is in the skipped set. A new job
without the condition passes, so new jobs run by default.
`docs/source/contributor-guide/testing.md` gets one sentence on the new
check.
## What is the testing strategy for this PR?
`python3 ci/scripts/check_asf_yaml_status_checks.py` passes on this
branch. The required `asf-yaml-check` job in `dev.yml` runs it on every
PR and merge-group run. The job fails if a listed job loses the
condition, if another job gains it, if an artifact step loses its guard,
or if a job outside the skipped set downloads the `cargo-check`
artifact.
This PR adds no unit tests for the validator. The validator checks the
real workflow in a required job, and a test suite for the validator is
out of scope for this change.
Hosted CI on this PR runs all 26 Rust jobs. The first push to `main`
after the merge is the check of the reduced path: seven Rust jobs run
and 19 skip.
## Are there any user-facing changes?
No. This PR changes CI and the contributor guide only.1 parent 4780aec commit 8e0c668
3 files changed
Lines changed: 194 additions & 8 deletions
File tree
- .github/workflows
- ci/scripts
- docs/source/contributor-guide
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
33 | 44 | | |
34 | 45 | | |
35 | 46 | | |
| |||
80 | 91 | | |
81 | 92 | | |
82 | 93 | | |
| 94 | + | |
83 | 95 | | |
84 | 96 | | |
85 | 97 | | |
| |||
91 | 103 | | |
92 | 104 | | |
93 | 105 | | |
94 | | - | |
| 106 | + | |
95 | 107 | | |
96 | 108 | | |
97 | 109 | | |
| |||
109 | 121 | | |
110 | 122 | | |
111 | 123 | | |
| 124 | + | |
112 | 125 | | |
113 | 126 | | |
114 | 127 | | |
| |||
139 | 152 | | |
140 | 153 | | |
141 | 154 | | |
| 155 | + | |
142 | 156 | | |
143 | 157 | | |
144 | 158 | | |
| |||
185 | 199 | | |
186 | 200 | | |
187 | 201 | | |
| 202 | + | |
188 | 203 | | |
189 | 204 | | |
190 | 205 | | |
| |||
247 | 262 | | |
248 | 263 | | |
249 | 264 | | |
| 265 | + | |
250 | 266 | | |
251 | 267 | | |
252 | 268 | | |
| |||
367 | 383 | | |
368 | 384 | | |
369 | 385 | | |
| 386 | + | |
370 | 387 | | |
371 | 388 | | |
372 | 389 | | |
| |||
448 | 465 | | |
449 | 466 | | |
450 | 467 | | |
| 468 | + | |
451 | 469 | | |
452 | 470 | | |
453 | 471 | | |
| |||
525 | 543 | | |
526 | 544 | | |
527 | 545 | | |
| 546 | + | |
528 | 547 | | |
529 | 548 | | |
530 | 549 | | |
| |||
547 | 566 | | |
548 | 567 | | |
549 | 568 | | |
| 569 | + | |
550 | 570 | | |
551 | 571 | | |
552 | 572 | | |
| |||
562 | 582 | | |
563 | 583 | | |
564 | 584 | | |
| 585 | + | |
565 | 586 | | |
566 | 587 | | |
567 | 588 | | |
| |||
589 | 610 | | |
590 | 611 | | |
591 | 612 | | |
| 613 | + | |
592 | 614 | | |
593 | 615 | | |
594 | 616 | | |
| |||
621 | 643 | | |
622 | 644 | | |
623 | 645 | | |
| 646 | + | |
624 | 647 | | |
625 | 648 | | |
626 | 649 | | |
| |||
658 | 681 | | |
659 | 682 | | |
660 | 683 | | |
| 684 | + | |
661 | 685 | | |
662 | 686 | | |
663 | 687 | | |
| |||
718 | 742 | | |
719 | 743 | | |
720 | 744 | | |
| 745 | + | |
721 | 746 | | |
722 | 747 | | |
723 | 748 | | |
| |||
735 | 760 | | |
736 | 761 | | |
737 | 762 | | |
| 763 | + | |
738 | 764 | | |
739 | 765 | | |
740 | 766 | | |
| |||
750 | 776 | | |
751 | 777 | | |
752 | 778 | | |
| 779 | + | |
753 | 780 | | |
754 | 781 | | |
755 | 782 | | |
| |||
787 | 814 | | |
788 | 815 | | |
789 | 816 | | |
| 817 | + | |
790 | 818 | | |
791 | 819 | | |
792 | 820 | | |
| |||
810 | 838 | | |
811 | 839 | | |
812 | 840 | | |
| 841 | + | |
813 | 842 | | |
814 | 843 | | |
815 | 844 | | |
| |||
845 | 874 | | |
846 | 875 | | |
847 | 876 | | |
| 877 | + | |
848 | 878 | | |
849 | 879 | | |
850 | 880 | | |
| |||
877 | 907 | | |
878 | 908 | | |
879 | 909 | | |
| 910 | + | |
880 | 911 | | |
881 | 912 | | |
882 | 913 | | |
| |||
0 commit comments