Skip to content

Commit 8e0c668

Browse files
kumarUjjawalOmega359
authored andcommitted
ci: skip queue-verified Rust jobs on pushes to main (apache#25365)
## Which issue does this PR close? - Part of apache#25148. This PR does not close the issue. ## Rationale for this change Every commit reaches upstream `main` through the merge queue. The queue runs the Rust workflow at that commit. The push to `main` then runs the same 26 jobs at the same SHA. Only seven of those jobs give new output after the merge: - `linux-build-lib`, `linux-cargo-check-datafusion-functions`, `linux-test`, `linux-test-example`, `clippy`, and `macos-aarch64` save the caches that later PRs restore. The workflow saves caches only on pushes to `main`. - `linux-test` publishes coverage. - `linux-datafusion-ffi-features` is not a required check in `.asf.yaml`, so a PR can merge when it fails. The post-merge run shows that failure on `main`. From September 9 to September 16, 2026, the Rust workflow ran 116 times for pushes to `main`. Each of those commits had a successful merge-queue run at the same SHA before the push. The ten most recent pushes used a mean of 71.7 runner-minutes each. The 19 skipped jobs account for 47.6 of those minutes: 19.5 on GitHub-hosted Linux runners and 28.0 on RunsOn. At that merge rate, this change saves about 2,200 GitHub-hosted Linux runner-minutes and 3,100 RunsOn runner-minutes per week. The GitHub-hosted minutes count against the shared ASF limit that the issue describes. Tradeoff: a direct push to `main` that bypasses the merge queue also skips the 19 jobs. A manual `workflow_dispatch` on `main` runs all 26 jobs. `extended.yml` already uses the same approach for `main`. Post-merge runs also catch intermittent failures. In the sample week, three post-merge runs failed. Each commit had passed the same jobs in the queue. The logs show environment errors: a test memory-pool limit, a ChromeDriver port bind, and a MinIO image pull. Two of the three failed jobs are in the skipped set. ## What changes are included in this PR? `.github/workflows/rust.yml`: - 19 jobs get `if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}`. The condition has no status function, so jobs that depend on `linux-build-lib` still require that job to succeed. - The Cargo check artifact archive and upload steps in `linux-build-lib` get the same condition. Both consumers of that artifact are in the skipped set. - A header comment explains the behavior. | Event | Rust jobs | | ---------------------------------------- | --------: | | Push to upstream `main` | 7 | | PR or merge group | 26 | | `workflow_dispatch` | 26 | | Push to another branch, or a fork push | 26 | `ci/scripts/check_asf_yaml_status_checks.py` gets a second check. It requires that exactly the jobs in `POST_MERGE_SKIPPED_JOBS` carry the condition, that the artifact steps carry it, and that every job that downloads the `cargo-check` artifact is in the skipped set. A new job without the condition passes, so new jobs run by default. `docs/source/contributor-guide/testing.md` gets one sentence on the new check. ## What is the testing strategy for this PR? `python3 ci/scripts/check_asf_yaml_status_checks.py` passes on this branch. The required `asf-yaml-check` job in `dev.yml` runs it on every PR and merge-group run. The job fails if a listed job loses the condition, if another job gains it, if an artifact step loses its guard, or if a job outside the skipped set downloads the `cargo-check` artifact. This PR adds no unit tests for the validator. The validator checks the real workflow in a required job, and a test suite for the validator is out of scope for this change. Hosted CI on this PR runs all 26 Rust jobs. The first push to `main` after the merge is the check of the reduced path: seven Rust jobs run and 19 skip. ## Are there any user-facing changes? No. This PR changes CI and the contributor guide only.
1 parent 4780aec commit 8e0c668

3 files changed

Lines changed: 194 additions & 8 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,17 @@
3030
# It is being rolled out incrementally across all workflows.
3131
# Track progress at https://github.com/apache/datafusion/issues/24487.
3232
#
33+
# # Pushes to `main`
34+
#
35+
# Commits reach upstream `main` through the merge queue, which runs this
36+
# workflow at the same SHA. A push to upstream `main` therefore runs only the
37+
# jobs that save caches, publish coverage, or run the non-required FFI check.
38+
# The other jobs carry an `if:` that skips them for that event alone. PRs, the
39+
# merge queue, `workflow_dispatch`, and every other branch run all jobs. A
40+
# direct push that bypasses the queue is not re-checked; dispatch the workflow
41+
# on `main` for a full run. `ci/scripts/check_asf_yaml_status_checks.py` keeps
42+
# the skipped set explicit.
43+
#
3344
name: Rust
3445

3546
concurrency:
@@ -80,6 +91,7 @@ jobs:
8091
run: cargo xtask ci step check workspace
8192
- name: Archive Cargo check artifacts
8293
id: archive-check
94+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
8395
continue-on-error: true
8496
shell: bash
8597
run: |
@@ -91,7 +103,7 @@ jobs:
91103
--exclude='registry/src/*/*/false' \
92104
-cf "$RUNNER_TEMP/cargo-check/check-registry.tar" -C "$CARGO_HOME" registry
93105
- name: Upload Cargo check artifacts
94-
if: steps.archive-check.outcome == 'success'
106+
if: ${{ steps.archive-check.outcome == 'success' && !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
95107
continue-on-error: true
96108
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
97109
with:
@@ -109,6 +121,7 @@ jobs:
109121
linux-datafusion-common-features:
110122
name: cargo check datafusion-common features
111123
needs: linux-build-lib
124+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
112125
runs-on: ubuntu-latest
113126
container:
114127
image: amd64/rust
@@ -139,6 +152,7 @@ jobs:
139152
linux-datafusion-substrait-features:
140153
name: cargo check datafusion-substrait features
141154
needs: linux-build-lib
155+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
142156
env:
143157
CARGO_INCREMENTAL: "0"
144158
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -185,6 +199,7 @@ jobs:
185199
linux-datafusion-proto-features:
186200
name: cargo check datafusion-proto features
187201
needs: linux-build-lib
202+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
188203
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
189204
container:
190205
image: amd64/rust
@@ -247,6 +262,7 @@ jobs:
247262
linux-cargo-check-datafusion:
248263
name: cargo check datafusion features
249264
needs: linux-build-lib
265+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
250266
env:
251267
CARGO_INCREMENTAL: "0"
252268
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -367,6 +383,7 @@ jobs:
367383
linux-cargo-check-datafusion-spark:
368384
name: cargo check datafusion-spark features
369385
needs: linux-build-lib
386+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
370387
runs-on: ubuntu-latest
371388
container:
372389
image: amd64/rust
@@ -448,6 +465,7 @@ jobs:
448465
linux-test-datafusion-cli:
449466
name: cargo test datafusion-cli (amd64)
450467
needs: linux-build-lib
468+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
451469
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
452470
steps:
453471
- uses: runs-on/action@efac073ea2507ec18797de3a81704201ade11d9d # v2.3.1
@@ -525,6 +543,7 @@ jobs:
525543
linux-test-doc:
526544
name: cargo test doc (amd64)
527545
needs: linux-build-lib
546+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
528547
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
529548
container:
530549
image: amd64/rust
@@ -547,6 +566,7 @@ jobs:
547566
linux-rustdoc:
548567
name: cargo doc
549568
needs: linux-build-lib
569+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
550570
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
551571
container:
552572
image: amd64/rust
@@ -562,6 +582,7 @@ jobs:
562582

563583
linux-wasm-pack:
564584
name: build and run with wasm-pack
585+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
565586
runs-on: ubuntu-24.04
566587
steps:
567588
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -589,6 +610,7 @@ jobs:
589610
verify-benchmark-results:
590611
name: verify benchmark results (amd64)
591612
needs: linux-build-lib
613+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
592614
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
593615
container:
594616
image: amd64/rust
@@ -621,6 +643,7 @@ jobs:
621643
sqllogictest-postgres:
622644
name: "Run sqllogictest with Postgres runner"
623645
needs: linux-build-lib
646+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
624647
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
625648
container:
626649
image: amd64/rust
@@ -658,6 +681,7 @@ jobs:
658681
sqllogictest-substrait:
659682
name: "Run sqllogictest in Substrait round-trip mode"
660683
needs: linux-build-lib
684+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
661685
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
662686
container:
663687
image: amd64/rust
@@ -718,6 +742,7 @@ jobs:
718742

719743
vendor:
720744
name: Verify Vendored Code
745+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
721746
runs-on: ubuntu-latest
722747
container:
723748
image: amd64/rust
@@ -735,6 +760,7 @@ jobs:
735760

736761
check-fmt:
737762
name: Check cargo fmt
763+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
738764
runs-on: ubuntu-latest
739765
container:
740766
image: amd64/rust
@@ -750,6 +776,7 @@ jobs:
750776
751777
check-workflow-tool-installs:
752778
name: Check GitHub Actions install tooling
779+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
753780
runs-on: ubuntu-latest
754781
steps:
755782
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -787,6 +814,7 @@ jobs:
787814
cargo-toml-formatting-checks:
788815
name: check Cargo.toml formatting
789816
needs: linux-build-lib
817+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
790818
runs-on: ubuntu-latest
791819
container:
792820
image: amd64/rust
@@ -810,6 +838,7 @@ jobs:
810838
config-docs-check:
811839
name: check configs.md and ***_functions.md is up-to-date
812840
needs: linux-build-lib
841+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
813842
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
814843
container:
815844
image: amd64/rust
@@ -845,6 +874,7 @@ jobs:
845874
examples-docs-check:
846875
name: check example README is up-to-date
847876
needs: linux-build-lib
877+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
848878
runs-on: ubuntu-latest
849879
container:
850880
image: amd64/rust
@@ -877,6 +907,7 @@ jobs:
877907
# - datafusion-cli
878908
msrv:
879909
name: Verify MSRV (Min Supported Rust Version)
910+
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
880911
runs-on: ubuntu-latest
881912
container:
882913
image: amd64/rust

0 commit comments

Comments
 (0)