Repository navigation
Commit d1684ab
authored
ci: cancel obsolete merge queue runs (apache#26760)
## Which issue does this PR close?
Related to apache#25148. Inspired by apache/airflow#75619.
## Rationale for this change
Merge queue runs continue using runners after their group is removed or
replaced. The Rust workflow gives each merge-group SHA a separate
concurrency group, so a replacement does not cancel the old run. Jobs
still waiting for a runner can start long after the queue has stopped
using their results.
A read-only analysis of runs created on October 19–25, 2026 (UTC) found:
| Measurement | Result |
| --- | ---: |
| Merge-group workflow runs inspected | 1,170 |
| Distinct merge-group commits / PRs | 195 / 154 |
| Removed or superseded groups with subsequent runner use | 36 |
| Workflows / jobs with subsequent runner use | 91 / 473 |
| Runner time after removal or replacement | 20.3 hours |
| Jobs that started after removal or replacement | 312, using 12.7
runner-hours |
For the Rust and extended-test workflows, this was 19.0 of 369.4
runner-hours, or 5.2% of their sampled runner time. Cancellation taking
effect one minute after obsolescence would have avoided 17.9
runner-hours; at five minutes, 11.9 hours; at a fixed fifteen-minute
delay, 9.4 hours. A separate model using the next exact quarter-hour
boundary estimates 11.2 hours.
Examples:
- [apache#26562](apache#26562): after
manual queue removal, 34 jobs used another 121.6 runner-minutes in
parallel. They finished within about 14 minutes. [Rust
run](https://github.com/apache/datafusion/actions/runs/40069843292).
- [apache#26373](apache#26373): the group
was removed for failed checks at 21:21 UTC on October 22, but its MSRV
job did not start until 23:20 UTC. [Rust
run](https://github.com/apache/datafusion/actions/runs/39855621556).
The cutoff was an exact-SHA queue-removal event or the creation of a
replacement group for the same PR. For each job, the analysis summed
time after the later of its start and the cutoff plus the assumed
cancellation delay. It excluded skipped jobs, deduplicated successful
jobs copied into rerun records, and excluded seven rerun attempts
started after obsolescence that could have been deliberate debugging.
All Rust and extended-test jobs were fetched for the denominator; other
workflows were inspected when their run metadata showed activity after
the cutoff. Merged groups and all rerun attempts contributed zero to the
reported avoidable time, so preserving both leaves these estimates
unchanged.
These are historical runner times and estimated avoidable portions, not
deployed savings or billing estimates. Of the 312 late-starting jobs, 42
used Runs-On (2.4 runner-hours); 270 used GitHub Linux runners. We have
not measured whether cancelling queued jobs immediately terminates
instances already starting. The models also exclude ref-deletion
latency, cancellation cleanup, scheduler delays, and cleanup-workflow
overhead. Recent runner changes make this sample unsuitable as an annual
forecast.
## What changes are included in this PR?
- Scan when a new merge group is created and every 15 minutes. Ref
deletion can lag the event, so prompt cancellation is not guaranteed.
- Ship in observation mode. Cancellation requires the repository Actions
variable `MERGE_QUEUE_CLEANUP_ENABLED=true`. Document at least one week
of observation before enabling it; there is no automatic activation.
- Log candidate run IDs, SHAs, attempts, reasons, and UTC observation
times for comparison with eventual outcomes. Each merge-group scan has
its own concurrency group, keyed by the group SHA, so a waiting scan on
a group commit is never replaced and shown as cancelled. Scheduled and
manual scans keep one group per mode, so a manual dry run cannot replace
a waiting cancellation scan.
- Target only first-attempt, unfinished `merge_group` runs in
`apache/datafusion` whose exact queue ref is missing or points
elsewhere. Recheck the run and ref, then compare with its target branch
to preserve commits already merged. Preserve reruns, other event types,
and cleanup itself.
- Keep exact-ref detection while the GraphQL queue-entry-to-group-SHA
mapping remains unvalidated. A changed ref target is a safeguard for a
reused queue branch; the suffix is not treated as the run SHA.
- If an Actions run listing reaches the 1,000-result cap, stop the scan
without cancelling any runs. Active merge-group runs stay far below the
cap.
- Share the ref lookup and the target-branch comparison among the
workflows of one group.
- API errors preserve affected runs and allow other candidates to be
checked. Merge-group scans warn; scheduled/manual scans fail visibly.
Recheck HTTP 409 responses and retry unresolved cancellations on later
scans.
- Check out the default branch's script. The workflow definition itself
still comes from the triggering revision.
- Add behavioral tests to Dev CI and the local lint suite; remove the
test that repeated YAML literals.
## What is the testing strategy for this PR?
- 35 cleanup tests cover active/merged groups, missing/replaced refs,
target branches containing slashes, reruns, races, dry runs, API errors,
error reporting by trigger, pagination, the 1,000-result cap, and shared
comparisons. Together with existing fork-CI tests, 59 Node tests pass.
- Mutation checks confirm the merged-commit, rerun, listing-cap and
shared-comparison regressions are detected.
- `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D
warnings`, and the complete `./dev/rust_lint.sh` suite pass, including
generated docs, links, dependency checks, Rust docs, and the HTML
documentation build.
- actionlint 1.7.12 passes for both changed workflows.
- A live read-only probe completed five listing calls with no eligible
candidates. No workflow was cancelled during validation. The observation
period, production cancellation latency, and instance shutdown behavior
remain to be validated after deployment.
## Are there any user-facing changes?
Initially the workflow reports candidates only. Once enabled by a
maintainer, it cancels obsolete first-attempt merge-queue runs while
preserving merged commits and deliberate reruns. Contributors can rerun
obsolete jobs for debugging. No query behavior or public API changes.1 parent c1013ec commit d1684ab
6 files changed
Lines changed: 813 additions & 1 deletion
File tree
- .github/workflows
- ci/scripts
- dev
- docs/source/contributor-guide
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
0 commit comments