Repository navigation
CI: Merge queue rejects PR even after all tests pass #23400
Description
Describe the bug
When merging a PR, the merge queue should:
- Process PRs one by one.
- For each PR, apply its diff on top of
main, ensure there are no merge conflicts, verify that all CI tests pass, and then merge it.
However, it seems possible for the merge queue to reject a PR even when all CI tests have passed. It would be helpful to understand why this happens.
The failure case:
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
Phoenix500526 commentedon Jul 10, 2026
I investigated the merge-group run for #23324.
The merge queue did not remove the PR after a successful CI run. It removed it after attempt 1 of the Rust workflow failed on merge-group commit 301e684.
At 19:16:38 UTC, four required jobs were marked as failed:
Their logs show that the RunsOn runners received shutdown signals. GitHub removed the PR from the merge queue at 19:17:05.
The RunsOn bot started attempt 2 at 19:22:40, after the PR had already been removed. That retry passed, and GitHub now displays the latest attempt, which is why the checks currently appear green.
The affected runners were Spot instances, so this was likely a transient runner interruption, although RunsOn or CloudWatch logs would be needed to confirm the underlying infrastructure event.
Therefore, the merge queue did not reject a successful run. It reacted to the failed first attempt, while the UI now shows the successful retry. Re-adding the PR created a new merge-group run, which passed and merged successfully.
Phoenix500526 commentedon Jul 10, 2026
🤔 Maybe we can add an automatic requeue bot for it.
2010YOUY01 commentedon Jul 11, 2026
Thank you for the investigation! @Phoenix500526
I think further investigation requires admin access, which I don’t have (perhaps in Github Web UI -> top-right setting button 🤔 ) @alamb
alamb commentedon Jul 12, 2026
Unfortunately I don't have GUI access to github settings either -- it is all controlled by the ASF INfra team (and we self serve via
.asf.yaml)I thought the way the merge queue works is that several PRs are queued up together and if any one fails it boots the whole thing out of of the queue. I think @blaginin helped set up the merge queue and knows more.
From my perspective the merge queue is doing a great job of avoiding logical conflicts from landing on main, even though it sometimes does cause a PR to be removed from the uqeue
blaginin commentedon Jul 16, 2026
Hey! This may be happening because we are using spot instances- those are much cheaper although sometimes AWS may take them back during the run
rluvaton commentedon Sep 8, 2026
merge queue related - maybe it is the spot instances
Having:
and it is happening again
While no checks are running, either all pending or all passing
blaginin commentedon Sep 8, 2026
let's see if this time it passes
blaginin commentedon Sep 8, 2026
i think something is wrong with the runners
blaginin commentedon Sep 8, 2026
I pinged @gmcdonald - i think the issue is around ASF CI in general, not related to merge queue
blaginin commentedon Sep 8, 2026
moved to #25088
alamb commentedon Sep 8, 2026
Closing this one as I think it has been resolved
alamb commentedon Oct 14, 2026
We are seeing the same symptom. I filed a new issue: