Repository navigation
Jobs not being picked up occasionally #48
Description
I'm seeing a new failure mode recently.
In mldsa-native/mlkem-native, we start 4 relatively fast jobs on the RISE runners for each PR. Since recently (maybe yesterday), I often see 3 of the 4 jobs being picked up very quickly, but one does not get picked up for a long time even after the other 3 finish. Sometimes it helps to cancel the job and restart it.
For example here the 4th job wasn't picked up for almost 2 hours until I cancelled it and restarted it. The restarted one is not yet picked up after 30 minutes even though I think you are not under heavy load right now according to the dashboard.
The dashboard shows the job as pending and a corresponding worker as "active":
Activity
mkannwischer commentedon Jun 5, 2026
The same pattern occured now here: https://github.com/pq-code-package/mlkem-native/actions/runs/27008198416/job/79726517694?pr=1648 with 3 jobs being picked up instantly, while the 4th one remained pending.
Eventually the mldsa-native job I mentioned above got picked up, so maybe this is a problem with the start-up of the runner being slow?
mkannwischer commentedon Jun 5, 2026
Btw, we have recently changed our CI, so the RISE jobs only run if the base CI passes to reduce load for you.
If there are still load issues, we could reduce the number of tests.
mkannwischer commentedon Jun 7, 2026
This one was waiting for almost 6 hours before I cancelled it: https://github.com/pq-code-package/mlkem-native/actions/runs/27063469189/job/79898328014?pr=1677
We are not the only ones affected by this: https://github.com/kairos-io/hadron/actions/runs/27069994409/job/79910245453?pr=406 has been waiting for 2.5h even though there is only one job running on the RISE runners.
luhenry commentedon Jun 18, 2026
Just seeing you issue here, let me have a look.
mkannwischer commentedon Jun 19, 2026
Thanks. I have cut our CI down to run just one job for each PR on the rise runners. Since then I have not seen any more issues.
Bump peter-evans/create-pull-request from 7 to 8 (riseproject-dev#48)