Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners#4059
Merged
nikola-jokic merged 2 commits intomasterfrom May 14, 2025
Merged
Conversation
…failed ephemeral runners
nikola-jokic
requested review from
a team,
mumoshu,
rentziass and
toast-gear
as code owners
nikola-jokic
added
the
gha-runner-scale-set
|
This would be a big help to us. We're having some issues with our Kubevirt runners that we need to resolve, but this would help us out until we can nail down the precise problem. |
|
Could we please prioritize this PR? We are currently experiencing the same problem as described in #2721, and it is impacting our workflow. |
Collaborator
Author
|
Hey @rr-krupesh-savaliya, I just want to say that depending on the root cause of the failures happening on your cluster, you might still want to inspect them and fix them. This is just a measure that would allow self-healing, but underlying problems should still be fixed, so you can get all the benefits of autoscaling |
Link-
reviewed
May 14, 2025
| } | ||
| return updated.Status.Reason, nil | ||
| }, ephemeralRunnerTimeout, ephemeralRunnerInterval).Should(BeEquivalentTo("TooManyPodFailures"), "Reason should be TooManyPodFailures") | ||
| for i := range 5 { |
Member
There was a problem hiding this comment.
Do we have coverage for partial failures? Do we want to add one more test case to cover behaviour for less than < 5 failures?
Collaborator
Author
There was a problem hiding this comment.
There is a test that covers it on eviction, which kind of covers the case for re-creation, and we have the case that checks the exit code 0 when the runner exists in the service. I feel like these tests are covering the less than 5 failures.
Link-
approved these changes
May 14, 2025
Member
Link-
left a comment
Link-
left a comment
There was a problem hiding this comment.
We already discussed this at length, left a non blocking comment
|
At @nikola-jokic @Link- customer mentioned this PR in a recent ticket: I hope this message finds you well. We are writing with regard to GitHub Self-Hosted Runners. We have noticed that the bug fix we have been awaiting was recently merged into the main branch. Could you please let us know the approximate ETA for a release that includes this fix? Thank you very much for your assistance. |
4 tasks
4 tasks
Merged
4 tasks
Okabe-Junya
added a commit
to mercari/actions-runner-controller
that referenced
this pull request
Jan 5, 2026
* Updates: runner to v2.318.0 container-hooks to v0.6.1 (actions#3684) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.319.0 (actions#3702) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.319.1 (actions#3708) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <[email protected]> * Add exponential backoff when generating runner reg tokens (actions#3724) * Updates: runner to v2.320.0 (actions#3763) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.321.0 container-hooks to v0.6.2 (actions#3809) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ARC e2e tests (actions#3836) * Make EphemeralRunnerController MaxConcurrentReconciles configurable (actions#3832) Co-authored-by: Bassem Dghaidi <[email protected]> * Make EphemeralRunnerReconciler create runner pods earlier (actions#3831) Co-authored-by: Bassem Dghaidi <[email protected]> * Bump github.com/bradleyfalzon/ghinstallation/v2 from 2.8.0 to 2.12.0 (actions#3837) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <[email protected]> * Update docs with details for the dashboard visualizations (actions#3696) Co-authored-by: Bassem Dghaidi <[email protected]> * Make k8s client rate limiter parameters configurable (actions#3848) Co-authored-by: Taketoshi Fujiwara <[email protected]> * Bump golang.org/x/crypto from 0.22.0 to 0.31.0 (actions#3844) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <[email protected]> * Prepare `0.10.0` release (actions#3849) * Fix helm chart bug related to `runnerMaxConcurrentReconciles` (actions#3858) * Prepare `0.10.1` release (actions#3859) * Update dependabot config to group packages (& include actions eco) (actions#3880) * Fix template tests and add go test on gha-validate-chart (actions#3886) * cmd/ghalistener/config: export Validate (actions#3870) Co-authored-by: Han-Wen Nienhuys <[email protected]> * Updated dead link (actions#3830) Co-authored-by: Nikola Jokic <[email protected]> * docs: end markdown code block correctly (actions#3736) * Clarify syntax for `githubConfigSecret` (actions#3812) Co-authored-by: Bassem Dghaidi <[email protected]> * Bump golang.org/x/net from 0.25.0 to 0.33.0 (actions#3881) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nikola Jokic <[email protected]> * Updates: runner to v2.322.0 (actions#3893) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Sanitize labels ending in hyphen, underscore, and dot (actions#3664) * metrics cardinality for ghalistener (actions#3671) Co-authored-by: Bassem Dghaidi <[email protected]> Co-authored-by: Nikola Jokic <[email protected]> * Rename log from target/actual to build/autoscalingRunnerSet version (actions#3957) * Use Ready from the pod conditions when setting it to the EphemeralRunner (actions#3891) * AutoscalingRunnerSet env: not Rendering correctly (actions#3826) Co-authored-by: Nikola Jokic <[email protected]> * Drop verbose flag from runner scale set init-dind-externals copy (actions#3805) * Include custom annotations and labels to all resources created by `gha-runner-scale-set` chart (actions#3934) * Remove old githubrunnerscalesetlistener, remove warning and fix config bug (actions#3937) * Wrap errors in controller helper methods and swap logic in cleanups (actions#3960) * Clean up as much as possible in a single pass for the EphemeralRunner reconciler (actions#3941) * Use gha-runner-scale-set-controller.chart instead of .Chart.Version (actions#3729) Co-authored-by: Nikola Jokic <[email protected]> * Trim volume and container helpers in gha-runner-scale-set (actions#3807) Co-authored-by: Bassem Dghaidi <[email protected]> * Small readme updates for readability (actions#3860) * Update all dependencies, conforming to the new controller-runtime API (actions#3949) * feat: allow namespace overrides (actions#3797) Signed-off-by: Jesús Fernández <[email protected]> Co-authored-by: Nikola Jokic <[email protected]> * chore: Added `OwnerReferences` during resource creation for `EphemeralRunnerSet`, `EphemeralRunner`, and `EphemeralRunnerPod` (actions#3575) * Updates: runner to v2.323.0 (actions#3976) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2 (actions#3984) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add events role permission to leader_election_role (actions#3988) * Create configurable metrics (actions#3975) * Prepare 0.11.0 release (actions#3992) * Fix busy runners metric (actions#4016) * Bump the gomod group across 1 directory with 7 updates (actions#4008) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nikola Jokic <[email protected]> * Include more context to errors raised by github/actions client (actions#4032) Co-authored-by: Copilot <[email protected]> * Pin third party actions (actions#3981) * upgrade(golangci-lint): v2.1.2 (actions#4023) Signed-off-by: karamaru-alpha <[email protected]> * Revised dashboard (actions#4022) * feat(helm): move dind to sidecar (actions#3842) * Fix code block fences (actions#3140) Co-authored-by: Mosè Giordano <[email protected]> * Add missing backtick to metrics.serviceMonitor.namespace line to correct formatting (actions#3790) * Bump go version (actions#4075) * Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners (actions#4059) * Updates: runner to v2.324.0 container-hooks to v0.7.0 (actions#4086) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix docker lint warnings (actions#4074) * Relax version requirements to allow patch version mismatch (actions#4080) Co-authored-by: Copilot <[email protected]> * Refactor resource naming removing unnecessary calculations (actions#4076) * Allow use of client id as an app id (actions#4057) * Updates: runner to v2.325.0 (actions#4109) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add job_workflow_ref label to listener metrics (actions#4054) Signed-off-by: rskmm0chang <[email protected]> * Bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 (actions#4118) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add startup probe to dind side-car (actions#4117) * Avoid nil point when config.Metrics is nil and expose all metrics if none are configured (actions#4101) Co-authored-by: Nikola Jokic <[email protected]> * Add response body to error when fetching access token (actions#4005) Co-authored-by: mluffman <[email protected]> Co-authored-by: Nikola Jokic <[email protected]> * Delete config secret when listener pod gets deleted (actions#4033) Co-authored-by: Nikola Jokic <[email protected]> * Azure Key Vault integration to resolve secrets (actions#4090) * Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 (actions#4120) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Prepare 0.12.0 release (actions#4122) * Bump build-push-action to 6.18.0 (actions#4123) * Remove cache for build-push-action (actions#4124) * Fix indentation of startupProbe attributes in dind sidecar (actions#4126) * Fix dind sidecar template (actions#4128) * Remove duplicate float64 call (actions#4139) * Remove check if runner exists after exit code 0 (actions#4142) * Explicitly requeue during backoff ephemeral runner (actions#4152) * Prepare 0.12.1 release (actions#4153) * Update CodeQL workflow for v3 (global-run-codeql.yaml) (actions#4157) * Bump the actions group across 1 directory with 5 updates (actions#4160) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(runner): add ubuntu 24.04 support (actions#3598) * Fix image pull secrets list arguments in the chart (actions#4164) * Remove workflow actions version comments since upgrades are done via dependabot (actions#4161) * Updates: runner to v2.326.0 (actions#4176) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update example GitHub URLs in values.yaml to include an example for enterprise account-level runners (actions#4181) * Add Missing Languages to CodeQL Advanced Configuration (actions#4179) * Updates: runner to v2.327.0 (actions#4185) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove deprecated preserveUnknownFields from CRDs (actions#4135) * Updates: runner to v2.327.1 (actions#4188) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove JIT config from ephemeral runner status field (actions#4191) * Fix usage of underscore in Runner Scale Set name (actions#3545) Co-authored-by: Nikola Jokic <[email protected]> * Bump docker/login-action from 3.4.0 to 3.5.0 in the actions group (actions#4196) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 4 to 5 in the actions group (actions#4205) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Updates: runner to v2.328.0 (actions#4209) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Requeue if create pod returns already exists error (actions#4201) * docs: fix repo path typo (actions#4229) * Update CODEOWNERS (actions#4251) * Update CODEOWNERS to include new maintainer (actions#4253) * Remove ephemeral runner when exit code != 0 and is patched with the job (actions#4239) * Add workflow name and target labels (actions#4240) * Bump the actions group across 1 directory with 5 updates (actions#4262) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Introduce new kubernetes-novolume mode (actions#4250) Co-authored-by: Copilot <[email protected]> * Ensure ephemeral runner is deleted from the service on exit != 0 (actions#4260) * docs: fix broken Grafana dashboard JSON path (actions#4270) * Potential fix for code scanning alert no. 3: Workflow does not contain permissions (actions#4273) Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 1: Workflow does not contain permissions (actions#4274) Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <[email protected]> Co-authored-by: jiaren-wu <[email protected]> Co-authored-by: Copilot <[email protected]> * Bump all dependencies (actions#4266) * Bump the gomod group across 1 directory with 4 updates (actions#4277) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Prepare 0.13.0 release (actions#4280) * Revert "gha: customize client-go rate limiter params (#4)" This reverts commit 8728190. Keep several instrumentations * Revert "gha: make MaxConcurrentReconciles for each reconciler configurable (#1)" This reverts commit 057a1e7. * chore(chart): bump version to 0.13.0-rc.1 for testing --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Jesús Fernández <[email protected]> Signed-off-by: karamaru-alpha <[email protected]> Signed-off-by: rskmm0chang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <[email protected]> Co-authored-by: Yusuke Kuoka <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ken Muse <[email protected]> Co-authored-by: Taketoshi Fujiwara <[email protected]> Co-authored-by: Rob Herley <[email protected]> Co-authored-by: Nikola Jokic <[email protected]> Co-authored-by: Han-Wen Nienhuys <[email protected]> Co-authored-by: Han-Wen Nienhuys <[email protected]> Co-authored-by: Matteo Bianchi <[email protected]> Co-authored-by: James Ward <[email protected]> Co-authored-by: John Wesley Walker III <[email protected]> Co-authored-by: &es <[email protected]> Co-authored-by: Chris Johnston <[email protected]> Co-authored-by: thinkbiggerltd <[email protected]> Co-authored-by: Cees-Jan Kiewiet <[email protected]> Co-authored-by: Mikey Smet <[email protected]> Co-authored-by: Patrick Vickery <[email protected]> Co-authored-by: Salman Chishti <[email protected]> Co-authored-by: J. Fernández <[email protected]> Co-authored-by: kahirokunn <[email protected]> Co-authored-by: David Maxwell <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Ryosei Karaki <[email protected]> Co-authored-by: Borislav Velkov <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: scodef <[email protected]> Co-authored-by: Ryo Sakamoto <[email protected]> Co-authored-by: Tingluo Huang <[email protected]> Co-authored-by: Nash Luffman <[email protected]> Co-authored-by: mluffman <[email protected]> Co-authored-by: Wim Fournier <[email protected]> Co-authored-by: Jeev B <[email protected]> Co-authored-by: Mark Huijgen <[email protected]> Co-authored-by: calx <[email protected]> Co-authored-by: adjn <[email protected]> Co-authored-by: Ho Kim <[email protected]> Co-authored-by: Cory Calahan <[email protected]> Co-authored-by: Kylie Stradley <[email protected]> Co-authored-by: Alex Hatzenbuhler <[email protected]> Co-authored-by: clechevalli <[email protected]> Co-authored-by: zkpepe <[email protected]> Co-authored-by: Dennis Stone <[email protected]> Co-authored-by: Berat Postalcioglu <[email protected]> Co-authored-by: Jiaren Wu <[email protected]> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <[email protected]> Co-authored-by: jiaren-wu <[email protected]> Co-authored-by: Junya Okabe <[email protected]>
unpollito
pushed a commit
to DistruApp/actions-runner-controller
that referenced
this pull request
Jan 21, 2026
…failed ephemeral runners (actions#4059)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2721
Fixes #3685
Fixes #3700