Repository navigation
Fix EKS bootstrap readiness, tag enforcement, and failure cleanup - #3448
Merged
gregschohn merged 3 commits intoOct 15, 2026
Merged
gregschohn merged 3 commits into
gregschohn merged 3 commits into
Conversation
AndreKurait
requested review from
Jeremydupras,
akshay2000,
awagen,
gregschohn,
jugal-chauhan,
jzonthemtn,
k-rooot,
nagarajg17,
niravpi,
oeyh,
peternied and
sumobrian
as code owners
October 13, 2026 18:46
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:46 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:54 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:54 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:54 — with
GitHub Actions
Inactive
AndreKurait
had a problem deploying
to
migrations-cicd
October 13, 2026 18:54 — with
GitHub Actions
Failure
AndreKurait
temporarily deployed
to
migrations-cicd
October 13, 2026 18:54 — with
GitHub Actions
Inactive
AndreKurait
marked this pull request as draft
October 13, 2026 21:16
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 03:28 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 03:28 — with
GitHub Actions
Inactive
AndreKurait
had a problem deploying
to
migrations-cicd
October 14, 2026 03:28 — with
GitHub Actions
Error
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 03:28 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 03:28 — with
GitHub Actions
Inactive
AndreKurait
marked this pull request as ready for review
October 14, 2026 03:58
Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 04:21 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 12:59 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 12:59 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 12:59 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 12:59 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
October 14, 2026 12:59 — with
GitHub Actions
Inactive
gregschohn
approved these changes
Oct 15, 2026
gregschohn
left a comment
Collaborator
There was a problem hiding this comment.
As long as this clears all the EKS tests, this LGTM
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix three EKS bootstrap failure paths: premature tag enforcement, accepting an inactive BuildKit builder, and skipping EKS cleanup when bootstrap fails after CloudFormation has already created the cluster.
Tag-enforcement handoff — CDC k6 build #237
Bootstrap attached
MigrationsDenyUntaggedCreatesForTestswhile the built-indefaultNodeClass was still reconciling. Its final untaggedRunInstancesrequests were denied, causing post-test CloudTrail verification to fail despite a successful migration.CloudTrail for CDC k6 build #237 shows the policy attached at 22:04:23 UTC on October 12, the compute update still
InProgressat 22:04:25, denied default-NodeClass requests at 22:04:51–52, and deletion of its launch template at 22:04:55. The tagged replacement's permissions check succeeded during the same interval.The bootstrap now polls the returned EKS update ID until
Successful, then waits for Kubernetes to removesystem,general-purpose, anddefaultbefore enabling enforcement. The replacement pool is applied first. Interrupted reruns also wait for deletion. The tag verifier and deny-policy contents are unchanged.BuildKit readiness and Jenkins compatibility
In EKS integration build #1398, ARM64 startup reported zero ready replicas and a backend TLS error. Bootstrap nevertheless continued with both builders
inactive, andbuildKitBakeAllfailed twice. This run did not enable tagging, so the tag-enforcement fix alone would not address it.running; an inspect command's zero exit status alone is insufficient. Apply the same validation when reusing an existing builder.timeout/gtimeout. This preserves compatibility with the older Jenkins Buildx Kubernetes driver.In CDC + AOSS build #733 and AOSS search build #436, Jenkins rejected the Kubernetes driver option
timeoutduring builder creation. The shared backend now uses the outer deadline and readiness retries, which both jobs already call. The regression harness reproduces that driver rejection and exercises a cold-start retry.Cleanup after a partial bootstrap
bootstrapMApreviously read stack exports only after the entire bootstrap succeeded. A failed image build therefore leftenv.eksClusterNameunset, andeksPostCleanupskipped its application and EKS teardown steps before CloudFormation attempted to delete the VPC.Recover stack exports on bootstrap failure so those existing cleanup steps run, then rethrow the original bootstrap error. If exports are unavailable, log that failure without hiding the original error. Build #1398 ended with a VPC dependency error; its logs do not identify the exact remaining dependency, so this change fixes the demonstrated skipped-cleanup path without assuming which resource remained.
Testing
Based on upstream
mainatb735ab15a.libraries/testAutomationtests pass, including 11 tag-handoff cases and 12 BuildKit cases.invalid driver option timeout for driver kubernetesagainst88841a90cbefore the correction, and now passes.test/awsBootstrapGrantEksAccessOnly.shpasses. Source and assembled bootstrap passbash -n; assembly succeeds.git diff --checkpass. BuildKit backend passes ShellCheck; bootstrap has no new ShellCheck findings versusmain.88841a90c, but failed before image builds because of the unsupported driver option. A full EKS integration run with the compatibility correction is still pending.Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.