Skip to content

Fix EKS bootstrap readiness, tag enforcement, and failure cleanup - #3448

Merged
gregschohn merged 3 commits into
opensearch-project:mainfrom
AndreKurait:fix/eks-tag-enforcement-bootstrap
Oct 15, 2026
Merged

gregschohn merged 3 commits into
opensearch-project:mainfrom
AndreKurait:fix/eks-tag-enforcement-bootstrap

Conversation

@AndreKurait

@AndreKurait AndreKurait commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Description

Fix three EKS bootstrap failure paths: premature tag enforcement, accepting an inactive BuildKit builder, and skipping EKS cleanup when bootstrap fails after CloudFormation has already created the cluster.

Tag-enforcement handoff — CDC k6 build #237

Bootstrap attached MigrationsDenyUntaggedCreatesForTests while the built-in default NodeClass was still reconciling. Its final untagged RunInstances requests were denied, causing post-test CloudTrail verification to fail despite a successful migration.

CloudTrail for CDC k6 build #237 shows the policy attached at 22:04:23 UTC on October 12, the compute update still InProgress at 22:04:25, denied default-NodeClass requests at 22:04:51–52, and deletion of its launch template at 22:04:55. The tagged replacement's permissions check succeeded during the same interval.

The bootstrap now polls the returned EKS update ID until Successful, then waits for Kubernetes to remove system, general-purpose, and default before enabling enforcement. The replacement pool is applied first. Interrupted reruns also wait for deletion. The tag verifier and deny-policy contents are unchanged.

BuildKit readiness and Jenkins compatibility

In EKS integration build #1398, ARM64 startup reported zero ready replicas and a backend TLS error. Bootstrap nevertheless continued with both builders inactive, and buildKitBakeAll failed twice. This run did not enable tagging, so the tag-enforcement fix alone would not address it.

  • Require both named native builders to report running; an inspect command's zero exit status alone is insufficient. Apply the same validation when reusing an existing builder.
  • Enforce the configured cold-start budget around the inspect/retry loop, bounding each inspect by the remaining time with timeout/gtimeout. This preserves compatibility with the older Jenkins Buildx Kubernetes driver.
  • Retry unsuccessful readiness checks within the overall deadline and print pod/NodeClaim diagnostics when attempts or time run out.

In CDC + AOSS build #733 and AOSS search build #436, Jenkins rejected the Kubernetes driver option timeout during builder creation. The shared backend now uses the outer deadline and readiness retries, which both jobs already call. The regression harness reproduces that driver rejection and exercises a cold-start retry.

Cleanup after a partial bootstrap

bootstrapMA previously read stack exports only after the entire bootstrap succeeded. A failed image build therefore left env.eksClusterName unset, and eksPostCleanup skipped its application and EKS teardown steps before CloudFormation attempted to delete the VPC.

Recover stack exports on bootstrap failure so those existing cleanup steps run, then rethrow the original bootstrap error. If exports are unavailable, log that failure without hiding the original error. Build #1398 ended with a VPC dependency error; its logs do not identify the exact remaining dependency, so this change fixes the demonstrated skipped-cleanup path without assuming which resource remained.

Testing

Based on upstream main at b735ab15a.

  • All 150 libraries/testAutomation tests pass, including 11 tag-handoff cases and 12 BuildKit cases.
  • Four Groovy regression scenarios pass: normal bootstrap, failed bootstrap with recoverable exports, failed bootstrap without exports, and export failure after successful bootstrap. Recoverable cases verify application/EKS cleanup precedes stack deletion.
  • The inactive-builder and missing-cleanup-metadata regression cases fail against the original code and pass with these changes. The older-driver regression reproduced invalid driver option timeout for driver kubernetes against 88841a90c before the correction, and now passes.
  • test/awsBootstrapGrantEksAccessOnly.sh passes. Source and assembled bootstrap pass bash -n; assembly succeeds.
  • Python lint and git diff --check pass. BuildKit backend passes ShellCheck; bootstrap has no new ShellCheck findings versus main.
  • Builds Improve elasticsearch test console #733 and Add Apache 2.0 license headers to Fetch Migration python code #436 verified failure-time export recovery and completion of CloudFormation cleanup at 88841a90c, but failed before image builds because of the unsupported driver option. A full EKS integration run with the compatibility correction is still pending.

Check List

  • New functionality includes testing
  • Public documentation issue/PR created, if applicable. Not applicable: bootstrap synchronization and failure handling; the BuildKit development guide is updated here.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@AndreKurait
AndreKurait marked this pull request as draft October 13, 2026 21:16
@AndreKurait
AndreKurait marked this pull request as ready for review October 14, 2026 03:58

@gregschohn gregschohn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as this clears all the EKS tests, this LGTM

This branch was previously deployed

1 inactive deployment
migrations-cicd — 6806ace5 Deployed Oct 15, 2026 by AndreKurait via eks-aoss-integ-test #12457
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants