Skip to content

[CASCL-1386] (9/11) Evict standalone EC2 nodes#3176

Merged
L3n41c merged 4 commits into
mainfrom
lenaic/CASCL-1386-evict-09-standalone
Jul 3, 2026
Merged

[CASCL-1386] (9/11) Evict standalone EC2 nodes#3176
L3n41c merged 4 commits into
mainfrom
lenaic/CASCL-1386-evict-09-standalone

Conversation

@L3n41c

@L3n41c L3n41c commented Jun 18, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Implements standalone EC2 node eviction: cordon, drain and terminate the instances (reusing the drain primitives from part 8).

Motivation

PR #3026 is too large to review as a single change. This is part 9 of 11 of a stack that splits it into small, individually-reviewable pieces that each build and pass tests on their own. See #3026 for the full feature context and end-to-end QA.

Additional Notes

Stacked PR — the base branch is lenaic/CASCL-1386-evict-08-drain-and-asg, not main, so the diff shows only this step's change. Reuses the drain primitives introduced in part 8. The code is taken verbatim from #3026 (rebased onto main); there are no logic changes versus #3026.

Minimum Agent Versions

N/A — kubectl-datadog plugin only.

Describe your test plan

go test ./cmd/kubectl-datadog/autoscaling/cluster/... passes at this commit. Full end-to-end QA is tracked on #3026.

Checklist

@codecov-commenter

codecov-commenter commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.41%. Comparing base (bc40e36) to head (a29e4e3).

Additional details and impacted files

Impacted file tree graph

@@                             Coverage Diff                              @@
##           lenaic/CASCL-1386-evict-08-drain-and-asg    #3176      +/-   ##
============================================================================
+ Coverage                                     44.36%   44.41%   +0.04%     
============================================================================
  Files                                           391      391              
  Lines                                         31296    31316      +20     
============================================================================
+ Hits                                          13886    13908      +22     
+ Misses                                        16511    16509       -2     
  Partials                                        899      899              
Flag Coverage Δ
unittests 44.41% <100.00%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...tl-datadog/autoscaling/cluster/evict/standalone.go 100.00% <100.00%> (+100.00%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc40e36...a29e4e3. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@datadog-official

This comment has been minimized.

@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 22c07f1 to 5bad6e2 Compare June 19, 2026 14:21
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 97fffd2 to c3fcaa0 Compare June 19, 2026 14:21
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 5bad6e2 to e2962ee Compare June 19, 2026 14:45
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch 2 times, most recently from 403189a to ae903f8 Compare June 22, 2026 13:24
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from e2962ee to 0b04574 Compare June 22, 2026 13:25
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from ae903f8 to 4e1f9b8 Compare June 23, 2026 08:49
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 1e9f157 to ef93860 Compare June 25, 2026 08:18
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 4e1f9b8 to e551e5a Compare June 25, 2026 08:18
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from ef93860 to bc40e36 Compare June 26, 2026 12:16
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch 2 times, most recently from a29e4e3 to 05eef42 Compare June 26, 2026 14:45
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch 2 times, most recently from 1b6d08b to 9b63eda Compare June 29, 2026 15:41
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 05eef42 to 06cff60 Compare June 29, 2026 15:41
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 9b63eda to 7dc54d8 Compare July 1, 2026 17:09
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 06cff60 to 37b4085 Compare July 1, 2026 21:47
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 7dc54d8 to 21505f9 Compare July 1, 2026 22:58
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 37b4085 to 5304834 Compare July 1, 2026 22:58
L3n41c added 2 commits July 2, 2026 11:38
Introduce the reusable node-cordon and pod-eviction/drain primitives that the
per-manager evictors (EKS managed node groups, ASG, Karpenter, standalone)
build on:

- cordonNodes/cordonNode: idempotent, conflict-retrying cordon that treats an
  already-gone node as a silent skip and returns the cordoned Node objects.
- drainNode: evicts every evictable pod (skipping mirror, DaemonSet, completed
  and terminating pods), then waits for the node to empty.
- evictPodWithRetry: PDB-aware eviction that retries on 429 and treats 404 as
  success.
- listPodsOnNode, waitForNodeEmpty and the pod-classification predicates
  (podOccupiesNode/shouldSkipEviction/isMirrorPod/isDaemonSetPod/isCompleted).

These primitives have no production caller yet; the first (evictEKSManagedNode-
Group) lands in the next PR in the stack. A temporary blank-identifier
reference keeps golangci-lint's `unused` quiet until then (the linter runs with
tests = false, so the unit tests do not count as use) and is removed by that PR.
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from 21505f9 to b346c58 Compare July 2, 2026 09:43
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 5304834 to 61f3139 Compare July 2, 2026 09:43
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-08-drain-and-asg branch from b346c58 to 3757a85 Compare July 2, 2026 12:53
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch 2 times, most recently from 3accbec to 627768a Compare July 3, 2026 09:17
@L3n41c
L3n41c marked this pull request as ready for review July 3, 2026 09:19
@L3n41c
L3n41c requested review from a team as code owners July 3, 2026 09:19
Comment on lines +23 to +25
id, hasInstanceID := commonaws.ExtractEC2InstanceID(node)
if !hasInstanceID {
log.Printf("Warning: node %s has unexpected providerID %q; cannot terminate the underlying instance", nodeName, node.Spec.ProviderID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe move this after the drain since it does not need either id? Otherwise it will show warning and then hand on long-running drain...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitively makes a lot of sense. I’ve just moved that part in 34876a8.

Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 627768a to 34876a8 Compare July 3, 2026 09:32
Base automatically changed from lenaic/CASCL-1386-evict-08-drain-and-asg to main July 3, 2026 09:35
@L3n41c
L3n41c merged commit e95a33a into main Jul 3, 2026
27 of 36 checks passed
@L3n41c
L3n41c deleted the lenaic/CASCL-1386-evict-09-standalone branch July 3, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants