[CASCL-1386] (9/11) Evict standalone EC2 nodes#3176
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## lenaic/CASCL-1386-evict-08-drain-and-asg #3176 +/- ##
============================================================================
+ Coverage 44.36% 44.41% +0.04%
============================================================================
Files 391 391
Lines 31296 31316 +20
============================================================================
+ Hits 13886 13908 +22
+ Misses 16511 16509 -2
Partials 899 899
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
June 19, 2026 14:21
22c07f1 to
5bad6e2
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
June 19, 2026 14:21
97fffd2 to
c3fcaa0
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
June 19, 2026 14:45
5bad6e2 to
e2962ee
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
2 times, most recently
from
June 22, 2026 13:24
403189a to
ae903f8
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
June 22, 2026 13:25
e2962ee to
0b04574
Compare
5 tasks
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
June 23, 2026 08:49
ae903f8 to
4e1f9b8
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
June 25, 2026 08:18
1e9f157 to
ef93860
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
June 25, 2026 08:18
4e1f9b8 to
e551e5a
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
June 26, 2026 12:16
ef93860 to
bc40e36
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
2 times, most recently
from
June 26, 2026 14:45
a29e4e3 to
05eef42
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
2 times, most recently
from
June 29, 2026 15:41
1b6d08b to
9b63eda
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
June 29, 2026 15:41
05eef42 to
06cff60
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
July 1, 2026 17:09
9b63eda to
7dc54d8
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
July 1, 2026 21:47
06cff60 to
37b4085
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
July 1, 2026 22:58
7dc54d8 to
21505f9
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
July 1, 2026 22:58
37b4085 to
5304834
Compare
Introduce the reusable node-cordon and pod-eviction/drain primitives that the per-manager evictors (EKS managed node groups, ASG, Karpenter, standalone) build on: - cordonNodes/cordonNode: idempotent, conflict-retrying cordon that treats an already-gone node as a silent skip and returns the cordoned Node objects. - drainNode: evicts every evictable pod (skipping mirror, DaemonSet, completed and terminating pods), then waits for the node to empty. - evictPodWithRetry: PDB-aware eviction that retries on 429 and treats 404 as success. - listPodsOnNode, waitForNodeEmpty and the pod-classification predicates (podOccupiesNode/shouldSkipEviction/isMirrorPod/isDaemonSetPod/isCompleted). These primitives have no production caller yet; the first (evictEKSManagedNode- Group) lands in the next PR in the stack. A temporary blank-identifier reference keeps golangci-lint's `unused` quiet until then (the linter runs with tests = false, so the unit tests do not count as use) and is removed by that PR.
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
July 2, 2026 09:43
21505f9 to
b346c58
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
July 2, 2026 09:43
5304834 to
61f3139
Compare
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-08-drain-and-asg
branch
from
July 2, 2026 12:53
b346c58 to
3757a85
Compare
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
2 times, most recently
from
July 3, 2026 09:17
3accbec to
627768a
Compare
L3n41c
marked this pull request as ready for review
July 3, 2026 09:19
AlexanderYastrebov
approved these changes
Jul 3, 2026
Comment on lines
+23
to
+25
| id, hasInstanceID := commonaws.ExtractEC2InstanceID(node) | ||
| if !hasInstanceID { | ||
| log.Printf("Warning: node %s has unexpected providerID %q; cannot terminate the underlying instance", nodeName, node.Spec.ProviderID) |
Contributor
There was a problem hiding this comment.
nit: maybe move this after the drain since it does not need either id? Otherwise it will show warning and then hand on long-running drain...
Member
Author
There was a problem hiding this comment.
Definitively makes a lot of sense. I’ve just moved that part in 34876a8.
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
L3n41c
force-pushed
the
lenaic/CASCL-1386-evict-09-standalone
branch
from
July 3, 2026 09:32
627768a to
34876a8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Implements standalone EC2 node eviction: cordon, drain and terminate the instances (reusing the drain primitives from part 8).
Motivation
PR #3026 is too large to review as a single change. This is part 9 of 11 of a stack that splits it into small, individually-reviewable pieces that each build and pass tests on their own. See #3026 for the full feature context and end-to-end QA.
Additional Notes
Stacked PR — the base branch is
lenaic/CASCL-1386-evict-08-drain-and-asg, notmain, so the diff shows only this step's change. Reuses the drain primitives introduced in part 8. The code is taken verbatim from #3026 (rebased ontomain); there are no logic changes versus #3026.Minimum Agent Versions
N/A —
kubectl-datadogplugin only.Describe your test plan
go test ./cmd/kubectl-datadog/autoscaling/cluster/...passes at this commit. Full end-to-end QA is tracked on #3026.Checklist
qa/skip-qalabel applied (not independently shippable; QA tracked on [CASCL-1386] Add evict-legacy-nodes subcommand to drain non-Datadog node groups #3026)