[CASCL-1386] (10/11) Evict Karpenter user NodePools#3177
Conversation
🛑 Gate Violations
ℹ️ Info🎯 Code Coverage (details) Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 72d803c | Docs | Datadog PR Page | Give us feedback! |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## lenaic/CASCL-1386-evict-09-standalone #3177 +/- ##
=========================================================================
+ Coverage 44.41% 44.42% +0.01%
=========================================================================
Files 391 391
Lines 31316 31322 +6
=========================================================================
+ Hits 13908 13914 +6
+ Misses 16509 16508 -1
- Partials 899 900 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
e6ebd52 to
c0b814f
Compare
97fffd2 to
c3fcaa0
Compare
c0b814f to
08904e2
Compare
403189a to
ae903f8
Compare
08904e2 to
43a0d78
Compare
ae903f8 to
4e1f9b8
Compare
43a0d78 to
1505efe
Compare
4e1f9b8 to
e551e5a
Compare
045f39b to
f21be8d
Compare
e551e5a to
a29e4e3
Compare
f21be8d to
4fbbbac
Compare
a29e4e3 to
05eef42
Compare
4fbbbac to
4ba4925
Compare
06cff60 to
37b4085
Compare
e5f9e42 to
bc86be8
Compare
37b4085 to
5304834
Compare
Introduce the reusable node-cordon and pod-eviction/drain primitives that the per-manager evictors (EKS managed node groups, ASG, Karpenter, standalone) build on: - cordonNodes/cordonNode: idempotent, conflict-retrying cordon that treats an already-gone node as a silent skip and returns the cordoned Node objects. - drainNode: evicts every evictable pod (skipping mirror, DaemonSet, completed and terminating pods), then waits for the node to empty. - evictPodWithRetry: PDB-aware eviction that retries on 429 and treats 404 as success. - listPodsOnNode, waitForNodeEmpty and the pod-classification predicates (podOccupiesNode/shouldSkipEviction/isMirrorPod/isDaemonSetPod/isCompleted). These primitives have no production caller yet; the first (evictEKSManagedNode- Group) lands in the next PR in the stack. A temporary blank-identifier reference keeps golangci-lint's `unused` quiet until then (the linter runs with tests = false, so the unit tests do not count as use) and is removed by that PR.
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
5304834 to
61f3139
Compare
bc86be8 to
a6e1990
Compare
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
61f3139 to
3accbec
Compare
a6e1990 to
768f105
Compare
3accbec to
627768a
Compare
768f105 to
af9257f
Compare
| errs = append(errs, fmt.Errorf("drain node %s: %w", node.Name, err)) | ||
| } | ||
| } | ||
| if !drainOpts.DryRun && len(errs) == 0 { |
There was a problem hiding this comment.
Should this log rather be at the call site?
There was a problem hiding this comment.
Good point.
The “policy” that should be applied to all node group types is to:
- in case of success, log directly what was done inside the
evictXXXXXfunction. - in case of failure, return the errors and let the caller log them.
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
Part of a stack splitting #3026 (too large to review in one piece) into small pieces that each build and pass tests on their own. The command is fully functional only once the whole stack lands.
627768a to
34876a8
Compare
af9257f to
72d803c
Compare
What does this PR do?
Implements Karpenter user NodePool eviction: cordon and drain (Karpenter reclaims the emptied instances).
Motivation
PR #3026 is too large to review as a single change. This is part 10 of 11 of a stack that splits it into small, individually-reviewable pieces that each build and pass tests on their own. See #3026 for the full feature context and end-to-end QA.
Additional Notes
Stacked PR — the base branch is
lenaic/CASCL-1386-evict-09-standalone, notmain, so the diff shows only this step's change. Reuses the drain primitives introduced in part 8. The code is taken verbatim from #3026 (rebased ontomain); there are no logic changes versus #3026.Minimum Agent Versions
N/A —
kubectl-datadogplugin only.Describe your test plan
go test ./cmd/kubectl-datadog/autoscaling/cluster/...passes at this commit. Full end-to-end QA is tracked on #3026.Checklist
qa/skip-qalabel applied (not independently shippable; QA tracked on [CASCL-1386] Add evict-legacy-nodes subcommand to drain non-Datadog node groups #3026)