Skip to content

[CASCL-1386] (10/11) Evict Karpenter user NodePools#3177

Merged
L3n41c merged 5 commits into
mainfrom
lenaic/CASCL-1386-evict-10-karpenter-user
Jul 3, 2026
Merged

[CASCL-1386] (10/11) Evict Karpenter user NodePools#3177
L3n41c merged 5 commits into
mainfrom
lenaic/CASCL-1386-evict-10-karpenter-user

Conversation

@L3n41c

@L3n41c L3n41c commented Jun 18, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Implements Karpenter user NodePool eviction: cordon and drain (Karpenter reclaims the emptied instances).

Motivation

PR #3026 is too large to review as a single change. This is part 10 of 11 of a stack that splits it into small, individually-reviewable pieces that each build and pass tests on their own. See #3026 for the full feature context and end-to-end QA.

Additional Notes

Stacked PR — the base branch is lenaic/CASCL-1386-evict-09-standalone, not main, so the diff shows only this step's change. Reuses the drain primitives introduced in part 8. The code is taken verbatim from #3026 (rebased onto main); there are no logic changes versus #3026.

Minimum Agent Versions

N/A — kubectl-datadog plugin only.

Describe your test plan

go test ./cmd/kubectl-datadog/autoscaling/cluster/... passes at this commit. Full end-to-end QA is tracked on #3026.

Checklist

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 18, 2026

Copy link
Copy Markdown

Code Coverage

Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

Patch coverage: 71.43% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
Patch Coverage: 71.43%
Overall Coverage: 45.38% (+0.01%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 72d803c | Docs | Datadog PR Page | Give us feedback!

@codecov-commenter

codecov-commenter commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.42857% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.42%. Comparing base (a29e4e3) to head (f21be8d).

Files with missing lines Patch % Lines
...atadog/autoscaling/cluster/evict/karpenter_user.go 71.42% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@                            Coverage Diff                            @@
##           lenaic/CASCL-1386-evict-09-standalone    #3177      +/-   ##
=========================================================================
+ Coverage                                  44.41%   44.42%   +0.01%     
=========================================================================
  Files                                        391      391              
  Lines                                      31316    31322       +6     
=========================================================================
+ Hits                                       13908    13914       +6     
+ Misses                                     16509    16508       -1     
- Partials                                     899      900       +1     
Flag Coverage Δ
unittests 44.42% <71.42%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...atadog/autoscaling/cluster/evict/karpenter_user.go 75.00% <71.42%> (+75.00%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a29e4e3...f21be8d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from e6ebd52 to c0b814f Compare June 19, 2026 14:21
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 97fffd2 to c3fcaa0 Compare June 19, 2026 14:21
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from c0b814f to 08904e2 Compare June 19, 2026 14:45
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch 2 times, most recently from 403189a to ae903f8 Compare June 22, 2026 13:24
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from 08904e2 to 43a0d78 Compare June 22, 2026 13:25
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from ae903f8 to 4e1f9b8 Compare June 23, 2026 08:49
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from 43a0d78 to 1505efe Compare June 23, 2026 08:49
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 4e1f9b8 to e551e5a Compare June 25, 2026 08:18
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch 2 times, most recently from 045f39b to f21be8d Compare June 26, 2026 12:16
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from e551e5a to a29e4e3 Compare June 26, 2026 12:16
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from f21be8d to 4fbbbac Compare June 26, 2026 14:45
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from a29e4e3 to 05eef42 Compare June 26, 2026 14:45
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from 4fbbbac to 4ba4925 Compare June 29, 2026 15:41
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 06cff60 to 37b4085 Compare July 1, 2026 21:47
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch 2 times, most recently from e5f9e42 to bc86be8 Compare July 1, 2026 22:58
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 37b4085 to 5304834 Compare July 1, 2026 22:58
L3n41c added 2 commits July 2, 2026 11:38
Introduce the reusable node-cordon and pod-eviction/drain primitives that the
per-manager evictors (EKS managed node groups, ASG, Karpenter, standalone)
build on:

- cordonNodes/cordonNode: idempotent, conflict-retrying cordon that treats an
  already-gone node as a silent skip and returns the cordoned Node objects.
- drainNode: evicts every evictable pod (skipping mirror, DaemonSet, completed
  and terminating pods), then waits for the node to empty.
- evictPodWithRetry: PDB-aware eviction that retries on 429 and treats 404 as
  success.
- listPodsOnNode, waitForNodeEmpty and the pod-classification predicates
  (podOccupiesNode/shouldSkipEviction/isMirrorPod/isDaemonSetPod/isCompleted).

These primitives have no production caller yet; the first (evictEKSManagedNode-
Group) lands in the next PR in the stack. A temporary blank-identifier
reference keeps golangci-lint's `unused` quiet until then (the linter runs with
tests = false, so the unit tests do not count as use) and is removed by that PR.
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 5304834 to 61f3139 Compare July 2, 2026 09:43
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from bc86be8 to a6e1990 Compare July 2, 2026 09:43
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 61f3139 to 3accbec Compare July 2, 2026 12:53
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from a6e1990 to 768f105 Compare July 2, 2026 12:53
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 3accbec to 627768a Compare July 3, 2026 09:17
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from 768f105 to af9257f Compare July 3, 2026 09:17
@L3n41c
L3n41c marked this pull request as ready for review July 3, 2026 09:19
@L3n41c
L3n41c requested review from a team as code owners July 3, 2026 09:19
errs = append(errs, fmt.Errorf("drain node %s: %w", node.Name, err))
}
}
if !drainOpts.DryRun && len(errs) == 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this log rather be at the call site?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.
The “policy” that should be applied to all node group types is to:

  • in case of success, log directly what was done inside the evictXXXXX function.
  • in case of failure, return the errors and let the caller log them.

L3n41c added 2 commits July 3, 2026 11:31
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
Part of a stack splitting #3026 (too large to review in one piece) into
small pieces that each build and pass tests on their own. The command is
fully functional only once the whole stack lands.
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-09-standalone branch from 627768a to 34876a8 Compare July 3, 2026 09:32
@L3n41c
L3n41c force-pushed the lenaic/CASCL-1386-evict-10-karpenter-user branch from af9257f to 72d803c Compare July 3, 2026 09:32
Base automatically changed from lenaic/CASCL-1386-evict-09-standalone to main July 3, 2026 09:53
@L3n41c
L3n41c merged commit 0442077 into main Jul 3, 2026
41 of 42 checks passed
@L3n41c
L3n41c deleted the lenaic/CASCL-1386-evict-10-karpenter-user branch July 3, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants