test/new-e2e/tests/autoscaling/spot: add spot scheduling local kind cluster e2e tests by AlexanderYastrebov · Pull Request #48821 · DataDog/datadog-agent

AlexanderYastrebov · 2026-04-02T18:39:24Z

What does this PR do?

Add spot scheduling e2e test suite using a locally-built cluster-agent image loaded into kind.
In the subsequent steps I plan to expand it into CI-compatible test in EKS cluster.

Motivation

Verify spot scheduling works.

Describe how you validated your changes

See README.md

Additional Notes

Updates https://datadoghq.atlassian.net/browse/CASCL-1222

Exclude test package from bazel build for now similra to other e2e packages.

dd-octo-sts · 2026-04-20T11:39:58Z

Files inventory check summary

File checks results against ancestor 3eb519ee:

Results for datadog-agent_7.80.0~devel.git.258.435a3e4.pipeline.109943923-1_amd64.deb:

No change detected

…tion (#48684) ### What does this PR do? Add support for customizing kind cluster topology and pre-loading Docker images into the cluster before the agent is deployed. ### Motivation Local e2e tests for Spot Scheduling #47429 ### Describe how you validated your changes I am testing spot scheduling via #48821 and extracted these changes into a separate PR. ### Additional Notes Co-authored-by: alexander.yastrebov <[email protected]>

AlexanderYastrebov · 2026-04-21T13:19:02Z

+	image := os.Getenv("DD_TEST_CLUSTER_AGENT_IMAGE")
+	if image == "" {
+		t.Skip("DD_TEST_CLUSTER_AGENT_IMAGE not set; skipping spot scheduling e2e tests")
+	}


This is necessary atm because spot scheduling is not released yet. After a while I plan to relax this check and allow testing both custom built image for development as well as running tests against published image.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6934a8e1fa

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-04-21T13:21:00Z

+		switch node.Labels[karpenterCapacityTypeLabel] {
+		case karpenterCapacityTypeSpot:
+			s.spotNode = node.Name
+		default:
+			s.onDemandNode = node.Name


Exclude control-plane nodes when selecting on-demand worker

identifyNodes treats every non-spot node as the on-demand target, so in a kind cluster it can select the control-plane node (which is also non-spot) instead of the labeled on-demand worker. Because the control-plane is typically NoSchedule, expectRunningOnDemand then counts zero pods on s.onDemandNode and the suite fails/flakes depending on node list ordering. Filter to worker nodes (or explicitly require karpenter.sh/capacity-type=on-demand) when assigning s.onDemandNode to make placement assertions deterministic.

Useful? React with 👍 / 👎.

Nah, its spot vs non-spot really because spot scheduler by design does not modify non-spot pods and they can end up on control plane node as well which is OK.

chouetz

Minor suggestion

…luster e2e tests Add spot scheduling e2e test suite using a locally-built cluster-agent image loaded into kind. Exclude test package from bazel build for now similar to other e2e packages. Co-Authored-By Claude Sonnet 4.6 <[email protected]>

…ter e2e tests Add a CI path for the spot scheduling e2e test suite using an AWS kindvm provisioner (kind running on an EC2 VM), alongside the existing local kind path. - Add `TestSpotSchedulingKindCI` — skips unless `E2E_PIPELINE_ID` is set; uses the AWS kindvm provisioner with the `cluster-agent-qa` ECR image auto-resolved by the framework from pipeline/commit env vars. - Extend `scenarios/aws/kindvm` with `workerNodes` support (`WithKindWorkerNodes` option + `NewKindClusterWithConfig` call) so the cloud kindvm path can provision spot/on-demand node topology, matching what `localkubernetes.Provisioner` already supported. - Add `new-e2e-autoscaling` GitLab CI job (depends on `qa_dca` for the ECR image, `allow_failure: true`). - Add `.on_autoscaling_or_e2e_changes` trigger rule covering the spot test directory and `pkg/clusteragent/autoscaling/cluster/`. The local-only e2e tests introduced in #48821 cannot catch regressions introduced on the main branch because they require a Docker daemon and a locally-built image — neither of which is available on CI pod runners. Running on AWS kindvm closes this gap: every change to the spot scheduler or its tests now triggers a real cluster verification automatically. Updates https://datadoghq.atlassian.net/browse/CASCL-1222 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…ter e2e tests (#49985) ### What does this PR do? Add a CI path for the spot scheduling e2e test suite using an AWS kindvm provisioner (kind running on an EC2 VM), alongside the existing local kind path. ### Motivation The local-only e2e tests introduced in #48821 cannot catch regressions introduced on the main branch because they require a Docker daemon and a locally-built image — neither of which is available on CI pod runners. Running on AWS kindvm closes this gap: every change to the spot scheduler or its tests now triggers a real cluster verification automatically. ### Describe how you validated your changes Newly added `new-e2e-autoscaling` pipeline passed. ### Additional Notes - Add `TestSpotSchedulingKindCI` — skips unless `E2E_PIPELINE_ID` is set; uses the AWS kindvm provisioner with the `cluster-agent-qa` ECR image auto-resolved by the framework from pipeline/commit env vars. - Extend `scenarios/aws/kindvm` with `workerNodes` support (`WithKindWorkerNodes` option + `NewKindClusterWithConfig` call) so the cloud kindvm path can provision spot/on-demand node topology, matching what `localkubernetes.Provisioner` already supported. - Add `new-e2e-autoscaling` GitLab CI job (depends on `qa_dca` for the ECR image, `allow_failure: true`). - Add `.on_autoscaling_or_e2e_changes` trigger rule covering the spot test directory and `pkg/clusteragent/autoscaling/cluster/`. Updates https://datadoghq.atlassian.net/browse/CASCL-1222 Co-authored-by: alexander.yastrebov <[email protected]>

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from 5502ac2 to 118bc84 Compare April 2, 2026 18:40

AlexanderYastrebov changed the title ~~test/e2e-framework: add spot scheduling local kind cluster e2e tests~~ test/new-e2e/tests/autoscaling/spot: add spot scheduling local kind cluster e2e tests Apr 2, 2026

Base automatically changed from alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-2 to main April 16, 2026 23:02

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from 118bc84 to ed544e3 Compare April 20, 2026 11:15

dd-octo-sts Bot added internal Identify a non-fork PR team/agent-devx labels Apr 20, 2026

github-actions Bot added the long review PR is complex, plan time to review it label Apr 20, 2026

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from ed544e3 to 2b74a50 Compare April 20, 2026 11:16

AlexanderYastrebov mentioned this pull request Apr 20, 2026

test/e2e-framework: support kind cluster topology and image customization #48684

Merged

AlexanderYastrebov added changelog/no-changelog No changelog entry needed qa/done QA done before merge and regressions are covered by tests labels Apr 20, 2026

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch 3 times, most recently from c033ca3 to eb77f83 Compare April 20, 2026 15:31

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from eb77f83 to 6934a8e Compare April 21, 2026 12:45

AlexanderYastrebov marked this pull request as ready for review April 21, 2026 13:16

AlexanderYastrebov requested a review from a team as a code owner April 21, 2026 13:16

AlexanderYastrebov commented Apr 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 21, 2026

View reviewed changes

chouetz approved these changes Apr 23, 2026

View reviewed changes

Comment thread test/new-e2e/tests/autoscaling/spot/deployment_test.go Outdated

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch 2 times, most recently from 2a9104c to bc58e9f Compare April 24, 2026 16:17

AlexanderYastrebov requested a review from a team as a code owner April 24, 2026 16:17

dd-octo-sts Bot added the team/agent-build label Apr 24, 2026

aiuto approved these changes Apr 27, 2026

View reviewed changes

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch 2 times, most recently from 315c83e to 50ae720 Compare April 27, 2026 09:05

This comment has been minimized.

Sign in to view

L3n41c reviewed Apr 27, 2026

View reviewed changes

Comment thread test/new-e2e/tests/autoscaling/spot/suite_test.go

Comment thread test/new-e2e/tests/autoscaling/spot/suite_test.go

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from 50ae720 to fb13f55 Compare April 27, 2026 14:18

AlexanderYastrebov force-pushed the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch from fb13f55 to 435a3e4 Compare April 27, 2026 15:55

gh-worker-dd-mergequeue-cf854d Bot merged commit f77729f into main Apr 27, 2026
223 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local branch April 27, 2026 18:26

github-actions Bot added this to the 7.80.0 milestone Apr 27, 2026

AlexanderYastrebov mentioned this pull request Apr 28, 2026

test/new-e2e/tests/autoscaling/spot: add spot scheduling CI kind cluster e2e tests #49985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test/new-e2e/tests/autoscaling/spot: add spot scheduling local kind cluster e2e tests#48821

test/new-e2e/tests/autoscaling/spot: add spot scheduling local kind cluster e2e tests#48821
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
alexander.yastrebov/pkg/clusteragent/autoscaling/spot-scheduling-e2e-local

AlexanderYastrebov commented Apr 2, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Apr 20, 2026 •

edited

Loading

Uh oh!

AlexanderYastrebov Apr 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Uh oh!

AlexanderYastrebov Apr 21, 2026 •

edited

Loading

Uh oh!

chouetz left a comment

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AlexanderYastrebov commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

dd-octo-sts Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files inventory check summary

Results for datadog-agent_7.80.0~devel.git.258.435a3e4.pipeline.109943923-1_amd64.deb:

Uh oh!

AlexanderYastrebov Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

AlexanderYastrebov Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chouetz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlexanderYastrebov commented Apr 2, 2026 •

edited

Loading

dd-octo-sts Bot commented Apr 20, 2026 •

edited

Loading

AlexanderYastrebov Apr 21, 2026 •

edited

Loading