Pod level in place pod resize - alpha by ndixita · Pull Request #132919 · kubernetes/kubernetes

ndixita · 2025-07-13T22:22:19Z

What type of PR is this?

/kind feature

/kind api-change

What this PR does / why we need it:

This PR adds support for IPPR + Pod Level resources to allow in place resizing of pod-level resources in alpha stage.

Which issue(s) this PR is related to:

Issue: kubernetes/enhancements#5419

Reference KEPs when applicable in addition to specific issues.

KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/5419-pod-level-resources-in-place-resize/README.md
https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2837-pod-level-resource-spec/README.md

Does this PR introduce a user-facing change?

This change allows In Place Resize of Pod Level Resources 
- Add Resources in PodStatus to capture resources set at pod-level cgroup
- Add AllocatedResources in PodStatus to capture resources requested in the PodSpec

k8s-ci-robot · 2025-07-13T22:22:21Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-07-13T22:22:28Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ndixita · 2025-07-13T23:22:26Z

/ok-to-test

tallclair · 2025-11-11T19:26:34Z

The WiFi isn't working at the conference, so I won't be able to review this until I get back to my hotel room tonight.

@natasha41575 can you help review today? If you lgtm I'll try to just approve tonight.

natasha41575 · 2025-11-11T19:23:49Z

+// comparison (DeepEqual) against oldPod.Spec. If the specs match after masking,
+// it guarantees that no changes were made other than the allowed resource modifications
+// needed for the resize.
+func validatePodLevelResourcesResize(newPod, oldPod *core.Pod, podSpecToMutate *core.PodSpec, specPath *field.Path, opts PodValidationOptions) field.ErrorList {


Add some of these cases to the e2e tests in test/e2e/common/node/pod_resize.go, in doPodResizePatchErrorTests

natasha41575 · 2025-11-11T19:27:41Z

+	// owner: @ndixita
+	// kep: https://kep.k8s.io/5419
+	//
+	// Enables specifying resources at pod-level.


update the description

natasha41575 · 2025-11-11T19:33:45Z

 	GetContainerResources(podUID types.UID, containerName string) (v1.ResourceRequirements, bool)
 	GetPodResourceInfoMap() PodResourceInfoMap
 	GetPodResourceInfo(podUID types.UID) (PodResourceInfo, bool)
+	GetPodLevelResources(podUID types.UID) (*v1.ResourceRequirements, bool)


(tiny optional nit) the analogous GetContainerResources returns (v1.ResourceRequirements, bool), should we keep it consistent and have the same return values?

natasha41575 · 2025-11-11T19:33:55Z

 type writer interface {
 	SetContainerResources(podUID types.UID, containerName string, resources v1.ResourceRequirements) error
 	SetPodResourceInfo(podUID types.UID, resourceInfo PodResourceInfo) error
+	SetPodLevelResources(podUID types.UID, alloc *v1.ResourceRequirements) error


similar comment here

natasha41575 · 2025-11-11T19:39:27Z

+
+	podInfo, ok := s.podResources[podUID]
+	if !ok {
+		podInfo.PodLevelResources = &v1.ResourceRequirements{}


does this line do anything? You are immediately overwriting it in the next line

natasha41575 · 2025-11-11T19:50:40Z

 			expectPodSyncTriggered: "true",
 		},
+		{
+			name:                         "pod-level: Request CPU and memory increase - expect InProgress",


Thanks for adding these cases. They look good. I would like to request one or two cases with both pod level and container level resources

natasha41575 · 2025-11-11T19:51:28Z

+				if tt.inPlacePodLevelResizeEnabled {
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InPlacePodLevelResourcesVerticalScaling, true)
+				}


I see a few other places actually where the same comment applies, you can update them all

Suggested change

if tt.inPlacePodLevelResizeEnabled {

featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InPlacePodLevelResourcesVerticalScaling, true)

}

featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InPlacePodLevelResourcesVerticalScaling, tt.inPlacePodLevelResizeEnabled)

natasha41575 · 2025-11-11T20:05:10Z

+	// testPods[0] has the highest priority, as it doesn't increase resource requests (pod-level).
+	// testPods[1] has the next highest priority, as it doesn't increase resource requests (container-level).


(nonblocking)

Is the pod with pod-level resources being prioritied over the pod with container-level resources intentional? (I think the answer is and should be no, they are equal priority and either being first is acceptable) Let's leave a comment to state as such

What I want this test to cover is that the resize of pod-level resources is taken into account when determining priority. As in, resizes that do not increase any requests are prioritized over resizes that do increase pod-level requests. To that end, I think some of the other pods should be converted to using pod-level resources. One option to ensure all our bases are covered:

testPods[2] should have both pod and container-level resources specified, with only pod-level resources being increased

testPods[3] should be converted to having only pod-level resources, with the resize increasing them

testPods[4] should have both pod and container-level resources, with the resize increasing only container-level resources

testPods[5] and testPods[6] cover the case of only container-level resources being specified and resized.

natasha41575 · 2025-11-11T20:11:29Z

-	currentResources := containerResourcesFromRequirements(&actuatedResources)
+	var actuatedPodResources *v1.ResourceRequirements
+	if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodLevelResourcesVerticalScaling) {
+		actuatedPodResources, _ = m.actuatedState.GetPodLevelResources(pod.UID)


do we need to check the return value of found here (similar to what we do with container resources above)

It's just used for logging, but probably good to know?

natasha41575 · 2025-11-11T20:14:35Z

+	if m.isContainerResourceResizeInProgress(allocatedPod, podStatus) {
+		return true
+	}
+
+	return m.isPodLevelResourcesResizeInProgress(allocatedPod, podStatus)


optional nit, just a personal style preference

Suggested change

if m.isContainerResourceResizeInProgress(allocatedPod, podStatus) {

return true

}

return m.isPodLevelResourcesResizeInProgress(allocatedPod, podStatus)

return m.isContainerResourceResizeInProgress(allocatedPod, podStatus) || m.isPodLevelResourcesResizeInProgress(allocatedPod, podStatus)

natasha41575 · 2025-11-11T20:29:52Z

 //	   b) api-server in services doesn't start with --enable-admission-plugins=ResourceQuota
 //	      and is not possible to start it from TEST_ARGS
 //	Above tests in test/e2e/node/pod_resize.go
-var _ = SIGDescribe("Pod InPlace Resize Container", func() {


#132919 (comment)

I don't want to change this here because this description is included in the name of the test in conformance.yaml (see https://github.com/kubernetes/kubernetes/pull/135067/files)

i.e. make sure there's no diff when you run ./hack/update-conformance.sh

natasha41575

The WiFi isn't working at the conference, so I won't be able to review this until I get back to my hotel room tonight.

@natasha41575 can you help review today? If you lgtm I'll try to just approve tonight.

I'm out of time today. I have a few outstanding comments here and also above but besides this I think this largely LGTM if those are addressed

I will also add a caveat that I did not get around to looking at the e2e tests super carefully after they were updated to make everything pass - so that still needs another eye on it.

natasha41575 · 2025-11-11T23:32:09Z

 		return resizeResult
 	}

+	updateActuatedPodLevelResources := func(resourceName v1.ResourceName) error {


Is it possible to move this function definition to right before it's called (L903) so that the code block is within the FG?

tallclair · 2025-11-12T01:05:52Z

/hold
Just to gate the merge. I'll remove the hold with my LGTM.

Signed-off-by: ndixita <[email protected]>

tallclair

Were you going to pull #135203 into this PR?

I don't think any of the outstanding feedback is blocking for alpha. I'll take another look in a couple hours. In the meantime, try to track down the missing approvals.

@liggitt I think this is ready for your approval.

/approve

tallclair · 2025-11-12T00:14:05Z

+
+	podWithoutContainerResources := pod.DeepCopy()
+	podWithoutContainerResources.Spec.Containers[0].Resources = v1.ResourceRequirements{}
+	podWithoutContainerResources.Spec.InitContainers[0].Resources = v1.ResourceRequirements{}


I think the pod has 2 regular containers and 2 init containers. Also, did you mean for this pod to have pod-level resources?

tallclair · 2025-11-12T00:25:55Z

+// The implementation of CPU/Memory limit calculation is duplicated here and in
+// pkg/kubelet/kuberuntime/kuberuntime_container_linux.go (getCPULimits/getMemoryLimits).
+// Refactor into a shared utility function.
+func GetLimits(res *ResourceOpts) v1.ResourceList {


nit: why the ResourceOpts struct, rather than just passing 2 parameters?

While testing I swapped the two params :| so for clarity or I could pass container resources as the v1.ResourceRequirements and pod resources as *v1.ResourceRequirements.

tallclair · 2025-11-12T00:50:14Z

-	currentResources := containerResourcesFromRequirements(&actuatedResources)
+	var actuatedPodResources *v1.ResourceRequirements
+	if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodLevelResourcesVerticalScaling) {
+		actuatedPodResources, _ = m.actuatedState.GetPodLevelResources(pod.UID)


It's just used for logging, but probably good to know?

tallclair · 2025-11-12T00:55:31Z

+
+		if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodLevelResourcesVerticalScaling) {
+			if err = updateActuatedPodLevelResources(rName); err != nil {
+				logger.Error(err, "Failed to update pod-level actuated resources", "resource", rName, "pod", klog.KObj(pod))


Still need to address this.

tallclair · 2025-11-12T00:58:54Z

+		return nil
+	}
+
+	if actuatedPod.Spec.Resources == nil {


nit: this and the nil check below seem unnecessary.

tallclair · 2025-11-12T01:15:14Z

+	if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodLevelResourcesVerticalScaling) {
+		apiPodStatus.Resources = kl.convertToAPIPodLevelResourcesStatus(pod, oldPodStatus)
+		opts := resourcehelper.PodResourcesOptions{
+			SkipPodLevelResources: !utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources),


This should exclude overhead, right?

tallclair · 2025-11-12T01:16:45Z

+}
+
+func (kl *Kubelet) convertToAPIPodLevelResourcesStatus(allocatedPod *v1.Pod, oldPodStatus v1.PodStatus) *v1.ResourceRequirements {
+	if allocatedPod.Status.Phase != v1.PodRunning {


I think we discussed just leaving the actual resources unset when the pod isn't running?

tallclair · 2025-11-12T01:18:26Z

 	if !opts.SkipPodLevelResources && IsPodLevelRequestsSet(pod) {
+
+		var effectiveReqs v1.ResourceList
+		if opts.InPlacePodLevelResourcesVerticalScalingEnabled && opts.UseStatusResources {


These changes should have unit test coverage.

tallclair · 2025-11-12T01:19:22Z

 		},
 	}
+
+	if podResources != nil {


nit: nil check is unnecessary

liggitt · 2025-11-12T01:52:19Z

/approve
for API bits

looks like lint / verify has some cleanup to make happy

ndixita · 2025-11-12T06:09:52Z

/test pull-kubernetes-unit-windows-master

Signed-off-by: ndixita <[email protected]>

ndixita · 2025-11-12T06:22:33Z

/retest

ndixita · 2025-11-12T07:14:32Z

/retest

ndixita · 2025-11-12T07:21:39Z

/test pull-kubernetes-node-e2e-containerd

k8s-ci-robot · 2025-11-12T07:30:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dom4ha, liggitt, ndixita, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~api/OWNERS~~ [liggitt]
~~pkg/api/OWNERS~~ [liggitt]
~~pkg/apis/OWNERS~~ [liggitt]
~~pkg/features/OWNERS~~ [dom4ha,liggitt,tallclair]
~~pkg/generated/openapi/OWNERS~~ [liggitt]
~~pkg/kubelet/OWNERS~~ [liggitt,tallclair]
~~pkg/registry/core/pod/OWNERS~~ [liggitt,tallclair]
~~pkg/scheduler/OWNERS~~ [dom4ha,liggitt]
~~staging/src/k8s.io/api/OWNERS~~ [liggitt]
~~staging/src/k8s.io/client-go/applyconfigurations/OWNERS~~ [liggitt]
~~staging/src/k8s.io/component-helpers/OWNERS~~ [liggitt]
~~staging/src/k8s.io/component-helpers/resource/OWNERS~~ [liggitt]
~~test/compatibility_lifecycle/reference/OWNERS~~ [dom4ha,liggitt,tallclair]
~~test/e2e/common/OWNERS~~ [liggitt,tallclair]
~~test/e2e/feature/OWNERS~~ [dom4ha,liggitt,tallclair]
~~test/e2e/node/OWNERS~~ [liggitt,tallclair]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ndixita · 2025-11-12T07:53:07Z

/test pull-kubernetes-e2e-gce

Signed-off-by: ndixita <[email protected]>

ndixita · 2025-11-12T08:50:46Z

/retest

k8s-ci-robot · 2025-11-12T09:12:35Z

@ndixita: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-node-e2e-restartallcontainers	`c2f8fec`	link	false	`/test pull-kubernetes-node-e2e-restartallcontainers`
pull-kubernetes-node-kubelet-inplace-pod-level-resources-resize	`e687175`	link	false	`/test pull-kubernetes-node-kubelet-inplace-pod-level-resources-resize`
pull-kubernetes-conformance-image-test	`eed667f`	link	false	`/test pull-kubernetes-conformance-image-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ndixita · 2025-11-12T09:15:12Z

/test pull-kubernetes-e2e-gce

tallclair · 2025-11-12T13:54:08Z

+	if oldPodInfo.Spec.Resources == nil && newPodInfo.Spec.Resources == nil {
+		return false
+	}
+	return !reflect.DeepEqual(oldPodInfo.Spec.Resources, newPodInfo.Spec.Resources)


I didn't catch this before, but I think we forbid updating a pod with pod-level-resources before? So can this logic just be:

Suggested change

if oldPodInfo.Spec.Resources == nil && newPodInfo.Spec.Resources == nil {

return false

}

return !reflect.DeepEqual(oldPodInfo.Spec.Resources, newPodInfo.Spec.Resources)

return oldPodInfo.Spec.Resources != nil || newPodInfo.Spec.Resources != nil

If we do need the resource comparison, it should use api.Semantic.DeepEqual instead.

tallclair · 2025-11-12T13:55:56Z

/lgtm

Some follow-up to do, but I think this is good enough for alpha.

k8s-ci-robot · 2025-11-12T13:56:04Z

LGTM label has been added.

Details

Git tree hash: 87219c958ab2fc2dea5777e8039fae6ced1de7d4

tallclair · 2025-11-12T14:48:10Z

/hold cancel

github-project-automation Bot added this to SIG Apps Jul 13, 2025

github-project-automation Bot moved this to Needs Triage in SIG Apps Jul 13, 2025

k8s-ci-robot requested review from apelisse and derekwaynecarr July 13, 2025 22:22

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jul 13, 2025

ndixita force-pushed the pod-level-in-place-pod-resize branch 4 times, most recently from 1c21c5b to b2b7220 Compare July 14, 2025 17:39

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. area/test sig/node Categorizes an issue or PR as relevant to SIG Node. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 14, 2025

verioned_feature_list update

0f88a9a

natasha41575 reviewed Nov 11, 2025

View reviewed changes

Comment thread test/e2e/common/node/pod_level_resources_resize.go Outdated

Comment thread test/e2e/node/pod_resize.go Outdated

Test fixes

21920bb

Signed-off-by: ndixita <[email protected]>

tallclair reviewed Nov 12, 2025

View reviewed changes

Comment thread test/e2e/common/node/pod_level_resources_resize.go Outdated

Comment thread test/e2e/common/node/pod_level_resources_resize.go Outdated

Test fixes

10b73f8

Signed-off-by: ndixita <[email protected]>

Add InPlacePodLevelResourcesVerticalScaling declared feature.

0e88308

Enabling NodeDeclaredFeatures in unit tests

5ac2ffc

Signed-off-by: ndixita <[email protected]>

tallclair reviewed Nov 12, 2025

View reviewed changes

renovate Bot mentioned this pull request Feb 14, 2026

chore(deps)!: Update kubernetes group (minor) sebtiz13/homecluster#85

Merged

1 task

renovate Bot mentioned this pull request Apr 8, 2026

fix(deps): update python dependencies (major) canonical/mongodb-k8s-operator#483

Open

1 task

natasha41575 mentioned this pull request Apr 21, 2026

OWNERS: nominate natasha41575 as pod-resize subpackage approver #138508

Merged

		// testPods[0] has the highest priority, as it doesn't increase resource requests (pod-level).
		// testPods[1] has the next highest priority, as it doesn't increase resource requests (container-level).

Conversation

ndixita commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR is related to:

Does this PR introduce a user-facing change?

Uh oh!

k8s-ci-robot commented Jul 13, 2025

Uh oh!

k8s-ci-robot commented Jul 13, 2025

Uh oh!

ndixita commented Jul 13, 2025

Uh oh!

tallclair commented Nov 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natasha41575 Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natasha41575 Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natasha41575 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tallclair commented Nov 12, 2025

Uh oh!

tallclair left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ndixita commented Jul 13, 2025 •

edited

Loading

natasha41575 Nov 11, 2025 •

edited

Loading

natasha41575 Nov 11, 2025 •

edited

Loading

natasha41575 left a comment •

edited

Loading

k8s-ci-robot commented Nov 12, 2025 •

edited

Loading