Repository navigation
Description
Context
"KEP-3521: Pod Scheduling Readiness" had graduated to GA in 1.30. It allows gating the scheduling of a pod, hence delaying it until the gate is removed.
This feature is useful for multiple use-cases, one of them being to allow out-of-tree quota management solutions to integrate well with Kubernetes. In fact, in a post in the Kubernetes blog the outlined use-case is the following:
An important use case this feature enables is dynamic quota management. Kubernetes supports ResourceQuota, however the API Server enforces quota at the time you attempt Pod creation.
...
Scheduling gates allows an external quota manager to address the above limitation of ResourceQuota.
In addition, in the "What problem does it solve?" section, it writes:
Clearing the gates is the responsibility of external controllers with knowledge of when the Pod should be considered for scheduling (e.g., a quota manager).
As can be seen from the above, the Pod Scheduling Readiness feature was specifically designed to allow out-of-tree quota management solutions to integrate well with Kubernetes.
The problem
The In-place Update of Pod Resources KEP allows to modify pod's resources without re-scheduling them. While this feature is absolutely useful, it removes the assumption in which every pod that uses new resources would be scheduled.
The current behavior contradicts one of the main goals of Pod Scheduling Readiness, which is a GAed feature, and is effectively breaking existing out-of-tree quota solutions.
Possible solutions
Idea 1: Introduce a resize gate, which operates similarly to a scheduling gate, meaning it will gate resizes until removed. This will allow external quota managements to become aware of the resize request, being able to perform custom logic.
Idea 2: Re-use the current scheduling gates as resize gates. In other words, delay resize requests until the scheduling gate is removed. (As far as I remember, scheduling gates are immutable, which could challenge this approach).
Idea 3: ??
Effected projects
I'm aware of two projects that are effected by this, being
- Kueue: https://github.com/kubernetes-sigs/kueue.
- AAQ (Application Aware Quota): https://github.com/kubevirt/application-aware-quota.
Related discussions
Issue opened to Kueue: kubernetes-sigs/kueue#5257.
Old issue about the same subject: kubernetes/enhancements#4304.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
- StatusShow more project fieldsDone
Activity
k8s-ci-robot commentedon May 18, 2025
This issue is currently awaiting triage.
If a SIG or subproject determines this is a relevant issue, they will accept it by applying the
triage/acceptedlabel and provide further guidance.The
triage/acceptedlabel can be added by org members by writing/triage acceptedin a comment.Details
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
iholder101 commentedon May 18, 2025
/sig node
/sig scheduling
iholder101 commentedon May 18, 2025
/cc @esotsal @kannon92
kannon92 commentedon May 18, 2025
I still don't really follow the problem you are asking to resolve.
If a pod has not been scheduled due to a gate, I think resizing should be fine. If I understand Scheduling gates, the binding to a node is paused until the gate is removed. Does this mean that one can resize resources and the scheduler will pick the correct resource size? Or does it pick the old one?
Could you come up with a repro case?
And for ApplicationAwareResourceQuota, what is your exact problem with this integration?
ffromani commentedon May 18, 2025
/cc
iholder101 commentedon May 18, 2025
@kannon92 As shown in the PR description, one of the main goals of scheduling gates were to allow external quotas to operate.
To simplify, what AAQ (or any other out-of-tree quota mechanism) does, is:
Being able to implement an external quota mechanism in this way relies on the assumption that resources can be allocated solely through scheduling. in-place updates effectively breaks this scheme, since now it's possible to:
kannon92 commentedon May 18, 2025
I see. Yea that is a similar problem for Kueue.
Does anyone know how this is handled with ResourceQuotas? I would think that there is potentially the same problem with ResourceQuota. Pods with low resources could be admitted, bypassing resourcequota, and then scaled up.
kannon92 commentedon May 18, 2025
Maybe @Huang-Wei has some thoughts from scheduling gate side?
lmktfy commentedon May 18, 2025
If you want to enforce external quota, one option you have is to configure your cluster to deny resizes that grow requests (you can, if you want, also block vertical scale down).
We can document this route. For example, you could use a ValidatingAdmissionPolicy as the means to restrict resizing. You can restrict for all namespaces or just some.
If you want to use an external mechanism that prevents vertical scale up, or that does math on this, you can do that providing the implementation is thorough enough. So far as I know, only Kubernetes ResourceQuota has support for restricting scale-up today, but that may change.
41 remaining items
natasha41575 commentedon Aug 26, 2026
From the node side of things, I am also in favor of the idea of some sort of resize gate that could allow a scheduler like kueue to approve the resize before kubelet actuates it.
natasha41575 commentedon Aug 26, 2026
Looks like broad agreement to somehow solve the issue
/triage accepted
/kind feature
kannon92 commentedon Aug 28, 2026
Now we just need an owner to lead this work?
gshaibi commentedon Aug 30, 2026
I've drafted a short KEP outline and I'll be happy to own this work: https://docs.google.com/document/d/1xlXgVAGK48HDillOHAdAtIiYLdfErZlJKWXyeM5QYFw/edit?usp=sharing
I'll ask for a slot at an upcoming SIG Scheduling meeting as a starting point - guidance welcome on whether this belongs in SIG Scheduling or SIG Node. Comments welcome on the doc meanwhile.
iholder101 commentedon Aug 31, 2026
Thanks everyone for your willingness to promote this effort!
I'd be more than happy to help push this forward. Will gladly review your KEP draft @gshaibi, and would be happy to be involved when it's turned out to a formal KEP.
Thank you!
natasha41575 commentedon Aug 31, 2026
Thanks! I think both SIGs should be involved in review, so would recommend taking it to a SIG Node meeting as well.
The code changes would be in Kubelet which is owned by Node.
iholder101 commentedon Sep 8, 2026
Thanks a lot @gshaibi for this sketch and the work you put into it! Much appreciated!
Reviewed and left comment.
I really like the direction that this is taking. I do think we need to seriously consider the alternative that's laid out in the doc, as it's very interesting and solves some serious problems.
Let me know how I can help push this forward!
tallclair commentedon Sep 23, 2026
@iholder101 or @gshaibi, can you open an enhancement issue in kubernetes/enhancements to track this? I think it's too late for v1.38, but I'd like to try and make progress on this in v1.39.
gshaibi commentedon Sep 24, 2026
Sure, will do.
iholder101 commentedon Sep 24, 2026
Thanks @gshaibi!
I'd be happy if you can add me as a co-owner of the KEP, would be happy to collaborate on this one 👍
gshaibi commentedon Sep 28, 2026
Opened the enhancement issue: kubernetes/enhancements#6438. Will follow with a KEP PR soon.
gshaibi commentedon Oct 6, 2026
KEP PR: kubernetes/enhancements#6459