Skip to content

[KEP-3521]: Integrate well with out-of-tree resource quota solutions #131835

Open

Description

@iholder101

Context

"KEP-3521: Pod Scheduling Readiness" had graduated to GA in 1.30. It allows gating the scheduling of a pod, hence delaying it until the gate is removed.

This feature is useful for multiple use-cases, one of them being to allow out-of-tree quota management solutions to integrate well with Kubernetes. In fact, in a post in the Kubernetes blog the outlined use-case is the following:

An important use case this feature enables is dynamic quota management. Kubernetes supports ResourceQuota, however the API Server enforces quota at the time you attempt Pod creation.
...
Scheduling gates allows an external quota manager to address the above limitation of ResourceQuota.

In addition, in the "What problem does it solve?" section, it writes:

Clearing the gates is the responsibility of external controllers with knowledge of when the Pod should be considered for scheduling (e.g., a quota manager).

As can be seen from the above, the Pod Scheduling Readiness feature was specifically designed to allow out-of-tree quota management solutions to integrate well with Kubernetes.

The problem

The In-place Update of Pod Resources KEP allows to modify pod's resources without re-scheduling them. While this feature is absolutely useful, it removes the assumption in which every pod that uses new resources would be scheduled.

The current behavior contradicts one of the main goals of Pod Scheduling Readiness, which is a GAed feature, and is effectively breaking existing out-of-tree quota solutions.

Possible solutions

Idea 1: Introduce a resize gate, which operates similarly to a scheduling gate, meaning it will gate resizes until removed. This will allow external quota managements to become aware of the resize request, being able to perform custom logic.

Idea 2: Re-use the current scheduling gates as resize gates. In other words, delay resize requests until the scheduling gate is removed. (As far as I remember, scheduling gates are immutable, which could challenge this approach).

Idea 3: ??

Effected projects

I'm aware of two projects that are effected by this, being

Related discussions

Issue opened to Kueue: kubernetes-sigs/kueue#5257.
Old issue about the same subject: kubernetes/enhancements#4304.

Activity

  1. added
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on May 18, 2025
  2. k8s-ci-robot commented on May 18, 2025

    @k8s-ci-robot
    Contributor

    This issue is currently awaiting triage.

    If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

    The triage/accepted label can be added by org members by writing /triage accepted in a comment.

    Details

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

  3. iholder101 commented on May 18, 2025

    @iholder101
    ContributorAuthor

    /sig node
    /sig scheduling

  4. added
    sig/nodeCategorizes an issue or PR as relevant to SIG Node.
    sig/schedulingCategorizes an issue or PR as relevant to SIG Scheduling.
    and removed
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on May 18, 2025
  5. iholder101 commented on May 18, 2025

    @iholder101
    ContributorAuthor
  6. kannon92 commented on May 18, 2025

    @kannon92
    Contributor

    I still don't really follow the problem you are asking to resolve.

    If a pod has not been scheduled due to a gate, I think resizing should be fine. If I understand Scheduling gates, the binding to a node is paused until the gate is removed. Does this mean that one can resize resources and the scheduler will pick the correct resource size? Or does it pick the old one?

    Could you come up with a repro case?

    And for ApplicationAwareResourceQuota, what is your exact problem with this integration?

  7. ffromani commented on May 18, 2025

    @ffromani
    Contributor

    /cc

  8. iholder101 commented on May 18, 2025

    @iholder101
    ContributorAuthor

    I still don't really follow the problem you are asking to resolve.

    If a pod has not been scheduled due to a gate, I think resizing should be fine. If I understand Scheduling gates, the binding to a node is paused until the gate is removed. Does this mean that one can resize resources and the scheduler will pick the correct resource size? Or does it pick the old one?

    Could you come up with a repro case?

    And for ApplicationAwareResourceQuota, what is your exact problem with this integration?

    @kannon92 As shown in the PR description, one of the main goals of scheduling gates were to allow external quotas to operate.
    To simplify, what AAQ (or any other out-of-tree quota mechanism) does, is:

    1. Adds a scheduling gate to relevant pods using a webhook.
    2. Whenever a pod is created, its scheduling is blocked until AAQ decides to accept it.
    3. AAQ checks if there are enough resources for the pod, and if so, it removes the scheduling gate and updates its internal state.

    Being able to implement an external quota mechanism in this way relies on the assumption that resources can be allocated solely through scheduling. in-place updates effectively breaks this scheme, since now it's possible to:

    1. Create a very small quota.
    2. Create a pod that uses very little resources.
    3. AAQ would accept this pod and update its internal state accordingly.
    4. The pod gets in-place updated to use much more resources.
    5. AAQ is not aware of this update and cannot block it, hence is both broken and has a wrong idea of the amount of resources used.
  9. kannon92 commented on May 18, 2025

    @kannon92
    Contributor

    I see. Yea that is a similar problem for Kueue.

    Does anyone know how this is handled with ResourceQuotas? I would think that there is potentially the same problem with ResourceQuota. Pods with low resources could be admitted, bypassing resourcequota, and then scaled up.

  10. kannon92 commented on May 18, 2025

    @kannon92
    Contributor

    Maybe @Huang-Wei has some thoughts from scheduling gate side?

  11. lmktfy commented on May 18, 2025

    @lmktfy
    Member

    If you want to enforce external quota, one option you have is to configure your cluster to deny resizes that grow requests (you can, if you want, also block vertical scale down).

    We can document this route. For example, you could use a ValidatingAdmissionPolicy as the means to restrict resizing. You can restrict for all namespaces or just some.

    If you want to use an external mechanism that prevents vertical scale up, or that does math on this, you can do that providing the implementation is thorough enough. So far as I know, only Kubernetes ResourceQuota has support for restricting scale-up today, but that may change.

  12. 41 remaining items

  13. natasha41575 commented on Aug 26, 2026

    @natasha41575
    Contributor

    From the node side of things, I am also in favor of the idea of some sort of resize gate that could allow a scheduler like kueue to approve the resize before kubelet actuates it.

  14. natasha41575 commented on Aug 26, 2026

    @natasha41575
    Contributor

    Looks like broad agreement to somehow solve the issue

    /triage accepted
    /kind feature

  15. added
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    kind/featureCategorizes issue or PR as related to a new feature.
    and removed
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Aug 26, 2026
  16. kannon92 commented on Aug 28, 2026

    @kannon92
    Contributor

    Now we just need an owner to lead this work?

  17. gshaibi commented on Aug 30, 2026

    @gshaibi
    Contributor

    I've drafted a short KEP outline and I'll be happy to own this work: https://docs.google.com/document/d/1xlXgVAGK48HDillOHAdAtIiYLdfErZlJKWXyeM5QYFw/edit?usp=sharing

    I'll ask for a slot at an upcoming SIG Scheduling meeting as a starting point - guidance welcome on whether this belongs in SIG Scheduling or SIG Node. Comments welcome on the doc meanwhile.

  18. iholder101 commented on Aug 31, 2026

    @iholder101
    ContributorAuthor

    Thanks everyone for your willingness to promote this effort!

    I'd be more than happy to help push this forward. Will gladly review your KEP draft @gshaibi, and would be happy to be involved when it's turned out to a formal KEP.

    Thank you!

  19. natasha41575 commented on Aug 31, 2026

    @natasha41575
    Contributor

    I've drafted a short KEP outline and I'll be happy to own this work: https://docs.google.com/document/d/1xlXgVAGK48HDillOHAdAtIiYLdfErZlJKWXyeM5QYFw/edit?usp=sharing

    I'll ask for a slot at an upcoming SIG Scheduling meeting as a starting point - guidance welcome on whether this belongs in SIG Scheduling or SIG Node. Comments welcome on the doc meanwhile.

    Thanks! I think both SIGs should be involved in review, so would recommend taking it to a SIG Node meeting as well.

    The code changes would be in Kubelet which is owned by Node.

  20. iholder101 commented on Sep 8, 2026

    @iholder101
    ContributorAuthor

    Thanks a lot @gshaibi for this sketch and the work you put into it! Much appreciated!

    Reviewed and left comment.

    I really like the direction that this is taking. I do think we need to seriously consider the alternative that's laid out in the doc, as it's very interesting and solves some serious problems.

    Let me know how I can help push this forward!

  21. tallclair commented on Sep 23, 2026

    @tallclair
    Member

    @iholder101 or @gshaibi, can you open an enhancement issue in kubernetes/enhancements to track this? I think it's too late for v1.38, but I'd like to try and make progress on this in v1.39.

  22. gshaibi commented on Sep 24, 2026

    @gshaibi
    Contributor

    @iholder101 or @gshaibi, can you open an enhancement issue in kubernetes/enhancements to track this? I think it's too late for v1.38, but I'd like to try and make progress on this in v1.39.

    Sure, will do.

  23. iholder101 commented on Sep 24, 2026

    @iholder101
    ContributorAuthor

    @iholder101 or @gshaibi, can you open an enhancement issue in kubernetes/enhancements to track this? I think it's too late for v1.38, but I'd like to try and make progress on this in v1.39.

    Sure, will do.

    Thanks @gshaibi!
    I'd be happy if you can add me as a co-owner of the KEP, would be happy to collaborate on this one 👍

  24. gshaibi commented on Sep 28, 2026

    @gshaibi
    Contributor

    Opened the enhancement issue: kubernetes/enhancements#6438. Will follow with a KEP PR soon.

  25. gshaibi commented on Oct 6, 2026

    @gshaibi
    Contributor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.sig/nodeCategorizes an issue or PR as relevant to SIG Node.sig/schedulingCategorizes an issue or PR as relevant to SIG Scheduling.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions