Skip to content

Scale up does not happen when current node count is smaller than min size #4942

Closed

Description

@ovstuckrad

Which component are you using?: Cluster Autoscaler with Cluster API

What version of the component are you using?: 1.21.0

What k8s version are you using (kubectl version)?: 1.21.3

kubectl version Output
$ kubectl version
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?: OpenStack

What did you expect to happen?:
When the number of nodes is under the minimum size and I have unschedulable pods, new nodes are created.

What happened instead?:
No new nodes were created, pods stay in Pending.

scale_up.go:472] Estimated 2 nodes needed in MachineDeployment/kube-system/diligent-worker
scale_up.go:586] Final scale-up plan: [{MachineDeployment/kube-system/diligent-worker 2->4 (max: 10)}]
static_autoscaler.go:427] Failed to scale up: failed to increase node group size: size decrease too large - desired:4 min:6size decrease too large - desired:4 min:6 

The following code prevents a scale up from happening:

https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.21.0/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go#L102-L104

	case nreplicas < r.minSize:
		return fmt.Errorf("size decrease too large - desired:%d min:%d", nreplicas, r.minSize)
	}

How to reproduce it (as minimally and precisely as possible):
Create a cluster with 2 nodes. Set the min size to something higher. In my example min size to 4, max size to 10.

Create pods that would trigger a scale up by 1 or 2 nodes.

The replicas are smaller than the min size, no scale up happens.

Anything else we need to know?:

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on Jun 3, 2022
  2. squ94wk commented on Jun 3, 2022

    @squ94wk

    By the same logic (SetSize()), scale downs are currently probably prevented if current - diff > max.

  3. k8s-triage-robot commented on Sep 15, 2022

    @k8s-triage-robot

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle stale
    • Mark this issue or PR as rotten with /lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale

  4. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Sep 15, 2022
  5. k8s-triage-robot commented on Oct 15, 2022

    @k8s-triage-robot

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle rotten

  6. added
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/cluster-autoscalerIssues or PRs related to the Cluster Autoscaler componentkind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions