Skip to content

clusterapi: an increase that leaves a node group below its min size is refused, and the group never scales up #10400

Open

Description

@lexfrei

Which component are you using?:

/area cluster-autoscaler

/kind bug

What version of the component are you using?:

Component version: cluster-autoscaler 1.32.0, clusterapi provider. The clusterapi provider code below is unchanged on master.

What k8s version are you using (kubectl version)?:

Workload cluster on Kubernetes 1.35.6, with a Kamaji control plane and KubeVirt workers. Cluster API v1.10.1.

What environment is this in?:

Cozystack, which runs tenant Kubernetes clusters through Cluster API and one cluster-autoscaler per tenant cluster, with --cloud-provider=clusterapi and --enforce-node-group-min-size=true.

What did you expect to happen?:

A MachineDeployment whose min-size annotation is above its current replicas grows to the min size, either for pending pods or through --enforce-node-group-min-size.

What happened instead?:

The group never grows. The scale-up for a pending pod asks for fewer nodes than the min size and is refused by the provider, even though it is an increase:

executor.go:166] Scale-up: setting group MachineDeployment/tenant-cplane/computeplane-cluster-md0 size to 1
clusterstate.go:313] Disabling scale-up for node group MachineDeployment/tenant-cplane/computeplane-cluster-md0 until 2026-10-06 18:53:42.863101496 +0000 UTC m=+326.766182875; errorClass=Other; errorCode=cloudProviderError
static_autoscaler.go:518] Failed to scale up: failed to increase node group size: size decrease too large - desired:1 min:2
orchestrator.go:328] ScaleUpToNodeGroupMinSize: node group is ready to scale up: &{[in backoff after failed scale-up]}

The refusal puts the group in backoff, and ScaleUpToNodeGroupMinSize skips a group in backoff. When the backoff expires, the pending-pod scale-up runs first again, asks for one node again, and is refused again. The group never leaves zero.

The refusal comes from SetSize, which rejects any nreplicas < r.minSize regardless of direction. IncreaseSize passes size + delta straight to it. The skip in backoff is in ScaleUpToNodeGroupMinSize.

How to reproduce it (as minimally and precisely as possible):

  1. Create a MachineDeployment with cluster-api-autoscaler-node-group-min-size: "0" and max size 2, at 0 replicas.
  2. Raise both annotations to "2". Cluster API keeps the live spec.replicas: 0 on that update, because the object already carries replicas.
  3. Leave a pod pending that one node of the group would fit.

The autoscaler logs the lines above every backoff period, and the MachineDeployment stays at 0.

Anything else we need to know?:

This is the bug in #4942, which the triage bot closed as stale. The code is still there.

Refusing only a decrease below the min size fixes it: fetch the scale first, and return the error when nreplicas < r.minSize && nreplicas < current replicas. DecreaseTargetSize keeps its guard that way, and DeleteNodes already checks the min size itself. I have the change with tests for both directions and can send it as a PR.

I hit this in Cozystack, a CNCF project I maintain. The fix ships there as a patch on its autoscaler image, and the patch goes away once this is fixed here.

Activity

  1. added
    area/cluster-autoscalerIssues or PRs related to the Cluster Autoscaler component
    kind/bugCategorizes issue or PR as related to a bug.
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Oct 6, 2026
  2. Siddhartha-singh01 commented on Oct 17, 2026

    @Siddhartha-singh01

    I checked this against current kubernetes/autoscaler master and kubernetes-sigs/cluster-autoscaler main. The loop in the report is still in the code, and the issue is unassigned.

    IncreaseSize forwards replicas + delta straight to SetSize:

    return ng.scalableResource.SetSize(size + delta)

    SetSize rejects any target below minSize, including an increase:

    case nreplicas < r.minSize:
        return fmt.Errorf("size decrease too large - desired:%d min:%d", nreplicas, r.minSize)

    A pending pod that needs one node, on a MachineDeployment at 0 with min 2, calls IncreaseSize(1). SetSize(1) returns that error. The core records it as a cloud-provider error and puts the group in backoff. ScaleUpToNodeGroupMinSize would ask for minSize - targetSize nodes, which SetSize would accept, but IsNodeGroupReadyToScaleUp returns the backoff reason and the group is skipped. When the backoff expires, the pending-pod scale-up runs first and fails the same way, so the group stays at 0.

    Proposed fix, only in (*nodegroup).IncreaseSize: if size+delta is still below MinSize(), raise the target to MinSize() before calling SetSize. A request to grow stays a grow, and the first successful scale-up lands on the minimum instead of being rejected. SetSize should keep rejecting targets below min, so scale-down cannot pass it. DecreaseTargetSize stays as it is.

    Test in clusterapi_nodegroup_test.go:

    • replicas 0, min 2, max 4, IncreaseSize(1) patches spec.replicas to 2 and returns nil
    • replicas 2, min 2, IncreaseSize(1) still targets 3
    • a target above max is still rejected by SetSize

    Please take a look at this approach.

    /assign
    /cc @enxebre @elmiko @jackfrancis

  3. elmiko commented on Oct 20, 2026

    @elmiko
    Contributor

    thanks for reporting this, i'm surprised that the group does not increase to its minimum size.

  4. elmiko commented on Oct 20, 2026

    @elmiko
    Contributor

    fwiw, i think we should not be increasing the replicas to something lower than the minimum when --enforce-node-group-min-size is set. if we see an IncreaseSize request happen, and the flag is set, we should increase the replicas to the minimum. i think.

  5. elmiko commented on Oct 20, 2026

    @elmiko
    Contributor

    /triage accepted

  6. added
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    and removed
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/cluster-autoscalerIssues or PRs related to the Cluster Autoscaler componentkind/bugCategorizes issue or PR as related to a bug.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions