Skip to content

Helm upgrade for charts that contain statefulsets with .spec.updateStrategy.rollingUpdate.partition is incorrect. #11773

Closed
#11774
Closed
Helm upgrade for charts that contain statefulsets with .spec.updateStrategy.rollingUpdate.partition is incorrect.#11773
#11774
Labels
bugCategorizes issue or PR as related to a bug.in progress

Description

@amannijhawan
amannijhawan
opened on Jan 29, 2023

Output of helm version:

[centos@dev-server-centos helm]$ helm version
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.18.9"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.14-dispatcher-dirty", GitCommit:"009f3ac5f1d09b1ec5d4dc811b6032271fe61f75", GitTreeState:"dirty", BuildDate:"2022-09-19T01:31:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15-gke.2500", GitCommit:"fbf2f97a417bed01195ce67ab3c222d68f06f8b7", GitTreeState:"clean", BuildDate:"2022-10-26T09:24:45Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE

Hi Helm Maintainers.

In our testing we found that helm upgrade for charts that contain statefulsets with .spec.updateStrategy.rollingUpdate.partition does not function as expected. When using with a --wait and --timeout parameter the following command always hangs
helm upgrade helm-test ./ --debug --timeout=900s --wait -n test-helm

Here is a simple chart that recreates this issue.
https://github.com/amannijhawan/helm-test-case-anijhawan/blob/gh-pages/example-0.1.0.tgz

Steps to reproduce:

  1. Create a local scratch dir
    mkdir repro
  2. Download the helm package
    cd repro && wget https://github.com/amannijhawan/helm-test-case-anijhawan/raw/gh-pages/example-0.1.0.tgz
  3. Unarchive Package
    tar -xvzf example-0.1.0.tgz
  4. Create an isolated namesapce
    kubectl create ns repro
  5. Helm install the chart.
    helm install helm-test . -n repro
  6. Attempt Helm Upgrade
    helm upgrade helm-test ./ --debug --timeout=100s --wait --values=upgrade.yaml -n repro
    We see that the helm upgrade fails with the following logs.

Edited to included the full output that shows the correct logs

[centos@dev-server-centos helm-test-case-anijhawan]$ helm upgrade helm-test  ./ --debug --timeout=100s --wait   -n repro
upgrade.go:142: [debug] preparing upgrade for helm-test
upgrade.go:150: [debug] performing update for helm-test
upgrade.go:322: [debug] creating upgraded release for helm-test
client.go:229: [debug] checking 2 resources for changes
client.go:512: [debug] Looks like there are no changes for Service "nginx"
client.go:521: [debug] Patch StatefulSet "web" in namespace repro
upgrade.go:394: [debug] waiting for release helm-test resources (created: 0 updated: 2  deleted: 0)
wait.go:66: [debug] beginning wait for 2 resources with timeout of 1m40s
ready.go:362: [debug] StatefulSet is not ready: repro/web. update has not yet been observed
ready.go:393: [debug] StatefulSet is not ready: repro/web. 2 out of 3 expected pods are ready
ready.go:393: [debug] StatefulSet is not ready: repro/web. 2 out of 3 expected pods are ready
ready.go:393: [debug] StatefulSet is not ready: repro/web. 2 out of 3 expected pods are ready
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
ready.go:398: [debug] StatefulSet is not ready: repro/web. currentRevision web-6b4fd5c9f6 does not yet match updateRevision web-545c9585b7
upgrade.go:434: [debug] warning: Upgrade "helm-test" failed: timed out waiting for the condition
Error: UPGRADE FAILED: timed out waiting for the condition
helm.go:84: [debug] timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:990
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:918
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571
[centos@dev-server-centos helm-test-case-anijhawan]$ ^C

However at this point we can check that the upgrade has been successfully applied to one pod matching the current value of partition.

[centos@dev-server-centos helm-test-case-anijhawan]$ kubectl get pod web-2 -o yaml   -nrepro | grep -a5 resource
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: web
    uid: 416718f4-f8bd-4f3e-b193-439571269248
  resourceVersion: "816068910"
  uid: dcd81ea4-376e-419f-b437-087a58f34819
spec:
  containers:
  - image: k8s.gcr.io/nginx-slim:0.8
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      name: web
      protocol: TCP
    resources:
      requests:
        memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
[centos@dev-server-centos helm-test-case-anijhawan]$ kubectl get pod web-1 -o yaml   -nrepro | grep -a5 resource
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: web
    uid: 416718f4-f8bd-4f3e-b193-439571269248
  resourceVersion: "816067417"
  uid: 7522e2ae-1578-40b7-8bcc-aba3a56b685c
spec:
  containers:
  - image: k8s.gcr.io/nginx-slim:0.8
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      name: web
      protocol: TCP
    resources:
      requests:
        memory: 500Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:

The issue seems to be this check in ready.go thats assuming the upgrade hasn't been applied yet.

helm/pkg/kube/ready.go

Line 397 in 76157c6

if sts.Status.CurrentRevision != sts.Status.UpdateRevision {

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugCategorizes issue or PR as related to a bug.in progress

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions