Skip to content

[BUG] Restarting a rollout with workloadRef crashes the operator pod #751

@jorgelon

Description

@jorgelon

Describe the bug
There are 2 ways to create a rollout in argo rollouts:

  • The first one is with a single resource "rollout". This includes both logics: the deployment spec and the rollout spec. With the new 1.1.0 reloader release, restarting this rollout works great (deleting the pods and not creating a new replicaset)

  • The second one is with 2 resources. A rollout including the rollout spec and calling an existing deployment with spec.workloadRef. And the referenced deployment with the deployment spec.
    Restarting this rollout makes the controller crash.

To Reproduce
Change the secret that reloads the rollout deployed via workloadRef

Expected behavior
It works same ways as with a single rollout resource

Environment

  • Operator Version: v1.1.0
  • Kubernetes/OpenShift Version: 1.29.6

Log (via stern)
reloader-7df5bc9f58-phgjj reloader E0918 08:33:15.113227 1 runtime.go:79] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]:
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19f31e0?, 0xc001912c48})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?})
reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}})
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).processNextItem(0xc000432d80)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).runWorker(...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(*Controller).Run in goroutine 66
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259
reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0 [recovered]
reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0
reloader-7df5bc9f58-phgjj reloader
reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]:
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?})
reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}})
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).processNextItem(0xc000432d80)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).runWorker(...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(*Controller).Run in goroutine 66
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259

Metadata

Metadata

Assignees

Labels

good-first-issueA good starting point for new contributorskind/bugSomething isn't workingkind/enhancementNew feature or requestkind/help wantedExtra attention is needed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions