Repository navigation
helm-controller panics on hooks with helm.sh/hook-output-log-policy (nil HookOutputFunc) #1562
Description
Describe the bug
helm-controller v1.6.3 panics during Helm upgrade when a chart hook Pod/Job has
helm.sh/hook-output-log-policy. The process exits 2 (SIGSEGV), the release is
left pending-upgrade, and the next reconcile unlocks and retries.
Not OOM. lastState.terminated.reason=Error, exitCode=2.
Stack
panic: runtime error: invalid memory address or nil pointer dereference helm.sh/helm/v4/pkg/kube.(*Client).OutputContainerLogsForPodList helm.sh/helm/v4/pkg/action.(*Configuration).outputLogsByPolicy helm.sh/helm/v4/pkg/action.(*Configuration).execHook
OutputContainerLogsForPodList is called with a nil writerFunc
(cfg.HookOutputFunc).
Why
Helm CLI Init() sets HookOutputFunc to io.Discard.
NewConfiguration() does not. helm-controller ConfigFactory.Build() uses
NewConfiguration() and never sets HookOutputFunc.
Charts that set helm.sh/hook-output-log-policy: hook-succeeded,hook-failed
work with helm upgrade and crash under Flux.
Version
- helm-controller: v1.6.3 (
fluxcd/helm-controller) - Helm SDK: v4.2.2 (replace
github.com/fluxcd/helm/v4 v4.2.4-flux.1)
Repro
- HelmRelease for a chart with a post-upgrade hook Pod annotated
helm.sh/hook-output-log-policy: hook-succeeded,hook-failed
(example: ArchestranetworkPolicyProbe). - Let helm-controller reconcile an upgrade.
- Controller panics; release stays
pending-upgrade.
Workaround
Disable the hook in values (archestra.networkPolicyProbe.enabled: false)
so the annotation is not rendered.
Expected
No panic. Default HookOutputFunc to io.Discard (or nil-check in Helm)
and complete the upgrade.
Helm controller log
Details
{"level":"info","ts":"2026-08-18T19:02:54.xxxZ","msg":"release in-sync with desired state","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":""} {"level":"error","ts":"2026-08-18T19:02:54.xxxZ","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":"","error":"terminal error: exceeded maximum retries: cannot remediate failed release"} {"level":"info","ts":"2026-08-18T19:02:54.xxxZ","msg":"release locked: release with status 'pending-upgrade'","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":""} {"level":"info","ts":"2026-08-18T19:02:54.xxxZ","msg":"running 'unlock' action with timeout of 5m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":""} {"level":"info","ts":"2026-08-18T19:02:54.xxxZ","msg":"release not managed by controller: found existing release in storage","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":""} {"level":"info","ts":"2026-08-18T19:02:54.xxxZ","msg":"running 'upgrade' action with timeout of 5m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"","namespace":"flux-system"},"namespace":"flux-system","name":"","reconcileID":""} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1add3f0]goroutine 1895 [running]:
helm.sh/helm/v4/pkg/kube.(*Client).OutputContainerLogsForPodList(...)
helm.sh/helm/[email protected]/pkg/kube/client.go:1267 +0x250
helm.sh/helm/v4/pkg/action.(*Configuration).outputContainerLogsForListOptions(...)
helm.sh/helm/[email protected]/pkg/action/hooks.go:274 +0xfe
helm.sh/helm/v4/pkg/action.(*Configuration).outputLogsByPolicy(...)
helm.sh/helm/[email protected]/pkg/action/hooks.go:262 +0x3b0
helm.sh/helm/v4/pkg/action.(*Configuration).execHookWithDelayedShutdown.func2()
helm.sh/helm/[email protected]/pkg/action/hooks.go:155 +0x107
helm.sh/helm/v4/pkg/action.(*Configuration).execHook(...)
helm.sh/helm/[email protected]/pkg/action/hooks.go:49 +0x85
helm.sh/helm/v4/pkg/action.(*Upgrade).releasingUpgrade(...)
helm.sh/helm/[email protected]/pkg/action/upgrade.go:510 +0x665
created by helm.sh/helm/v4/pkg/action.(*Upgrade).performUpgrade in goroutine 288
helm.sh/helm/[email protected]/pkg/action/upgrade.go:418 +0x1685
Activity
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
lubronzhan commentedon Oct 4, 2026
Looks like #1563 should've fixed the issue? Not sure why @pujitha24 closed the PR
matheuscscp commentedon Oct 4, 2026
Instead of discarding it looks like the right thing would be trying to wire the hook's logs into our debug log, that we surface in events. Isn't exactly that the intent of
helm.sh/hook-output-log-policy? Surfacing the hook logs?fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
lubronzhan commentedon Oct 5, 2026
Ok created this to forward the log #1604
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
artem-nefedov commentedon Oct 11, 2026
Debatable. I understand that this annotation was added so helm CLI users can see job logs, which makes sense. But in kubernetes, if you already have established log collection setup (as most of us do), I really don't want the hook job logs to be duplicated in the controller. I understand potential benefit for cases where the job is deleted and the cluster has no log forwarding. But even so, I would prefer discard approach.
edit: if these are only seen with log-level=debug, I guess it's not a problem
matheuscscp commentedon Oct 11, 2026
Good points @artem-nefedov!
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (…