-
Notifications
You must be signed in to change notification settings - Fork 539
Operator Version 0.25.0 and 0.25.1 are producing "invalid memory address" error #1749
Copy link
Copy link
Closed
Description
I'm able to see that there are no restarts happening on my operator when i am in 0.24.3 docker tag of clickhouse operator. However i see that the operator restarts multiple times when i am in 0.25.0 or in 0.25.1 version.
I0618 10:08:48.319164 1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 0-2
I0618 10:08:48.319391 1 stateful-set-application.go:176] stsSetupLogContainer():clickhouse/clickhouse-server/c25cff67-c577-410a-9ff2-61792518067b:add log container for host: 1-0
I0618 10:08:48.518054 1 statefulset-reconciler.go:108] clickhouse/chi-clickhouse-server-mycluster-02-1-0:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse/chi-clickhouse-server-mycluster-02-1-0
I0618 10:08:48.518090 1 object-status.go:54] GetObjectStatusFromMetas():clickhouse/chi-clickhouse-server-mycluster-02-1-0:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse/chi-clickhouse-server-mycluster-02-1-0. Cur: ff779a51ad47b1fbe51fa094c7970dd6eb2c4935 New: e8327fe75e27492df5fbca057b48dd0e648f4657
I0618 10:08:48.518158 1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 1-0
I0618 10:08:48.518384 1 stateful-set-application.go:176] stsSetupLogContainer():clickhouse/clickhouse-server/c25cff67-c577-410a-9ff2-61792518067b:add log container for host: 1-1
I0618 10:08:48.718331 1 statefulset-reconciler.go:108] clickhouse/chi-clickhouse-server-mycluster-02-1-1:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse/chi-clickhouse-server-mycluster-02-1-1
I0618 10:08:48.718368 1 object-status.go:54] GetObjectStatusFromMetas():clickhouse/chi-clickhouse-server-mycluster-02-1-1:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse/chi-clickhouse-server-mycluster-02-1-1. Cur: a6230b5c09accdf8f85eb5f8d383b7bf7fe85916 New: 02ec6d3585a01c74dc27b959df0a9c3e10c8c8f8
I0618 10:08:48.718437 1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 1-1
E0618 10:08:48.718564 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 257 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x16d0ee0, 0x2be93f0})
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x400057b340?})
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x78
panic({0x16d0ee0?, 0x2be93f0?})
/usr/local/go/src/runtime/panic.go:791 +0x124
github.com/altinity/clickhouse-operator/pkg/model/k8s.StatefulSetContainerGet(0x0, {0x4000c48c48?, 0x4000c48ca8?, 0x139ef78?})
/clickhouse-operator/pkg/model/k8s/stateful_set.go:31 +0x80
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getContainerClickHouse(...)
/clickhouse-operator/pkg/model/chi/creator/container.go:69
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getImageTagClickHouse(0x4000c48cb8?, 0x23014?)
/clickhouse-operator/pkg/model/chi/creator/container.go:79 +0x60
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).GetAppImageTag(0x19e578c?, 0x4000c2ff80?)
/clickhouse-operator/pkg/model/chi/creator/container.go:46 +0x1c
github.com/altinity/clickhouse-operator/pkg/model/common/creator.(*Creator).GetAppImageTag(0x4000c48cf8?, 0x23284?)
/clickhouse-operator/pkg/model/common/creator/stateful-set-application.go:186 +0x2c
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).getHostSoftwareVersion(0x4000848600, {0x1ca2ad8, 0x4000c76910}, 0x40012e0b60, {0x0?, 0x10?, 0x80?})
/clickhouse-operator/pkg/controller/chi/worker-reconciler-helper.go:43 +0x78
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions.func1(0x40012e0b60)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:201 +0x78
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).WalkHosts(0x4000ae11e0, 0x4000c49148)
/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:669 +0x170
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions(0x4000848600?, {0x1ca2ad8?, 0x4000c76910?}, 0x4000ae11e0)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:199 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).buildFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00, 0x40010a0b60}, 0x0)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:171 +0xa8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00?, 0x40010a0b60?}, 0x78?, {{{0x0, 0x0, 0x1, 0x0, ...}, ...}, ...}, ...)
/clickhouse-operator/pkg/controller/chi/worker.go:265 +0x38
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateEndpoints(...)
/clickhouse-operator/pkg/controller/chi/worker.go:241
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileEndpoints(0x100000003?, {0x1ca2ad8?, 0x4000c76910?}, 0x0?)
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:120 +0xc0
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x16f40e0, 0x4000f5cb10})
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:170 +0x3a8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0x4000848600)
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:53 +0x2b8
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000c2e8b0, {0x1c838c0, 0x4000b79890}, 0x1, 0x4000656d20)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000c2e8b0, 0x3b9aca00, 0x0, 0x1, 0x4000656d20)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(...)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run in goroutine 99
/clickhouse-operator/pkg/controller/chi/controller.go:526 +0x4d8
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x238 pc=0x120ffe0]
goroutine 257 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x400057b340?})
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x16d0ee0?, 0x2be93f0?})
/usr/local/go/src/runtime/panic.go:791 +0x124
github.com/altinity/clickhouse-operator/pkg/model/k8s.StatefulSetContainerGet(0x0, {0x4000c48c48?, 0x4000c48ca8?, 0x139ef78?})
/clickhouse-operator/pkg/model/k8s/stateful_set.go:31 +0x80
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getContainerClickHouse(...)
/clickhouse-operator/pkg/model/chi/creator/container.go:69
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getImageTagClickHouse(0x4000c48cb8?, 0x23014?)
/clickhouse-operator/pkg/model/chi/creator/container.go:79 +0x60
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).GetAppImageTag(0x19e578c?, 0x4000c2ff80?)
/clickhouse-operator/pkg/model/chi/creator/container.go:46 +0x1c
github.com/altinity/clickhouse-operator/pkg/model/common/creator.(*Creator).GetAppImageTag(0x4000c48cf8?, 0x23284?)
/clickhouse-operator/pkg/model/common/creator/stateful-set-application.go:186 +0x2c
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).getHostSoftwareVersion(0x4000848600, {0x1ca2ad8, 0x4000c76910}, 0x40012e0b60, {0x0?, 0x10?, 0x80?})
/clickhouse-operator/pkg/controller/chi/worker-reconciler-helper.go:43 +0x78
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions.func1(0x40012e0b60)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:201 +0x78
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).WalkHosts(0x4000ae11e0, 0x4000c49148)
/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:669 +0x170
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions(0x4000848600?, {0x1ca2ad8?, 0x4000c76910?}, 0x4000ae11e0)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:199 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).buildFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00, 0x40010a0b60}, 0x0)
/clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:171 +0xa8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00?, 0x40010a0b60?}, 0x78?, {{{0x0, 0x0, 0x1, 0x0, ...}, ...}, ...}, ...)
/clickhouse-operator/pkg/controller/chi/worker.go:265 +0x38
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateEndpoints(...)
/clickhouse-operator/pkg/controller/chi/worker.go:241
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileEndpoints(0x100000003?, {0x1ca2ad8?, 0x4000c76910?}, 0x0?)
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:120 +0xc0
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x16f40e0, 0x4000f5cb10})
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:170 +0x3a8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0x4000848600)
/clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:53 +0x2b8
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000c2e8b0, {0x1c838c0, 0x4000b79890}, 0x1, 0x4000656d20)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000c2e8b0, 0x3b9aca00, 0x0, 0x1, 0x4000656d20)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(...)
/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run in goroutine 99
/clickhouse-operator/pkg/controller/chi/controller.go:526 +0x4d8
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels