Skip to content

Operator Version 0.25.0 and 0.25.1 are producing "invalid memory address" error #1749

@karthik-thiyagarajan

Description

@karthik-thiyagarajan

I'm able to see that there are no restarts happening on my operator when i am in 0.24.3 docker tag of clickhouse operator. However i see that the operator restarts multiple times when i am in 0.25.0 or in 0.25.1 version.

I0618 10:08:48.319164       1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 0-2 
I0618 10:08:48.319391       1 stateful-set-application.go:176] stsSetupLogContainer():clickhouse/clickhouse-server/c25cff67-c577-410a-9ff2-61792518067b:add log container for host: 1-0
I0618 10:08:48.518054       1 statefulset-reconciler.go:108] clickhouse/chi-clickhouse-server-mycluster-02-1-0:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse/chi-clickhouse-server-mycluster-02-1-0
I0618 10:08:48.518090       1 object-status.go:54] GetObjectStatusFromMetas():clickhouse/chi-clickhouse-server-mycluster-02-1-0:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse/chi-clickhouse-server-mycluster-02-1-0. Cur: ff779a51ad47b1fbe51fa094c7970dd6eb2c4935 New: e8327fe75e27492df5fbca057b48dd0e648f4657
I0618 10:08:48.518158       1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 1-0 
I0618 10:08:48.518384       1 stateful-set-application.go:176] stsSetupLogContainer():clickhouse/clickhouse-server/c25cff67-c577-410a-9ff2-61792518067b:add log container for host: 1-1
I0618 10:08:48.718331       1 statefulset-reconciler.go:108] clickhouse/chi-clickhouse-server-mycluster-02-1-1:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse/chi-clickhouse-server-mycluster-02-1-1
I0618 10:08:48.718368       1 object-status.go:54] GetObjectStatusFromMetas():clickhouse/chi-clickhouse-server-mycluster-02-1-1:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse/chi-clickhouse-server-mycluster-02-1-1. Cur: a6230b5c09accdf8f85eb5f8d383b7bf7fe85916 New: 02ec6d3585a01c74dc27b959df0a9c3e10c8c8f8
I0618 10:08:48.718437       1 worker-reconciler-helper.go:52] getHostSoftwareVersion():Need to report version from the tag. Tag: 25.5.2.47 Host: 1-1 
E0618 10:08:48.718564       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 257 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x16d0ee0, 0x2be93f0})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x400057b340?})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x78
panic({0x16d0ee0?, 0x2be93f0?})
        /usr/local/go/src/runtime/panic.go:791 +0x124
github.com/altinity/clickhouse-operator/pkg/model/k8s.StatefulSetContainerGet(0x0, {0x4000c48c48?, 0x4000c48ca8?, 0x139ef78?})
        /clickhouse-operator/pkg/model/k8s/stateful_set.go:31 +0x80
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getContainerClickHouse(...)
        /clickhouse-operator/pkg/model/chi/creator/container.go:69
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getImageTagClickHouse(0x4000c48cb8?, 0x23014?)
        /clickhouse-operator/pkg/model/chi/creator/container.go:79 +0x60
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).GetAppImageTag(0x19e578c?, 0x4000c2ff80?)
        /clickhouse-operator/pkg/model/chi/creator/container.go:46 +0x1c
github.com/altinity/clickhouse-operator/pkg/model/common/creator.(*Creator).GetAppImageTag(0x4000c48cf8?, 0x23284?)
        /clickhouse-operator/pkg/model/common/creator/stateful-set-application.go:186 +0x2c
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).getHostSoftwareVersion(0x4000848600, {0x1ca2ad8, 0x4000c76910}, 0x40012e0b60, {0x0?, 0x10?, 0x80?})
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-helper.go:43 +0x78
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions.func1(0x40012e0b60)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:201 +0x78
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).WalkHosts(0x4000ae11e0, 0x4000c49148)
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:669 +0x170
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions(0x4000848600?, {0x1ca2ad8?, 0x4000c76910?}, 0x4000ae11e0)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:199 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).buildFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00, 0x40010a0b60}, 0x0)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:171 +0xa8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00?, 0x40010a0b60?}, 0x78?, {{{0x0, 0x0, 0x1, 0x0, ...}, ...}, ...}, ...)
        /clickhouse-operator/pkg/controller/chi/worker.go:265 +0x38
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateEndpoints(...)
        /clickhouse-operator/pkg/controller/chi/worker.go:241
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileEndpoints(0x100000003?, {0x1ca2ad8?, 0x4000c76910?}, 0x0?)
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:120 +0xc0
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x16f40e0, 0x4000f5cb10})
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:170 +0x3a8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0x4000848600)
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:53 +0x2b8
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000c2e8b0, {0x1c838c0, 0x4000b79890}, 0x1, 0x4000656d20)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000c2e8b0, 0x3b9aca00, 0x0, 0x1, 0x4000656d20)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run in goroutine 99
        /clickhouse-operator/pkg/controller/chi/controller.go:526 +0x4d8
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x238 pc=0x120ffe0]

goroutine 257 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x400057b340?})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x16d0ee0?, 0x2be93f0?})
        /usr/local/go/src/runtime/panic.go:791 +0x124
github.com/altinity/clickhouse-operator/pkg/model/k8s.StatefulSetContainerGet(0x0, {0x4000c48c48?, 0x4000c48ca8?, 0x139ef78?})
        /clickhouse-operator/pkg/model/k8s/stateful_set.go:31 +0x80
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getContainerClickHouse(...)
        /clickhouse-operator/pkg/model/chi/creator/container.go:69
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).getImageTagClickHouse(0x4000c48cb8?, 0x23014?)
        /clickhouse-operator/pkg/model/chi/creator/container.go:79 +0x60
github.com/altinity/clickhouse-operator/pkg/model/chi/creator.(*ContainerManager).GetAppImageTag(0x19e578c?, 0x4000c2ff80?)
        /clickhouse-operator/pkg/model/chi/creator/container.go:46 +0x1c
github.com/altinity/clickhouse-operator/pkg/model/common/creator.(*Creator).GetAppImageTag(0x4000c48cf8?, 0x23284?)
        /clickhouse-operator/pkg/model/common/creator/stateful-set-application.go:186 +0x2c
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).getHostSoftwareVersion(0x4000848600, {0x1ca2ad8, 0x4000c76910}, 0x40012e0b60, {0x0?, 0x10?, 0x80?})
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-helper.go:43 +0x78
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions.func1(0x40012e0b60)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:201 +0x78
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).WalkHosts(0x4000ae11e0, 0x4000c49148)
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:669 +0x170
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).findMinMaxVersions(0x4000848600?, {0x1ca2ad8?, 0x4000c76910?}, 0x4000ae11e0)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:199 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).buildFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00, 0x40010a0b60}, 0x0)
        /clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go:171 +0xa8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateFromMeta(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x1cbaf00?, 0x40010a0b60?}, 0x78?, {{{0x0, 0x0, 0x1, 0x0, ...}, ...}, ...}, ...)
        /clickhouse-operator/pkg/controller/chi/worker.go:265 +0x38
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateEndpoints(...)
        /clickhouse-operator/pkg/controller/chi/worker.go:241
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileEndpoints(0x100000003?, {0x1ca2ad8?, 0x4000c76910?}, 0x0?)
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:120 +0xc0
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0x4000848600, {0x1ca2ad8, 0x4000c76910}, {0x16f40e0, 0x4000f5cb10})
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:170 +0x3a8
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0x4000848600)
        /clickhouse-operator/pkg/controller/chi/worker-boilerplate.go:53 +0x2b8
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000c2e8b0, {0x1c838c0, 0x4000b79890}, 0x1, 0x4000656d20)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000c2e8b0, 0x3b9aca00, 0x0, 0x1, 0x4000656d20)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run in goroutine 99
        /clickhouse-operator/pkg/controller/chi/controller.go:526 +0x4d8

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions