Repository navigation
Closed
Labels
wontfixThis will not be worked onThis will not be worked on
Description
2024-07-15T11:19:08Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: admissionregistration.k8s.io/v1: Get \"https://kubernetes-test0.tenant-test0.svc:6443/apis/admissionregistration.k8s.io/v1?timeout=10s\": context deadline exceeded"}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:68
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56
2024-07-15T11:19:09Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-konnectivity-certificate","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-konnectivity-certificate", "reconcileID": "ade0284e-a277-4c70-848a-5138b70dddfb"}
2024-07-15T11:19:09Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-konnectivity-certificate","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-konnectivity-certificate", "reconcileID": "ade0284e-a277-4c70-848a-5138b70dddfb", "after": "87460h59m37.007380834s"}
{"level":"warn","ts":"2024-07-15T11:19:09.490703Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d82380/etcd-0.etcd-headless.tenant-root.svc:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-07-15T11:19:11Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-front-proxy-client-certificate","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-front-proxy-client-certificate", "reconcileID": "b6da9941-a533-461d-bd5d-bb5245a328c5"}
2024-07-15T11:19:12Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-front-proxy-client-certificate","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-front-proxy-client-certificate", "reconcileID": "b6da9941-a533-461d-bd5d-bb5245a328c5", "after": "8572h56m3.319115679s"}
2024-07-15T11:19:13Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-api-server-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-api-server-certificate", "reconcileID": "9455cd48-738a-4578-93a4-8aaff668e902"}
{"level":"warn","ts":"2024-07-15T11:19:11.887367Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d82380/etcd-0.etcd-headless.tenant-root.svc:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-07-15T11:19:14Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-api-server-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-api-server-certificate", "reconcileID": "9455cd48-738a-4578-93a4-8aaff668e902", "after": "7438h49m57.410941272s"}
2024-07-15T11:19:13Z ERROR unable to delete datastore data {"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kubernetes-test0","namespace":"tenant-georg"}, "namespace": "tenant-georg", "name": "kubernetes-test0", "reconcileID": "e1226656-9803-4468-87f4-735077b3dd88", "resource": "datastore-setup", "error": "unable to delete the datastore: cannot delete database: context deadline exceeded", "errorVerbose": "context deadline exceeded\ncannot delete database\ngithub.com/clastix/kamaji/internal/datastore/errors.NewCannotDeleteDatabaseError\n\t/workspace/internal/datastore/errors/errors.go:29\ngithub.com/clastix/kamaji/internal/datastore.(*EtcdClient).DeleteDB\n\t/workspace/internal/datastore/etcd.go:133\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:211\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nunable to delete the datastore\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:212\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete
/workspace/internal/resources/datastore/datastore_setup.go:147
github.com/clastix/kamaji/internal/resources.HandleDeletion
/workspace/internal/resources/resource.go:88
github.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile
/workspace/controllers/tenantcontrolplane_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231
2024-07-15T11:19:15Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-api-server-kubelet-client-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-api-server-kubelet-client-certificate", "reconcileID": "1df91c30-3fbe-4d91-8796-d79c011b8c18"}
2024-07-15T11:19:15Z ERROR resource deletion failed {"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kubernetes-test0","namespace":"tenant-georg"}, "namespace": "tenant-georg", "name": "kubernetes-test0", "reconcileID": "e1226656-9803-4468-87f4-735077b3dd88", "resource": "datastore-setup", "error": "unable to delete the datastore: cannot delete database: context deadline exceeded", "errorVerbose": "context deadline exceeded\ncannot delete database\ngithub.com/clastix/kamaji/internal/datastore/errors.NewCannotDeleteDatabaseError\n\t/workspace/internal/datastore/errors/errors.go:29\ngithub.com/clastix/kamaji/internal/datastore.(*EtcdClient).DeleteDB\n\t/workspace/internal/datastore/etcd.go:133\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:211\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nunable to delete the datastore\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:212\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile
/workspace/controllers/tenantcontrolplane_controller.go:152
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231
2024-07-15T11:19:15Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-api-server-kubelet-client-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-api-server-kubelet-client-certificate", "reconcileID": "1df91c30-3fbe-4d91-8796-d79c011b8c18", "after": "7438h49m56.211676939s"}
2024-07-15T11:19:16Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-admin-kubeconfig","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-admin-kubeconfig", "reconcileID": "7cbc598a-d89e-4ce6-9082-35d2c11a0d5f"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "validatingwebhookconfiguration", "controllerGroup": "admissionregistration.k8s.io", "controllerKind": "ValidatingWebhookConfiguration", "source": "kind source: *v1.ValidatingWebhookConfiguration"}
2024-07-15T11:19:18Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "kind source: *v1.DaemonSet"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "kind source: *v1.ServiceAccount"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRoleBinding"}
2024-07-15T11:19:18Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "channel source: 0xc001b8ee00"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "validatingwebhookconfiguration", "controllerGroup": "admissionregistration.k8s.io", "controllerKind": "ValidatingWebhookConfiguration", "source": "channel source: 0xc001afcb40"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "validatingwebhookconfiguration", "controllerGroup": "admissionregistration.k8s.io", "controllerKind": "ValidatingWebhookConfiguration"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "channel source: 0xc001b8e840"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.Secret"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "channel source: 0xc001b8f600"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2024-07-15T11:19:20Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRoleBinding"}
2024-07-15T11:19:21Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRoleBinding"}
2024-07-15T11:19:21Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "kind source: *v1.ClusterRoleBinding"}
2024-07-15T11:19:19Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRole"}
2024-07-15T11:19:21Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "channel source: 0xc001b8ff00"}
2024-07-15T11:19:21Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ServiceAccount"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "channel source: 0xc001afd300"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ServiceAccount"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.Service"}
2024-07-15T11:19:23Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ConfigMap"}
2024-07-15T11:19:22Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-admin-kubeconfig","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-admin-kubeconfig", "reconcileID": "7cbc598a-d89e-4ce6-9082-35d2c11a0d5f", "after": "8572h56m10.304161855s"}
2024-07-15T11:19:22Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.Role"}
2024-07-15T11:19:23Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.RoleBinding"}
2024-07-15T11:19:23Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.Deployment"}
2024-07-15T11:19:24Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ConfigMap"}
2024-07-15T11:19:24Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "channel source: 0xc001b8e380"}
2024-07-15T11:19:24Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding"}
2024-07-15T11:19:24Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-datastore-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-datastore-certificate", "reconcileID": "b8c0393d-c525-4a20-94d3-888de01f3b99"}
2024-07-15T11:19:25Z ERROR Reconciler error {"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kubernetes-test0","namespace":"tenant-georg"}, "namespace": "tenant-georg", "name": "kubernetes-test0", "reconcileID": "e1226656-9803-4468-87f4-735077b3dd88", "error": "unable to delete the datastore: cannot delete database: context deadline exceeded", "errorVerbose": "context deadline exceeded\ncannot delete database\ngithub.com/clastix/kamaji/internal/datastore/errors.NewCannotDeleteDatabaseError\n\t/workspace/internal/datastore/errors/errors.go:29\ngithub.com/clastix/kamaji/internal/datastore.(*EtcdClient).DeleteDB\n\t/workspace/internal/datastore/etcd.go:133\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:211\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nunable to delete the datastore\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).deleteDB\n\t/workspace/internal/resources/datastore/datastore_setup.go:212\ngithub.com/clastix/kamaji/internal/resources/datastore.(*Setup).Delete\n\t/workspace/internal/resources/datastore/datastore_setup.go:146\ngithub.com/clastix/kamaji/internal/resources.HandleDeletion\n\t/workspace/internal/resources/resource.go:88\ngithub.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile\n\t/workspace/controllers/tenantcontrolplane_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:333
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:270
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231
2024-07-15T11:19:26Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.DaemonSet"}
2024-07-15T11:19:26Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting EventSource {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "channel source: 0xc001afdbc0"}
2024-07-15T11:19:26Z INFO soot_tenant-leotolstoi_kubernetes-test5 Starting Controller {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding"}
2024-07-15T11:19:27Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-datastore-certificate","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-datastore-certificate", "reconcileID": "b8c0393d-c525-4a20-94d3-888de01f3b99", "after": "1842h32m50.20592689s"}
2024-07-15T11:19:29Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-scheduler-kubeconfig","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-scheduler-kubeconfig", "reconcileID": "6dbf9d38-869a-40f4-b1fc-c206bf6a8097"}
2024-07-15T11:19:32Z INFO certificate is still valid, enqueuing back {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test5-scheduler-kubeconfig","namespace":"tenant-leotolstoi"}, "namespace": "tenant-leotolstoi", "name": "kubernetes-test5-scheduler-kubeconfig", "reconcileID": "6dbf9d38-869a-40f4-b1fc-c206bf6a8097", "after": "8572h57m15.009970538s"}
2024-07-15T11:19:32Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: apps/v1: Get \"https://kubernetes-test5.tenant-leotolstoi.svc:6443/apis/apps/v1?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:68
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56
2024-07-15T11:19:33Z INFO starting CertificateLifecycle handling {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"kubernetes-test0-controller-manager-kubeconfig","namespace":"tenant-test0"}, "namespace": "tenant-test0", "name": "kubernetes-test0-controller-manager-kubeconfig", "reconcileID": "f1f2aaa1-2320-4565-b8c3-056465a8a2c6"}
kubectl get pods -n cozy-kamaji
NAME READY STATUS RESTARTS AGE
kamaji-c7448f786-v6cmt 0/1 CrashLoopBackOff 388 (2m14s ago) 6d19h
details will be below @kvaps
Reacted by George Gaál
Metadata
Metadata
Assignees
Labels
wontfixThis will not be worked onThis will not be worked on
Activity
prometherion commentedon Jul 15, 2024
I'm a bit confused here, @gecube. May I ask you a precise way to replicate this and which Namespace have been deleted?
kvaps commentedon Jul 15, 2024
Hi @prometherion, I'm currently ivestigating this issue:
I found that namespace has been stuck in terminating state:
in describe I can see that it is because of kamaji finalizer:
Inside this namespace I can see that secret is not deleted:
kvaps commentedon Jul 15, 2024
from the kamaji logs it's only seen:
gecube commentedon Jul 15, 2024
Some additional details. The issue was observed on https://github.com/aenix-io/cozystack installation.
The actions lead to the issue:
tenant-georg03)tenant-georg03namespace. Wait till it is initialised, work with cluster several days.Checking the pods I found that they could not start because of absence of kubeconfig:
...
prometherion commentedon Jul 15, 2024
We added the finalizer to the Datastore secret since this is required to delete the Datastore data, such as key prefixes for etcd, and schemas for RDBMS.
I'm a bit lost with the Cozystack terminology, thanks for the patience here, please, may I ask you to confirm these are the right steps to reproduce?
kvaps commentedon Jul 15, 2024
Unfortunately I cannot reproduce this behavior :-(
I just see this secret is created, and kamaji is not trying to remove it nor finalizer from it
prometherion commentedon Jul 15, 2024
@gecube reading again the reported logs, it seems to me Kamaji is not able to delete the given Tenant since the connection with the related etcd is broken (
context deadline exceeded).Where is the Datastore located? Furthermore, what's the error causing the CrashLoopbackoff for the Kamaji pod? I wonder about some health checks, or is it a nil pointer dereference?
kvaps commentedon Jul 15, 2024
It seems the problem occurrs only when datastore is not available, I was able to reproduce it:
check that namespace is still holding the secret for accessing database:
it has finalizer which is blocking namespace removal.
Kamaji removes
tenantcontrolplanes.kamaji.clastix.iofrom the namespace even if datastore is not available.So if you recover datastore later the orphan secret won't be reconciled
prometherion commentedon Jul 15, 2024
I was able to reproduce this, but Kamaji is not in
CrashLoopBackOff(v1.0.0)I see in the logs kamaji tries to connect to the given Datastore, and that's ok: the problem here is that Kamaji is not aware of your business logic.
Not sure if it's the case, but let's take for granted you're deleting the Datastore/etcd in the same Namespace where the Tenant Control Plane resides: we know the Tenant Control Plane has a dependency with the Datastore that must be finalized prior the deletion of the Datastore itself. Kamaji is not aware you're deleting the entire Namespace and the etcd is gone, so it tries to constantly reconcile the finalizer by performing the clean-up.
I would suggest you, if it's possible, to have an order in the actions, as we have with the creation of a Tenant Control Plane where:
With the same principle, the deletion requires:
It could be your
etcdis unreachable for a specific reason as I did in my test (kubectl scale sts --replicas=0): even here, once scaled up to normal, Kamaji has been able to connect to the etcd, performing clean-up, and then the TCP has been removed, as well as the secrets, and the Namespace.I don't think this is a bug report we have to address, it sounds more an edge case where you have to orchestrate better your platform on top of Kamaji.
gecube commentedon Jul 15, 2024
@prometherion thanks for the reproduction. I think that we could not rely on removal order anyway. If we can implement the order for applying objects - there are many mechanism for it, particularly in Helm itself or FluxCD, but for the removal we can expect anything. Like user comes and removes the namespace completely, because he is not aware of complex logic under the hood. And we can do nothing on platform level with it. The only option (as I believe) - is to write all controllers in such a manner that:
prometherion commentedon Jul 15, 2024
I'd like to help here, but unfortunately, it's out of our control.
If the user makes the Datastore unavailable for any reason, and user deletes the TenantControlPlane, Kamaji still relies on the clean-up of those resources. It makes sense since we don't want to have etcd with orphaned keys, and given the context here (such as having etcd unreachable, and the user deleting the Namespace) it sounds like an edge case.
The addressable bug here is the CrashLoopBackOff which is non-reproducible, at least with v1.0.0.
As I said before, without being nasty, it's not a bug per se, but the typical Kubernetes scenario where there's a chain of dependencies that must be known by the user, or if it's orchestrated by a third-party platform, it must know and orchestrated accordingly.
I'm going to close this issue but:
CrashLoopBackOffsuho922 commentedon Oct 30, 2024
@prometherion Why does it use a finalizer, unlike other secret resources?
Predictable Reproduction Flow
As I understand, tenant data is deleted with admin privileges. Is datastore secret necessary before deleting a tenant(#376)?
prometherion commentedon Oct 30, 2024
All the Datastore actions, besides creation, are achieved using the limited account.
We could switch over the root credentials given by the Datastore resource, and retrieve the scheme from the Tenant Control Plane status.
It would require a refactoring tho, something that I'm not able to manage right now: happy to receive contributions tho, as well as providing guidance through the code base.
suho922 commentedon Oct 30, 2024
Do you consider it a bug that the datastore-secret still remains even after the tenantControlPlane has been deleted?
It is related to the finalizer(finalizer.kamaji.clastix.io/datastore-secret)
These are logs about it.
In my view, The current TenantControlPlane controller cannot guarantee the removal of that finalizer consistently.
prometherion commentedon Oct 30, 2024
I'm unable to replicate the issue.
JFI I'm running on Kamaji on
latest.suho922 commentedon Oct 30, 2024
Have there been any efforts or work related to this issue? If not, this behavior does not occur consistently.
This behavior occurs due to the following reason:
"The object has been modified; please apply your changes to the latest version and try again."
gecube commentedon Oct 30, 2024
@jds9090 agree, this occurs not every time, so need a lot of tries to confirm the existence of the issue.
suho922 commentedon Oct 30, 2024
For example, it might be necessary to add a finalizer to the tenantControlPlane to ensure the deletion of the datastore-secret. However, it's still unclear whether this approach is appropriate.
prometherion commentedon Oct 30, 2024
@jds9090 the
The object has been modified; please apply your changes to the latest version and try again.is the root issue, I agree, may I ask you to share if there's any trace of an update action in the annotations, such aslast-applied-configuration?Trying to understand who's changing that
Secretis the key!prometherion commentedon Oct 30, 2024
Wondering if it eventually succeeds or not, since I'm not able to replicate.
And I was thinking that maybe we could wrap this portion of code in a
Retryfunction.kamaji/internal/resources/datastore/datastore_storage_config.go
Lines 70 to 92 in fdd0035
suho922 commentedon Oct 31, 2024
Unfortunately, there is no information regarding the last-applied-configuration.
suho922 commentedon Oct 31, 2024
A retry mechanism seems like it would be helpful here.
suho922 commentedon Oct 31, 2024
I believe it could occur in a scenario like this.
(Action 1)
(Action 2)
suho922 commentedon Oct 31, 2024
I’ve been reviewing the approach for Tenant Kubernetes API server access to etcd, which relies solely on TLS-based authentication without using user and password information. This setup appears to apply to the Operator as well, as it also establishes etcd connections without utilizing user and password credentials. I understand that schema information is being used for tenant separation. if there are any scenarios where user and password information would be necessary for etcd connections.
prometherion commentedon Nov 12, 2024
@jds9090 just merged #631 which performs a retry mechanism, hope this will fix this race condition!
suho922 commentedon Nov 12, 2024
Thank you! I will let you know if it happens again.