You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Missing ClusterRoleBinding kubeadm:apiserver-kubelet-client since 26.5.5-edge / k8s v1.36.1 #1171
In Kubernetes v1.36.1 the group kubeadm:cluster-admins was dropped from the apiserver-kubelet-client certificate.
Instead a ClusterRoleBinding kubeadm:apiserver-kubelet-client is created in a new boostraptoken funktion in kubeadmin AllowAPIServerToAccessKubeletAPI. The change was done in commit b0b25e8.
The call of AllowAPIServerToAccessKubeletAPI is missing in Kamajis of the kubeadmin function runBootstrapToken.
Steps to reproduce
Create a tenant cluster with Kamaji and Kubernetes v1.36.1.
Check if the ClusterRoleBinding exists: kubectl get clusterrolebindings.rbac.authorization.k8s.io kubeadm:apiserver-kubelet-client
Results
Kublet bootstrapping with the client certificate
I would guess with the Kamaji 26.5.5-edge and later, you cannot use the certificate from the secret <tenant>-api-server-kubelet-client-certificate to join nodes to the cluster. But we are using bootstrap tokens instead so this is unverified. And if the proposed change would fix this is equally unverified
Authentication for monitoring with the client certificate
In the monitoring guide the example code suggests using the certificate from the secret <tenant>-api-server-kubelet-client-certificate to access the metrics endpoints of the controlplane pods. Starting with Kubernetes v1.36.1 this will no longer work. Even with the proposed change the user kube-apiserver-kubelet-client no longer has permissions to access the metrics endpoints.
Fix
I will create a PR shortly to call AllowAPIServerToAccessKubeletAPI.
I think the issue with monitoring permissions requires some discussion first on how you would like to handle it.
Maybe a dedicated certificate with the group system:monitoring could be created.
Prometheus → control-plane component /metrics (kube-apiserver, kube-controller-manager :10257, kube-scheduler :10259): a different path. system:kubelet-api-admin does not grant the /metrics nonResourceURL, so even with fix(kubeadm): call AllowAPIServerToAccessKubeletAPI in BootstrapToken added in k8s v1.36.1 #1172 the apiserver-kubelet-client cert gets 403 on those endpoints (it lost its privileged group kubeadm:cluster-admins in v1.36.1).
The monitoring guide currently reuses the apiserver-kubelet-client cert, which only worked because that cert used to be over-privileged. The clean fix is a credential in the system:monitoring group: the system:monitoring ClusterRole and its ClusterRoleBinding to the system:monitoring group are created by the RBAC bootstrap on every cluster (present at least since v1.27), and the role grants get on the nonResourceURLs /metrics, /metrics/slis, /livez, /readyz, /healthz. Because Kamaji runs kube-controller-manager and kube-scheduler with --authentication-kubeconfig / --authorization-kubeconfig (delegated authn+authz to the tenant API server), a system:monitoring identity is honored on all three components — over both bearer-token and client-cert auth.
Note Kamaji itself needs no metrics access here — this is purely the user's monitoring identity — so I'd suggest keeping it user-managed (doc change), not auto-provisioned by Kamaji. Two options for the guide; I'd document both:
Option A — ServiceAccount + token (recommended)
No CA key handling, rotatable, k8s-native. In the tenant cluster:
Mint a token (kubectl create token prometheus-metrics -n kube-system --duration=..., or a long-lived kubernetes.io/service-account-token Secret), put it in a Secret next to Prometheus, and switch the ServiceMonitor/PodMonitor from tlsConfig.cert/keySecret to bearer auth:
authorization:
credentials:
name: <tenant>-metrics-tokenkey: tokentlsConfig:
insecureSkipVerify: true # or supply the tenant CA via `ca`
Option B — client certificate with O=system:monitoring
Fits the current cert-based guide structure. The user issues a client cert signed by the tenant CA (the <tenant>-ca Secret) with CN=<anything>, O=system:monitoring, extended key usage clientAuth, then keeps the existing tlsConfig.cert/keySecret wiring pointed at that cert instead of apiserver-kubelet-client. Caveat: this requires handling the tenant CA private key to sign, which is more sensitive than a token.
If maintainers would rather Kamaji provide this out of the box, an opt-in generated system:monitoring client cert (e.g. behind a spec.controlPlane.* flag) is feasible and would mirror the existing apiserver-kubelet-client cert resource — but given Kamaji has no internal need for it, doc-only/user-managed seems like the right default. Happy to take whichever direction you prefer.
The kubeadm:apiserver-kubelet-client binding from #1172 is created in the BootstrapToken phase, which is gated by the checksum of the kube-public/cluster-info ConfigMap. New tenants get it on first bootstrap, but already-provisioned clusters upgrading to v1.36.1 won't — the phase is skipped because cluster-info doesn't change on a version bump. A one-time nudge (clearing status.kubeadmPhase.bootstrapToken.checksum, or applying the binding manually) heals them. Might be worth a note in the upgrade docs, or a follow-up to make that phase re-run.
Happy to open the doc PR (both options) once there's agreement on the direction.
🤖 This proposal was drafted with the help of Claude Code.
@Jakob3xD asking to you, not the AI 🙃 Option B sounds feasible, especially considering we implemented the KubeconfigGenerator that can be used along with Kamaji to generate kubeconfigs with their certificates.
I just understand the actual issue and was a bit trigger-happy sending the proposal.
After looking into it, Option A is not a good solution as it would require copying the token from the tenant cluster to the seed cluster.
You are right that with KubeconfigGenerator we already have the options for this, but I need to look into Prometheus if it can handle the key and cert being under one secret key and not split into cert and key.
Activity
prometherion commentedon Jun 11, 2026
Thanks for the detailed bug report with all the references!
I'll try to take a look at this and come back with a plan.
Jakob3xD commentedon Jun 23, 2026
Proposal for the monitoring-permissions part
Splitting this out from the
kubeadm:apiserver-kubelet-clientClusterRoleBinding fix (#1172), since the two are separate paths:kubectl logs/exec/top, node proxy): fixed by fix(kubeadm): call AllowAPIServerToAccessKubeletAPI in BootstrapToken added in k8s v1.36.1 #1172 viaAllowAPIServerToAccessKubeletAPI, which bindskube-apiserver-kubelet-clientto thesystem:kubelet-api-adminClusterRole./metrics(kube-apiserver, kube-controller-manager:10257, kube-scheduler:10259): a different path.system:kubelet-api-admindoes not grant the/metricsnonResourceURL, so even with fix(kubeadm): call AllowAPIServerToAccessKubeletAPI in BootstrapToken added in k8s v1.36.1 #1172 theapiserver-kubelet-clientcert gets403on those endpoints (it lost its privileged groupkubeadm:cluster-adminsin v1.36.1).The monitoring guide currently reuses the
apiserver-kubelet-clientcert, which only worked because that cert used to be over-privileged. The clean fix is a credential in thesystem:monitoringgroup: thesystem:monitoringClusterRole and its ClusterRoleBinding to thesystem:monitoringgroup are created by the RBAC bootstrap on every cluster (present at least since v1.27), and the role grantsgeton the nonResourceURLs/metrics,/metrics/slis,/livez,/readyz,/healthz. Because Kamaji runs kube-controller-manager and kube-scheduler with--authentication-kubeconfig/--authorization-kubeconfig(delegated authn+authz to the tenant API server), asystem:monitoringidentity is honored on all three components — over both bearer-token and client-cert auth.Note Kamaji itself needs no metrics access here — this is purely the user's monitoring identity — so I'd suggest keeping it user-managed (doc change), not auto-provisioned by Kamaji. Two options for the guide; I'd document both:
Option A — ServiceAccount + token (recommended)
No CA key handling, rotatable, k8s-native. In the tenant cluster:
Mint a token (
kubectl create token prometheus-metrics -n kube-system --duration=..., or a long-livedkubernetes.io/service-account-tokenSecret), put it in a Secret next to Prometheus, and switch theServiceMonitor/PodMonitorfromtlsConfig.cert/keySecretto bearer auth:Option B — client certificate with
O=system:monitoringFits the current cert-based guide structure. The user issues a client cert signed by the tenant CA (the
<tenant>-caSecret) withCN=<anything>,O=system:monitoring, extended key usageclientAuth, then keeps the existingtlsConfig.cert/keySecretwiring pointed at that cert instead ofapiserver-kubelet-client. Caveat: this requires handling the tenant CA private key to sign, which is more sensitive than a token.Existing clusters (heads-up for #1172)
The
kubeadm:apiserver-kubelet-clientbinding from #1172 is created in theBootstrapTokenphase, which is gated by the checksum of thekube-public/cluster-infoConfigMap. New tenants get it on first bootstrap, but already-provisioned clusters upgrading to v1.36.1 won't — the phase is skipped becausecluster-infodoesn't change on a version bump. A one-time nudge (clearingstatus.kubeadmPhase.bootstrapToken.checksum, or applying the binding manually) heals them. Might be worth a note in the upgrade docs, or a follow-up to make that phase re-run.Happy to open the doc PR (both options) once there's agreement on the direction.
🤖 This proposal was drafted with the help of Claude Code.
prometherion commentedon Jun 24, 2026
@Jakob3xD asking to you, not the AI 🙃 Option B sounds feasible, especially considering we implemented the
KubeconfigGeneratorthat can be used along with Kamaji to generate kubeconfigs with their certificates.Jakob3xD commentedon Jun 24, 2026
I just understand the actual issue and was a bit trigger-happy sending the proposal.
After looking into it, Option A is not a good solution as it would require copying the token from the tenant cluster to the seed cluster.
You are right that with
KubeconfigGeneratorwe already have the options for this, but I need to look into Prometheus if it can handle the key and cert being under one secret key and not split into cert and key.Jakob3xD commentedon Jun 25, 2026
Opened #1212 to follow up on the monitoring-cert idea from this thread.