Skip to content

Missing ClusterRoleBinding kubeadm:apiserver-kubelet-client since 26.5.5-edge / k8s v1.36.1 #1171

Closed

Description

@s-diez

Summary

In Kubernetes v1.36.1 the group kubeadm:cluster-admins was dropped from the apiserver-kubelet-client certificate.
Instead a ClusterRoleBinding kubeadm:apiserver-kubelet-client is created in a new boostraptoken funktion in kubeadmin AllowAPIServerToAccessKubeletAPI. The change was done in commit b0b25e8.

The call of AllowAPIServerToAccessKubeletAPI is missing in Kamajis of the kubeadmin function runBootstrapToken.

Steps to reproduce

  1. Create a tenant cluster with Kamaji and Kubernetes v1.36.1.
  2. Check if the ClusterRoleBinding exists: kubectl get clusterrolebindings.rbac.authorization.k8s.io kubeadm:apiserver-kubelet-client

Results

Kublet bootstrapping with the client certificate

I would guess with the Kamaji 26.5.5-edge and later, you cannot use the certificate from the secret <tenant>-api-server-kubelet-client-certificate to join nodes to the cluster. But we are using bootstrap tokens instead so this is unverified. And if the proposed change would fix this is equally unverified

Authentication for monitoring with the client certificate

In the monitoring guide the example code suggests using the certificate from the secret <tenant>-api-server-kubelet-client-certificate to access the metrics endpoints of the controlplane pods. Starting with Kubernetes v1.36.1 this will no longer work. Even with the proposed change the user kube-apiserver-kubelet-client no longer has permissions to access the metrics endpoints.

Fix

I will create a PR shortly to call AllowAPIServerToAccessKubeletAPI.

I think the issue with monitoring permissions requires some discussion first on how you would like to handle it.
Maybe a dedicated certificate with the group system:monitoring could be created.

Activity

  1. prometherion commented on Jun 11, 2026

    @prometherion
    Member

    Thanks for the detailed bug report with all the references!

    I'll try to take a look at this and come back with a plan.

  2. Jakob3xD commented on Jun 23, 2026

    @Jakob3xD
    Contributor

    Proposal for the monitoring-permissions part

    Splitting this out from the kubeadm:apiserver-kubelet-client ClusterRoleBinding fix (#1172), since the two are separate paths:

    The monitoring guide currently reuses the apiserver-kubelet-client cert, which only worked because that cert used to be over-privileged. The clean fix is a credential in the system:monitoring group: the system:monitoring ClusterRole and its ClusterRoleBinding to the system:monitoring group are created by the RBAC bootstrap on every cluster (present at least since v1.27), and the role grants get on the nonResourceURLs /metrics, /metrics/slis, /livez, /readyz, /healthz. Because Kamaji runs kube-controller-manager and kube-scheduler with --authentication-kubeconfig / --authorization-kubeconfig (delegated authn+authz to the tenant API server), a system:monitoring identity is honored on all three components — over both bearer-token and client-cert auth.

    Note Kamaji itself needs no metrics access here — this is purely the user's monitoring identity — so I'd suggest keeping it user-managed (doc change), not auto-provisioned by Kamaji. Two options for the guide; I'd document both:

    Option A — ServiceAccount + token (recommended)

    No CA key handling, rotatable, k8s-native. In the tenant cluster:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: prometheus-metrics
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: prometheus-metrics:system-monitoring
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:monitoring
    subjects:
    - kind: ServiceAccount
      name: prometheus-metrics
      namespace: kube-system

    Mint a token (kubectl create token prometheus-metrics -n kube-system --duration=..., or a long-lived kubernetes.io/service-account-token Secret), put it in a Secret next to Prometheus, and switch the ServiceMonitor/PodMonitor from tlsConfig.cert/keySecret to bearer auth:

    authorization:
      credentials:
        name: <tenant>-metrics-token
        key: token
    tlsConfig:
      insecureSkipVerify: true   # or supply the tenant CA via `ca`

    Option B — client certificate with O=system:monitoring

    Fits the current cert-based guide structure. The user issues a client cert signed by the tenant CA (the <tenant>-ca Secret) with CN=<anything>, O=system:monitoring, extended key usage clientAuth, then keeps the existing tlsConfig.cert/keySecret wiring pointed at that cert instead of apiserver-kubelet-client. Caveat: this requires handling the tenant CA private key to sign, which is more sensitive than a token.

    If maintainers would rather Kamaji provide this out of the box, an opt-in generated system:monitoring client cert (e.g. behind a spec.controlPlane.* flag) is feasible and would mirror the existing apiserver-kubelet-client cert resource — but given Kamaji has no internal need for it, doc-only/user-managed seems like the right default. Happy to take whichever direction you prefer.

    Existing clusters (heads-up for #1172)

    The kubeadm:apiserver-kubelet-client binding from #1172 is created in the BootstrapToken phase, which is gated by the checksum of the kube-public/cluster-info ConfigMap. New tenants get it on first bootstrap, but already-provisioned clusters upgrading to v1.36.1 won't — the phase is skipped because cluster-info doesn't change on a version bump. A one-time nudge (clearing status.kubeadmPhase.bootstrapToken.checksum, or applying the binding manually) heals them. Might be worth a note in the upgrade docs, or a follow-up to make that phase re-run.

    Happy to open the doc PR (both options) once there's agreement on the direction.


    🤖 This proposal was drafted with the help of Claude Code.

  3. prometherion commented on Jun 24, 2026

    @prometherion
    Member

    @Jakob3xD asking to you, not the AI 🙃 Option B sounds feasible, especially considering we implemented the KubeconfigGenerator that can be used along with Kamaji to generate kubeconfigs with their certificates.

  4. Jakob3xD commented on Jun 24, 2026

    @Jakob3xD
    Contributor

    I just understand the actual issue and was a bit trigger-happy sending the proposal.
    After looking into it, Option A is not a good solution as it would require copying the token from the tenant cluster to the seed cluster.

    You are right that with KubeconfigGenerator we already have the options for this, but I need to look into Prometheus if it can handle the key and cert being under one secret key and not split into cert and key.

  5. Jakob3xD commented on Jun 25, 2026

    @Jakob3xD
    Contributor

    Opened #1212 to follow up on the monitoring-cert idea from this thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions