Skip to content

kubelet EndpointSlice only contains IPv4 addresses on dual-stack clusters where IPv6 is the primary IP family #8681

Description

@ckrause

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Description

On a dual-stack Kubernetes cluster configured with ipFamilies: [IPv6, IPv4] (IPv6-first), prometheus-operator creates only an addressType: IPv4 EndpointSlice for the kubelet Service. No IPv6 EndpointSlice is created. This causes all kubelet scrape targets to be silently dropped since Prometheus 3.11.0.

Steps to Reproduce

  1. Deploy prometheus-operator v0.92.1 with --kubelet-endpointslice=true on a dual-stack cluster where the shoot/cluster networking has ipFamilies: [IPv6, IPv4] (IPv6 as primary family).
  2. Inspect the EndpointSlices created for the kubelet Service:
    kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kubelet
    
  3. Observe that only an addressType: IPv4 EndpointSlice exists, despite nodes having both IPv4 and IPv6 InternalIP addresses in node.Status.Addresses.
  4. Inspect a node to confirm both address families are present:
    kubectl get node <node-name> -o jsonpath='{.status.addresses}' | jq .
    
    This returns two entries with "type": "InternalIP" — one IPv4 and one IPv6.

Expected Result

prometheus-operator should create both an addressType: IPv4 and an addressType: IPv6 EndpointSlice for the kubelet Service when nodes report both address families under InternalIP in node.Status.Addresses.

Actual Result

Only an addressType: IPv4 EndpointSlice is created. On clusters where svc.Spec.IPFamilies[0] is IPv6 (because the kubelet Service is created without an explicit ipFamilyPolicy, so Kubernetes assigns the cluster's default), Prometheus 3.11.0+ drops the IPv4 EndpointSlice via the deduplication logic introduced in prometheus/prometheus#18192, resulting in 0 active kubelet scrape targets.

Prometheus Operator Version

Prometheus operator version: 0.92.1
Args:
   --kubelet-service=kube-system/kubelet
   --kubelet-endpoints=true
   --kubelet-endpointslice=true

Kubernetes Version

clientVersion:
  buildDate: "2025-11-11T19:00:37Z"
  gitVersion: v1.34.2
  platform: darwin/arm64
serverVersion:
  buildDate: "2026-04-15T18:01:23Z"
  gitVersion: v1.35.4
  platform: linux/amd64

Kubernetes Cluster Type

Other (please comment)

How did you deploy Prometheus-Operator?

yaml manifests

Manifests

# kubelet Service as created by prometheus-operator (no ipFamilyPolicy set)
apiVersion: v1
kind: Service
metadata:
  name: kubelet
  namespace: kube-system
  labels:
    k8s-app: kubelet
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: https-metrics
      port: 10250
    - name: cadvisor
      port: 4194

# Resulting EndpointSlice (IPv4 only — IPv6 slice is never created)
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: kubelet-xxxxx
  namespace: kube-system
  labels:
    kubernetes.io/service-name: kubelet
addressType: IPv4
endpoints:
  - addresses: ["10.x.x.x"]   # IPv4 only, no IPv6 slice exists

prometheus-operator log output

# No errors logged — the operator successfully creates the IPv4 EndpointSlice.
# The bug is silent: the IPv6 slice is simply never created.
level=info msg="Nodes converted to endpoint addresses" num_addresses=12
level=info msg="Updating endpointslice object"

Anything else?

Kubernetes Cluster Type: Gardener-managed AWS shoot cluster with dual-stack networking (ipFamilies: [IPv6, IPv4])

This bug became visible with Prometheus 3.11.0 which introduced prometheus/prometheus#18192 — a correct fix for duplicate scraping on dual-stack services. That PR added nonPrimaryIPFamilySlice() which drops EndpointSlices whose addressType does not match svc.Spec.IPFamilies[0]. Before 3.11, the IPv4 EndpointSlice was used silently even on IPv6-primary clusters; after 3.11 it is correctly dropped, exposing the missing IPv6 EndpointSlice. The issue affects any dual-stack cluster where IPv6 is the primary IP family and prometheus-operator is deployed with --kubelet-endpointslice=true.

Workaround (until upstream fix is available): set --kubelet-endpointslice=false in the prometheus-operator deployment and serviceDiscoveryRole: Endpoints in the Prometheus CRD. This falls back to the classic v1.Endpoints object which is populated with IPv4 addresses only and is not subject to the Prometheus 3.11 deduplication filter.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions