Is there an existing issue for this?
What happened?
Description
On a dual-stack Kubernetes cluster configured with ipFamilies: [IPv6, IPv4] (IPv6-first), prometheus-operator creates only an addressType: IPv4 EndpointSlice for the kubelet Service. No IPv6 EndpointSlice is created. This causes all kubelet scrape targets to be silently dropped since Prometheus 3.11.0.
Steps to Reproduce
- Deploy prometheus-operator v0.92.1 with
--kubelet-endpointslice=true on a dual-stack cluster where the shoot/cluster networking has ipFamilies: [IPv6, IPv4] (IPv6 as primary family).
- Inspect the EndpointSlices created for the kubelet Service:
kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kubelet
- Observe that only an
addressType: IPv4 EndpointSlice exists, despite nodes having both IPv4 and IPv6 InternalIP addresses in node.Status.Addresses.
- Inspect a node to confirm both address families are present:
kubectl get node <node-name> -o jsonpath='{.status.addresses}' | jq .
This returns two entries with "type": "InternalIP" — one IPv4 and one IPv6.
Expected Result
prometheus-operator should create both an addressType: IPv4 and an addressType: IPv6 EndpointSlice for the kubelet Service when nodes report both address families under InternalIP in node.Status.Addresses.
Actual Result
Only an addressType: IPv4 EndpointSlice is created. On clusters where svc.Spec.IPFamilies[0] is IPv6 (because the kubelet Service is created without an explicit ipFamilyPolicy, so Kubernetes assigns the cluster's default), Prometheus 3.11.0+ drops the IPv4 EndpointSlice via the deduplication logic introduced in prometheus/prometheus#18192, resulting in 0 active kubelet scrape targets.
Prometheus Operator Version
Prometheus operator version: 0.92.1
Args:
--kubelet-service=kube-system/kubelet
--kubelet-endpoints=true
--kubelet-endpointslice=true
Kubernetes Version
clientVersion:
buildDate: "2025-11-11T19:00:37Z"
gitVersion: v1.34.2
platform: darwin/arm64
serverVersion:
buildDate: "2026-04-15T18:01:23Z"
gitVersion: v1.35.4
platform: linux/amd64
Kubernetes Cluster Type
Other (please comment)
How did you deploy Prometheus-Operator?
yaml manifests
Manifests
# kubelet Service as created by prometheus-operator (no ipFamilyPolicy set)
apiVersion: v1
kind: Service
metadata:
name: kubelet
namespace: kube-system
labels:
k8s-app: kubelet
spec:
type: ClusterIP
clusterIP: None
ports:
- name: https-metrics
port: 10250
- name: cadvisor
port: 4194
# Resulting EndpointSlice (IPv4 only — IPv6 slice is never created)
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: kubelet-xxxxx
namespace: kube-system
labels:
kubernetes.io/service-name: kubelet
addressType: IPv4
endpoints:
- addresses: ["10.x.x.x"] # IPv4 only, no IPv6 slice exists
prometheus-operator log output
# No errors logged — the operator successfully creates the IPv4 EndpointSlice.
# The bug is silent: the IPv6 slice is simply never created.
level=info msg="Nodes converted to endpoint addresses" num_addresses=12
level=info msg="Updating endpointslice object"
Anything else?
Kubernetes Cluster Type: Gardener-managed AWS shoot cluster with dual-stack networking (ipFamilies: [IPv6, IPv4])
This bug became visible with Prometheus 3.11.0 which introduced prometheus/prometheus#18192 — a correct fix for duplicate scraping on dual-stack services. That PR added nonPrimaryIPFamilySlice() which drops EndpointSlices whose addressType does not match svc.Spec.IPFamilies[0]. Before 3.11, the IPv4 EndpointSlice was used silently even on IPv6-primary clusters; after 3.11 it is correctly dropped, exposing the missing IPv6 EndpointSlice. The issue affects any dual-stack cluster where IPv6 is the primary IP family and prometheus-operator is deployed with --kubelet-endpointslice=true.
Workaround (until upstream fix is available): set --kubelet-endpointslice=false in the prometheus-operator deployment and serviceDiscoveryRole: Endpoints in the Prometheus CRD. This falls back to the classic v1.Endpoints object which is populated with IPv4 addresses only and is not subject to the Prometheus 3.11 deduplication filter.
Is there an existing issue for this?
What happened?
Description
On a dual-stack Kubernetes cluster configured with
ipFamilies: [IPv6, IPv4](IPv6-first), prometheus-operator creates only anaddressType: IPv4EndpointSlice for the kubelet Service. No IPv6 EndpointSlice is created. This causes all kubelet scrape targets to be silently dropped since Prometheus 3.11.0.Steps to Reproduce
--kubelet-endpointslice=trueon a dual-stack cluster where the shoot/cluster networking hasipFamilies: [IPv6, IPv4](IPv6 as primary family).addressType: IPv4EndpointSlice exists, despite nodes having both IPv4 and IPv6InternalIPaddresses innode.Status.Addresses."type": "InternalIP"— one IPv4 and one IPv6.Expected Result
prometheus-operator should create both an
addressType: IPv4and anaddressType: IPv6EndpointSlice for the kubelet Service when nodes report both address families underInternalIPinnode.Status.Addresses.Actual Result
Only an
addressType: IPv4EndpointSlice is created. On clusters wheresvc.Spec.IPFamilies[0]isIPv6(because the kubelet Service is created without an explicitipFamilyPolicy, so Kubernetes assigns the cluster's default), Prometheus 3.11.0+ drops the IPv4 EndpointSlice via the deduplication logic introduced in prometheus/prometheus#18192, resulting in 0 active kubelet scrape targets.Prometheus Operator Version
Kubernetes Version
Kubernetes Cluster Type
Other (please comment)
How did you deploy Prometheus-Operator?
yaml manifests
Manifests
prometheus-operator log output
Anything else?
Kubernetes Cluster Type: Gardener-managed AWS shoot cluster with dual-stack networking (
ipFamilies: [IPv6, IPv4])This bug became visible with Prometheus 3.11.0 which introduced prometheus/prometheus#18192 — a correct fix for duplicate scraping on dual-stack services. That PR added
nonPrimaryIPFamilySlice()which drops EndpointSlices whoseaddressTypedoes not matchsvc.Spec.IPFamilies[0]. Before 3.11, the IPv4 EndpointSlice was used silently even on IPv6-primary clusters; after 3.11 it is correctly dropped, exposing the missing IPv6 EndpointSlice. The issue affects any dual-stack cluster where IPv6 is the primary IP family and prometheus-operator is deployed with--kubelet-endpointslice=true.Workaround (until upstream fix is available): set
--kubelet-endpointslice=falsein the prometheus-operator deployment andserviceDiscoveryRole: Endpointsin the Prometheus CRD. This falls back to the classicv1.Endpointsobject which is populated with IPv4 addresses only and is not subject to the Prometheus 3.11 deduplication filter.