You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feat]: Use port 10250 on the Metrics Server pod to use existing firewall rules #1026
What would you like to be added:
If we change the default port for the Metrics Server pod to 10250 then it would be covered by default firewall rules and work on most clusters out of the box. This is exactly what projects such as Cert Manager and Prometheus Operator do with their web hooks to make them just work.
The general pattern for cluster to pod communication seems to be to use port 443 where possible, so Service, and to use port 10250 where that isn't possible. The exception to this rule is where the host network needs to be used, in which case the port needs changing and the cluster firewall rules need modifying to support the new port.
For the Helm chart this would mean changing the default containerPort value to 10250.
I have tested this on an EKS cluster and can confirm that it functions correctly.
Why is this needed:
This would make Metrics Server just work on most Kubernetes clusters without any additional configuration.
I also suspect that the removal of support for running the pod with port 443 is why the Metrics Server version in use is often far older than expected.
So just for clarity, I'm expecting to change the default value of --secure-port to 10250 but not make any other changes to the Go code. In the manifests and Helm chart I'll be changing Metrics server to use 10250 for --secure-port and the deployment port but I'll be leaving the service port as 443, which is in line with the other users of this pattern.
No changing Go code sgtm, however I was thinking about checking Service port to 10250. Reason is that depending on configuration apiserver might want to communicate using service port instead of container port, and depend on kube-proxy to do the loadbalancing.
@serathius I'm pretty sure that 443 will be open by default in most if not all clusters, for example GKE has it open for Metrics Server. This is how the other projects operate and they are activly using the service to load balance the requests from the control plane.
I'm not sure if we can about what crazy things people do, so definitely don't think that people have port 443 opened. I also would be worried about assuming that people follow GKE as example, got burned on this to many times (FYI I work on GKE :P).
@serathius I agree with the sentiment but does port 443 need to be open anywhere to use it on a Service? My mental model has kube-proxy translate the call to the service kube-system/metrics-server:443 to an EndpointSlice which uses the container port (which is why services can always use low ports)?
My understanding is (definetly not k8s network expert), is that kube-system/metrics-server service is translated into domain metrics-server.kube-system (or something like that). Domain is then resolved by kube-dns to service IP, which is separate IP pool than container one. So the request is send by application to Service IP, only when request gets to host VM, IP forwarding rules controlled by kube-proxy, translate the Service IP to container IP allowing loadbalancing between multiple containers.
I'm pretty sure that a request to the metrics server service would be resolved to metrics-server.kube-system.svc.cluster.local:443 and when the request is made it would be intercepted by iptable, IPVS or eBPF rules which would redirect the traffic to a backend on the container port (iptables reference). Assuming that this is correct the service port doesn't matter as it's only used for discovery from the source node.
This port is in used by the kubelet? That's why it's already open. Now with setting the containerPort to 10250, metrics-server startup fails with panic: failed to create listener: failed to listen on 0.0.0.0:10250: listen tcp 0.0.0.0:10250: bind: address already in use.
Solution:
Leave the containerPort: 4443 and modify the node sg to include an ingress definition for 4443. See below if using terraform-aws-eks module:
Once this is in place, you can check the status of the metrics server with kubectl get apiservice v1beta1.metrics.k8s.io -o json | jq '.status' & should see:
10250 will be in use for small k8s installs (k0s, k3s) when control nodes also run as workers, I'm not sure it's such a great idea to have clashing ports between kubelet and metrics-server in case metrics-server needs to run with hostNetwork: true
@artem-zinnatullin this is a common pattern across the Kubernetes ecosystem. Using host network is non-standard in this context so I don't think it's much extra complexity to set a second value to change the port. This could be automated when host network is set but that might make documentation harder.
Activity
stevehipwell commentedon May 12, 2022
CC @serathius
serathius commentedon May 12, 2022
This is what we did in kubernetes/kubernetes#105957 so +1 from me.
Making MS configuration simpler by reducing ports needed to be enabled is definitely a good goal.
stevehipwell commentedon May 12, 2022
@serathius are you happy if I open a PR to do this?
serathius commentedon May 12, 2022
Sure, please be sure to:
stevehipwell commentedon May 12, 2022
So just for clarity, I'm expecting to change the default value of
--secure-portto10250but not make any other changes to the Go code. In the manifests and Helm chart I'll be changing Metrics server to use10250for--secure-portand the deployment port but I'll be leaving the service port as 443, which is in line with the other users of this pattern.serathius commentedon May 12, 2022
No changing Go code sgtm, however I was thinking about checking Service port to
10250. Reason is that depending on configuration apiserver might want to communicate using service port instead of container port, and depend on kube-proxy to do the loadbalancing.stevehipwell commentedon May 12, 2022
@serathius I'm pretty sure that
443will be open by default in most if not all clusters, for example GKE has it open for Metrics Server. This is how the other projects operate and they are activly using the service to load balance the requests from the control plane.serathius commentedon May 12, 2022
I'm not sure if we can about what crazy things people do, so definitely don't think that people have port 443 opened. I also would be worried about assuming that people follow GKE as example, got burned on this to many times (FYI I work on GKE :P). There is a surprising amount of people that setup Kubernetes the hard way and expect everything to work out of the box (just read https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md :P)
My hope with that change that we can not only change, but also reduce amount of ports we require Metrics Server to run.
stevehipwell commentedon May 13, 2022
@serathius I agree with the sentiment but does port
443need to be open anywhere to use it on aService? My mental model has kube-proxy translate the call to the servicekube-system/metrics-server:443to anEndpointSlicewhich uses the container port (which is why services can always use low ports)?serathius commentedon May 13, 2022
My understanding is (definetly not k8s network expert), is that
kube-system/metrics-serverservice is translated into domainmetrics-server.kube-system(or something like that). Domain is then resolved bykube-dnsto service IP, which is separate IP pool than container one. So the request is send by application to Service IP, only when request gets to host VM, IP forwarding rules controlled bykube-proxy, translate the Service IP to container IP allowing loadbalancing between multiple containers.stevehipwell commentedon May 13, 2022
I'm pretty sure that a request to the metrics server service would be resolved to
metrics-server.kube-system.svc.cluster.local:443and when the request is made it would be intercepted by iptable, IPVS or eBPF rules which would redirect the traffic to a backend on the container port (iptables reference). Assuming that this is correct the service port doesn't matter as it's only used for discovery from the source node.jtgorny commentedon Dec 14, 2022
This port is in used by the kubelet? That's why it's already open. Now with setting the
containerPortto 10250, metrics-server startup fails withpanic: failed to create listener: failed to listen on 0.0.0.0:10250: listen tcp 0.0.0.0:10250: bind: address already in use.Solution:
containerPort: 4443and modify the node sg to include an ingress definition for 4443. See below if usingterraform-aws-eksmodule:kubectl get apiservice v1beta1.metrics.k8s.io -o json | jq '.status'& should see:stevehipwell commentedon Dec 14, 2022
@jtgorny port
10250is only in use if you're running Metrics Server on the host network?artem-zinnatullin commentedon Mar 29, 2023
10250will be in use for small k8s installs (k0s, k3s) when control nodes also run as workers, I'm not sure it's such a great idea to have clashing ports between kubelet and metrics-server in case metrics-server needs to run withhostNetwork: truestevehipwell commentedon Mar 29, 2023
@artem-zinnatullin this is a common pattern across the Kubernetes ecosystem. Using host network is non-standard in this context so I don't think it's much extra complexity to set a second value to change the port. This could be automated when host network is set but that might make documentation harder.
fix(metrics-server): move the listener off the kubelet's port (#3361)