Conversation
Signed-off-by: Alex Leong <[email protected]>
mateiidavid
left a comment
mateiidavid
left a comment
There was a problem hiding this comment.
Nice! Simple and straight to the point. I tested it and it works well.
One tiny suggestion. The change is made in getEndpointByHostname() which in turn will be called from getOrNewPodPublisher. In the latter, we are already formatting the error (see the linked snippet)
linkerd2/controller/api/destination/watcher/pod_watcher.go
Lines 277 to 283 in 29e6683
This yields the following error message:
:; go run controller/script/destination-client/main.go -token "{}" -path 10-23-0-37.nginx-svc.default.svc.cluster.local:80 -method getProfile
FATA[0000] rpc error: code = NotFound desc = failed to get pod for hostname 10-23-0-37: rpc error: code = NotFound desc = no pod found in Endpoints default/nginx-svc for hostname 10-23-0-37
exit status 1
If we take the formatting out of the calling function (pod watcher one) the error should look nicer:
:; go run controller/script/destination-client/main.go -token "{}" -path 10-23-0-37.nginx-svc.default.svc.cluster.local:80 -method getProfile
FATA[0000] rpc error: code = NotFound desc = no pod found in Endpoints default/nginx-svc for hostname 10-23-0-37
exit status 1
Signed-off-by: Alex Leong <[email protected]>
This edge release fixes two bugs in the Destination controller that could cause outbound connections to hang indefinitely. * helm: Introduce configurable values for protocol detection ([#11536]) * destination: Fix GetProfiles error when address is opaque and unmeshed ([#11556]) * destination: Return NotFound for unknown pod names ([#11540]) * proxy: Log controller errors at WARN * proxy: Fix grpc_status metric labels for inbound traffic [#11536]: #11536 [#11556]: #11556 [#11540]: #11540
This edge release fixes two bugs in the Destination controller that could cause outbound connections to hang indefinitely. * helm: Introduce configurable values for protocol detection ([#11536]) * destination: Fix GetProfiles error when address is opaque and unmeshed ([#11556]) * destination: Return NotFound for unknown pod names ([#11540]) * proxy: Log controller errors at WARN * proxy: Fix grpc_status metric labels for inbound traffic [#11536]: #11536 [#11556]: #11556 [#11540]: #11540
|
Could this fix be backported to / included in the 2.14 release branch? Since linkerd/linkerd2-proxy#2499 was merged and released in v2.14.3 we're now experiencing a lot of log spam when using headless services (as explained in the description above). |
|
@annismckenzie Unfortunately we are no longer backporting to 2.14 except for critical security patches. |
5 participants
Fixes #11065
When an inbound proxy receives a request with a canonical name of the form
hostname.service.namespace.svc.cluster.domain, we assume thathostnameis the hostname of the pod as described in https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields. However, pods are also addressable withpod-ip.service.namespace.svc.cluster.domain. When the destination controller gets a profile request of this form, we attempt to find a pod with hostname ofpod-ipand return an error with gRPC statusUnknownsince this will not exist.It is expected that this profile lookup will fail since we cannot have service profiles for individual pods. However, returning a gRPC status
Unknownfor these requests brings the reported success rate of the destination controller down. Instead we should return these as gRPC statusNotFoundso that these responses don't get reported as server errors.