Owner Metadata for Spans. #4008
Description
Feature Request
Right Now, For metrics Prometheus automatically adds the pod's labels as metric labels, giving us the capability to link grafana dashboard for that component directly in the linkerd dashboard and allowing users to jump there seamlessly.
A similar model with Jaeger would be great i.e jumping directly from the linkerd dashboard to the jaeger dashboard and showing relevant spans of the component. This would require some labels to be present that allows us to filter spans based on the selected component.
We already send some owner Metadata, for the spans as seen in the picture.
The hostname label allows us to categorize spans based on the pod that the spans came from, allowing us to directly specify a link with the query hostname label for pods. But we would also want a similar case for the kind and name of workload.
Sadly, This Information is actually not passed into the pod's env variables right now. The hostname is injected by k8s currently and we use that. But A similar thing is not present for workloads.
What problem are you trying to solve?
More metadata for spans.
How should the problem be solved?
We already find the workload, type information during injection and add them as pod labels.
We can use the downward API https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/ to make those labels attached as volumes for the linkerd-proxy to attach them as labels here.
https://github.com/linkerd/linkerd2-proxy/blob/9faff18c8a6ffabe92afcd2a47ab169b99baa1b0/linkerd/trace-context/src/layer.rs#L142
Any alternatives you've considered?
- In Open Telemetry, there is a k8s processor which attaches those labels as per https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/processor/k8sprocessor by doing a reverse pod lookup of the pod. But openTelemtry is not 1.0 yet. I'm giving this a try and will psot the results.
Further reading:
https://developers.soundcloud.com/blog/using-kubernetes-pod-metadata-to-improve-zipkin-traces