What did you do?
Prometheus has undocumented support for emitting a spanID attribute in the query log when OpenTelemetry tracing is active.
It omits the trace-id, making log/trace correlation much more difficult and less useful.
Add the trace-id to the query log too.
This does not affect the main Prometheus log or the scrape-failure log.
What did you expect to see?
In a query log like
{"time":"2026-02-25T12:05:53.126237084+13:00","level":"INFO","source":"engine.go:710","msg":"promql query logged","params":{"end":"2026-02-24T23:05:53.124Z","query":"up","start":"2026-02-24T23:05:53.124Z","step":0},"stats":{"timings":{"evalTotalTime":0.000078765,"resultSortTime":0,"queryPreparationTime":0.000016221,"innerEvalTime":0.000034725,"execQueueTime":0.000136209,"execTotalTime":0.000112741},"samples":{"totalQueryableSamples":0,"peakSamples":0}},"spanID":"28159b73b9a9a426","httpRequest":{"clientIP":"127.0.0.1","method":"POST","path":"/api/v1/query"}}
with tracing enabled like in the attached config, I expected to see both a traceID and spanID element in the log messages.
What did you see instead? Under which circumstances?
In reality there is no traceID, only a spanID
jq -s '.|first|pick(.spanID, .traceID)' queries.log
{
"spanID": "475af3586496d227",
"traceID": null
}
System information
Linux 6.17.0-14-generic x86_64
Prometheus version
prometheus, version (branch: , revision: 8f1f1f31461d3f86386985ef309c8c5b2213b3bb-modified)
build user:
build date:
go version: go1.25.5
platform: linux/amd64
tags: unknown
Prometheus configuration file
# prometheus.yml
# you don't need an actual working destination for the traces to test this
tracing:
client_type: grpc
endpoint: "localhost:4317"
insecure: true
compression: gzip
timeout: 1s
sampling_fraction: 1
global:
query_log_file: queries.log
Alertmanager version
Alertmanager configuration file
Logs
What did you do?
Prometheus has undocumented support for emitting a
spanIDattribute in the query log when OpenTelemetry tracing is active.It omits the trace-id, making log/trace correlation much more difficult and less useful.
Add the trace-id to the query log too.
This does not affect the main Prometheus log or the scrape-failure log.
What did you expect to see?
In a query log like
with tracing enabled like in the attached config, I expected to see both a traceID and spanID element in the log messages.
What did you see instead? Under which circumstances?
In reality there is no traceID, only a spanID
System information
Linux 6.17.0-14-generic x86_64
Prometheus version
Prometheus configuration file
Alertmanager version
Alertmanager configuration file
Logs