fix(ddtrace/tracer): make traceID hex cache safe for concurrent reads#4726
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 01957d8 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-06-25 10:44:31 Comparing candidate commit 01957d8 in PR branch Found 0 performance improvements and 2 performance regressions! Performance is the same for 295 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
|
@rarguelloF could you have a look at this? |
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
devflow unqueued this merge request: It did not become mergeable within the expected time |
|
@codex review |
Split TestConcurrentInjectTraceIDHex into cold-cache (local span) and hot-cache (extracted context) sub-tests. The cold-cache case now asserts hexEncoded is empty as a precondition instead of forcing it, so the test fails loudly if newSpanContext ever starts populating the cache. The new hot-cache case round-trips through Inject/Extract so extractTextMap finalizes the traceID via cacheHex, exercising the cached branch of HexEncoded under -race.
…ests Use the public HexEncoded() accessor instead of reaching into the unexported traceID.value field. This compares trace IDs by their semantic hex value regardless of hex-cache state (a local span has a cold cache, an extracted context a hot one), which is more intention-revealing than a raw byte-field comparison.
newSpanContext deliberately does not finalize the traceID hex cache, so locally started spans use the non-caching HexEncoded fallback and allocate on each HexEncoded/UpperHex call (e.g. per Inject). This is an intentional choice to keep span creation allocation-free at the cost of per-call hex allocation on the propagation path. Document it at newSpanContext and correct the HexEncoded doc so the cold-cache path is described as the normal state for local spans rather than an edge case.
dbd325f
into
main
What does this PR do?
Makes the lazy
traceID.hexEncodedcache (introduced in #4481) safe under concurrent reads.(*traceID).HexEncodedno longer writes the cache from a read path; the cache is now populated eagerly at construction finalization for the context-extraction paths, andHexEncodedfalls back to a non-cachinghex.EncodeToStringif the cache is empty (so concurrent readers can never race on the string write).Concrete changes:
(*traceID).HexEncodedis now a pure read with a non-caching fallback.computeAndCacheHex→cacheHex, with a doc comment stating it must be called at construction finalization, before the enclosingSpanContextis shared.cacheHex()at the trace-ID extraction finalization sites, where the resultingSpanContextis about to be returned and may then be injected concurrently:FromGenericCtx(all three return paths)extractTextMap(Datadog propagator)extractTraceID128(covers B3 single/multi and W3CparseTraceparent)spanContextFromTraceComment(sqlcomment Extract)newSpanContext(the localStartSpanpath) intentionally does not callcacheHex(): caching there would add an allocation to every span creation, including spans that are never injected. Local spans rely on the non-cachingHexEncodedfallback instead, which is race-free because the fallback performs no write.traceIDFrom64Bits/traceIDFrom128Bitsfinalize viacacheHex.TestConcurrentInjectTraceIDHexreproduces the race under-race(fan-outInjecton a sharedSpanContextwith an empty hex cache); fails onmain, passes with this fix.Motivation
PR #4481 cached the hex form of
traceIDto avoid millions ofhex.EncodeToStringallocations per second on the propagation hot path. The cache used a check-then-act lazy write with no synchronization:A
KafkaWriterfan-out crashed withpanic: runtime error: invalid memory address or nil pointer dereferenceinisValidPropagatableTag. Two goroutines callInjecton the sameSpanContext; both see an empty cache, both write tot.hexEncoded, and a torn read of the Go string header ({ptr,len}is 16 bytes, not atomic) yields{nil, 16}. The downstreamfor _, ch := range vthen dereferences the nil pointer.Stack from the crash:
tracer/util.go:41←textmap.go:481←propagating_tags.go:95←textmap.go:463(injectTextMap→setPropagatingTag(ctx, keyTraceID128, ctx.traceID.UpperHex())).-raceconfirmed the same race directly:Read at spancontext.go:52(check) vs.Previous write at spancontext.go:106(computeAndCacheHex).All
traceIDmutation sites are construction-time (before theSpanContextis shared across goroutines), so the read path can be made a pure, race-free read: extracted contexts populate the cache eagerly at finalization (keeping the inject hot path zero-alloc for propagated spans), while locally created spans fall back to a non-caching computation that performs no write. Affected released versions:v2.8.0-dev,v2.8.0-rc.1,v2.8.0-rc.2, andv2.8.0.Reviewer's Checklist
make lintlocally.make testlocally.make generatelocally.make fix-moduleslocally.