otel: propagate context across forks + root spans at queue/gRPC entry points#414
Conversation
… points
Hasql/inSpan spans emitted from background work were orphan because
hs-opentelemetry-api keys its Context on Haskell ThreadId; every Ki.fork /
async starts on a fresh ThreadId with empty context. Two fixes:
1. System.Tracing.forkWithCtx — Effectful.Ki.fork variant that captures
the parent's OTel context and re-attaches it inside the child fiber.
Swept the 5 Ki.fork sites in BackgroundJobs.hs (project fan-out,
dual-write, endpoint discovery fan-out, gitsync).
2. Root spans at message-driven entry points so downstream spans have
something to attach to:
- OtlpServer.processList → withSpan_ "otlp.processList"
(covers Kafka + Pubsub OTLP batches)
- OtlpServer.grpcRunBackground → withSpan_ "otlp.grpc.<label>"
(covers all three gRPC RpcHandlers — traces/logs/metrics)
- ProcessMessage.processMessages → withSpan_ "pubsub.processMessages"
processList/processMessages signatures gain Tracing :> es. Both run
inside runBackground / runTestBg, which already wire runTracing.
HTTP path was already correct (newOpenTelemetryWaiMiddleware in
Server.hs sets up the request span). Top-level service async fibers
in Server.hs intentionally stay plain async — they are root services,
not children of any caller context.
Code ReviewOverviewClean, targeted fix: propagate OTel context across Issues1.
Prefer bracket-style cleanup, which is also shorter since it avoids the nested forkWithCtx scope action = do
ctx <- liftIO Context.getContext
Ki.fork scope do
detach <- liftIO $ Context.attachContext ctx
action `finally` liftIO detach( 2.
pgThread <- Ki.fork scope $ tryAny $ void $ bulkInsertOtelLogsAndSpans records
tfThread <- Ki.fork scope do
res <- tryAny (retryTimefusion 10 records)These are the Hasql-intensive paths — the exact calls that produce orphan spans. The PR swept the 5
3.
Nits
SummaryTwo things worth fixing before merge:
|
- forkWithCtx: capture previous context from attachContext and restore it via adjustContext on the way out (finally). Matters once the thread is pooled or otherwise outlives the fork — today's per-fork OS threads make it cosmetic but the prior code silently dropped the token. - Telemetry.bulkInsertOtelLogsAndSpansTF: convert both Ki.fork sites (PG writer + TF writer). These are the dual-write hot path so their hasql spans were the most visible orphans.
The PR 'Build and Test PR' workflow runs hlint with fail-on=warning; the 'restricted alias' rule wants Data.HashMap.Strict imported as HM. master's Deploy workflow skips hlint, which is why this slipped past.
Code ReviewGood fix overall — the OTel context-propagation problem is real and the 1. Incomplete sweep —
|
Code Review: otel context propagation + root spansCorrect fix for orphan spans.
forkWithCtx scope action = do
ctx <- liftIO Context.getContext
Ki.fork scope $
bracket
(liftIO $ Context.attachContext ctx)
(\prev -> liftIO $ Context.adjustContext (const prev))
(const action)Pre-existing exception-safety gap in Span naming: OTel semantic conventions use Constraint propagation: Adding Test coverage: The test plan is manual. A small unit test forking under Summary:
|
1. Complete the fork sweep: - Pages/LogExplorer/Log.hs (6 sites) — log explorer prefetch fan-out - Pages/Bots/Discord.hs (2 sites) — discord thread fetch 2. forkWithCtx: switch from finally-with-loose-state to bracket so the acquire / release / use shape is explicit. 3. Add attributes to root spans so they're useful in production: - processList: messaging.batch.message_count, ce.type - processMessages: same - grpc handlers: otlp.signal (= traces|logs|metrics)
otlp.processList → otlp.process_list pubsub.processMessages → pubsub.process_messages otlp.grpc.<label> unchanged (label is already snake-case)
|
Code Review PR 414 otel context propagation - Overview: Clean well-scoped fix. The core mechanism forkWithCtx is correct: capture ThreadId-keyed context before the fork, attach it inside the child, and restore on exit. Root spans at processList, grpcRunBackground, and processMessages are the right entry points. Issues: (1) Use HM.findWithDefault instead of fromMaybe lookup at the two ce-type attribute sites in processList and processMessages. (2) Trailing do without dollar in processMessages - should match processList style with ] dollar do. (3) forkWithCtx bracket cleanup is a no-op in practice - GHC RTS does not pool Ki/forkIO threads, each gets a fresh ThreadId, the restore is dead code. (4) const action in bracket - minor style nit, backslash_ -> action is clearer. (5) Discord.hs forks propagate context to plain HTTP calls that do not emit OTel spans - harmless overhead, worth a note. (6) Log.hs forks (6 sites) not mentioned in PR description - Pages/LogExplorer/Log.hs has 6 Ki.fork -> forkWithCtx changes absent from the summary, please add a line to the PR body. (7) length msgs in span attributes is O(n) - low severity but worth noting for high-throughput ingestion paths. (8) Data.Effectful.Hasql import style change is unrelated and belongs in its own commit. Correctness looks good: bracket handles async cancellation correctly, processList short-circuit before withSpan_ is correct, grpcRunBackground withSpan_ wraps correctly, awaitAll scope semantics preserved. Summary: mechanism is sound, main asks before merging are (1) HM.findWithDefault at the two ce-type sites, (2) add Log.hs forks to PR description, (3) clarify or drop the bracket restore comment in forkWithCtx. |
…ontext attachContext returns Token (not Context). Restoring via adjustContext type-errors. Pair with detachContext which is the matched API.
These were untracked working files that got swept in by the initial git add -A; not part of the otel context-propagation change.
Code ReviewOverall: Well-targeted fix. Correctness
Missed Potential missed sites: The PR description says "swept the 5 to confirm the sweep is complete. Code quality
-- OtlpServer.hs and ProcessMessage.hs
("ce.type", OA.toAttribute (fromMaybe "" (HM.lookup "ce-type" attrs)))Setting an attribute to let baseAttrs = [("messaging.batch.message_count", OA.toAttribute (length msgs))]
ceAttr = [("ce.type", OA.toAttribute t) | t <- maybeToList (HM.lookup "ce-type" attrs)]
withSpan_ "..." (baseAttrs <> ceAttr) ...Span name inconsistency — the PR description calls the span
tryAny (runBackground appLogger appCtx tp $ withSpan_ ("otlp.grpc." <> label) [...] task) >>= \caseExtracting the wrapped task into a
Succinctness / packages
-- current
bracket acquire release (const action)
-- slightly cleaner (matches local style)
bracket acquire release \_ -> actionBoth are fine; this is style-only. No reimplementation concerns — Test coverageNo tests added for
…would prevent silent regressions if the OTel API changes its thread-local semantics. Not blocking, but worth a TODO. Performance / securityNo concerns. Context capture is a single Summary
|
Code ReviewSummary: Fixes orphan OTel spans from forked fibers by capturing/restoring thread-local context in
|
| Core fix | ✅ Correct — forkWithCtx is the right approach |
Exception safety in WithSpan |
|
| Context API consistency | adjustContext vs attach/detach mismatch |
Duplicate fromMaybe "" pattern |
|
| Unrelated deletion | ❌ Should be separate PR |
|
test |
|
Code Review: otel context propagation Overview: Clean, well-scoped fix. forkWithCtx is minimal and correct; root spans at processList/processMessages/grpcRunBackground give downstream Hasql spans a parent. The Ki.fork sweep looks comprehensive. Issues:
Minor: grpcRunBackground puts withSpan_ inside runBackground effect stack (correct, runTracing is wired there). A one-line comment helps future readers. Span name otlp.process_list in code vs otlp.processList in description -- align for dashboard clarity. What is good: forkWithCtx is the right abstraction. bracket for context restore is correct. Root span placement is correct. Attribute choice follows OTel conventions. Incident/plan docs are useful. Verdict: Approve with the comment fix and a Ki.fork grep before merge. |
Code ReviewGood fix for a real correctness problem — OTel orphan spans from forked threads are genuinely hard to debug. The approach (propagate context at fork time via Bug:
|
1. runTracing: close span + restore context in `finally` so exceptions in the body don't leak open spans. Pre-existed but the 3 new root spans amplified the blast radius. 2. Discord forks revert to plain Ki.fork — the getWith calls don't produce OTel spans, so context propagation was pollution. Drops the IOE :> es constraint from getThreadStarterMessage. 3. Use fold instead of fromMaybe \"\" for Text Monoid lookup. 4. fromIntegral @int @int64 on length so the attribute type matches OTel's 64-bit integer convention.
Code ReviewSummarySolid fix for a real OTel correctness problem. Positives
Issues1. Redundant OA.toAttribute (fromIntegral @Int @Int64 $ length msgs)
("batch_size", OA.toAttribute $ V.length projectIds)Both call sites can just write 2. Duplicated attribute pair
[ ("messaging.batch.message_count", OA.toAttribute ...)
, ("ce.type", OA.toAttribute @Text $ fold (HM.lookup "ce-type" attrs))
]Consider extracting a small helper or at least sharing the definition. Given these two functions sit in different modules this is borderline, but the duplication is verbatim. 3.
Minor Nits
Test CoverageNo automated tests — acknowledged in the PR description with a manual verification plan. Given this is IO/thread-local behaviour, a property test that spawns child fibers and asserts Overall this is a well-reasoned fix with good rollout (sweeping all relevant |
Matches BackgroundJobs.hs:939 and other call sites.
Code ReviewOverviewThis PR fixes orphan OTel spans in background/async work by: (1) introducing IssuesMissed
|
- runTracing: set Error status on exception (was endSpan Nothing → failed spans appeared green in trace UIs). - ce.type: omit the attribute entirely when missing instead of emitting empty string; foldMap on Maybe keeps the list inline. - Discord.hs: comment explaining why Ki.fork stays plain — getWith from Data.Effectful.Wreq is not OTel-instrumented so context propagation has nothing to attach.
Code ReviewOverviewThis PR fixes two real problems: orphan OTel spans from background fibers (context is ThreadId-keyed and isn't inherited across Bug fix in
|
…ages (#415) * otel: extract batchSpanAttrs helper to dedupe processList/processMessages Both root-span sites built the same attribute list inline. Pulled it into System.Tracing.batchSpanAttrs so the two callers reduce to a single helper call. Uses toList over Maybe to drop the ce.type attribute entirely when the CloudEvents header is absent (instead of emitting an empty string). Follow-up from PR #414 review — the dedupe landed on the branch a few seconds after the squash merge, so it didn't make the merge. * address PR #415 review: maybeToList + drop unused rmAckId binding * drop redundant @text annotation in batchSpanAttrs * batchSpanAttrs: switch to catMaybes form for uniform optional-attr handling * address PR #415 review: _ackId → _ and clarify ce-type/ce.type haddock
Summary
Hasql/inSpan spans emitted from background work were orphan because
hs-opentelemetry-apikeys itsContexton HaskellThreadId; everyKi.fork/asyncstarts on a freshThreadIdwith empty context. Plus an exception-safety bug inrunTracingitself. Fixes:System.Tracing.forkWithCtx—Effectful.Ki.forkvariant that captures the parent's OTel context and re-attaches it inside the child fiber viabracket attachContext detachContext. Swept allKi.forksites that emit OTel spans downstream:BackgroundJobs.hs(5),Telemetry.hs(2 — dual-write hot path),Pages/LogExplorer/Log.hs(6).Pages/Bots/Discord.hsleft as plainKi.forkwith a comment —getWithis not OTel-instrumented.OtlpServer.processList→"otlp.process_list"(Kafka + Pubsub OTLP batches)OtlpServer.grpcRunBackground→"otlp.grpc.<label>"(all 3 gRPC RpcHandlers)ProcessMessage.processMessages→"pubsub.process_messages"messaging.batch.message_count+ce.typevia sharedbatchSpanAttrs.runTracingexception safety — the old handler skippedendSpan+ context restore on exception, silently leaking unclosed spans. Now useswithException(to markErrorstatus) +finally(to always close + restore).processList/processMessagessignatures gainTracing :> es. Both run insiderunBackground/runTestBg, which already wirerunTracing.HTTP path was already correct (
newOpenTelemetryWaiMiddleware). Top-level service async fibers inServer.hsintentionally stay plainasync— they're root services.Test plan
otlp.process_listparent in monoscope's own tracesotlp.grpc.traces/.logs/.metricsrootsErrorstatus (no green-failed spans)