Allow embedding a Dag run under an external trace via run conf #67210

Context

DagRun spans are the head of an Airflow trace: new_dagrun_trace_carrier mints a fresh root W3C traceparent when the run is created, and _emit_dagrun_span (in airflow-core/src/airflow/models/dagrun.py) emits the dag_run.* span with context=context.Context(), forcing a root span unconditionally. Every downstream span (task_run, worker) rides on that carrier, which is also the head-sampling decision point (see #68591, #68592, #68860).

This is the correct default: an Airflow-initiated run (scheduler, timetable, asset, backfill, UI/API trigger) is a workflow root and should own its own trace. Reviewers on #67424 — which attempted to auto-nest based on trigger type — confirmed this: the Dag run span should not become a child of the scheduler loop or the API request span that happened to create it.

Problem

There is a legitimate, currently-impossible use case: a run triggered by an external system that is already emitting a trace (an upstream orchestrator, an event pipeline, a CI job, another Airflow deployment). The operator may want that Airflow run to appear inside the external trace rather than as a disconnected root, so the whole workflow reads end-to-end.

The original spec here proposed nesting under the "currently active parent span," chosen from the trigger source. That is wrong: _emit_dagrun_span runs in the scheduler, so the active in-process span is never the external workflow — it is a scheduler-internal span, exactly what reviewers rejected. The parent must be supplied explicitly, as data, not inferred from ambient process context.

On the "spans always nest under the current context" convention

That rule is the synchronous RPC convention: it assumes the code doing the work received a propagated context and executes inside the logical operation that owns it, so "current context = logical parent" holds. A Dag run is not that shape — it is async / triggered / fan-out. The trigger (API call, timetable tick, asset event, another operator on a worker) is decoupled from execution in both process and time: the run is created server-side and the dag_run.* span is emitted in the scheduler at completion, across many worker processes. OpenTelemetry's own semantic conventions for messaging / batch / "triggered-by" causality that crosses an async boundary use new root + span link back to the producer, precisely because the consumer's work is not a synchronous child and may be delayed, retried, batched, or fanned out. So default-root (with optional explicit embed / link) is the correct convention for this shape — not an Airflow-specific exception.

Note also that tracer.start_span() with no explicit context is already a root span when no span is active in the current context; default-nest and default-root only diverge when a span is ambiently active at emit time — and in the scheduler that span is an internal artifact. "Default root" here means "do not attach to the incidental scheduler span," which is exactly why the current code force-roots.

Proposal

Make external-trace attachment opt-in, carried as a reserved run conf key — consistent with the existing per-run trace controls airflow/trace_sampled (#68860) and airflow/task_span_detail_level:

• Default (no key): unchanged — the Dag run is a root trace.
• Opt-in: a reserved conf key (e.g. airflow/parent_trace_context) carrying a W3C traceparent (and optionally tracestate). When present, the run attaches to that external trace.

Apply the decision at carrier-creation time (new_dagrun_trace_carrier), not at span-emit time, so that (a) the whole run — Dag run span and all task/worker spans — consistently lives in the external trace, and (b) parent-based samplers (parentbased_*) inherit the external SAMPLED decision.

This does not preclude a future step where the API server auto-fills the conf key from an inbound traceparent header — at which point genuinely-propagated runs nest automatically while everything else stays root. The requirement here is only that the parent be real (carried as data), not the scheduler's accidental ambient span.

Open question — attach how? (needs resolving on the PR)

1. Parent-child nesting (suggested default): the Dag run span becomes a child of the external span; run + tasks share the external trace_id. True embedding; enables sampling propagation. Downside: the run is no longer a standalone root trace.
2. Span link: the Dag run stays its own root trace but carries an OTel link to the external span (suggested by a reviewer on Fix Dag run span parent context handling #67424, and the OTel-conventional choice for async producer→consumer causality). Keeps traces separable; does not propagate sampling, and the run is not visually inside the external trace.
3. Both: nest when an explicit external parent is supplied, and also record a link. Most flexible, most to build/test/document.

Recommendation: start with nesting, leave the door open to a link. Final call on the PR.

Acceptance criteria

• _emit_dagrun_span no longer unconditionally forces a root span but remains root by default.
• Attachment is driven by an explicit reserved run-conf key (a supplied W3C traceparent), not by trigger source or ambient in-process context.
• Airflow-initiated runs with no such conf remain root traces (no regression).
• The chosen mechanism is implemented and documented alongside the other airflow/* trace conf keys.
• The inline TODO at _emit_dagrun_span referencing this issue is removed.

Refs

• Original PR review comment: Add detailed task spans #63568 (comment)
• Attempted implementation + reviewer feedback: Fix Dag run span parent context handling #67424
• Related per-run trace controls: Make head sampling possible in airflow #68591, Forward run-identity attributes to the trace sampler #68592, Allow forcing trace head-sampling per DAG run via conf #68860

Drafted-by: Claude Code (Opus 4.8); reviewed by @dstandish before posting