Skip to content

Add detailed task spans - #63568

Merged
dstandish merged 13 commits into
apache:mainfrom
astronomer:add-detailed-task-spans
May 22, 2026
Merged

dstandish merged 13 commits into
apache:mainfrom
astronomer:add-detailed-task-spans

Conversation

@dstandish

@dstandish dstandish commented Mar 13, 2026 •

Copy link
Copy Markdown
Contributor

Builds on top of #63839

Add spans for detailed debugging info during the task execution process

These are only emitted if you add {"airflow/task_span_detail_level": 2} to dag run conf.

@dstandish
dstandish force-pushed the add-detailed-task-spans branch 3 times, most recently from 2cbc72a to 4b37205 Compare March 20, 2026 16:36
@dstandish
dstandish force-pushed the add-detailed-task-spans branch from a1445df to 935f819 Compare April 1, 2026 10:15
@ashb
ashb force-pushed the add-detailed-task-spans branch from f472203 to d83ac4f Compare April 9, 2026 14:53
@dstandish
dstandish force-pushed the add-detailed-task-spans branch from d83ac4f to 6cef7fc Compare April 9, 2026 16:09
@kaxil
kaxil requested a review from Copilot April 10, 2026 19:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds optional “detail spans” to OpenTelemetry traces to improve task execution debugging, gated by a DAG run conf setting (airflow/task_span_detail_level), and extends tests to validate propagation and span hierarchy.

Changes:

  • Introduces detail_span decorator/context manager in the Task SDK task runner and wraps key execution steps with nested spans when detail level > 1.
  • Propagates task_span_detail_level via W3C tracestate in the DAG run’s trace carrier, and preserves it when clearing task instances.
  • Adds unit/integration tests to verify detail-level parsing/propagation and expected span structure.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
task-sdk/src/airflow/sdk/execution_time/task_runner.py Adds detail_span and instruments task lifecycle with nested spans + exception/status recording
task-sdk/tests/task_sdk/execution_time/test_task_runner.py Adds tests for detail_span behavior at different detail levels
shared/observability/src/airflow_shared/observability/traces/init.py Encodes detail level into tracestate and adds API to read it back
shared/observability/tests/observability/test_traces.py Adds tests for tracestate entries, carrier creation, and level extraction
airflow-core/src/airflow/models/dagrun.py Seeds dagrun carrier with detail level; wraps _emit_dagrun_span call with error handling
airflow-core/src/airflow/models/taskinstance.py Preserves detail level when regenerating dagrun carrier during clear_task_instances
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py Makes task instance id attribute string + guards span emission
airflow-core/tests/unit/models/test_dagrun.py Tests that detail level in conf is embedded and readable
airflow-core/tests/unit/models/test_taskinstance.py Tests that clearing task instances preserves detail level
airflow-core/tests/unit/listeners/test_listeners.py Updates expected listener error wording
airflow-core/tests/integration/otel/test_otel.py Parameterizes integration test to validate default vs detail span hierarchies
scripts/ci/docker-compose/integration-otel.yml Adds deprecation note comment for env var
dev/* Adds internal review/demo/trace analysis dev artifacts

dstandish added 13 commits May 21, 2026 15:52
Avoid relying on `TraceState.get(key, default=...)` (not supported in all
OpenTelemetry versions) by fetching the value and applying the default in
Python. Narrow the exception handler to `(TypeError, ValueError)` so
unexpected errors aren't silently swallowed.
The outer `span` could be referenced before assignment if control flow
skipped the assignment path; initialize it to `INVALID_SPAN` up front.
Also drop the inner `as span` binding on the `detail_span("run")` block
to prevent it from shadowing the outer `span` name.
Replace the ambiguous "maybe need to make optional!!!" inline note with
a concrete TODO explaining the intended behavior — nest the dag run
span under the active parent by default, and only force a root span
for Airflow-initiated runs (API trigger, scheduler, backfill). Links
to the public tracking issue.
`_make_provider_with_detail_level` was added but never referenced by
any test in the class — drop it.
- Remove duplicate `except KeyboardInterrupt:` left over from the partial
  revert (the file no longer parsed)
- Re-add @detail_span("run") on run()
- Revert test_listener_suppresses_exceptions assertion to match the
  restored "error calling listener" log message
…n sites

The initial PR keeps only 12 of the originally-planned detail spans in
task_runner.py; the rest are deferred to follow-up work tracked at
https://linear.app/astronomer/issue/ACD-157. Trim expected_hierarchy to
the spans that are actually emitted today, and re-parent finalize under
worker.task1 (it's called after run() returns, so run's span is closed
by then).
When more than one dag run has emitted spans before the assertion (e.g.
default_spans runs before detail_spans in the same suite), picking
data["data"][-1] from Jaeger silently selected the wrong trace and made
the detail_spans assertion compare against the earlier default_spans
hierarchy. Filter the response by airflow.dag_run.run_id so the test
always inspects the trace it actually triggered.
@dstandish
dstandish force-pushed the add-detailed-task-spans branch from f9aa271 to 5b75384 Compare May 21, 2026 23:21
@dstandish
dstandish merged commit 2e9cbc2 into apache:main May 22, 2026
143 checks passed
@dstandish
dstandish deleted the add-detailed-task-spans branch May 22, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants