Repository navigation
Add detailed task spans - #63568
Merged
Merged
Add detailed task spans#63568
Conversation
dstandish
force-pushed
the
add-detailed-task-spans
branch
3 times, most recently
from
March 20, 2026 16:36
2cbc72a to
4b37205
Compare
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
ashb
reviewed
Mar 23, 2026
dstandish
marked this pull request as ready for review
March 23, 2026 15:23
dstandish
requested review from
XD-DENG,
amoghrajesh,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
hussein-awala,
jason810496,
kaxil,
o-nikolas,
pierrejeambrun,
potiuk,
rawwar and
shubhamraj-git
as code owners
March 23, 2026 15:23
dstandish
force-pushed
the
add-detailed-task-spans
branch
from
March 24, 2026 02:18
a6e6add to
e7cdd54
Compare
dstandish
force-pushed
the
add-detailed-task-spans
branch
from
April 1, 2026 10:15
a1445df to
935f819
Compare
jedcunningham
approved these changes
Apr 2, 2026
ashb
force-pushed
the
add-detailed-task-spans
branch
from
April 9, 2026 14:53
f472203 to
d83ac4f
Compare
dstandish
force-pushed
the
add-detailed-task-spans
branch
from
April 9, 2026 16:09
d83ac4f to
6cef7fc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds optional “detail spans” to OpenTelemetry traces to improve task execution debugging, gated by a DAG run conf setting (airflow/task_span_detail_level), and extends tests to validate propagation and span hierarchy.
Changes:
- Introduces
detail_spandecorator/context manager in the Task SDK task runner and wraps key execution steps with nested spans when detail level > 1. - Propagates
task_span_detail_levelvia W3Ctracestatein the DAG run’s trace carrier, and preserves it when clearing task instances. - Adds unit/integration tests to verify detail-level parsing/propagation and expected span structure.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| task-sdk/src/airflow/sdk/execution_time/task_runner.py | Adds detail_span and instruments task lifecycle with nested spans + exception/status recording |
| task-sdk/tests/task_sdk/execution_time/test_task_runner.py | Adds tests for detail_span behavior at different detail levels |
| shared/observability/src/airflow_shared/observability/traces/init.py | Encodes detail level into tracestate and adds API to read it back |
| shared/observability/tests/observability/test_traces.py | Adds tests for tracestate entries, carrier creation, and level extraction |
| airflow-core/src/airflow/models/dagrun.py | Seeds dagrun carrier with detail level; wraps _emit_dagrun_span call with error handling |
| airflow-core/src/airflow/models/taskinstance.py | Preserves detail level when regenerating dagrun carrier during clear_task_instances |
| airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py | Makes task instance id attribute string + guards span emission |
| airflow-core/tests/unit/models/test_dagrun.py | Tests that detail level in conf is embedded and readable |
| airflow-core/tests/unit/models/test_taskinstance.py | Tests that clearing task instances preserves detail level |
| airflow-core/tests/unit/listeners/test_listeners.py | Updates expected listener error wording |
| airflow-core/tests/integration/otel/test_otel.py | Parameterizes integration test to validate default vs detail span hierarchies |
| scripts/ci/docker-compose/integration-otel.yml | Adds deprecation note comment for env var |
| dev/* | Adds internal review/demo/trace analysis dev artifacts |
dstandish
force-pushed
the
add-detailed-task-spans
branch
from
May 19, 2026 14:28
d56232b to
9e0c0d9
Compare
Avoid relying on `TraceState.get(key, default=...)` (not supported in all OpenTelemetry versions) by fetching the value and applying the default in Python. Narrow the exception handler to `(TypeError, ValueError)` so unexpected errors aren't silently swallowed.
The outer `span` could be referenced before assignment if control flow
skipped the assignment path; initialize it to `INVALID_SPAN` up front.
Also drop the inner `as span` binding on the `detail_span("run")` block
to prevent it from shadowing the outer `span` name.
Replace the ambiguous "maybe need to make optional!!!" inline note with a concrete TODO explaining the intended behavior — nest the dag run span under the active parent by default, and only force a root span for Airflow-initiated runs (API trigger, scheduler, backfill). Links to the public tracking issue.
`_make_provider_with_detail_level` was added but never referenced by any test in the class — drop it.
- Remove duplicate `except KeyboardInterrupt:` left over from the partial
revert (the file no longer parsed)
- Re-add @detail_span("run") on run()
- Revert test_listener_suppresses_exceptions assertion to match the
restored "error calling listener" log message
…n sites The initial PR keeps only 12 of the originally-planned detail spans in task_runner.py; the rest are deferred to follow-up work tracked at https://linear.app/astronomer/issue/ACD-157. Trim expected_hierarchy to the spans that are actually emitted today, and re-parent finalize under worker.task1 (it's called after run() returns, so run's span is closed by then).
When more than one dag run has emitted spans before the assertion (e.g. default_spans runs before detail_spans in the same suite), picking data["data"][-1] from Jaeger silently selected the wrong trace and made the detail_spans assertion compare against the earlier default_spans hierarchy. Filter the response by airflow.dag_run.run_id so the test always inspects the trace it actually triggered.
dstandish
force-pushed
the
add-detailed-task-spans
branch
from
May 21, 2026 23:21
f9aa271 to
5b75384
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on top of #63839
Add spans for detailed debugging info during the task execution process
These are only emitted if you add
{"airflow/task_span_detail_level": 2}to dag run conf.