Skip to content

feat: [SVLS-9168] add aws.durable.operation_attempt tag to durable operations span#18191

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 10 commits into
mainfrom
yiming.luo/durable-retry-attempt
Jun 4, 2026
Merged

feat: [SVLS-9168] add aws.durable.operation_attempt tag to durable operations span#18191
gh-worker-dd-mergequeue-cf854d[bot] merged 10 commits into
mainfrom
yiming.luo/durable-retry-attempt

Conversation

@lym953

@lym953 lym953 commented May 20, 2026

Copy link
Copy Markdown
Contributor

Description

Add aws.durable.operation_attempt to aws.durable.step and aws.durable.wait_for_condition spans. The value is 0-indexed: 0 = original attempt, 1 = first retry, 2 = second retry, etc.

Sourced from the Durable Execution SDK's StepDetails.attempt field in the operation checkpoint, passed through directly. When no checkpoint exists yet (first execution, no prior failures), the tag defaults to 0.

This tag will be used by UI to display attempt count and group attempts for the same operation.

Testing

Installed the tracer on a durable function and invoked it.

The aws.durable.step span for order_verify_inventory operation failed twice the succeeded.

For the original attempt, the tag has a value of 0.
image

1st retry (2nd attempt): aws.durable.operation_attempt is 1.
image

2nd retry (3rd attempt): aws.durable.operation_attempt is 2.
image

Link for you to check them yourself

Why only step and wait_for_condition spans?

The SDK has six OperationExecutor subclasses; only two use the StepDetails.attempt retry mechanism:

Executor Detail field on checkpoint Has retry?
StepOperationExecutor step_details yes
WaitForConditionOperationExecutor step_details (polling iterations) yes
CallbackOperationExecutor callback_details no
InvokeOperationExecutor chained_invoke_details no
ChildOperationExecutor no
WaitOperationExecutor wait_details no

wait_for_callback isn't its own executor — it's a helper that internally calls create_callback + step, so any retries appear on the inner aws.durable.step child span (already covered). map and parallel also have no executor; their work is decomposed into MAP_ITERATION / PARALLEL_BRANCH child operations, which _is_top_level_for_span filters out anyway.

Add aws.durable.operation_retry_attempt to aws.durable.step and
aws.durable.wait_for_condition spans, sourced from the SDK's
StepDetails.attempt checkpoint field. 0 = original attempt, N = Nth
retry. Set as a numeric metric so it supports range queries.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented May 20, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/_trace/subscribers/aws_durable.py                               @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/_events/aws_durable.py                                  @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/aws_durable_execution_sdk_python/patch.py      @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/ext/aws_durable.py                                              @DataDog/apm-core-python @DataDog/apm-idm-python
releasenotes/notes/aws-durable-operation-retry-attempt-tag-553311e75641ec9d.yaml  @DataDog/apm-python
tests/snapshots/tests.contrib.aws_durable_execution_sdk_python.test_aws_durable_execution_sdk_python.test_parallel_propagates_trace_context.json  @DataDog/apm-python
tests/snapshots/tests.contrib.aws_durable_execution_sdk_python.test_aws_durable_execution_sdk_python.test_step_with_retry.json  @DataDog/apm-python
tests/snapshots/tests.contrib.aws_durable_execution_sdk_python.test_aws_durable_execution_sdk_python.test_workflow_failed_status.json  @DataDog/apm-python

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]   View in Datadog   GitLab

See error Failed to generate artifact: missing required input parameters for the wheel generation.

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]   View in Datadog   GitLab

See error ImportError: NotImplementedError: This version of CPython is not supported yet during the wheel testing process.

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]   View in Datadog   GitLab

See error NotImplementedError: This version of CPython is not supported yet during ddtrace import.

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4a69c14 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented May 20, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-03 22:58:13

Comparing candidate commit 4a69c14 in PR branch yiming.luo/durable-retry-attempt with baseline commit ba59c36 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 382 metrics, 9 unstable metrics.

scenario:span-start

  • 🟥 execution_time [+1.337ms; +1.500ms] or [+8.559%; +9.606%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+174.554ns; +211.341ns] or [+8.218%; +9.950%]

lym953 and others added 4 commits May 20, 2026 14:36
…try_attempt

Match the style of the other span-attribute setters in the same
subscriber. The wire result is identical — _set_attribute dispatches by
value type, so an int still lands in the metrics dict as a float.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…-indexed in prod

The AWS Lambda Durable service reports `step_details.attempt` 1-indexed
(1 for the first attempt, 2 after the first retry), not 0-indexed like
the SDK's own documented semantic. Subtract 1 (clamped to 0) so the tag
emits the user-facing retry count: 0 for the original attempt, 1 for the
first retry, etc.

Updates the test_step_with_retry snapshot: the retry-success span's
value drops from 1 to 0 because the SDK testing framework already
returns step_details.attempt as the retry count directly. The test
framework and prod disagree on the semantics; the AIDEV-NOTE in
patch.py captures the discrepancy.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ttempt

Rename `aws.durable.operation_retry_attempt` to `aws.durable.operation_attempt`.
Semantics remain 0-indexed: 0 = original attempt, 1 = first retry, etc.

- TAG_OPERATION_RETRY_ATTEMPT renamed to TAG_OPERATION_ATTEMPT
- AwsDurableOperationEvent field renamed accordingly
- Snapshot keys updated; values unchanged

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…indexed

Match the AWS UI's attempt-count convention: 1 = original attempt,
2 = first retry, etc. Pass step_details.attempt through directly (it's
already 1-indexed in production); default to 1 when no checkpoint
exists yet, and clamp with max(1, …) to handle the SDK testing
framework's 0-indexed values.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lym953 lym953 changed the title feat(aws_durable_execution_sdk_python): add operation_retry_attempt tag [SVLS-9168] feat: add operation_attempt tag to durable operations span May 20, 2026
@lym953 lym953 changed the title [SVLS-9168] feat: add operation_attempt tag to durable operations span [SVLS-9168] feat: add aws.durable.operation_attempt tag to durable operations span May 20, 2026
@lym953
lym953 marked this pull request as ready for review May 20, 2026 21:42
@lym953
lym953 requested review from a team as code owners May 20, 2026 21:42

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c363d36cfd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/contrib/internal/aws_durable_execution_sdk_python/patch.py Outdated
@lym953 lym953 changed the title [SVLS-9168] feat: add aws.durable.operation_attempt tag to durable operations span feat: [SVLS-9168] add aws.durable.operation_attempt tag to durable operations span May 21, 2026
@lym953
lym953 marked this pull request as draft May 26, 2026 15:24
…t tag

Switch aws.durable.operation_attempt to 0-indexed semantics: 0 = original
attempt, 1 = first retry, 2 = second retry, etc.

Production logging confirmed the server stores step_details.attempt as the
count of prior checkpointed attempts (0 when no checkpoint exists, 1 after
the first failure, etc.). Passing it through directly avoids the collision
where both the first and second attempt emitted 1 under the previous
max(1, attempt) logic. Both prod and the test framework use the same
underlying convention, so test and prod are now consistent.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@lym953
lym953 marked this pull request as ready for review May 26, 2026 17:13
Comment thread ddtrace/contrib/internal/aws_durable_execution_sdk_python/patch.py
Comment thread ddtrace/contrib/internal/aws_durable_execution_sdk_python/patch.py Outdated
@lym953

lym953 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@wantsui Could you review this PR on behalf of apm-idm-python?

@wantsui wantsui left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM since it looks like you're doing what already has been approved for JS at DataDog/dd-trace-js#8595

@lym953

lym953 commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Merge failed again. Let me remove Pablo from reviewers and retry.

@lym953
lym953 removed the request for review from pablomartinezbernardo June 3, 2026 22:28
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 374acd7 into main Jun 4, 2026
1290 of 1292 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the yiming.luo/durable-retry-attempt branch June 4, 2026 14:33
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 5, 2026
…nts (#18384)

## What does this PR do?

Extracts all `"aws.durable.*"` span name string literals from `patch.py` into named constants in `ddtrace/ext/aws_durable.py`, then replaces every bare literal with the corresponding constant.

**New constants added to `ddtrace/ext/aws_durable.py`:**
- `SPAN_STEP`
- `SPAN_WAIT`
- `SPAN_WAIT_FOR_CONDITION`
- `SPAN_WAIT_FOR_CALLBACK`
- `SPAN_CREATE_CALLBACK`
- `SPAN_MAP`
- `SPAN_PARALLEL`
- `SPAN_CHILD_CONTEXT`

## Motivation

Addresses [reviewer feedback](#18191 (comment)) on #18191: span name strings should be constants rather than bare literals so they are defined in one place and typos are caught at import time.

## Testing

No behavior change — span names are identical. Existing integration tests cover the affected code paths.

Co-authored-by: yiming.luo <[email protected]>
brettlangdon pushed a commit that referenced this pull request Jun 10, 2026
…erations span (#18191)

## Description

Add `aws.durable.operation_attempt` to `aws.durable.step` and `aws.durable.wait_for_condition` spans. The value is 0-indexed: 0 = original attempt, 1 = first retry, 2 = second retry, etc.

Sourced from the Durable Execution SDK's `StepDetails.attempt` field in the operation checkpoint, passed through directly. When no checkpoint exists yet (first execution, no prior failures), the tag defaults to `0`.

This tag will be used by UI to display attempt count and group attempts for the same operation.

## Testing
Installed the tracer on a durable function and invoked it.

The `aws.durable.step` span for `order_verify_inventory` operation failed twice the succeeded.

For the original attempt, the tag has a value of 0.
<img width="653" height="209" alt="image" src="https://github.com/user-attachments/assets/e58c4904-1d9c-4665-9ef5-0e4a56712144" />

1st retry (2nd attempt): `aws.durable.operation_attempt` is 1.
<img width="639" height="207" alt="image" src="https://github.com/user-attachments/assets/9bdf61f0-d842-4370-ab4b-4fe0238fc34e" />

2nd retry (3rd attempt): `aws.durable.operation_attempt` is 2.
<img width="649" height="194" alt="image" src="https://github.com/user-attachments/assets/c6e8f398-8a82-4513-a3e5-016379017888" />

[Link for you to check them yourself](https://dd.datad0g.com/apm/traces?query=service%3Ayiming-durable-py-custom-tracer&agg_m=count&agg_m_source=base&agg_t=count&cols=service%2Cresource_name%2C%40duration%2C%40http.method%2C%40http.status_code%2C%40_span.count%2C%40_duration.by_service&fromUser=false&graphType=flamegraph&historicalData=true&messageDisplay=inline&query_translation_version=v0&refresh_mode=paused&shouldShowLegend=true&sort=desc&spanType=all&storage=hot&traceQuery=&view=spans&start=1779812280000&end=1779812340000&paused=true)

### Why only `step` and `wait_for_condition` spans?

The SDK has six `OperationExecutor` subclasses; only two use the `StepDetails.attempt` retry mechanism:

| Executor | Detail field on checkpoint | Has retry? |
|---|---|---|
| `StepOperationExecutor` | `step_details` | yes |
| `WaitForConditionOperationExecutor` | `step_details` (polling iterations) | yes |
| `CallbackOperationExecutor` | `callback_details` | no |
| `InvokeOperationExecutor` | `chained_invoke_details` | no |
| `ChildOperationExecutor` | — | no |
| `WaitOperationExecutor` | `wait_details` | no |

`wait_for_callback` isn't its own executor — it's a helper that internally calls `create_callback` + `step`, so any retries appear on the inner `aws.durable.step` child span (already covered). `map` and `parallel` also have no executor; their work is decomposed into MAP_ITERATION / PARALLEL_BRANCH child operations, which `_is_top_level_for_span` filters out anyway.

Co-authored-by: yiming.luo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants