feat: [SVLS-9168] add aws.durable.operation_attempt tag to durable operations span#18191
Conversation
Add aws.durable.operation_retry_attempt to aws.durable.step and aws.durable.wait_for_condition spans, sourced from the SDK's StepDetails.attempt checkpoint field. 0 = original attempt, N = Nth retry. Set as a numeric metric so it supports range queries. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Codeowners resolved as |
|
BenchmarksBenchmark execution time: 2026-06-03 22:58:13 Comparing candidate commit 4a69c14 in PR branch Found 0 performance improvements and 2 performance regressions! Performance is the same for 382 metrics, 9 unstable metrics. scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
|
…try_attempt Match the style of the other span-attribute setters in the same subscriber. The wire result is identical — _set_attribute dispatches by value type, so an int still lands in the metrics dict as a float. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…-indexed in prod The AWS Lambda Durable service reports `step_details.attempt` 1-indexed (1 for the first attempt, 2 after the first retry), not 0-indexed like the SDK's own documented semantic. Subtract 1 (clamped to 0) so the tag emits the user-facing retry count: 0 for the original attempt, 1 for the first retry, etc. Updates the test_step_with_retry snapshot: the retry-success span's value drops from 1 to 0 because the SDK testing framework already returns step_details.attempt as the retry count directly. The test framework and prod disagree on the semantics; the AIDEV-NOTE in patch.py captures the discrepancy. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ttempt Rename `aws.durable.operation_retry_attempt` to `aws.durable.operation_attempt`. Semantics remain 0-indexed: 0 = original attempt, 1 = first retry, etc. - TAG_OPERATION_RETRY_ATTEMPT renamed to TAG_OPERATION_ATTEMPT - AwsDurableOperationEvent field renamed accordingly - Snapshot keys updated; values unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…indexed Match the AWS UI's attempt-count convention: 1 = original attempt, 2 = first retry, etc. Pass step_details.attempt through directly (it's already 1-indexed in production); default to 1 when no checkpoint exists yet, and clamp with max(1, …) to handle the SDK testing framework's 0-indexed values. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c363d36cfd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…t tag Switch aws.durable.operation_attempt to 0-indexed semantics: 0 = original attempt, 1 = first retry, 2 = second retry, etc. Production logging confirmed the server stores step_details.attempt as the count of prior checkpointed attempts (0 when no checkpoint exists, 1 after the first failure, etc.). Passing it through directly avoids the collision where both the first and second attempt emitted 1 under the previous max(1, attempt) logic. Both prod and the test framework use the same underlying convention, so test and prod are now consistent. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… AIDEV-NOTE Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…t AIDEV-NOTE Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
@wantsui Could you review this PR on behalf of |
wantsui
left a comment
There was a problem hiding this comment.
LGTM since it looks like you're doing what already has been approved for JS at DataDog/dd-trace-js#8595
|
Merge failed again. Let me remove Pablo from reviewers and retry. |
374acd7
into
main
…nts (#18384) ## What does this PR do? Extracts all `"aws.durable.*"` span name string literals from `patch.py` into named constants in `ddtrace/ext/aws_durable.py`, then replaces every bare literal with the corresponding constant. **New constants added to `ddtrace/ext/aws_durable.py`:** - `SPAN_STEP` - `SPAN_WAIT` - `SPAN_WAIT_FOR_CONDITION` - `SPAN_WAIT_FOR_CALLBACK` - `SPAN_CREATE_CALLBACK` - `SPAN_MAP` - `SPAN_PARALLEL` - `SPAN_CHILD_CONTEXT` ## Motivation Addresses [reviewer feedback](#18191 (comment)) on #18191: span name strings should be constants rather than bare literals so they are defined in one place and typos are caught at import time. ## Testing No behavior change — span names are identical. Existing integration tests cover the affected code paths. Co-authored-by: yiming.luo <[email protected]>
…erations span (#18191) ## Description Add `aws.durable.operation_attempt` to `aws.durable.step` and `aws.durable.wait_for_condition` spans. The value is 0-indexed: 0 = original attempt, 1 = first retry, 2 = second retry, etc. Sourced from the Durable Execution SDK's `StepDetails.attempt` field in the operation checkpoint, passed through directly. When no checkpoint exists yet (first execution, no prior failures), the tag defaults to `0`. This tag will be used by UI to display attempt count and group attempts for the same operation. ## Testing Installed the tracer on a durable function and invoked it. The `aws.durable.step` span for `order_verify_inventory` operation failed twice the succeeded. For the original attempt, the tag has a value of 0. <img width="653" height="209" alt="image" src="https://github.com/user-attachments/assets/e58c4904-1d9c-4665-9ef5-0e4a56712144" /> 1st retry (2nd attempt): `aws.durable.operation_attempt` is 1. <img width="639" height="207" alt="image" src="https://github.com/user-attachments/assets/9bdf61f0-d842-4370-ab4b-4fe0238fc34e" /> 2nd retry (3rd attempt): `aws.durable.operation_attempt` is 2. <img width="649" height="194" alt="image" src="https://github.com/user-attachments/assets/c6e8f398-8a82-4513-a3e5-016379017888" /> [Link for you to check them yourself](https://dd.datad0g.com/apm/traces?query=service%3Ayiming-durable-py-custom-tracer&agg_m=count&agg_m_source=base&agg_t=count&cols=service%2Cresource_name%2C%40duration%2C%40http.method%2C%40http.status_code%2C%40_span.count%2C%40_duration.by_service&fromUser=false&graphType=flamegraph&historicalData=true&messageDisplay=inline&query_translation_version=v0&refresh_mode=paused&shouldShowLegend=true&sort=desc&spanType=all&storage=hot&traceQuery=&view=spans&start=1779812280000&end=1779812340000&paused=true) ### Why only `step` and `wait_for_condition` spans? The SDK has six `OperationExecutor` subclasses; only two use the `StepDetails.attempt` retry mechanism: | Executor | Detail field on checkpoint | Has retry? | |---|---|---| | `StepOperationExecutor` | `step_details` | yes | | `WaitForConditionOperationExecutor` | `step_details` (polling iterations) | yes | | `CallbackOperationExecutor` | `callback_details` | no | | `InvokeOperationExecutor` | `chained_invoke_details` | no | | `ChildOperationExecutor` | — | no | | `WaitOperationExecutor` | `wait_details` | no | `wait_for_callback` isn't its own executor — it's a helper that internally calls `create_callback` + `step`, so any retries appear on the inner `aws.durable.step` child span (already covered). `map` and `parallel` also have no executor; their work is decomposed into MAP_ITERATION / PARALLEL_BRANCH child operations, which `_is_top_level_for_span` filters out anyway. Co-authored-by: yiming.luo <[email protected]>
Description
Add
aws.durable.operation_attempttoaws.durable.stepandaws.durable.wait_for_conditionspans. The value is 0-indexed: 0 = original attempt, 1 = first retry, 2 = second retry, etc.Sourced from the Durable Execution SDK's
StepDetails.attemptfield in the operation checkpoint, passed through directly. When no checkpoint exists yet (first execution, no prior failures), the tag defaults to0.This tag will be used by UI to display attempt count and group attempts for the same operation.
Testing
Installed the tracer on a durable function and invoked it.
The
aws.durable.stepspan fororder_verify_inventoryoperation failed twice the succeeded.For the original attempt, the tag has a value of 0.

1st retry (2nd attempt):

aws.durable.operation_attemptis 1.2nd retry (3rd attempt):

aws.durable.operation_attemptis 2.Link for you to check them yourself
Why only
stepandwait_for_conditionspans?The SDK has six
OperationExecutorsubclasses; only two use theStepDetails.attemptretry mechanism:StepOperationExecutorstep_detailsWaitForConditionOperationExecutorstep_details(polling iterations)CallbackOperationExecutorcallback_detailsInvokeOperationExecutorchained_invoke_detailsChildOperationExecutorWaitOperationExecutorwait_detailswait_for_callbackisn't its own executor — it's a helper that internally callscreate_callback+step, so any retries appear on the inneraws.durable.stepchild span (already covered).mapandparallelalso have no executor; their work is decomposed into MAP_ITERATION / PARALLEL_BRANCH child operations, which_is_top_level_for_spanfilters out anyway.