Skip to content

fix(aws-durable-execution-sdk-js): [SVLS-9168] handle FAILED checkpoints properly#9160

Merged
lym953 merged 1 commit into
masterfrom
yiming.luo/durable-failed-replay-attempt
Jul 1, 2026
Merged

fix(aws-durable-execution-sdk-js): [SVLS-9168] handle FAILED checkpoints properly#9160
lym953 merged 1 commit into
masterfrom
yiming.luo/durable-failed-replay-attempt

Conversation

@lym953

@lym953 lym953 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Treats a FAILED durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans:

  • aws.durable.replayed — now true when the operation is served from a terminal checkpoint (FAILED or SUCCEEDED)
  • aws.durable.operation_attempt — the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints.

Testing

Context

The always-fails operation runs/replays three times:

  • original attempt: fails
  • 2nd and final attempt (1st retry): fails
  • replay of the 2nd attempt

Result

Before

The last span for the always-fails operation has replay == false and operation_attempt span tag of 2, both of which are wrong.
image
image

After

The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off)
image

Related PR

The same fix for Python: #9160

A durable operation served from a FAILED checkpoint on replay (the SDK
re-raises the stored error without running the body) was not recognized as a
replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt`
normalization keyed only on `Status === 'SUCCEEDED'`.

As a result a caught-then-replayed failed operation reported
`replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose
2nd attempt failed) — one higher than, and inconsistent with, the live final
attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open
for FAILED.

Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so
the replay agrees with the live run (replayed=true, operation_attempt=1).
Verified in production on a durable Lambda.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@dd-octo-sts

dd-octo-sts Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.43 MB
Deduped: 7.49 MB
No deduping: 7.49 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 30, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 88.05% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ddeb831 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 30, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-30 19:55:25

Comparing candidate commit ddeb831 in PR branch yiming.luo/durable-failed-replay-attempt with baseline commit f51410e in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2253 metrics, 33 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-20

  • unstable execution_time [-187.882ms; +184.771ms] or [-5.042%; +4.959%]

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-185.637ms; +183.495ms] or [-7.076%; +6.994%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-204.195ms; +197.365ms] or [-8.100%; +7.829%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-152.396ms; +146.392ms] or [-5.294%; +5.085%]

scenario:appsec-control-20

  • unstable execution_time [-163.401ms; +158.303ms] or [-9.292%; +9.002%]

scenario:appsec-control-24

  • unstable execution_time [-100994.472µs; +101865.606µs] or [-8.310%; +8.382%]

scenario:appsec-control-26

  • unstable execution_time [-111.156ms; +108.833ms] or [-9.205%; +9.013%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-1880.707ms; +586.131ms] or [-20.243%; +6.309%]
  • unstable execution_time [-1885.865ms; +574.713ms] or [-18.844%; +5.743%]
  • unstable instructions [-16.7G instructions; +5.4G instructions] or [-21.627%; +6.947%]
  • unstable throughput [-124.962op/s; +424.802op/s] or [-3.803%; +12.929%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-1898.385ms; +3029.184ms] or [-23.870%; +38.089%]
  • unstable execution_time [-1912.669ms; +3065.774ms] or [-22.183%; +35.557%]
  • unstable instructions [-16.2G instructions; +25.8G instructions] or [-25.189%; +40.095%]
  • unstable max_rss_usage [-6.325MB; +10.852MB] or [-3.999%; +6.861%]
  • unstable throughput [-897.540op/s; +556.773op/s] or [-23.442%; +14.542%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-1630.311ms; +550.451ms] or [-20.663%; +6.976%]
  • unstable execution_time [-1678.002ms; +577.613ms] or [-19.538%; +6.725%]
  • unstable instructions [-14.2G instructions; +4.6G instructions] or [-22.034%; +7.140%]
  • unstable throughput [-176.794op/s; +491.740op/s] or [-4.597%; +12.786%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2208.442ms; +768.288ms] or [-23.261%; +8.092%]
  • unstable execution_time [-2234.213ms; +771.365ms] or [-21.922%; +7.569%]
  • unstable instructions [-19.8G instructions; +7.0G instructions] or [-25.091%; +8.834%]
  • unstable throughput [-163.999op/s; +442.947op/s] or [-5.043%; +13.622%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-410.888ms; +258.753ms] or [-8.415%; +5.299%]
  • unstable execution_time [-409.812ms; +253.559ms] or [-8.272%; +5.118%]
  • unstable throughput [-89101.928op/s; +138127.649op/s] or [-5.257%; +8.150%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-23.552MB; +26.744MB] or [-6.223%; +7.066%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-7747.566KB; +6448.709KB] or [-5.765%; +4.798%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-556.241ms; +629.982ms] or [-5.434%; +6.154%]
  • unstable execution_time [-555.579ms; +648.419ms] or [-5.283%; +6.166%]
  • unstable throughput [-4.433op/s; +3.932op/s] or [-5.793%; +5.139%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-5157.835KB; +3443.501KB] or [-6.422%; +4.287%]

@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.73%. Comparing base (f51410e) to head (ddeb831).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #9160   +/-   ##
=======================================
  Coverage   93.73%   93.73%           
=======================================
  Files         892      892           
  Lines       51250    51251    +1     
  Branches    11928    11928           
=======================================
+ Hits        48039    48040    +1     
  Misses       3211     3211           
Flag Coverage Δ
aiguard 34.87% <ø> (-0.08%) ⬇️
aiguard-integration 41.79% <ø> (ø)
apm-bucket-0 34.84% <ø> (-0.08%) ⬇️
apm-bucket-1 40.33% <ø> (-0.07%) ⬇️
apm-bucket-2 37.34% <ø> (-0.07%) ⬇️
apm-capabilities-tracing 48.67% <33.33%> (+<0.01%) ⬆️
apm-integrations-aerospike 33.12% <ø> (-0.08%) ⬇️
apm-integrations-confluentinc-kafka-javascript 40.02% <ø> (-0.08%) ⬇️
apm-integrations-couchbase 33.40% <ø> (-0.08%) ⬇️
apm-integrations-http 41.95% <ø> (-0.07%) ⬇️
apm-integrations-kafkajs 40.26% <ø> (-0.08%) ⬇️
apm-integrations-next 29.50% <ø> (-0.04%) ⬇️
apm-integrations-prisma 35.03% <ø> (-0.07%) ⬇️
apm-integrations-tedious 33.89% <ø> (-0.08%) ⬇️
appsec 57.62% <ø> (-0.15%) ⬇️
appsec-express_fastify_graphql 53.67% <ø> (-0.06%) ⬇️
appsec-integration 36.22% <ø> (ø)
appsec-kafka_ldapjs_lodash 43.58% <ø> (-0.07%) ⬇️
appsec-mongodb-core_mongoose_mysql 48.74% <ø> (-0.06%) ⬇️
appsec-next 28.00% <ø> (-0.07%) ⬇️
appsec-node-serialize_passport_postgres 47.91% <ø> (-0.06%) ⬇️
appsec-sourcing_stripe_template 45.47% <ø> (-0.06%) ⬇️
debugger 44.42% <ø> (+0.04%) ⬆️
instrumentations-bucket-0 28.15% <ø> (-0.08%) ⬇️
instrumentations-bucket-1 37.40% <ø> (-0.07%) ⬇️
instrumentations-bucket-10 40.35% <ø> (-0.07%) ⬇️
instrumentations-bucket-11 27.95% <ø> (-0.08%) ⬇️
instrumentations-bucket-12 28.67% <ø> (-0.08%) ⬇️
instrumentations-bucket-13 27.78% <ø> (-0.08%) ⬇️
instrumentations-bucket-2 30.23% <ø> (-0.07%) ⬇️
instrumentations-bucket-3 35.90% <ø> (-0.07%) ⬇️
instrumentations-bucket-4 28.56% <ø> (-0.08%) ⬇️
instrumentations-bucket-5 36.28% <ø> (-0.07%) ⬇️
instrumentations-bucket-6 38.25% <ø> (-0.08%) ⬇️
instrumentations-bucket-7 36.01% <ø> (-0.07%) ⬇️
instrumentations-bucket-8 36.95% <ø> (-0.07%) ⬇️
instrumentations-bucket-9 39.45% <ø> (-0.07%) ⬇️
instrumentations-instrumentation-couchbase 46.48% <ø> (ø)
instrumentations-integration-esbuild 24.91% <66.66%> (+<0.01%) ⬆️
llmobs-ai_anthropic_bedrock 39.53% <ø> (-0.07%) ⬇️
llmobs-google-genai_langchain_vertex-ai 36.97% <ø> (-0.06%) ⬇️
llmobs-openai 39.58% <ø> (-0.07%) ⬇️
llmobs-sdk 43.59% <ø> (-0.08%) ⬇️
master-coverage 93.73% <100.00%> (?)
openfeature 37.76% <ø> (ø)
openfeature-unit 50.39% <ø> (ø)
platform-core_esbuild_instrumentations-misc 23.36% <ø> (ø)
platform-integration 47.44% <ø> (+0.01%) ⬆️
platform-shimmer_unit-guardrails_webpack 18.91% <ø> (ø)
plugins-bucket-0 36.38% <100.00%> (-0.06%) ⬇️
plugins-bucket-1 39.60% <ø> (ø)
plugins-bucket-11 38.46% <ø> (-0.08%) ⬇️
plugins-bucket-17 39.15% <ø> (+0.08%) ⬆️
plugins-bucket-18 42.01% <ø> (-0.07%) ⬇️
plugins-bucket-19 39.56% <ø> (-0.07%) ⬇️
plugins-bucket-20 43.25% <ø> (-0.07%) ⬇️
plugins-bucket-4 37.71% <ø> (-0.07%) ⬇️
plugins-bullmq_cassandra_cookie 39.76% <ø> (-0.07%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 33.22% <ø> (-0.08%) ⬇️
plugins-fetch_fs_generic-pool 36.05% <ø> (-0.07%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 43.10% <ø> (-0.07%) ⬇️
plugins-hapi_hono_ioredis 37.78% <ø> (-0.07%) ⬇️
plugins-jest_knex_langgraph 32.60% <ø> (-0.08%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 27.91% <ø> (-0.08%) ⬇️
plugins-lodash_mariadb_memcached 35.17% <ø> (-0.08%) ⬇️
plugins-mongodb_mongodb-core_mongoose 36.32% <ø> (-0.06%) ⬇️
plugins-multer_mysql_mysql2 35.14% <ø> (-0.08%) ⬇️
plugins-nats_node-serialize_opensearch 37.14% <ø> (-0.08%) ⬇️
plugins-passport-http_pino_postgres 35.53% <ø> (-0.08%) ⬇️
plugins-process_pug_redis 34.26% <ø> (-0.08%) ⬇️
plugins-undici_url_valkey 35.88% <ø> (-0.08%) ⬇️
plugins-vm_winston_ws 37.55% <ø> (-0.08%) ⬇️
profiling 43.62% <ø> (-0.07%) ⬇️
serverless-aws-sdk-aws-sdk 33.29% <ø> (-0.07%) ⬇️
serverless-aws-sdk-bedrockruntime 32.16% <ø> (-0.07%) ⬇️
serverless-aws-sdk-client 37.16% <ø> (ø)
serverless-aws-sdk-dynamodb 34.09% <ø> (-0.07%) ⬇️
serverless-aws-sdk-eventbridge 27.25% <ø> (-0.07%) ⬇️
serverless-aws-sdk-kinesis 37.40% <ø> (-0.07%) ⬇️
serverless-aws-sdk-lambda 34.59% <ø> (-0.07%) ⬇️
serverless-aws-sdk-s3 32.59% <ø> (-0.07%) ⬇️
serverless-aws-sdk-serverless-peer-service 39.56% <ø> (+0.01%) ⬆️
serverless-aws-sdk-sns 38.25% <ø> (-0.07%) ⬇️
serverless-aws-sdk-sqs 38.00% <ø> (-0.07%) ⬇️
serverless-aws-sdk-stepfunctions 33.18% <ø> (-0.07%) ⬇️
serverless-aws-sdk-util 47.95% <ø> (ø)
serverless-bucket-0 39.46% <ø> (ø)
serverless-lambda 34.33% <ø> (-0.09%) ⬇️
test-optimization-cucumber 52.30% <ø> (+0.11%) ⬆️
test-optimization-cypress 49.41% <ø> (+0.06%) ⬆️
test-optimization-jest 55.37% <ø> (+0.07%) ⬆️
test-optimization-mocha 53.38% <ø> (+0.08%) ⬆️
test-optimization-playwright-playwright-atr 43.32% <ø> (-0.15%) ⬇️
test-optimization-playwright-playwright-efd 43.61% <ø> (+0.08%) ⬆️
test-optimization-playwright-playwright-final-status 43.73% <ø> (+0.08%) ⬆️
test-optimization-playwright-playwright-impacted-tests 43.15% <ø> (ø)
test-optimization-playwright-playwright-reporting 43.26% <ø> (+0.08%) ⬆️
test-optimization-playwright-playwright-test-management 44.71% <ø> (+0.08%) ⬆️
test-optimization-playwright-playwright-test-span 44.51% <ø> (+0.07%) ⬆️
test-optimization-selenium 45.28% <ø> (+0.06%) ⬆️
test-optimization-testopt 48.04% <ø> (+0.08%) ⬆️
test-optimization-vitest 50.62% <ø> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lym953 lym953 changed the title fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays fix(aws-durable-execution-sdk-js): [SVLS-9168] handle FAILED checkpoints properly Jun 30, 2026
@lym953
lym953 requested review from a team and pablomartinezbernardo June 30, 2026 20:39
@lym953
lym953 marked this pull request as ready for review June 30, 2026 20:40
@lym953
lym953 requested a review from a team as a code owner June 30, 2026 20:40
@lym953
lym953 requested review from bojbrook and removed request for a team June 30, 2026 20:40
@lym953
lym953 merged commit e035b64 into master Jul 1, 2026
1081 of 1083 checks passed
@lym953
lym953 deleted the yiming.luo/durable-failed-replay-attempt branch July 1, 2026 15:11
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 2, 2026
#9160)

A durable operation served from a FAILED checkpoint on replay (the SDK
re-raises the stored error without running the body) was not recognized as a
replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt`
normalization keyed only on `Status === 'SUCCEEDED'`.

As a result a caught-then-replayed failed operation reported
`replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose
2nd attempt failed) — one higher than, and inconsistent with, the live final
attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open
for FAILED.

Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so
the replay agrees with the live run (replayed=true, operation_attempt=1).
Verified in production on a durable Lambda.

Co-authored-by: Claude Opus 4.8 <[email protected]>
This was referenced Jul 2, 2026
rochdev pushed a commit that referenced this pull request Jul 2, 2026
#9160)

A durable operation served from a FAILED checkpoint on replay (the SDK
re-raises the stored error without running the body) was not recognized as a
replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt`
normalization keyed only on `Status === 'SUCCEEDED'`.

As a result a caught-then-replayed failed operation reported
`replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose
2nd attempt failed) — one higher than, and inconsistent with, the live final
attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open
for FAILED.

Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so
the replay agrees with the live run (replayed=true, operation_attempt=1).
Verified in production on a durable Lambda.

Co-authored-by: Claude Opus 4.8 <[email protected]>
rochdev pushed a commit that referenced this pull request Jul 2, 2026
#9160)

A durable operation served from a FAILED checkpoint on replay (the SDK
re-raises the stored error without running the body) was not recognized as a
replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt`
normalization keyed only on `Status === 'SUCCEEDED'`.

As a result a caught-then-replayed failed operation reported
`replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose
2nd attempt failed) — one higher than, and inconsistent with, the live final
attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open
for FAILED.

Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so
the replay agrees with the live run (replayed=true, operation_attempt=1).
Verified in production on a durable Lambda.

Co-authored-by: Claude Opus 4.8 <[email protected]>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/dd-trace-py that referenced this pull request Jul 10, 2026
…kpoints properly (#18824)

## What does this PR do?

Treats a **FAILED** durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans:

- `aws.durable.replayed` — now `true` when the operation is served from a terminal checkpoint (FAILED or SUCCEEDED)
- `aws.durable.operation_attempt` — the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints.

## Testing
### Context
The `always-fails` operation runs/replays three times:
- original attempt: fails
- 2nd and final attempt (1st retry): fails
- replay of the 2nd attempt

### Result
#### Before
The last span for the `always-fails` operation has `replay == false` and `operation_attempt` span tag of `2`. Operations table shows the operation has 3 attempts. All are wrong.
<img width="851" height="389" alt="image" src="https://github.com/user-attachments/assets/bca1691a-70dc-43a8-a80d-e63876f20bbb" />
<img width="304" height="223" alt="image" src="https://github.com/user-attachments/assets/507f594e-2c0d-44df-9fd2-01b0b3e8ac0a" />

#### After

The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off)

<img width="844" height="517" alt="image" src="https://github.com/user-attachments/assets/ede1bef8-63c4-44ae-9f45-4ec3eee9631b" />

It has span tag `aws.durable.replayed == true` and `aws.durable.operation_attempt == 1`, as expected.

Operations table shows the operation has 2 attempts, as expected.

## Staging link
[Link](https://dd.datad0g.com/serverless/aws/lambda?search=functionname%3Add-durable-attempt-py&fromUser=false&graphType=flamegraph&panel_end=1782850230200&panel_paused=false&panel_start=1782846630200&panel_tab=durableExecutions&refresh_mode=sliding&shouldShowLegend=true&sp=%5B%7B%22p%22%3A%7B%22entityId%22%3A%22aws-lambda-functions%2Bdd-durable-attempt-py%2Bus-east-2%2B425362996713%22%7D%2C%22i%22%3A%22lambda-panel%22%7D%2C%7B%22p%22%3A%7B%22traceID%22%3A%226a44216000000000ac65d4f099b859bf%22%2C%22selectedSpanID%22%3A%226869736249323812685%22%7D%2C%22i%22%3A%22trace-panel%22%7D%5D&spanID=5613198925804955800&spanViewType=infrastructure&traceID=6a44216000000000ac65d4f099b859bf&traceQuery=&start=1782677253459&end=1782850053459&paused=false)

## Next steps
Do the same fix for JS. DataDog/dd-trace-js#9160

Co-authored-by: yiming.luo <[email protected]>
vlad-scherbich pushed a commit to DataDog/dd-trace-py that referenced this pull request Jul 15, 2026
…kpoints properly (#18824)

## What does this PR do?

Treats a **FAILED** durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans:

- `aws.durable.replayed` — now `true` when the operation is served from a terminal checkpoint (FAILED or SUCCEEDED)
- `aws.durable.operation_attempt` — the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints.

## Testing
### Context
The `always-fails` operation runs/replays three times:
- original attempt: fails
- 2nd and final attempt (1st retry): fails
- replay of the 2nd attempt

### Result
#### Before
The last span for the `always-fails` operation has `replay == false` and `operation_attempt` span tag of `2`. Operations table shows the operation has 3 attempts. All are wrong.
<img width="851" height="389" alt="image" src="https://github.com/user-attachments/assets/bca1691a-70dc-43a8-a80d-e63876f20bbb" />
<img width="304" height="223" alt="image" src="https://github.com/user-attachments/assets/507f594e-2c0d-44df-9fd2-01b0b3e8ac0a" />

#### After

The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off)

<img width="844" height="517" alt="image" src="https://github.com/user-attachments/assets/ede1bef8-63c4-44ae-9f45-4ec3eee9631b" />

It has span tag `aws.durable.replayed == true` and `aws.durable.operation_attempt == 1`, as expected.

Operations table shows the operation has 2 attempts, as expected.

## Staging link
[Link](https://dd.datad0g.com/serverless/aws/lambda?search=functionname%3Add-durable-attempt-py&fromUser=false&graphType=flamegraph&panel_end=1782850230200&panel_paused=false&panel_start=1782846630200&panel_tab=durableExecutions&refresh_mode=sliding&shouldShowLegend=true&sp=%5B%7B%22p%22%3A%7B%22entityId%22%3A%22aws-lambda-functions%2Bdd-durable-attempt-py%2Bus-east-2%2B425362996713%22%7D%2C%22i%22%3A%22lambda-panel%22%7D%2C%7B%22p%22%3A%7B%22traceID%22%3A%226a44216000000000ac65d4f099b859bf%22%2C%22selectedSpanID%22%3A%226869736249323812685%22%7D%2C%22i%22%3A%22trace-panel%22%7D%5D&spanID=5613198925804955800&spanViewType=infrastructure&traceID=6a44216000000000ac65d4f099b859bf&traceQuery=&start=1782677253459&end=1782850053459&paused=false)

## Next steps
Do the same fix for JS. DataDog/dd-trace-js#9160

Co-authored-by: yiming.luo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants