fix(aws-durable-execution-sdk-js): [SVLS-9168] handle FAILED checkpoints properly#9160
Conversation
A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Overall package sizeSelf size: 6.43 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: ddeb831 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-06-30 19:55:25 Comparing candidate commit ddeb831 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2253 metrics, 33 unstable metrics.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9160 +/- ##
=======================================
Coverage 93.73% 93.73%
=======================================
Files 892 892
Lines 51250 51251 +1
Branches 11928 11928
=======================================
+ Hits 48039 48040 +1
Misses 3211 3211 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
…kpoints properly (#18824) ## What does this PR do? Treats a **FAILED** durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans: - `aws.durable.replayed` — now `true` when the operation is served from a terminal checkpoint (FAILED or SUCCEEDED) - `aws.durable.operation_attempt` — the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints. ## Testing ### Context The `always-fails` operation runs/replays three times: - original attempt: fails - 2nd and final attempt (1st retry): fails - replay of the 2nd attempt ### Result #### Before The last span for the `always-fails` operation has `replay == false` and `operation_attempt` span tag of `2`. Operations table shows the operation has 3 attempts. All are wrong. <img width="851" height="389" alt="image" src="https://github.com/user-attachments/assets/bca1691a-70dc-43a8-a80d-e63876f20bbb" /> <img width="304" height="223" alt="image" src="https://github.com/user-attachments/assets/507f594e-2c0d-44df-9fd2-01b0b3e8ac0a" /> #### After The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off) <img width="844" height="517" alt="image" src="https://github.com/user-attachments/assets/ede1bef8-63c4-44ae-9f45-4ec3eee9631b" /> It has span tag `aws.durable.replayed == true` and `aws.durable.operation_attempt == 1`, as expected. Operations table shows the operation has 2 attempts, as expected. ## Staging link [Link](https://dd.datad0g.com/serverless/aws/lambda?search=functionname%3Add-durable-attempt-py&fromUser=false&graphType=flamegraph&panel_end=1782850230200&panel_paused=false&panel_start=1782846630200&panel_tab=durableExecutions&refresh_mode=sliding&shouldShowLegend=true&sp=%5B%7B%22p%22%3A%7B%22entityId%22%3A%22aws-lambda-functions%2Bdd-durable-attempt-py%2Bus-east-2%2B425362996713%22%7D%2C%22i%22%3A%22lambda-panel%22%7D%2C%7B%22p%22%3A%7B%22traceID%22%3A%226a44216000000000ac65d4f099b859bf%22%2C%22selectedSpanID%22%3A%226869736249323812685%22%7D%2C%22i%22%3A%22trace-panel%22%7D%5D&spanID=5613198925804955800&spanViewType=infrastructure&traceID=6a44216000000000ac65d4f099b859bf&traceQuery=&start=1782677253459&end=1782850053459&paused=false) ## Next steps Do the same fix for JS. DataDog/dd-trace-js#9160 Co-authored-by: yiming.luo <[email protected]>
…kpoints properly (#18824) ## What does this PR do? Treats a **FAILED** durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans: - `aws.durable.replayed` — now `true` when the operation is served from a terminal checkpoint (FAILED or SUCCEEDED) - `aws.durable.operation_attempt` — the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints. ## Testing ### Context The `always-fails` operation runs/replays three times: - original attempt: fails - 2nd and final attempt (1st retry): fails - replay of the 2nd attempt ### Result #### Before The last span for the `always-fails` operation has `replay == false` and `operation_attempt` span tag of `2`. Operations table shows the operation has 3 attempts. All are wrong. <img width="851" height="389" alt="image" src="https://github.com/user-attachments/assets/bca1691a-70dc-43a8-a80d-e63876f20bbb" /> <img width="304" height="223" alt="image" src="https://github.com/user-attachments/assets/507f594e-2c0d-44df-9fd2-01b0b3e8ac0a" /> #### After The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off) <img width="844" height="517" alt="image" src="https://github.com/user-attachments/assets/ede1bef8-63c4-44ae-9f45-4ec3eee9631b" /> It has span tag `aws.durable.replayed == true` and `aws.durable.operation_attempt == 1`, as expected. Operations table shows the operation has 2 attempts, as expected. ## Staging link [Link](https://dd.datad0g.com/serverless/aws/lambda?search=functionname%3Add-durable-attempt-py&fromUser=false&graphType=flamegraph&panel_end=1782850230200&panel_paused=false&panel_start=1782846630200&panel_tab=durableExecutions&refresh_mode=sliding&shouldShowLegend=true&sp=%5B%7B%22p%22%3A%7B%22entityId%22%3A%22aws-lambda-functions%2Bdd-durable-attempt-py%2Bus-east-2%2B425362996713%22%7D%2C%22i%22%3A%22lambda-panel%22%7D%2C%7B%22p%22%3A%7B%22traceID%22%3A%226a44216000000000ac65d4f099b859bf%22%2C%22selectedSpanID%22%3A%226869736249323812685%22%7D%2C%22i%22%3A%22trace-panel%22%7D%5D&spanID=5613198925804955800&spanViewType=infrastructure&traceID=6a44216000000000ac65d4f099b859bf&traceQuery=&start=1782677253459&end=1782850053459&paused=false) ## Next steps Do the same fix for JS. DataDog/dd-trace-js#9160 Co-authored-by: yiming.luo <[email protected]>
What does this PR do?
Treats a FAILED durable checkpoint the same as a SUCCEEDED one, for two span attributes on durable operation spans:
aws.durable.replayed— nowtruewhen the operation is served from a terminal checkpoint (FAILED or SUCCEEDED)aws.durable.operation_attempt— the 0-indexing normalization (i.e. subtracting attempt value by 1) now applies to FAILED checkpoints too. Right now we only do this for SUCCEEDED checkpoints.Testing
Context
The
always-failsoperation runs/replays three times:Result
Before
The last span for the


always-failsoperation hasreplay == falseandoperation_attemptspan tag of2, both of which are wrong.After
The replay span shows as a separate row in the operations table, and is hidden by default (when "Show replays" is off)

Related PR
The same fix for Python: #9160