feat: [SVLS-9168] add aws.durable.operation_attempt tag to durable operation spans#8595
Merged
Conversation
…quired option
Drop the `retryable = false` default and the `= {}` fallback in
makeContextPlugin so every call site states retryability explicitly,
matching the named-boolean-option convention used elsewhere (e.g.
createCallbackInstrumentor's { captureResult }).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
…JSDoc Replace the "op" shorthand with "operation" in the addOpMeta and getOperationAttempt JSDoc prose. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…js JSDoc "op" is already the established shorthand across the durable plugin, so keep the JSDoc consistent with it. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Condense the attempt-indexing explanation while keeping the non-obvious parts: the pending-vs-SUCCEEDED dual indexing and the floor-at-0 rationale. Co-Authored-By: Claude Opus 4.8 <[email protected]>
crysmags
reviewed
Jun 24, 2026
crysmags
left a comment
crysmags
left a comment
Collaborator
There was a problem hiding this comment.
logic and structure are solid — replay normalization is correct, the static retryable flag approach is clean, and the step retry coverage at attempt=0 and attempt=1 is exactly right. one gap I'd close before merge: waitForCondition at attempt > 0. the replay test covers the first-check case, but there's no live multi-poll test where attempt=1, which is the only path that would catch a divergence if the SDK ever indexes waitForCondition's StepDetails.Attempt differently from step's. two P3 nits inline.
| function getOperationAttempt (ctxImpl) { | ||
| const stepId = ctxImpl?.getNextStepId?.() | ||
| if (!stepId) return 0 | ||
| const stepData = ctxImpl?._executionContext?.getStepData?.(stepId) |
Collaborator
There was a problem hiding this comment.
nit: addOpMeta already calls getNextStepId() + getStepData() on the same ctxImpl, and bindStart always calls both in sequence — so this is two traversals through the same SDK internals per retryable span start. not a hot loop so the cost is bounded, but a shared getStepDataForNext(ctxImpl) helper would eliminate the duplication if you want.
| createCallback: makeContextPlugin('createCallback', 'aws.durable.create_callback', { retryable: false }), | ||
| map: makeContextPlugin('map', 'aws.durable.map', { retryable: false }), | ||
| parallel: makeContextPlugin('parallel', 'aws.durable.parallel', { retryable: false }), | ||
| runInChildContext: RunInChildContextPlugin, |
Collaborator
There was a problem hiding this comment.
nit: RunInChildContextPlugin (line 88) doesn't declare static retryable, so this.constructor.retryable in super.bindStart() resolves to undefined — falsy, so no metrics are emitted, which is correct. an explicit static retryable = false on the class would make the intent clear and match the pattern the six makeContextPlugin calls above set.
| }) | ||
|
|
||
| return replayedCondition | ||
| }) |
Collaborator
There was a problem hiding this comment.
this covers replay normalization for a first-check success (attempt=0 on SUCCEEDED replay) — good. what's missing is a live multi-poll scenario: waitStrategy returns shouldContinue: true on the first check, forcing a second poll, and the second-poll span carries operation_attempt === 1. without it, getOperationAttempt for waitForCondition is only exercised at attempt=0, so if the SDK ever diverges waitForCondition's StepDetails.Attempt indexing from step's it would slip through undetected.
rough shape:
it('checkpoint plugin: waitForCondition reports operation_attempt=1 on second poll', async () => {
let polls = 0
const secondPollSpan = agent.assertSomeTraces(traces => {
const span = traces.flat().find(s =>
s.name === 'aws.durable.wait_for_condition' &&
s.resource === 'multi-poll-condition' &&
s.metrics?.['aws.durable.operation_attempt'] === 1
)
assert.ok(span, 'expected a second-poll span with operation_attempt=1')
}, { timeoutMs: 5000 })
await invokeHandler(async (event, ctx) => {
await ctx.waitForCondition('multi-poll-condition', async () => 'done', {
initialState: 'pending',
waitStrategy: () => ({ shouldContinue: polls++ < 1 }),
})
})
return secondPollSpan
})
Contributor
Author
There was a problem hiding this comment.
Good point! Added a test.
crysmags
reviewed
Jun 24, 2026
| */ | ||
| function getOperationAttempt (ctxImpl) { | ||
| const stepId = ctxImpl?.getNextStepId?.() | ||
| if (!stepId) return 0 |
Collaborator
There was a problem hiding this comment.
nit: typeof NaN === 'number' is true, so a NaN Attempt value slips past this guard and propagates to the formatted span, where span_format.js silently drops it via its !Number.isNaN() check — metric disappears instead of defaulting to 0. Number.isFinite(attempt) catches it explicitly:
if (!Number.isFinite(attempt)) return 0…pt=1 on second poll A multi-poll waitForCondition (first check returns shouldContinue:true) emits a separate span per poll, the second carrying operation_attempt=1. This exercises getOperationAttempt's live (non-replay) branch at attempt>0 for waitForCondition, the path that would catch the SDK ever indexing its StepDetails.Attempt differently from step's. No production change — verified the value is already correct on the current code. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…citly on RunInChildContextPlugin RunInChildContextPlugin is the one context plugin not built via makeContextPlugin, so it relied on this.constructor.retryable resolving to undefined. Declare it explicitly so every concrete plugin states its retryability, matching the required option on the makeContextPlugin calls. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… per span start Extract getStepDataForNext, which resolves the next stepId and its checkpoint entry in one pass. addOpMeta and getOperationAttempt now consume the pre-resolved data instead of each calling getNextStepId() + getStepData(), so a retryable span start no longer traverses the SDK internals twice. No behavior change. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…non-finite Attempt typeof NaN === 'number', so a NaN StepDetails.Attempt slipped past the old guard and propagated as a NaN metric, which span_format.js silently drops — the operation_attempt metric would vanish instead of defaulting to 0. Number.isFinite rejects NaN and Infinity too, so it falls back to 0 explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]>
crysmags
reviewed
Jun 29, 2026
| function getOperationAttempt (stepData) { | ||
| const attempt = stepData?.StepDetails?.Attempt | ||
| if (!Number.isFinite(attempt)) return 0 | ||
| return stepData.Status === 'SUCCEEDED' ? Math.max(0, attempt - 1) : attempt |
Collaborator
There was a problem hiding this comment.
Tracing this against the python version (#18191) and I think there's a cross-tracer mismatch worth confirming before this lands.
py passes step_details.attempt straight through with no SUCCEEDED special-case — its comment says the field is always the 0-indexed prior-failures count. This PR subtracts 1 on SUCCEEDED replays (Math.max(0, attempt - 1)). So for the same execution — a step that succeeds on its first try, observed on replay — py emits operation_attempt=1 and this emits 0. Since the whole point of the tag (per the py PR) is the UI grouping attempts for the same operation, the two tracers feeding different numbers for the identical run defeats that.
The interesting part: your replay test proves the JS SDK really does hand back attempt=1 on a first-try SUCCEEDED replay — otherwise the =0 assertion wouldn't pass. So the normalization here looks correct and py looks like it has a latent bug (a no-retry step showing as 1 on replay). But I can't tell from the code alone whether the two SDKs genuinely differ or whether one side is just wrong.
Can you check with the durable-execution folks which model is canonical? Two specific things:
- Does the py SDK actually report
0where the JS SDK reports1on a SUCCEEDED replay — i.e. is this a real SDK difference, or does one tracer have a bug? - Is it only SUCCEEDED replays that are 1-indexed? A FAILED op reloaded on replay wouldn't hit the
-1branch here, so if those are also 1-indexed we'd under-report them — and the comment already flags this is observed behavior, not an SDK guarantee.
Contributor
Author
There was a problem hiding this comment.
Good point. Actually I have a PR DataDog/dd-trace-py#18520 (after DataDog/dd-trace-py#18191) to change the Python logic to:
event.operation_attempt = (attempt - 1 if attempt > 0 else 0) if is_succeeded else attempt
which is consistent with the logic in this PR for JS. Sorry for not mentioning it in PR summary. Let me include it there.
Let me know if you have further questions.
Contributor
Author
There was a problem hiding this comment.
Findings (from Claude) are summarized on this page
Collaborator
There was a problem hiding this comment.
That resolves it — checked #18520, it's merged and the logic matches this PR exactly ((attempt - 1 if attempt > 0 else 0) == Math.max(0, attempt - 1) on SUCCEEDED). JS is the canonical model and py's consistent with it now.
The one thing still on "observed, not guaranteed" is whether non-SUCCEEDED replays could also be 1-indexed — but both tracers handle it identically now and it's documented on both sides, so I'm good shipping as-is and revisiting if it comes up.
crysmags
approved these changes
Jun 30, 2026
crysmags
left a comment
crysmags
left a comment
Collaborator
There was a problem hiding this comment.
Everything has been addressed, approved!
dd-octo-sts Bot
added a commit
that referenced
this pull request
Jul 1, 2026
…eration spans (#8595) * workflow(aws-durable-execution-sdk-js): install_package * workflow(aws-durable-execution-sdk-js): generate_app * workflow(aws-durable-execution-sdk-js): compile * workflow(aws-durable-execution-sdk-js): test:att1:iter1:fixer * workflow(aws-durable-execution-sdk-js): test:att1:iter2:fixer * workflow(aws-durable-execution-sdk-js): feature_implement * workflow(aws-durable-execution-sdk-js): get_lint_failures * workflow(aws-durable-execution-sdk-js): lint_and_fix:att1:iter1:fix_lint_errors * workflow(aws-durable-execution-sdk-js): review_cycle:att1:iter1:batch_fix * remove the unnecessary dd-api-key * clean up # Conflicts: # index.d.ts * yarn.lock changed... * fixing yarn.lock * remove the unintended finish() guard * update span names * use a fixed service name instead * update resource names * naming consistency * small fix * Python PR parity * Undo unnecessary changes * Finish error spans on asyncEnd * Simplify orchestrion file * Class/file name changes * Several simplifications and improvements * Do not explicitly set component * Remove includeReplayedTag * Smaller simplifications * Tests simplification * chore: update supported-integrations * More test simplification * Add aws.durable.operation_id and aws.durable.operation_name * Fix checks * Linter * Test simplifications * More test improvements * Lazy thenables + only close this integration's spans * Code simplifications * Fix rebase * Mirror changes in v5 * Test waitForCondition happy path * Comment improvements based on guidelines * supress child context for WaitForCallback * Increase tested version * Address review comments * Avoid patching on the plugin by creating a "settle" channel * Do not skipTime to avoid interfering with tracer's timers * Fix test flakiness * Test durable-execution-sdk-js only on node >=22 * Linter * feat(aws-durable-execution-sdk-js): trace-context checkpoint for cross-invocation continuity Persist the current trace context as a synthetic `_datadog_{N}` STEP operation when the SDK suspends to PENDING, so subsequent invocations (read by the upstream datadog-lambda-js wrapper) can resume the same trace. Files: - src/handler.js: install a hook on the SDK's terminationManager.terminate inside bindStart. Save fires only for resumable reasons (PENDING_TERMINATION_REASONS allow-list mirrors the SDK's TerminationReason enum entries that result in Status: PENDING). Gated by DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED (default on; opt out with 'false'/'0'). - src/trace-checkpoint.js: NEW. Datadog-only header inject (private TextMapPropagator with tracePropagationStyle.inject = ['datadog'], shadows the live tracer config), dedup against prior _datadog_N op via JSON.stringify-after-stripping-x-datadog-parent-id, deterministic blake2b stepId so the save is idempotent within an execution. - test/handler.checkpoint.spec.js: unit tests for the termination hook (pending vs non-pending reasons, env-var gate, idempotency, default reason). - test/trace-checkpoint.spec.js: unit tests for the save module (queue START+SUCCEED before terminating, dedup on parent-id-only changes). - test/index.spec.js: integration coverage for SDK safe-paths (single cycle, child-context, step-suspend-step). - packages/dd-trace/src/config/supported-configurations.json and generated-config-types.d.ts: register DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED. * small fixes: align format * test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug Skip wait_for_callback (happy path) and the entire invoke describe block (happy + error). All three fail deterministically in CI under @aws/durable-execution-sdk-js-testing's current TimerScheduler, whose hasScheduledFunction() undercounts in-flight scheduled functions and trips the test orchestrator's "Cannot return PENDING status with no pending operations." validation. Production (real AWS backend) is not affected — the validation is mock-only. Fix is open upstream as aws/aws-durable-execution-sdk-js#544; re-enable these tests once a release containing it is pinned in packages/dd-trace/test/plugins/versions/package.json. * refactor(aws-durable-execution-sdk-js): remove kTerminationHookInstalled guard The guard was defensive against a "same terminationManager passed to bindStart twice" scenario that cannot happen in the SDK as it stands — each Lambda invocation calls initializeExecutionContext, which constructs a fresh `new TerminationManager()`, so warm starts share the wrapper closure but not the terminationManager instance. Removing the Symbol + the guard + the explicit "twice across invocations" unit test that only covered a contrived re-entry. Drive-by: fix four pre-existing space-before-function-paren lint errors in the same file. * refactor(aws-durable-execution-sdk-js): anchor checkpoints at the execute span, not its parent Drop the `getParentSpanId` helper and inline the read directly during `state` initialization. While inlining, switch the anchor from the execute span's *parent* (typically `aws.lambda`'s id) to the execute span's *own* id (`span.context().toSpanId()`). Why anchor at the execute span: - It's a span this integration owns and just created, so always defined and never depends on what upstream context happened to be active when `bindStart` fired. - Topology becomes "resumed invocations are continuations of the first execute" — matching the user-facing model of a single durable execution. The old shape made resumes look like sibling Lambda invocations under whatever upstream span happened to be there. - In the no-upstream case the old code already fell through to the propagator default (= execute span's own id) via `if (parentId)` — so this just makes the behavior consistent across environments. Rename for clarity: - `saveTraceContextCheckpointIfUpdated`'s `checkpointAnchorSpanId` parameter -> `firstExecutionSpanId`. JSDoc spells out it's only consulted on the very first save; once a prior `_datadog_{N}` exists, the function reuses that checkpoint's `x-datadog-parent-id` verbatim. - The local `latestParentId` (the value carried forward across saves) -> `anchoredSpanId`, reflecting that it IS the anchor we've been using since the first save. - handler.js's `state.parentSpanId` -> `state.firstExecutionSpanId`. Note: dd-trace-py's `_resolve_override_parent_id` currently anchors at the execute span's parent (matching the old JS behavior). A follow-up should bring Python in line with this change so both languages produce the same trace shape. * Revert "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 8baa8ce. * Reapply "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 748a826. * force-disable legacyBaggageEnabled * Rename context variables * feat(aws-durable-execution-sdk-js): add aws.durable.operation_attempt tag Tag aws.durable.step and aws.durable.wait_for_condition spans with the 1-indexed attempt number (1 for the original attempt, 2 for the first retry, etc.), matching the AWS UI's attempt-count convention. Sourced from the SDK's checkpoint StepDetails.Attempt field; defaults to 1 when no checkpoint exists yet (the very first execution before the START checkpoint). Mirrors dd-trace-py #18191. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): use 0-indexed operation_attempt Pass StepDetails.Attempt through as-is (0 = original attempt, 1 = first retry, etc.) instead of remapping with max(1, …). The production AWS Lambda Durable service stores Attempt as "number of prior failed attempts" starting at 0, so the raw value already carries correct 0-indexed semantics. Default to 0 (not 1) when no checkpoint exists yet. Matches dd-trace-py after end-to-end verification on a deployed Lambda: CloudWatch logs confirmed server values 0/1/2 for original/retry/retry. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * Apply suggestions from code review Co-authored-by: Ruben Bridgewater <[email protected]> * make installTerminationCheckpointHook real private * simplify getDatadogOnlyPropagator * better names * lint * refactor(aws-durable-execution-sdk-js): use static retryable property instead of span name Set Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * refactor(aws-durable-execution-sdk-js): derive test RETRYABLE_SPAN_NAMES from plugin static property Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Lint * remove unnecessary try catch * return undefined instead of null * use timers.promises.setImmediate() instead * Add esbuild bundling acceptance test * ESM smoke tests * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * don't skip the established tests * also update buildPlugin * Use the normalized `tracer` getter, not the raw `_tracer`: the latter is the proxy, whose`_config` is undefined. trace-checkpoint reads `tracer._config`, so it needs the unwrapped tracer that actually carries the config. * lint * improve the coverage * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Address various review comments * Operation name * fix(aws-durable-execution-sdk-js): correct operation_attempt on replayed ops On a SUCCEEDED checkpoint (an op replayed from its stored result), StepDetails.Attempt holds the 1-indexed number of the attempt that succeeded rather than the "prior failed attempts" count used on a live run, so a replayed op reported its attempt one too high. Subtract 1 in that case, floored at 0, so a replay agrees with the original run. Mirrors dd-trace-py#18520. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): gate termination hook install in bindStart Extract #shouldInstallTerminationHook so bindStart decides whether to install the cross-invocation checkpoint hook, instead of the hook self-gating with early returns. The hook now assumes its preconditions and recomputes the handler args, execute span, and termination manager it wraps. Stub operationName() in the handler checkpoint spec: it reaches into the tracer's nomenclature, which the bare tracer stub doesn't provide. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor and move instrumentation logic to aws-durable-execution-sdk-js so that it's only done once instead of every invocation * Apply suggestion from @BridgeAR Co-authored-by: Ruben Bridgewater <[email protected]> * modify a inefficient check * simplify comparison * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * enhance getDatadogOnlyPropagator * docs(aws-durable-execution-sdk-js): avoid hyphen line-break in getOperationAttempt comment Reflow so "server-maintained" stays on one line, per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition replay attempt normalization waitForCondition is the other retryable op and shares step's StepDetails.Attempt convention (1-indexed on a SUCCEEDED replay), so add a replay test asserting its operation_attempt normalizes back to 0. Guards against the SDK diverging the two operations' attempt semantics. Per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): make retryable an explicit required option Drop the `retryable = false` default and the `= {}` fallback in makeContextPlugin so every call site states retryability explicitly, matching the named-boolean-option convention used elsewhere (e.g. createCallbackInstrumentor's { captureResult }). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): spell out "operation" in util.js JSDoc Replace the "op" shorthand with "operation" in the addOpMeta and getOperationAttempt JSDoc prose. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): revert to "op" shorthand in util.js JSDoc "op" is already the established shorthand across the durable plugin, so keep the JSDoc consistent with it. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): tighten getOperationAttempt JSDoc Condense the attempt-indexing explanation while keeping the non-obvious parts: the pending-vs-SUCCEEDED dual indexing and the floor-at-0 rationale. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition live attempt=1 on second poll A multi-poll waitForCondition (first check returns shouldContinue:true) emits a separate span per poll, the second carrying operation_attempt=1. This exercises getOperationAttempt's live (non-replay) branch at attempt>0 for waitForCondition, the path that would catch the SDK ever indexing its StepDetails.Attempt differently from step's. No production change — verified the value is already correct on the current code. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): declare retryable=false explicitly on RunInChildContextPlugin RunInChildContextPlugin is the one context plugin not built via makeContextPlugin, so it relied on this.constructor.retryable resolving to undefined. Declare it explicitly so every concrete plugin states its retryability, matching the required option on the makeContextPlugin calls. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): resolve next-step lookup once per span start Extract getStepDataForNext, which resolves the next stepId and its checkpoint entry in one pass. addOpMeta and getOperationAttempt now consume the pre-resolved data instead of each calling getNextStepId() + getStepData(), so a retryable span start no longer traverses the SDK internals twice. No behavior change. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): guard getOperationAttempt against non-finite Attempt typeof NaN === 'number', so a NaN StepDetails.Attempt slipped past the old guard and propagated as a NaN metric, which span_format.js silently drops — the operation_attempt metric would vanish instead of defaulting to 0. Number.isFinite rejects NaN and Infinity too, so it falls back to 0 explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Joey Zhao <[email protected]> Co-authored-by: Pablo Martínez Bernardo <[email protected]> Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Ruben Bridgewater <[email protected]> Co-authored-by: pablomartinezbernardo <[email protected]>
Merged
lym953
added a commit
that referenced
this pull request
Jul 1, 2026
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
dd-octo-sts Bot
pushed a commit
that referenced
this pull request
Jul 2, 2026
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
Merged
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
…eration spans (#8595) * workflow(aws-durable-execution-sdk-js): install_package * workflow(aws-durable-execution-sdk-js): generate_app * workflow(aws-durable-execution-sdk-js): compile * workflow(aws-durable-execution-sdk-js): test:att1:iter1:fixer * workflow(aws-durable-execution-sdk-js): test:att1:iter2:fixer * workflow(aws-durable-execution-sdk-js): feature_implement * workflow(aws-durable-execution-sdk-js): get_lint_failures * workflow(aws-durable-execution-sdk-js): lint_and_fix:att1:iter1:fix_lint_errors * workflow(aws-durable-execution-sdk-js): review_cycle:att1:iter1:batch_fix * remove the unnecessary dd-api-key * clean up # Conflicts: # index.d.ts * yarn.lock changed... * fixing yarn.lock * remove the unintended finish() guard * update span names * use a fixed service name instead * update resource names * naming consistency * small fix * Python PR parity * Undo unnecessary changes * Finish error spans on asyncEnd * Simplify orchestrion file * Class/file name changes * Several simplifications and improvements * Do not explicitly set component * Remove includeReplayedTag * Smaller simplifications * Tests simplification * chore: update supported-integrations * More test simplification * Add aws.durable.operation_id and aws.durable.operation_name * Fix checks * Linter * Test simplifications * More test improvements * Lazy thenables + only close this integration's spans * Code simplifications * Fix rebase * Mirror changes in v5 * Test waitForCondition happy path * Comment improvements based on guidelines * supress child context for WaitForCallback * Increase tested version * Address review comments * Avoid patching on the plugin by creating a "settle" channel * Do not skipTime to avoid interfering with tracer's timers * Fix test flakiness * Test durable-execution-sdk-js only on node >=22 * Linter * feat(aws-durable-execution-sdk-js): trace-context checkpoint for cross-invocation continuity Persist the current trace context as a synthetic `_datadog_{N}` STEP operation when the SDK suspends to PENDING, so subsequent invocations (read by the upstream datadog-lambda-js wrapper) can resume the same trace. Files: - src/handler.js: install a hook on the SDK's terminationManager.terminate inside bindStart. Save fires only for resumable reasons (PENDING_TERMINATION_REASONS allow-list mirrors the SDK's TerminationReason enum entries that result in Status: PENDING). Gated by DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED (default on; opt out with 'false'/'0'). - src/trace-checkpoint.js: NEW. Datadog-only header inject (private TextMapPropagator with tracePropagationStyle.inject = ['datadog'], shadows the live tracer config), dedup against prior _datadog_N op via JSON.stringify-after-stripping-x-datadog-parent-id, deterministic blake2b stepId so the save is idempotent within an execution. - test/handler.checkpoint.spec.js: unit tests for the termination hook (pending vs non-pending reasons, env-var gate, idempotency, default reason). - test/trace-checkpoint.spec.js: unit tests for the save module (queue START+SUCCEED before terminating, dedup on parent-id-only changes). - test/index.spec.js: integration coverage for SDK safe-paths (single cycle, child-context, step-suspend-step). - packages/dd-trace/src/config/supported-configurations.json and generated-config-types.d.ts: register DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED. * small fixes: align format * test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug Skip wait_for_callback (happy path) and the entire invoke describe block (happy + error). All three fail deterministically in CI under @aws/durable-execution-sdk-js-testing's current TimerScheduler, whose hasScheduledFunction() undercounts in-flight scheduled functions and trips the test orchestrator's "Cannot return PENDING status with no pending operations." validation. Production (real AWS backend) is not affected — the validation is mock-only. Fix is open upstream as aws/aws-durable-execution-sdk-js#544; re-enable these tests once a release containing it is pinned in packages/dd-trace/test/plugins/versions/package.json. * refactor(aws-durable-execution-sdk-js): remove kTerminationHookInstalled guard The guard was defensive against a "same terminationManager passed to bindStart twice" scenario that cannot happen in the SDK as it stands — each Lambda invocation calls initializeExecutionContext, which constructs a fresh `new TerminationManager()`, so warm starts share the wrapper closure but not the terminationManager instance. Removing the Symbol + the guard + the explicit "twice across invocations" unit test that only covered a contrived re-entry. Drive-by: fix four pre-existing space-before-function-paren lint errors in the same file. * refactor(aws-durable-execution-sdk-js): anchor checkpoints at the execute span, not its parent Drop the `getParentSpanId` helper and inline the read directly during `state` initialization. While inlining, switch the anchor from the execute span's *parent* (typically `aws.lambda`'s id) to the execute span's *own* id (`span.context().toSpanId()`). Why anchor at the execute span: - It's a span this integration owns and just created, so always defined and never depends on what upstream context happened to be active when `bindStart` fired. - Topology becomes "resumed invocations are continuations of the first execute" — matching the user-facing model of a single durable execution. The old shape made resumes look like sibling Lambda invocations under whatever upstream span happened to be there. - In the no-upstream case the old code already fell through to the propagator default (= execute span's own id) via `if (parentId)` — so this just makes the behavior consistent across environments. Rename for clarity: - `saveTraceContextCheckpointIfUpdated`'s `checkpointAnchorSpanId` parameter -> `firstExecutionSpanId`. JSDoc spells out it's only consulted on the very first save; once a prior `_datadog_{N}` exists, the function reuses that checkpoint's `x-datadog-parent-id` verbatim. - The local `latestParentId` (the value carried forward across saves) -> `anchoredSpanId`, reflecting that it IS the anchor we've been using since the first save. - handler.js's `state.parentSpanId` -> `state.firstExecutionSpanId`. Note: dd-trace-py's `_resolve_override_parent_id` currently anchors at the execute span's parent (matching the old JS behavior). A follow-up should bring Python in line with this change so both languages produce the same trace shape. * Revert "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 8baa8ce. * Reapply "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 748a826. * force-disable legacyBaggageEnabled * Rename context variables * feat(aws-durable-execution-sdk-js): add aws.durable.operation_attempt tag Tag aws.durable.step and aws.durable.wait_for_condition spans with the 1-indexed attempt number (1 for the original attempt, 2 for the first retry, etc.), matching the AWS UI's attempt-count convention. Sourced from the SDK's checkpoint StepDetails.Attempt field; defaults to 1 when no checkpoint exists yet (the very first execution before the START checkpoint). Mirrors dd-trace-py #18191. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): use 0-indexed operation_attempt Pass StepDetails.Attempt through as-is (0 = original attempt, 1 = first retry, etc.) instead of remapping with max(1, …). The production AWS Lambda Durable service stores Attempt as "number of prior failed attempts" starting at 0, so the raw value already carries correct 0-indexed semantics. Default to 0 (not 1) when no checkpoint exists yet. Matches dd-trace-py after end-to-end verification on a deployed Lambda: CloudWatch logs confirmed server values 0/1/2 for original/retry/retry. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * Apply suggestions from code review Co-authored-by: Ruben Bridgewater <[email protected]> * make installTerminationCheckpointHook real private * simplify getDatadogOnlyPropagator * better names * lint * refactor(aws-durable-execution-sdk-js): use static retryable property instead of span name Set Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * refactor(aws-durable-execution-sdk-js): derive test RETRYABLE_SPAN_NAMES from plugin static property Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Lint * remove unnecessary try catch * return undefined instead of null * use timers.promises.setImmediate() instead * Add esbuild bundling acceptance test * ESM smoke tests * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * don't skip the established tests * also update buildPlugin * Use the normalized `tracer` getter, not the raw `_tracer`: the latter is the proxy, whose`_config` is undefined. trace-checkpoint reads `tracer._config`, so it needs the unwrapped tracer that actually carries the config. * lint * improve the coverage * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Address various review comments * Operation name * fix(aws-durable-execution-sdk-js): correct operation_attempt on replayed ops On a SUCCEEDED checkpoint (an op replayed from its stored result), StepDetails.Attempt holds the 1-indexed number of the attempt that succeeded rather than the "prior failed attempts" count used on a live run, so a replayed op reported its attempt one too high. Subtract 1 in that case, floored at 0, so a replay agrees with the original run. Mirrors dd-trace-py#18520. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): gate termination hook install in bindStart Extract #shouldInstallTerminationHook so bindStart decides whether to install the cross-invocation checkpoint hook, instead of the hook self-gating with early returns. The hook now assumes its preconditions and recomputes the handler args, execute span, and termination manager it wraps. Stub operationName() in the handler checkpoint spec: it reaches into the tracer's nomenclature, which the bare tracer stub doesn't provide. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor and move instrumentation logic to aws-durable-execution-sdk-js so that it's only done once instead of every invocation * Apply suggestion from @BridgeAR Co-authored-by: Ruben Bridgewater <[email protected]> * modify a inefficient check * simplify comparison * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * enhance getDatadogOnlyPropagator * docs(aws-durable-execution-sdk-js): avoid hyphen line-break in getOperationAttempt comment Reflow so "server-maintained" stays on one line, per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition replay attempt normalization waitForCondition is the other retryable op and shares step's StepDetails.Attempt convention (1-indexed on a SUCCEEDED replay), so add a replay test asserting its operation_attempt normalizes back to 0. Guards against the SDK diverging the two operations' attempt semantics. Per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): make retryable an explicit required option Drop the `retryable = false` default and the `= {}` fallback in makeContextPlugin so every call site states retryability explicitly, matching the named-boolean-option convention used elsewhere (e.g. createCallbackInstrumentor's { captureResult }). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): spell out "operation" in util.js JSDoc Replace the "op" shorthand with "operation" in the addOpMeta and getOperationAttempt JSDoc prose. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): revert to "op" shorthand in util.js JSDoc "op" is already the established shorthand across the durable plugin, so keep the JSDoc consistent with it. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): tighten getOperationAttempt JSDoc Condense the attempt-indexing explanation while keeping the non-obvious parts: the pending-vs-SUCCEEDED dual indexing and the floor-at-0 rationale. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition live attempt=1 on second poll A multi-poll waitForCondition (first check returns shouldContinue:true) emits a separate span per poll, the second carrying operation_attempt=1. This exercises getOperationAttempt's live (non-replay) branch at attempt>0 for waitForCondition, the path that would catch the SDK ever indexing its StepDetails.Attempt differently from step's. No production change — verified the value is already correct on the current code. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): declare retryable=false explicitly on RunInChildContextPlugin RunInChildContextPlugin is the one context plugin not built via makeContextPlugin, so it relied on this.constructor.retryable resolving to undefined. Declare it explicitly so every concrete plugin states its retryability, matching the required option on the makeContextPlugin calls. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): resolve next-step lookup once per span start Extract getStepDataForNext, which resolves the next stepId and its checkpoint entry in one pass. addOpMeta and getOperationAttempt now consume the pre-resolved data instead of each calling getNextStepId() + getStepData(), so a retryable span start no longer traverses the SDK internals twice. No behavior change. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): guard getOperationAttempt against non-finite Attempt typeof NaN === 'number', so a NaN StepDetails.Attempt slipped past the old guard and propagated as a NaN metric, which span_format.js silently drops — the operation_attempt metric would vanish instead of defaulting to 0. Number.isFinite rejects NaN and Infinity too, so it falls back to 0 explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Joey Zhao <[email protected]> Co-authored-by: Pablo Martínez Bernardo <[email protected]> Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Ruben Bridgewater <[email protected]> Co-authored-by: pablomartinezbernardo <[email protected]>
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
…eration spans (#8595) * workflow(aws-durable-execution-sdk-js): install_package * workflow(aws-durable-execution-sdk-js): generate_app * workflow(aws-durable-execution-sdk-js): compile * workflow(aws-durable-execution-sdk-js): test:att1:iter1:fixer * workflow(aws-durable-execution-sdk-js): test:att1:iter2:fixer * workflow(aws-durable-execution-sdk-js): feature_implement * workflow(aws-durable-execution-sdk-js): get_lint_failures * workflow(aws-durable-execution-sdk-js): lint_and_fix:att1:iter1:fix_lint_errors * workflow(aws-durable-execution-sdk-js): review_cycle:att1:iter1:batch_fix * remove the unnecessary dd-api-key * clean up # Conflicts: # index.d.ts * yarn.lock changed... * fixing yarn.lock * remove the unintended finish() guard * update span names * use a fixed service name instead * update resource names * naming consistency * small fix * Python PR parity * Undo unnecessary changes * Finish error spans on asyncEnd * Simplify orchestrion file * Class/file name changes * Several simplifications and improvements * Do not explicitly set component * Remove includeReplayedTag * Smaller simplifications * Tests simplification * chore: update supported-integrations * More test simplification * Add aws.durable.operation_id and aws.durable.operation_name * Fix checks * Linter * Test simplifications * More test improvements * Lazy thenables + only close this integration's spans * Code simplifications * Fix rebase * Mirror changes in v5 * Test waitForCondition happy path * Comment improvements based on guidelines * supress child context for WaitForCallback * Increase tested version * Address review comments * Avoid patching on the plugin by creating a "settle" channel * Do not skipTime to avoid interfering with tracer's timers * Fix test flakiness * Test durable-execution-sdk-js only on node >=22 * Linter * feat(aws-durable-execution-sdk-js): trace-context checkpoint for cross-invocation continuity Persist the current trace context as a synthetic `_datadog_{N}` STEP operation when the SDK suspends to PENDING, so subsequent invocations (read by the upstream datadog-lambda-js wrapper) can resume the same trace. Files: - src/handler.js: install a hook on the SDK's terminationManager.terminate inside bindStart. Save fires only for resumable reasons (PENDING_TERMINATION_REASONS allow-list mirrors the SDK's TerminationReason enum entries that result in Status: PENDING). Gated by DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED (default on; opt out with 'false'/'0'). - src/trace-checkpoint.js: NEW. Datadog-only header inject (private TextMapPropagator with tracePropagationStyle.inject = ['datadog'], shadows the live tracer config), dedup against prior _datadog_N op via JSON.stringify-after-stripping-x-datadog-parent-id, deterministic blake2b stepId so the save is idempotent within an execution. - test/handler.checkpoint.spec.js: unit tests for the termination hook (pending vs non-pending reasons, env-var gate, idempotency, default reason). - test/trace-checkpoint.spec.js: unit tests for the save module (queue START+SUCCEED before terminating, dedup on parent-id-only changes). - test/index.spec.js: integration coverage for SDK safe-paths (single cycle, child-context, step-suspend-step). - packages/dd-trace/src/config/supported-configurations.json and generated-config-types.d.ts: register DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED. * small fixes: align format * test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug Skip wait_for_callback (happy path) and the entire invoke describe block (happy + error). All three fail deterministically in CI under @aws/durable-execution-sdk-js-testing's current TimerScheduler, whose hasScheduledFunction() undercounts in-flight scheduled functions and trips the test orchestrator's "Cannot return PENDING status with no pending operations." validation. Production (real AWS backend) is not affected — the validation is mock-only. Fix is open upstream as aws/aws-durable-execution-sdk-js#544; re-enable these tests once a release containing it is pinned in packages/dd-trace/test/plugins/versions/package.json. * refactor(aws-durable-execution-sdk-js): remove kTerminationHookInstalled guard The guard was defensive against a "same terminationManager passed to bindStart twice" scenario that cannot happen in the SDK as it stands — each Lambda invocation calls initializeExecutionContext, which constructs a fresh `new TerminationManager()`, so warm starts share the wrapper closure but not the terminationManager instance. Removing the Symbol + the guard + the explicit "twice across invocations" unit test that only covered a contrived re-entry. Drive-by: fix four pre-existing space-before-function-paren lint errors in the same file. * refactor(aws-durable-execution-sdk-js): anchor checkpoints at the execute span, not its parent Drop the `getParentSpanId` helper and inline the read directly during `state` initialization. While inlining, switch the anchor from the execute span's *parent* (typically `aws.lambda`'s id) to the execute span's *own* id (`span.context().toSpanId()`). Why anchor at the execute span: - It's a span this integration owns and just created, so always defined and never depends on what upstream context happened to be active when `bindStart` fired. - Topology becomes "resumed invocations are continuations of the first execute" — matching the user-facing model of a single durable execution. The old shape made resumes look like sibling Lambda invocations under whatever upstream span happened to be there. - In the no-upstream case the old code already fell through to the propagator default (= execute span's own id) via `if (parentId)` — so this just makes the behavior consistent across environments. Rename for clarity: - `saveTraceContextCheckpointIfUpdated`'s `checkpointAnchorSpanId` parameter -> `firstExecutionSpanId`. JSDoc spells out it's only consulted on the very first save; once a prior `_datadog_{N}` exists, the function reuses that checkpoint's `x-datadog-parent-id` verbatim. - The local `latestParentId` (the value carried forward across saves) -> `anchoredSpanId`, reflecting that it IS the anchor we've been using since the first save. - handler.js's `state.parentSpanId` -> `state.firstExecutionSpanId`. Note: dd-trace-py's `_resolve_override_parent_id` currently anchors at the execute span's parent (matching the old JS behavior). A follow-up should bring Python in line with this change so both languages produce the same trace shape. * Revert "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 8baa8ce. * Reapply "test(aws-durable-execution-sdk-js): skip 3 tests that race against TimerScheduler bug" This reverts commit 748a826. * force-disable legacyBaggageEnabled * Rename context variables * feat(aws-durable-execution-sdk-js): add aws.durable.operation_attempt tag Tag aws.durable.step and aws.durable.wait_for_condition spans with the 1-indexed attempt number (1 for the original attempt, 2 for the first retry, etc.), matching the AWS UI's attempt-count convention. Sourced from the SDK's checkpoint StepDetails.Attempt field; defaults to 1 when no checkpoint exists yet (the very first execution before the START checkpoint). Mirrors dd-trace-py #18191. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): use 0-indexed operation_attempt Pass StepDetails.Attempt through as-is (0 = original attempt, 1 = first retry, etc.) instead of remapping with max(1, …). The production AWS Lambda Durable service stores Attempt as "number of prior failed attempts" starting at 0, so the raw value already carries correct 0-indexed semantics. Default to 0 (not 1) when no checkpoint exists yet. Matches dd-trace-py after end-to-end verification on a deployed Lambda: CloudWatch logs confirmed server values 0/1/2 for original/retry/retry. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * Apply suggestions from code review Co-authored-by: Ruben Bridgewater <[email protected]> * make installTerminationCheckpointHook real private * simplify getDatadogOnlyPropagator * better names * lint * refactor(aws-durable-execution-sdk-js): use static retryable property instead of span name Set Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * refactor(aws-durable-execution-sdk-js): derive test RETRYABLE_SPAN_NAMES from plugin static property Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Lint * remove unnecessary try catch * return undefined instead of null * use timers.promises.setImmediate() instead * Add esbuild bundling acceptance test * ESM smoke tests * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * don't skip the established tests * also update buildPlugin * Use the normalized `tracer` getter, not the raw `_tracer`: the latter is the proxy, whose`_config` is undefined. trace-checkpoint reads `tracer._config`, so it needs the unwrapped tracer that actually carries the config. * lint * improve the coverage * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Address various review comments * Operation name * fix(aws-durable-execution-sdk-js): correct operation_attempt on replayed ops On a SUCCEEDED checkpoint (an op replayed from its stored result), StepDetails.Attempt holds the 1-indexed number of the attempt that succeeded rather than the "prior failed attempts" count used on a live run, so a replayed op reported its attempt one too high. Subtract 1 in that case, floored at 0, so a replay agrees with the original run. Mirrors dd-trace-py#18520. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): gate termination hook install in bindStart Extract #shouldInstallTerminationHook so bindStart decides whether to install the cross-invocation checkpoint hook, instead of the hook self-gating with early returns. The hook now assumes its preconditions and recomputes the handler args, execute span, and termination manager it wraps. Stub operationName() in the handler checkpoint spec: it reaches into the tracer's nomenclature, which the bare tracer stub doesn't provide. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor and move instrumentation logic to aws-durable-execution-sdk-js so that it's only done once instead of every invocation * Apply suggestion from @BridgeAR Co-authored-by: Ruben Bridgewater <[email protected]> * modify a inefficient check * simplify comparison * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/handler.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * Update packages/datadog-plugin-aws-durable-execution-sdk-js/src/trace-checkpoint.js Co-authored-by: Ruben Bridgewater <[email protected]> * enhance getDatadogOnlyPropagator * docs(aws-durable-execution-sdk-js): avoid hyphen line-break in getOperationAttempt comment Reflow so "server-maintained" stays on one line, per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition replay attempt normalization waitForCondition is the other retryable op and shares step's StepDetails.Attempt convention (1-indexed on a SUCCEEDED replay), so add a replay test asserting its operation_attempt normalizes back to 0. Guards against the SDK diverging the two operations' attempt semantics. Per review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * refactor(aws-durable-execution-sdk-js): make retryable an explicit required option Drop the `retryable = false` default and the `= {}` fallback in makeContextPlugin so every call site states retryability explicitly, matching the named-boolean-option convention used elsewhere (e.g. createCallbackInstrumentor's { captureResult }). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): spell out "operation" in util.js JSDoc Replace the "op" shorthand with "operation" in the addOpMeta and getOperationAttempt JSDoc prose. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): revert to "op" shorthand in util.js JSDoc "op" is already the established shorthand across the durable plugin, so keep the JSDoc consistent with it. Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(aws-durable-execution-sdk-js): tighten getOperationAttempt JSDoc Condense the attempt-indexing explanation while keeping the non-obvious parts: the pending-vs-SUCCEEDED dual indexing and the floor-at-0 rationale. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(aws-durable-execution-sdk-js): cover waitForCondition live attempt=1 on second poll A multi-poll waitForCondition (first check returns shouldContinue:true) emits a separate span per poll, the second carrying operation_attempt=1. This exercises getOperationAttempt's live (non-replay) branch at attempt>0 for waitForCondition, the path that would catch the SDK ever indexing its StepDetails.Attempt differently from step's. No production change — verified the value is already correct on the current code. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): declare retryable=false explicitly on RunInChildContextPlugin RunInChildContextPlugin is the one context plugin not built via makeContextPlugin, so it relied on this.constructor.retryable resolving to undefined. Declare it explicitly so every concrete plugin states its retryability, matching the required option on the makeContextPlugin calls. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(aws-durable-execution-sdk-js): resolve next-step lookup once per span start Extract getStepDataForNext, which resolves the next stepId and its checkpoint entry in one pass. addOpMeta and getOperationAttempt now consume the pre-resolved data instead of each calling getNextStepId() + getStepData(), so a retryable span start no longer traverses the SDK internals twice. No behavior change. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): guard getOperationAttempt against non-finite Attempt typeof NaN === 'number', so a NaN StepDetails.Attempt slipped past the old guard and propagated as a NaN metric, which span_format.js silently drops — the operation_attempt metric would vanish instead of defaulting to 0. Number.isFinite rejects NaN and Infinity too, so it falls back to 0 explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Joey Zhao <[email protected]> Co-authored-by: Pablo Martínez Bernardo <[email protected]> Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Ruben Bridgewater <[email protected]> Co-authored-by: pablomartinezbernardo <[email protected]>
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
#9160) A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-authored-by: Claude Opus 4.8 <[email protected]>
lym953
added a commit
that referenced
this pull request
Jul 6, 2026
…rless (#9162) * fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): set span type to serverless The durable plugins declare `static type = 'serverless'`, but `TracingPlugin.startSpan` only defaults `component` from the class — `type` and `kind` have no `this.constructor.*` fallback. The plugins forwarded `kind: this.constructor.kind` but omitted `type`, so every durable span was emitted with an empty `span.type`. Forward `type: this.constructor.type` in both bindStart calls so the spans carry `type: serverless`, matching the dd-trace-py integration. This lets serverless UIs (e.g. the Lambda trace panel's Durable Function Execution section) recognize the spans as Lambda/function spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
dd-octo-sts Bot
pushed a commit
that referenced
this pull request
Jul 7, 2026
…rless (#9162) * fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): set span type to serverless The durable plugins declare `static type = 'serverless'`, but `TracingPlugin.startSpan` only defaults `component` from the class — `type` and `kind` have no `this.constructor.*` fallback. The plugins forwarded `kind: this.constructor.kind` but omitted `type`, so every durable span was emitted with an empty `span.type`. Forward `type: this.constructor.type` in both bindStart calls so the spans carry `type: serverless`, matching the dd-trace-py integration. This lets serverless UIs (e.g. the Lambda trace panel's Durable Function Execution section) recognize the spans as Lambda/function spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
dd-octo-sts Bot
pushed a commit
that referenced
this pull request
Jul 7, 2026
…rless (#9162) * fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): set span type to serverless The durable plugins declare `static type = 'serverless'`, but `TracingPlugin.startSpan` only defaults `component` from the class — `type` and `kind` have no `this.constructor.*` fallback. The plugins forwarded `kind: this.constructor.kind` but omitted `type`, so every durable span was emitted with an empty `span.type`. Forward `type: this.constructor.type` in both bindStart calls so the spans carry `type: serverless`, matching the dd-trace-py integration. This lets serverless UIs (e.g. the Lambda trace panel's Durable Function Execution section) recognize the spans as Lambda/function spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
juan-fernandez
pushed a commit
that referenced
this pull request
Jul 8, 2026
…rless (#9162) * fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): set span type to serverless The durable plugins declare `static type = 'serverless'`, but `TracingPlugin.startSpan` only defaults `component` from the class — `type` and `kind` have no `this.constructor.*` fallback. The plugins forwarded `kind: this.constructor.kind` but omitted `type`, so every durable span was emitted with an empty `span.type`. Forward `type: this.constructor.type` in both bindStart calls so the spans carry `type: serverless`, matching the dd-trace-py integration. This lets serverless UIs (e.g. the Lambda trace panel's Durable Function Execution section) recognize the spans as Lambda/function spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
juan-fernandez
pushed a commit
that referenced
this pull request
Jul 8, 2026
…rless (#9162) * fix(aws-durable-execution-sdk-js): treat FAILED checkpoints as replays A durable operation served from a FAILED checkpoint on replay (the SDK re-raises the stored error without running the body) was not recognized as a replay: both `aws.durable.replayed` and the `aws.durable.operation_attempt` normalization keyed only on `Status === 'SUCCEEDED'`. As a result a caught-then-replayed failed operation reported `replayed=false` and a 1-indexed `operation_attempt` (e.g. 2 for a step whose 2nd attempt failed) — one higher than, and inconsistent with, the live final attempt (1). This is the same off-by-one #8595 fixed for SUCCEEDED, still open for FAILED. Treat a terminal checkpoint (SUCCEEDED or FAILED) as a replay for both tags, so the replay agrees with the live run (replayed=true, operation_attempt=1). Verified in production on a durable Lambda. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(aws-durable-execution-sdk-js): set span type to serverless The durable plugins declare `static type = 'serverless'`, but `TracingPlugin.startSpan` only defaults `component` from the class — `type` and `kind` have no `this.constructor.*` fallback. The plugins forwarded `kind: this.constructor.kind` but omitted `type`, so every durable span was emitted with an empty `span.type`. Forward `type: this.constructor.type` in both bindStart calls so the spans carry `type: serverless`, matching the dd-trace-py integration. This lets serverless UIs (e.g. the Lambda trace panel's Durable Function Execution section) recognize the spans as Lambda/function spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
aws.durable.operation_attempt(numeric metric) toaws.durable.stepandaws.durable.wait_for_conditionspans. The value is 0-indexed — the number of prior failed attempts:0— original attempt1— first retry2— second retrySourced from the Durable Execution SDK's
StepDetails.Attemptfield on the checkpoint, passed through as-is. When no checkpoint exists yet (the very first execution before any prior failures), the tag defaults to0.Motivation
Mirrors dd-trace-py #18191 and #18520 so the JS tracer surfaces the same retry-attempt signal.
Why only
stepandwait_for_conditionspans?The SDK has six durable operations; only two use the
StepDetails.Attemptretry mechanism:stepStepDetailswaitForConditionStepDetails(polling iterations)createCallbackCallbackDetailsinvokeChainedInvokeDetailsrunInChildContextwaitWaitDetailswaitForCallbackisn't its own retryable operation — it internally callscreateCallback+step, so any retries appear on the inneraws.durable.stepchild span (already covered).mapandparalleldecompose into MapIteration / ParallelBranch child operations, which therunInChildContextsuppression filter already strips.Additional Notes
Stacked on #8012 (Joey's aws-durable-execution-sdk-js instrumentation). Once that merges, retarget to
master.Testing
Installed the tracer on a Lambda durable function. The function has a step
step-1which fails twice then succeeds.Original attempt:
aws.durable.operation_attemptis 0 (link)2nd attempt (1st retry):
aws.durable.operation_attemptis 1 (link)3nd attempt (2nd retry):
aws.durable.operation_attemptis 2 (link)