fix(sdk): record per-item completion so summarized replay rebuilds FLAT map items by ParidelPooya · Pull Request #894 · aws/aws-durable-execution-sdk-js

ParidelPooya · 2026-09-02T21:27:20Z

context.map and context.parallel with nesting: FLAT rebuilt an empty result on replay when the batch was large enough to be summarized, even though every item had succeeded.

A batch whose aggregate result exceeds the checkpoint size limit is stored as a summary with ReplayChildren, and each later resumption reconstructs it from the per-item checkpoints in ConcurrencyController.replayItems. That reconstruction decided whether an item had finished by probing the item's context checkpoint — but with FLAT nesting the per-item contexts are virtual and never checkpointed, so every item read as unfinished and was skipped.

Not cosmetic: code deriving control flow from the batch result, such as a fan-out sized from the results, then creates a different set of operations on replay than it did live. The replay-consistency check detects the divergence and raises NonDeterministicExecutionError, which SDKs at or below 2.3.0 reported as a PENDING response rather than a failure. While other work is still draining PENDING is a legal answer, so the corruption is invisible; once nothing is left pending the service rejects the response with "Cannot return PENDING status with no pending operations" and the execution fails after deterministic retries. Batches whose result fits in one checkpoint are replayed by deserialization and never enter this path.

Change

The batch payload now records which items reached a terminal state, one character per item in index order. Replay reads that record instead of inferring it. Only terminality is recorded — an item's SUCCEEDED/FAILED outcome still comes from re-driving it.

Terminality resolves in order of authority:

the recorded statuses, with any context record the item has required to agree
the item's own context checkpoint, for payloads with no recorded statuses
a probe of the item's operations, for payloads predating the field

The agreement requirement in (1) is one-sided on purpose. A marker saying unfinished overrides a terminal context record, because the statuses are what the live BatchResult contained. The reverse is not honoured: re-driving an item whose context checkpoint is non-terminal hits checkForNonResolvingPromise and the reconstruction never settles (#751).

The probe in (3) walks every recorded operation and requires all of them to be terminal. Reading only the first is unsound — a mid-flight item can have a SUCCEEDED first operation, and re-driving it would execute its unfinished operations for real.

Two payload shapes are recognised, since ReplayChildren is reached two ways: the summary envelope carrying itemStatuses, and the full serialized result written when childOperationsDepth opts a context into child preservation, which has no summary envelope but does carry all: [{ index, status }]. Parsing is validated as strictly as parseRecordedCompletionReason.

There is deliberately no size handling on the record. Every batch item runs its body inside a context, and a context is meant to wrap at least one durable operation, so each item costs at least one. The service caps an execution at 30,000 operations, which bounds any batch at roughly that many items — about 30 KB of markers at one byte each, an order of magnitude inside the 256 KB checkpoint limit. Capping the marker count instead would be actively harmful: dropping the field sends replay back to probing, and probing cannot recover an item whose body created no durable operation, which is the corruption this record exists to prevent. A test pins the bound at the largest batch the service allows.

Behaviour change

A virtual item with no durable operation of its own is now re-driven on replay, so any part of its mapper body not wrapped in a durable operation runs again. This is the documented contract for virtual contexts and what NESTED already does, but it is new on this path.

That shape is against best practice regardless — a context is a container for durable operations, and one with nothing durable inside it is not checkpointed, so its work re-runs on every replay anyway. The TSDoc for DurableContext.runInChildContext and MapFunc now says so, and #896 tracks an eslint rule for it.

Testing

New examples/map/flat-summarized-replay covering the rebuild across a suspension for FLAT and NESTED, an item that checkpoints no durable operation, and a small-result control. It is localOnly: the suspension depends on the local runner's virtual clock, so deploying it would only produce a skipped cloud test. Every case asserts at least two invocations, so none can pass without actually suspending.

Against unpatched code both FLAT cases fail (replayedResultCount 0 against a live count of 8) while the controls pass. 14 new unit cases cover both directions of a statuses/record disagreement, the childOperationsDepth shape, a crafted itemStatuses value, the probe rejecting a mid-flight item, and the size bound at the service's operation limit.

The PENDING misreport that hid this was fixed separately on mainline by #867; this addresses the underlying corruption.

yaythomas

summary-generators.ts:20,64 allows 262,144 status markers, but the complete summary JSON is 262,297 bytes, which 153 bytes over the documented 256 KB maximum. The cap should reserve envelope headroom or check the final serialized size.

This only affects roughly quarter-million-item batches.

ParidelPooya · 2026-09-03T00:00:35Z

Good catch, and the arithmetic is exact — I reproduced 262,297 bytes against the 262,144 limit.

Fixed by checking the finished payload rather than reserving headroom, because a marker cap can only ever approximate the envelope, and composeSummaryGenerator appends a custom summaryGenerator's output afterwards, which can be any length. stringifyWithinLimit serializes the record and, if it exceeds CHECKPOINT_SIZE_LIMIT_BYTES, drops itemStatuses and re-serializes. Replay then falls back to probing, which is the degraded-but-correct path.

Where a custom summary is what pushes it over, the customer's string is kept and itemStatuses is dropped — the custom value is the documented contract, ours is an optimisation.

The old cap survives only as a cheap allocation guard (one byte per marker, so a run longer than the whole limit cannot fit inside any envelope) and is now expressed in terms of CHECKPOINT_SIZE_LIMIT_BYTES rather than a literal.

Three tests, asserting Buffer.byteLength of the finished payload:

markers exactly at the limit -> payload within limit, itemStatuses absent, completionReason retained
1,000 items -> markers present
large custom summary -> markers dropped, custom string intact

The first one fails at 262,297 against the previous behaviour, so it pins your case directly.

yaythomas · 2026-09-03T00:17:18Z

Dropping itemStatuses reintroduces the original replay corruption

summary-generators.ts:22-36 says dropping the field is safe because replay probes checkpoints. However, concurrent-execution-handler.ts:445-446 explicitly documents that probing cannot recover a completed FLAT item whose mapper created no durable operations.

I reproduced this with a valid dense 262,144-item FLAT batch:

Live result: 262,144 successful items
Final summary drops itemStatuses to remain under 256 KB
Replay finds no item contexts or child operations
Rebuilt result: 0 items

this replacement creates the same correctness failure at the size boundary.

…AT map items A batch whose aggregate result exceeds the checkpoint size limit is not stored whole. It is checkpointed as a summary with ContextOptions.ReplayChildren, and every later resumption rebuilds it from the per-item checkpoints in ConcurrencyController.replayItems. replayItems decided whether an item had finished by probing the item's CONTEXT checkpoint. With `nesting: FLAT` those per-item contexts are virtual and are never checkpointed, so the probe found nothing for every item, each one was skipped, and the rebuilt BatchResult came back EMPTY even though all items had succeeded. That is not cosmetic. Code deriving control flow from the batch result -- for example a fan-out sized from the results -- creates a different set of operations on replay than it did live. The replay-consistency check detects the divergence and raises NonDeterministicExecutionError, which SDKs at or below 2.3.0 reported as a PENDING response rather than a failure. While other work is still draining, PENDING is a legal answer and the corruption stays invisible; once nothing is left pending the service rejects the response outright with "Cannot return PENDING status with no pending operations", retries deterministically, and fails the execution. Only large batches are affected, since a result that fits in a single checkpoint is replayed by deserialization and never enters the rebuild path. The batch payload now records which items reached a terminal state, one character per item in index order (encodeItemStatuses). Only terminality is recorded: an item's SUCCEEDED/FAILED outcome comes from re-driving it. Terminality is resolved in order of authority: 1. The recorded per-item statuses, and any context record the item has, which must agree. The statuses are by construction what the live BatchResult contained, so an "unfinished" marker overrides even a terminal context record: a child that settles after the batch completed early has a terminal record but was never part of the live result, and rebuilding it would add an item the live run never reported. The agreement requirement is deliberately one-sided, because the two directions fail very differently. Marker unfinished against a terminal record means the item is skipped, which is what the live result did. Marker terminal against an unfinished record, if honoured, would re-drive the item -- and in ReplaySucceededContext runInChildContext goes through checkForNonResolvingPromise, which returns a never-resolving promise when the pending step id has a non-terminal checkpoint. The reconstruction would never settle, the invocation would time out, and the retry would land in the same state (issue #751). So a terminal marker is honoured only when the checkpoints do not contradict it. FLAT items have no context record, so this costs them nothing. 2. The item's own CONTEXT checkpoint, for payloads with no recorded statuses. 3. A probe of the item's operations, for payloads predating the field. It walks every recorded operation and requires all of them to be terminal, because an item that was mid-flight can have a SUCCEEDED first operation and must not be misread as finished -- re-driving it would execute its unfinished operations for real and duplicate their side effects. Recording the statuses is what makes the two cases no probe can settle decidable: the mid-flight multi-operation item above, and an item whose body creates no durable operation and therefore leaves no trace at all. Two payload shapes are recognised, because ReplayChildren is reached two ways: the summary envelope carrying `itemStatuses`, and the full serialized BatchResult written when childOperationsDepth opts a context into child preservation. That path never runs the summary generator, but its payload already carries `all: [{ index, status }]`, so it is covered rather than left on the probe. Parsing is validated as strictly as parseRecordedCompletionReason -- Object.hasOwn plus a marker-alphabet check -- since legacy payloads can carry arbitrary JSON and the consequence here is terminality rather than a display field. The encoding is omitted above a length budget so it cannot become a second size problem in the payload used precisely because the real result was too large; replay falls back to the probe there. One behaviour change on this path: a virtual item with no durable operation of its own is now re-driven on replay, so any part of its mapper body not wrapped in a durable operation runs again. This matches the documented contract for virtual contexts and what NESTED already does, but it is new here and is called out in the code and CHANGELOG. Adds examples/map/flat-summarized-replay covering the rebuild across a suspension for FLAT and NESTED, an item that checkpoints no durable operation, and a small-result control. It is localOnly: the suspension depends on the local runner's virtual clock, so deploying it would only produce a skipped cloud test. Against the unpatched code both FLAT cases fail (replayedResultCount 0 against a live count of 8) while the controls pass. Unit tests cover both directions of a statuses/record disagreement, the childOperationsDepth payload shape, a crafted itemStatuses value, the probe rejecting a mid-flight multi-operation item, and the size budget.

ParidelPooya · 2026-09-03T00:40:37Z

Removed the size handling rather than fixing it — you were right that dropping itemStatuses just trades one corruption for another, since probing cannot recover an item whose body created no durable operation.

The reason no size handling is needed: every batch item runs its body inside a context, and a context is a container for durable operations, so each item costs at least one. The service caps an execution at 30,000 operations, which bounds any batch at roughly that many items — about 30 KB of markers at one byte each, an order of magnitude inside the 256 KB checkpoint limit. A test pins that at the limit rather than leaving it as a comment.

For completeness on the shape that motivated the cap: a mapper doing nothing durable does cost 0 operations per item (both the START and SUCCEED checkpoints are skipped for a virtual context, so a pure-transform FLAT map is ~1 operation regardless of item count, and I measured 4 operations for 8 items versus 12 with a step). So the operation limit does not bound item count there. But that shape is against best practice — the work is not checkpointed and re-runs on every replay anyway — so it should not shape the design. The TSDoc for runInChildContext and MapFunc now documents the rule, and #896 tracks an eslint rule to enforce it.

For the record, the exact boundary was 261,992 items, not 262,144: the markers share the payload with a 153-byte envelope, which is why the old cap sat 152 items above the real threshold.

Bumps @aws/durable-execution-sdk-js from 2.3.0 to 2.3.1. This PR exists to cut the release tag for what has already landed on main; the only changes are the version field, the testing package's peer-dependency upper bound, and the lockfile entries for both. Patch rather than minor: all three SDK changes since sdk-2.3.0 are fixes, and none adds public API. The checkpoint payload gains one field (#894), which is compatible in both directions — an older SDK ignores it, and a newer SDK falls back to probing on payloads written without it. **Fixed** - Summarized replay of a FLAT-nested map or parallel rebuilt an EMPTY result even though every item had succeeded (#894). Such a batch is checkpointed as a summary and reconstructed from its per-item checkpoints on each later resumption, but the reconstruction decided whether an item had finished by probing the item's context checkpoint — and with FLAT nesting those per-item contexts are virtual and never checkpointed, so every item read as unfinished and was skipped. Code deriving control flow from the batch result then created a different set of operations on replay than it had live, which replay validation reported as a non-determinism failure. The batch payload now records which items reached a terminal state and replay reads that record. One behaviour change on this path: a virtual item that performs no durable operation of its own is now re-driven, so any part of its mapper body outside a durable operation runs again — the same as NESTED already does, and a shape the TSDoc now discourages. - Replay validation now answers FAILED instead of PENDING (#867). A NonDeterministicExecutionError is the only production error carrying TerminationReason.CUSTOM, and runHandler had no branch for it, so it fell through to the catch-all and answered {Status: "PENDING"} with no error and no log line. PENDING asks the service to keep the execution alive and invoke again, but the fault is deterministic: every replay reproduced it, the diagnostic the SDK had already built never reached the customer, and once nothing was actually pending the service rejected the response with "Cannot return PENDING status with no pending operations". Suspend reasons are now an allowlist and every other reason answers FAILED carrying the error, so a reason added later without a branch fails closed instead of silently claiming progress. - wait-for-condition surfaces serdes failures instead of silently restarting (#754). The handler used a bare catch around a native serdes.deserialize, so a checkpointed polling state that could not be deserialized was swallowed and the condition loop restarted from config.initialState. It now uses safeDeserialize, which logs, rethrows SerDesException and terminates through the termination manager, matching the Java and Python SDKs.

Bumps @aws/durable-execution-sdk-js from 2.3.0 to 2.3.1. This PR exists to cut the release tag for what has already landed on main; the only changes are the version field, the testing package's peer-dependency upper bound, the lockfile entries for both, and a CHANGELOG roll. The CHANGELOG had not been rolled since 2.0.0, so `[Unreleased]` was carrying three releases' worth of notes at once: the runtimes-without-AsyncLocalStorage entry that shipped in 2.3.0, the eslint-plugin 1.1.0 entries that shipped separately, and this release's fixes. Those are now under `[2.3.0]`, `[eslint-plugin 1.1.0]` and `[2.3.1]`, with `[Unreleased]` left empty. No wording changed; the one addition is an entry for #867, which had none. Note that 2.1.0 through 2.3.0 remain only partially documented here — the full notes for those releases live in their bump PR bodies (#822 and earlier) and were never copied into this file. Reconstructing them is left for a follow-up. Patch rather than minor: all three SDK changes since sdk-2.3.0 are fixes, and none adds public API. The checkpoint payload gains one field (#894), which is compatible in both directions — an older SDK ignores it, and a newer SDK falls back to probing on payloads written without it. **Fixed** - Summarized replay of a FLAT-nested map or parallel rebuilt an EMPTY result even though every item had succeeded (#894). Such a batch is checkpointed as a summary and reconstructed from its per-item checkpoints on each later resumption, but the reconstruction decided whether an item had finished by probing the item's context checkpoint — and with FLAT nesting those per-item contexts are virtual and never checkpointed, so every item read as unfinished and was skipped. Code deriving control flow from the batch result then created a different set of operations on replay than it had live, which replay validation reported as a non-determinism failure. The batch payload now records which items reached a terminal state and replay reads that record. One behaviour change on this path: a virtual item that performs no durable operation of its own is now re-driven, so any part of its mapper body outside a durable operation runs again — the same as NESTED already does, and a shape the TSDoc now discourages. - Replay validation now answers FAILED instead of PENDING (#867). A NonDeterministicExecutionError is the only production error carrying TerminationReason.CUSTOM, and runHandler had no branch for it, so it fell through to the catch-all and answered {Status: "PENDING"} with no error and no log line. PENDING asks the service to keep the execution alive and invoke again, but the fault is deterministic: every replay reproduced it, the diagnostic the SDK had already built never reached the customer, and once nothing was actually pending the service rejected the response with "Cannot return PENDING status with no pending operations". Suspend reasons are now an allowlist and every other reason answers FAILED carrying the error, so a reason added later without a branch fails closed instead of silently claiming progress. - wait-for-condition surfaces serdes failures instead of silently restarting (#754). The handler used a bare catch around a native serdes.deserialize, so a checkpointed polling state that could not be deserialized was swallowed and the condition loop restarted from config.initialState. It now uses safeDeserialize, which logs, rethrows SerDesException and terminates through the termination manager, matching the Java and Python SDKs.

ParidelPooya had a problem deploying to ai-pr-review September 2, 2026 21:27 — with GitHub Actions Error

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from b8000c0 to 6eabbe6 Compare September 2, 2026 22:03

ParidelPooya had a problem deploying to ai-pr-review September 2, 2026 22:03 — with GitHub Actions Error

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from 6eabbe6 to 3b01850 Compare September 2, 2026 22:53

ParidelPooya had a problem deploying to ai-pr-review September 2, 2026 22:53 — with GitHub Actions Error

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from 3b01850 to 51d0230 Compare September 2, 2026 23:10

ParidelPooya had a problem deploying to ai-pr-review September 2, 2026 23:10 — with GitHub Actions Error

ParidelPooya changed the title ~~fix(sdk): rebuild all items when replaying a summarized FLAT-nested map~~ fix(sdk): record per-item completion so summarized replay rebuilds FLAT map items Sep 2, 2026

yaythomas reviewed Sep 2, 2026

View reviewed changes

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from 51d0230 to 6da1d57 Compare September 2, 2026 23:59

ParidelPooya had a problem deploying to ai-pr-review September 2, 2026 23:59 — with GitHub Actions Error

ParidelPooya mentioned this pull request Sep 3, 2026

eslint-plugin: flag a child context whose body performs no durable operation #896

Closed

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from 6da1d57 to a712c3a Compare September 3, 2026 00:39

ParidelPooya had a problem deploying to ai-pr-review September 3, 2026 00:40 — with GitHub Actions Error

ParidelPooya force-pushed the fix/flat-map-summarized-replay-rebuild branch from a712c3a to ee4456b Compare September 3, 2026 00:40

ParidelPooya had a problem deploying to ai-pr-review September 3, 2026 00:40 — with GitHub Actions Error

yaythomas approved these changes Sep 3, 2026

View reviewed changes

ParidelPooya marked this pull request as ready for review September 3, 2026 00:54

ParidelPooya requested a deployment to ai-pr-review-runtime September 3, 2026 00:54 — with GitHub Actions Waiting

ParidelPooya had a problem deploying to ai-pr-review-runtime September 3, 2026 00:54 — with GitHub Actions Failure

Merge branch 'main' into fix/flat-map-summarized-replay-rebuild

3dfbe70

ParidelPooya merged commit 5db700c into main Sep 3, 2026
41 checks passed

ParidelPooya deleted the fix/flat-map-summarized-replay-rebuild branch September 3, 2026 02:30

ParidelPooya mentioned this pull request Sep 3, 2026

chore(sdk): bump core SDK to 2.3.1 #897

Merged

yaythomas mentioned this pull request Sep 3, 2026

fix(core): validate replay operation identity aws/aws-durable-execution-sdk-python#698

Merged

ParidelPooya had a problem deploying to ai-pr-review-runtime October 3, 2026 00:54 — with GitHub Actions Error

Navigation Menu

Repository navigation

fix(sdk): record per-item completion so summarized replay rebuilds FLAT map items - #894

fix(sdk): record per-item completion so summarized replay rebuilds FLAT map items#894
ParidelPooya merged 2 commits into
mainfrom
fix/flat-map-summarized-replay-rebuild

ParidelPooya commented Sep 2, 2026 •

edited

Loading

Uh oh!

yaythomas left a comment

Uh oh!

ParidelPooya commented Sep 3, 2026

Uh oh!

yaythomas commented Sep 3, 2026

Uh oh!

ParidelPooya commented Sep 3, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ParidelPooya commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Behaviour change

Testing

Uh oh!

yaythomas left a comment

Choose a reason for hiding this comment

Uh oh!

ParidelPooya commented Sep 3, 2026

Uh oh!

yaythomas commented Sep 3, 2026

Uh oh!

ParidelPooya commented Sep 3, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ParidelPooya commented Sep 2, 2026 •

edited

Loading