fix(sdk): suspend with PENDING when a checkpoint response has no token - #931

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Closes #918.

Description

SDK: suspend when a checkpoint response has no token

1. Each checkpoint response carries the token for the next checkpoint call.
2. A response without a token means the service accepts no further checkpoints from this invocation. For example, a newer invocation has taken over the execution.
3. Before this change, the checkpoint manager ignored the missing token. It kept the token it had just sent, and it sent that token again on the next call.
4. The service rejects a reused token with InvalidParameterValueException: Invalid checkpoint token. Since fix(sdk): classify stale checkpoint token as invocation failure #920, the SDK classifies that error as an invocation error.
5. So the invocation ended with a Lambda error one call later.

After this change, the checkpoint manager does the following when a response has no token:

• It sends no further checkpoint calls. It removes every queued checkpoint without sending it.
• It logs a warning through the customer logger. An absent token looks the same as a client that dropped the field. So the warning is the only record of which one happened.
• It terminates with a new termination reason, EXECUTION_SUSPENDED_BY_SERVICE. That reason is classified as a suspend. So withDurableExecution returns { Status: "PENDING" } through the existing suspend path.
• It does not apply NewExecutionState from that response. It does not resolve the callers of the accepted checkpoint. Either action would let the handler continue after the service has stopped recording. For example, the handler would start its next step. That step would run once in this invocation and again in the next one.

The checkpoint that received the token-less response was accepted. So its updates are durable. Updates that the SDK had not yet sent are lost. They replay on the next invocation.

One case is exempt:

1. The SDK checkpoints a result larger than 6 MB as an EXECUTION update after the handler returns.
2. That update finishes the execution.
3. So a token-less response to a batch that contains an EXECUTION update returns normally, and the invocation reports SUCCEEDED.

SDK: a token withdrawn after the handler returns

1. A handler can return, or throw, while a checkpoint it did not await is still in flight. The unfinished branches of a map or parallel with early completion are one example.
2. If the service answers that checkpoint without a token, it will record nothing further from this invocation. That includes the invocation's own result.
3. Before this change, the invocation still returned SUCCEEDED with the result, or FAILED. main has the same behaviour.
4. So withDurableExecution now checks for a withdrawn token before it returns SUCCEEDED or FAILED. If the token was withdrawn, it returns PENDING. The next invocation replays and reaches the outcome again.

The oversized-result checkpoint had a related defect:

1. withDurableExecution awaits that checkpoint after it has already chosen between the handler result and a termination. So a termination could not end that await.
2. The checkpoint can wait in the queue behind another batch. If that batch fails, or is answered without a token, the checkpoint manager clears the queue. The result is never sent, and the await never settles.
3. So the invocation ran until the Lambda timeout. main has the same behaviour for the failed case.
4. The checkpoint manager now reports whether it sent that update. An update the service accepted reports SUCCEEDED, even if a termination arrived while it was in flight, because the execution is then recorded as finished. An update it never sent answers from the termination: a withdrawn token gives PENDING, and a failed checkpoint throws its classified error.

SDK: one place decides the answer

Four answers disagreed depending on which exit the invocation took. One helper now decides them all, in a fixed order: a withdrawn token answers PENDING, a checkpoint failure throws its classified error, a serdes failure throws SerdesFailedError, and every other reason follows its class.

• A checkpoint that failed after the handler returned answered SUCCEEDED for a small result and threw for an oversized one. Both now throw.
• A serdes failure on the oversized-result path threw a plain Error, so the execution failed for good instead of the invocation being retried.
• A fault termination followed by a withdrawn token answered FAILED. It now answers PENDING.
• An unrecoverable invocation error was rethrown after the token was withdrawn, so Lambda retried with a token the service had already rejected. It now answers PENDING.

A PENDING that follows a handler error now carries that error to plugins.

Testing SDK: pause and resume (experimental)

LocalDurableTestRunner gains pauseExecution() and resumeExecution(). Both are marked @experimental.

• pauseExecution() marks the execution paused on the local checkpoint server. The server then accepts each checkpoint and returns it without a token. So the running invocation suspends with PENDING at its next checkpoint.
• While the execution is paused, the runner starts no invocation. Waits still elapse. Callbacks can still be sent. The runner defers the invocations that they would start.
• pauseExecution() resolves once no invocation is running.
• resumeExecution() clears the paused state. It then starts the deferred invocation, if one exists.

Both methods are also on the DurableTestRunner interface. So a test written against the interface compiles for either runner. CloudDurableTestRunner rejects both methods with a not-implemented error. It sends no request to the service, because the service has no API to withhold checkpoint tokens.

Adding the methods to the interface is a compile-time break for any external class that implements DurableTestRunner. Such a class must add both methods. The two runners in this repository are the only known implementations.

Examples

• A new example, src/examples/pause-resume, shows both methods. Its config sets localOnly: true. So the example is left out of the SAM template, and its tests are skipped under NODE_ENV=integration.
• tsc --noEmit in the examples package failed in any checkout where npm run generate-examples had not run. The cause is that examples-catalog.ts imports the generated, gitignored examples-catalog.json. A new declaration file, examples-catalog.d.json.ts, now gives tsc the types through allowArbitraryExtensions. The TypeScript Rollup plugin resolves the import to that declaration instead of the JSON. So rollup.config.mjs resolves the import to the real JSON before that plugin runs.

Open items from review

This pull request is a draft. An independent review found the items below. None of them blocks the SDK fix.

Review pass 1 found SUCCEEDED after a withdrawn token and an oversized result that hung. Commit a7b4dc4 fixes both. Review pass 2 found that a7b4dc4's own race abandoned a checkpoint the service accepted, plus four answers that disagreed between exits. Commit 0bb808b fixes those.

1. A pauseExecution() call made while resumeExecution() is in flight resolves without pausing the execution.
2. If the checkpoint server rejects a pause request, every later pauseExecution() and resumeExecution() call rejects. run() then does not settle.
3. tsc now reads the catalog type from examples-catalog.d.json.ts. So tsc no longer checks the generated catalog against ExamplesWithConfig.

Conformance

AGENTS.md places conformance tests in a separate repository. This behaviour warrants a conformance requirement: a checkpoint response without a token ends the invocation with PENDING, and the SDK makes no further checkpoint call.

Merge

Please squash-merge. The first commit adds a setup option, withholdCheckpointTokenOnCall. A later commit removes it and adds pauseExecution() and resumeExecution() instead.

Demo/Screenshots

Not applicable.

Checklist

• I have filled out every section of the PR template
• I have thoroughly tested this change

Testing

Unit Tests

Have unit tests been written for these changes?

Yes.

• checkpoint-termination.test.ts checks the termination reason and the warning. It checks that the manager makes no further call, including for a forced checkpoint. It checks that the caller of the accepted checkpoint stays unresolved.
• The testing SDK tests check the server's pause and resume handlers and their dispatch. They check that the server omits the token while the execution is paused and keeps the updates.
• A CloudDurableTestRunner test checks that both methods reject and send no request.

Integration Tests

Have integration tests been written for these changes?

Yes.

• checkpoint-token-revoked.composed.test.ts runs through withDurableExecution. It checks the PENDING result. It checks that the spent token is never sent again. It checks that accepted checkpoints are kept and that the next step does not start. It checks that an oversized result still reports SUCCEEDED. It checks that a token withdrawn after the handler returns or throws gives PENDING, including for an oversized result queued behind the withdrawn batch. It checks that an oversized result the service accepted reports SUCCEEDED even when a termination arrived while it was in flight. It checks that a fault termination followed by a withdrawn token answers PENDING.
• checkpoint-failure.composed.test.ts checks that a failed oversized-result checkpoint throws its classified error instead of waiting until the Lambda timeout. It checks that a failed checkpoint and a serdes failure give the same answer whatever the result size.
• with-durable-execution.plugin.test.ts checks that a PENDING carrying an error reports it to plugins.
• I reverted each fix in turn and confirmed that a test failed each time.
• checkpoint-composed.test.ts had a mock response with the field checkpointToken instead of CheckpointToken. main ignored a missing token, so the typo had no effect there. With this change, those tests waited for a checkpoint that never resolved. The mock now uses the field name of the real response.
• pause-resume-execution.integration.test.ts and pause-resume-execution-skip-time.integration.test.ts run against the local checkpoint server. They cover a pause during a step, a deferred callback invocation, a deferred wait invocation with skipTime, repeated calls, and the error when no run is in progress.
• I disabled each pause mechanism in the orchestrator in turn. At least one test failed each time.

Results:

• Core SDK: 1268 tests passed.
• Testing SDK: 1189 tests passed, 3 skipped.
• Examples: 187 Jest tests passed, 1 skipped. The skipped suite was already skipped on main. 2 Vitest tests passed.
• npm run lint reports no errors. tsc --noEmit passes in all three packages.

Examples

Has a new example been added for the change? (if applicable)

Yes. src/examples/pause-resume has two tests, each with a history file. One test pauses during a step. The other pauses while the workflow waits for a callback.