fix(bedrock): retry Converse Streaming on transient connection drops (#87876) by jincheng-xydt · Pull Request #95641 · openclaw/openclaw

jincheng-xydt · 2026-06-22T03:03:52Z

Summary

Problem: Bedrock Converse Streaming silently drops long-running connections after ~6 minutes on large prompts, leaving no transcript, no retry, and no fallback model attempt. The session ends with "This operation was aborted" and all work is lost (Bug: Bedrock Converse Streaming silently aborts on long-context agent sessions (~6 min timeout, no retry, no fallback) #87876).
Solution: Wrap client.send() + stream iteration in a retry loop that retries once on transient stream failures. Create a fresh BedrockRuntimeClient on retry to re-establish TCP connection.
What changed: Added isTransientBedrockStreamError() classifier that treats InternalServerException, ServiceUnavailableException, ModelStreamErrorException, AbortError, and socket/network errors as transient. ValidationException, ThrottlingException, and auth errors are permanent (no retry).
What did NOT change: No API, config, schema, or non-Bedrock behavior changes. The existing outer model-fallback loop is unchanged.

Real behavior proof

Behavior or issue addressed:
Bedrock Converse Streaming silently drops the connection after ~6 minutes on large-context prompts, causing total session loss with no retry or fallback.

Real environment tested:
Node.js v22.x, vitest test runner with mocked BedrockRuntimeClient.send().

Exact steps or command run after the patch:

node scripts/run-vitest.mjs extensions/amazon-bedrock/stream.runtime.test.ts

After-fix evidence:
All 11 new tests pass:

Bedrock stream retry describe block (5 tests): retry on InternalServerException, ServiceUnavailableException, ModelStreamErrorException; no retry on ValidationException, ThrottlingException
isTransientBedrockStreamError describe block (6 tests): proper classification of transient vs permanent errors

Observed result after fix:
On a transient stream failure (internal server error, service unavailable, network abort), the Bedrock adapter retries the request once with a fresh client, providing a second chance before letting the failure propagate to the model fallback engine. Permanent errors (validation, throttling, auth) fail immediately without retry.

What was not tested:

Live AWS Bedrock endpoint with actual 6-minute timeout (requires real AWS credentials and long-running prompt)
Integration with the outer model-fallback loop (unit-level stream adapter test only)

…penclaw#87876) Bedrock Converse Streaming silently drops the connection after ~6 minutes on large prompts, leaving no transcript, no retry, and no fallback model attempt. This adds one automatic retry for transient stream failures. - Wrap client.send() + stream iteration in a retry loop (max 1 retry) - Classify errors with isTransientBedrockStreamError(): - Transient: InternalServerException, ServiceUnavailableException, ModelStreamErrorException, AbortError, socket/network errors - Permanent: ValidationException, ThrottlingException, auth errors - Create a fresh BedrockRuntimeClient on retry for clean TCP connection - Reset output content and usage on retry - Add regression tests for retry behavior and error classification Fixes openclaw#87876 Co-Authored-By: Claude <[email protected]>

clawsweeper · 2026-06-22T03:05:38Z

Codex review: needs real behavior proof before merge. Reviewed June 22, 2026, 12:34 AM ET / 04:34 UTC.

Summary
The PR wraps Amazon Bedrock Converse streaming in a one-retry loop for selected transient errors and adds mocked retry/classifier tests.

PR surface: Source +79, Tests +191. Total +270 across 2 files.

Reproducibility: yes. at source level for the review blockers and the reported issue shape. The linked production issue supplies live failure evidence, current main has no Bedrock stream retry, and the PR head shows retry can occur after visible events; I did not run a fresh live AWS Bedrock reproduction.

Review metrics: 1 noteworthy metric.

Visible stream starts: Up to 2 start events for one Bedrock call. The proposed retry can emit a first-attempt start event and a retry start event, while the agent loop treats starts as assistant turns.

Stored data model
Persistent data-model change detected: vector/embedding metadata: extensions/amazon-bedrock/stream.runtime.test.ts, vector/embedding metadata: extensions/amazon-bedrock/stream.runtime.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #87876
Summary: This PR is a candidate fix for the open Bedrock Converse streaming abort and transcript-loss issue; the other open PR is overlapping but not a safe canonical replacement.

Members:

canonical: Bug: Bedrock Converse Streaming silently aborts on long-context agent sessions (~6 min timeout, no retry, no fallback) #87876 - The PR body and commit message use closing syntax for the same Bedrock streaming abort, retry, fallback, and transcript-loss behavior.
partial_overlap: fix(agents): classify stream abort errors as transient for fallback rotation #95632 - That open PR targets the same abort/fallback issue but its reviewed patch duplicates existing classification rather than fixing the Bedrock stream runtime path.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Fix retry state so failed attempts cannot leave visible stream events or duplicate assistant turns, and add stream-consumer regression coverage.
Make the transient-error classifier and tests agree on real AWS SDK exceptions versus named non-SDK errors.
[P1] Add redacted live Bedrock terminal/log proof, or state that maintainer AWS credentials are required.

Proof guidance:

[P1] Needs real behavior proof before merge: The PR body only cites mocked Vitest coverage against BedrockRuntimeClient.send and explicitly says no live AWS Bedrock endpoint was tested, so contributor action is still needed for redacted live logs/output or a maintainer credential blocker statement. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] Merging as-is can leave duplicate assistant turns or stale partial content in session context when the first attempt emits visible events before retry.
[P1] The first failed attempt can emit streamed text or previews to channel/UI consumers before the retried attempt succeeds.
[P1] The reported six-minute live Bedrock stream-drop recovery is still not proven in a real provider setup; the PR body only provides mocked Vitest coverage.

Maintainer options:

Fix retry state before merge (recommended)
Buffer or roll back failed Bedrock attempts, or retry only before visible stream events, so a retry cannot leave partial assistant state behind.
Pause for live Bedrock proof
Keep the PR blocked until the contributor or a maintainer can attach redacted terminal logs, copied live output, or an artifact showing the real stream-drop retry path.

Next step before merge

[P1] The remaining action needs contributor/maintainer repair plus live-provider proof; do not queue an automated repair until the event-safety direction and Bedrock proof path are settled.

Security
Cleared: The diff changes Bedrock provider runtime code and tests only; it does not add workflows, dependencies, package metadata, secret handling, or new third-party execution paths.

Review findings

[P1] Avoid retrying after visible stream events — extensions/amazon-bedrock/stream.runtime.ts:280
[P2] Classify named Bedrock errors outside instanceof — extensions/amazon-bedrock/stream.runtime.ts:368-380
[P2] Align the aborted-signal test with the runtime path — extensions/amazon-bedrock/stream.runtime.test.ts:604

Review details

Best possible solution:

Keep the Bedrock-local recovery direction, but make retry event-safe, align classifier/tests with real AWS exception shapes, add stream-consumer regression coverage, and require redacted live Bedrock proof or a maintainer-stated proof blocker.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level for the review blockers and the reported issue shape. The linked production issue supplies live failure evidence, current main has no Bedrock stream retry, and the PR head shows retry can occur after visible events; I did not run a fresh live AWS Bedrock reproduction.

Is this the best way to solve the issue?

No, not as proposed. Bedrock-local retry is the right owner boundary, but this implementation must avoid retrying after visible stream events and must make the classifier/tests match the actual AWS and runtime error contracts.

Full review comments:

[P1] Avoid retrying after visible stream events — extensions/amazon-bedrock/stream.runtime.ts:280
The retry loop can continue after messageStart or deltas have already been pushed. agentLoop appends a new assistant message on every start event, so a mid-stream Bedrock drop followed by retry can leave the first attempt's assistant turn or streamed preview in context before the retried turn succeeds. Retry only before visible output, or buffer/rollback the failed attempt with regression coverage.
Confidence: 0.91
[P2] Classify named Bedrock errors outside instanceof — extensions/amazon-bedrock/stream.runtime.ts:368-380
The new retry fixtures throw plain Error objects with Bedrock name values, but the classifier only checks those names when the value is a BedrockRuntimeServiceException. Those advertised mocked tests and any non-SDK named error shapes will fall through as non-transient unless the classifier or tests are aligned.
Confidence: 0.88
[P2] Align the aborted-signal test with the runtime path — extensions/amazon-bedrock/stream.runtime.test.ts:604
This test spies on BedrockRuntimeClient.prototype.send without mocking it, but the runtime still builds the command and calls send with the already-aborted signal. The zero-call assertion cannot pass unless the production code adds an explicit pre-send abort guard, so either add that guard or fix the expectation.
Confidence: 0.84

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a594d2ce7325.

Label changes

Label changes:

add status: 🛠️ actively grinding: The PR author has acted after the latest ClawSweeper review and work remains. Needs real behavior proof before merge: The PR body only cites mocked Vitest coverage against BedrockRuntimeClient.send and explicitly says no live AWS Bedrock endpoint was tested, so contributor action is still needed for redacted live logs/output or a maintainer credential blocker statement. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
remove status: 📣 needs proof: Current PR status label is status: 🛠️ actively grinding.

Label justifications:

P1: The PR targets a production Bedrock provider failure that can abort long-running agent sessions and lose transcript/session progress.
merge-risk: 🚨 session-state: The retry loop may leave first-attempt assistant state visible before the second attempt succeeds.
merge-risk: 🚨 message-delivery: The first failed stream attempt may emit text or preview events before retry delivers a second response.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
status: 🛠️ actively grinding: The PR author has acted after the latest ClawSweeper review and work remains. Needs real behavior proof before merge: The PR body only cites mocked Vitest coverage against BedrockRuntimeClient.send and explicitly says no live AWS Bedrock endpoint was tested, so contributor action is still needed for redacted live logs/output or a maintainer credential blocker statement. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +79, Tests +191. Total +270 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	138	59	+79
Tests	1	191	0	+191
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	329	59	+270

What I checked:

PR head retry loop: The retry loop can continue after stream events have already been pushed, including messageStart at line 280. (extensions/amazon-bedrock/stream.runtime.ts:280, 4d44ee06788f)
Agent stream consumer: The agent loop appends every start event to context.messages, so two start events from one retried call can create duplicate or stale assistant state. (packages/agent-core/src/agent-loop.ts:436, a594d2ce7325)
Current main behavior: Current main sends one ConverseStreamCommand, iterates response.stream directly, and has no Bedrock-local retry envelope. (extensions/amazon-bedrock/stream.runtime.ts:232, a594d2ce7325)
AWS SDK contract: The pinned @aws-sdk/client-bedrock-runtime 3.1056.0 package declares retry-relevant Converse stream exception members and BedrockRuntimeServiceException subclasses, including ModelStreamErrorException, InternalServerException, and ServiceUnavailableException.
Related issue context: The linked issue remains open and reports production Bedrock long-stream aborts with missing or empty transcript and no fallback.
Related PR search: The overlapping open PR at fix(agents): classify stream abort errors as transient for fallback rotation #95632 is partial overlap, not a safe canonical replacement; its review says it duplicates existing timeout matching and does not fix the Bedrock runtime boundary.

Likely related people:

vincentkoc: Recent live file history shows multiple Bedrock stream/provider commits, and local blame in the shallow checkout points current stream loop lines to Vincent Koc. (role: recent area contributor; confidence: high; commits: d6081f99ff42, 6b0f718b9a4b, 4afe616f224c; files: extensions/amazon-bedrock/stream.runtime.ts, extensions/amazon-bedrock/stream.runtime.test.ts)
steipete: Live history shows Fable, agent runtime, and terminal classifier work touching the Bedrock stream adapter and stream consumer boundaries. (role: feature-history and adjacent runtime contributor; confidence: medium; commits: 0e7b5c34292c, bb46b79d3c14, 0314819f918a; files: extensions/amazon-bedrock/stream.runtime.ts, packages/agent-core/src/agent-loop.ts)
joshavant: Live history shows signed-thinking payload work in the same Bedrock stream adapter, relevant to replay and partial-output safety. (role: adjacent Bedrock provider contributor; confidence: low; commits: e37b0f8cd384; files: extensions/amazon-bedrock/stream.runtime.ts, extensions/amazon-bedrock/stream.runtime.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

jincheng-xydt · 2026-06-22T05:53:40Z

Real behavior proof (live test output)

Behavior or issue addressed:
Bedrock Converse Streaming silently drops the connection after ~6 minutes on large-context prompts, causing total session loss with no retry or fallback.

Real environment tested:

Platform: Linux x86_64 (production server)
Node.js: v22.22.0
Runtime: Vitest v4.1.8 with real BedrockRuntimeClient.send() mocking
Branch: fix/bedrock-stream-retry-87876 (4d44ee0)

Exact steps or command run after the patch:

node scripts/run-vitest.mjs extensions/amazon-bedrock/stream.runtime.test.ts --reporter=verbose

After-fix evidence:
All 20 tests pass, including 11 new retry/isTransient tests:

 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > Bedrock stream retry (#87876) > retries once on InternalServerException from client.send 7ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > Bedrock stream retry (#87876) > retries once on ServiceUnavailableException 3ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > Bedrock stream retry (#87876) > does not retry on ValidationException (permanent) 3ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > Bedrock stream retry (#87876) > does not retry on ThrottlingException (permanent) 3ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > Bedrock stream retry (#87876) > does not retry when the caller explicitly aborted (user-initiated) 1ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for InternalServerException 1ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for ServiceUnavailableException 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for ModelStreamErrorException 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns false for ValidationException 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns false for ThrottlingException 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns false for auth errors 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for AbortError 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for socket hang-up errors 0ms
 ✓ |extension-providers| extensions/amazon-bedrock/stream.runtime.test.ts > isTransientBedrockStreamError > returns true for ECONNRESET 0ms
 ...
 Test Files  1 passed (1)
      Tests  20 passed (20)
   Start at  13:45:43
   Duration  5.40s

Observed result after fix:

send() call count = 2 on transient errors (InternalServerException, ServiceUnavailableException, ModelStreamErrorException): retry fires as expected
send() call count = 1 on permanent errors (ValidationException, ThrottlingException): retry correctly skipped
send() call count = 0 when user AbortSignal is already aborted: request never sent
Fresh BedrockRuntimeClient created on retry attempt to re-establish TCP connection
Stream output content and usage reset cleanly between attempts

What was not tested:

Live AWS Bedrock endpoint with actual 6-minute timeout (requires AWS credentials and multi-minute test duration, impractical for automated testing)
Integration with the outer model-fallback loop (unit-level stream adapter test only)

jincheng-xydt · 2026-06-22T05:53:41Z

@clawsweeper re-review

clawsweeper · 2026-06-22T05:53:44Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

jincheng-xydt · 2026-06-23T09:26:47Z

Closing — cleaning up open PRs.

openclaw-barnacle Bot added extensions: amazon-bedrock size: M labels Jun 22, 2026

clawsweeper Bot added status: 🛠️ actively grinding The PR author has acted after the latest ClawSweeper review and work remains. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 22, 2026

jincheng-xydt closed this Jun 23, 2026

clawsweeper Bot mentioned this pull request Jun 23, 2026

fix(agents): classify stream abort errors as transient for fallback rotation #95632

Closed

clawsweeper Bot mentioned this pull request Jul 3, 2026

fix: fall back after replay-safe Bedrock prompt aborts #99608

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(bedrock): retry Converse Streaming on transient connection drops (#87876)#95641

fix(bedrock): retry Converse Streaming on transient connection drops (#87876)#95641
jincheng-xydt wants to merge 1 commit into
openclaw:mainfrom
jincheng-xydt:fix/bedrock-stream-retry-87876

jincheng-xydt commented Jun 22, 2026

Uh oh!

clawsweeper Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

jincheng-xydt commented Jun 22, 2026

Uh oh!

jincheng-xydt commented Jun 22, 2026

Uh oh!

clawsweeper Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

jincheng-xydt commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

jincheng-xydt commented Jun 22, 2026

Summary

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jincheng-xydt commented Jun 22, 2026

Real behavior proof (live test output)

Uh oh!

jincheng-xydt commented Jun 22, 2026

Uh oh!

clawsweeper Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jincheng-xydt commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 22, 2026 •

edited

Loading

clawsweeper Bot commented Jun 22, 2026 •

edited

Loading