Skip to content

fix: fall back after replay-safe Bedrock prompt aborts#99608

Open
brian-bell wants to merge 5 commits into
openclaw:mainfrom
brian-bell:fix/bedrock-prompt-abort-fallback-87876
Open

fix: fall back after replay-safe Bedrock prompt aborts#99608
brian-bell wants to merge 5 commits into
openclaw:mainfrom
brian-bell:fix/bedrock-prompt-abort-fallback-87876

Conversation

@brian-bell

@brian-bell brian-bell commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Closes #87876

AI-assisted: yes

What Problem This Solves

Fixes an issue where users running long Bedrock agent sessions could lose the run when the Bedrock provider surfaced a provider-originated prompt abort. In the replay-safe prompt-abort shape from #87876, the run was marked aborted before the existing model fallback path could evaluate it, so configured fallback models were not tried.

Why This Change Was Made

The change lets only replay-safe provider prompt aborts enter the existing prompt failover path. It keeps user/external aborts, session timeouts, compaction timeouts, tool-execution timeouts, and attempts with possible side effects out of fallback so intentional cancellation and unsafe replay behavior stay unchanged.

User Impact

Bedrock users with configured fallback models can recover from this provider-originated prompt abort class instead of ending the whole agent run immediately. Runs that are not safe to replay still fail rather than duplicating side effects.

Evidence

Source-level proof:

  • node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts src/agents/model-fallback.run-embedded.e2e.test.ts
  • Structured autoreview on fix/bedrock-prompt-abort-fallback-87876: clean, no accepted/actionable findings.

Live Bedrock proof attempts, July 3, 2026:

  • AWS auth used profile-based resolution: AWS_PROFILE with AWS_SDK_LOAD_CONFIG=1. AWS key/session env was not exported to the child runs.
  • Direct Bedrock model smoke passed: node scripts/test-live.mjs --no-quiet -- src/agents/models.profiles.live.test.ts -t "completes across selected models" with OPENCLAW_LIVE_PROVIDERS=amazon-bedrock and OPENCLAW_LIVE_MODELS=amazon-bedrock/global.anthropic.claude-sonnet-4-6.
  • Gateway Bedrock smoke passed: node scripts/test-live.mjs --no-quiet -- src/gateway/gateway-models.profiles.live.test.ts -t "runs meaningful prompts across models with available keys" with OPENCLAW_LIVE_GATEWAY_PROVIDERS=amazon-bedrock and the same model. The prompt and tool-read probes both ended stopReason=stop.
  • Issue-specific long probe with Bedrock primary plus anthropic/claude-haiku-4-5 fallback, using a 66,188-byte replay-safe prompt: runId=29d923a4-4014-471a-bfa4-a3d6d78a02c6, status=ok, stopReason=stop; no embedded run failover decision, no model fallback decision, and no This operation was aborted.
  • Second issue-specific heavy probe with the same primary/fallback, using a 74,382-byte prompt: runId=2f21aaee-8341-4aab-a435-77735596e2f8, status=ok, stopReason=stop; again no abort/fallback.

Proof gap:

@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 20, 2026, 9:19 AM ET / 13:19 UTC.

Summary
The branch permits replay-safe, provider-owned Bedrock prompt aborts to enter existing model fallback while preserving user stops, timeouts, visible replies, and other replay-unsafe outcomes.

PR surface: Source +68, Tests +275. Total +343 across 5 files.

Reproducibility: no. high-confidence live reproduction is currently available: the source-level abort shape is covered by focused tests, but the contributor's 11 real Bedrock attempts did not trigger the provider-owned abort from the report.

Review metrics: none identified.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Provide a redacted trace showing provider-owned abort classification, configured fallback selection, and successful completion, or obtain explicit maintainer acceptance of the current proof standard.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The contributor supplied meaningful real Bedrock reachability and long-run evidence, but it does not show the changed provider-abort classification, fallback attempt, and recovered completion; add a redacted affected-environment trace or obtain a maintainer evidence-standard override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P2] Merging changes the terminal outcome for a narrow class of aborted agent runs; if the provider envelope is misclassified as replay-safe, a fallback could repeat an attempt after user-visible delivery or another side effect.
  • [P2] The PR's live Bedrock matrix proves reachability and normal long-run behavior, not the intended abort → fallback recovery chain, so green CI does not settle the session-state safety claim.

Maintainer options:

  1. Capture the actual recovery chain (recommended)
    Before merging, collect a redacted reporter or production trace that shows the provider-owned abort, replay-safe classification, fallback attempt, and successful terminal result.
  2. Accept the bounded replay contract
    A maintainer may accept the source-level guard and regression suite despite the missing real abort trace, explicitly owning the remaining provider-envelope uncertainty.

Next step before merge

  • [P1] The remaining blocker is a maintainer decision on whether source-level replay-safety proof can substitute for a rare affected-environment trace; no safe mechanical repair remains.

Maintainer decision needed

  • Question: Should this rare Bedrock provider-abort recovery require a real affected-environment abort-to-fallback trace before merge, or may the reviewed replay-safety guards plus focused tests and negative live probes satisfy the evidence bar?
  • Rationale: The contributor has exercised Bedrock organically across two platforms but cannot reproduce the provider-owned abort, and choosing whether to accept source-level safety proof is an evidence-standard decision rather than a mechanical repair.
  • Likely owner: steipete — Likely recent core-agent routing contact, though the available feature-history evidence is incomplete.
  • Options:
    • Require affected-environment trace (recommended): Keep the PR blocked until a redacted production or reporter trace shows provider-owned abort classification, fallback selection, and successful completion.
    • Accept bounded source proof: Accept the current predicate, replay-state coverage, focused tests, and negative live matrix as adequate for a rare envelope that cannot presently be reproduced.

Security
Cleared: The five-file TypeScript diff changes agent recovery and tests only; no dependency, workflow, secret, package-resolution, or supply-chain concern is evident.

Review details

Best possible solution:

Land only after a maintainer either obtains a redacted affected-environment trace proving the provider abort is replay-safe and completes on the configured fallback, or explicitly accepts the focused source guards and regression tests as sufficient for this rare provider envelope.

Do we have a high-confidence way to reproduce the issue?

No high-confidence live reproduction is currently available: the source-level abort shape is covered by focused tests, but the contributor's 11 real Bedrock attempts did not trigger the provider-owned abort from the report.

Is this the best way to solve the issue?

Yes, conditionally: reusing the existing model-fallback path with strict no-output, no-side-effect, no-timeout, and no-caller-stop gates is narrower than adding provider-specific retry logic, but the intended live envelope still needs maintainer acceptance or direct proof.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8d4d02a3cfa4.

Label changes

Label justifications:

  • P1: The unresolved behavior can end active Bedrock agent runs without trying a configured recovery model.
  • merge-risk: 🚨 session-state: This patch changes which aborted embedded-agent attempts may replay on a fallback, so a misclassification could repeat or alter a session outcome.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The contributor supplied meaningful real Bedrock reachability and long-run evidence, but it does not show the changed provider-abort classification, fallback attempt, and recovered completion; add a redacted affected-environment trace or obtain a maintainer evidence-standard override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +68, Tests +275. Total +343 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 3 73 5 +68
Tests 2 275 0 +275
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 348 5 +343

What I checked:

  • Current-main gap: Current main's embedded-session subscriber does not include the branch's delivered-visible-reply replay-state helpers or the heartbeat/silent-payload imports, so main does not contain the broader replay-safety implementation added by this PR. (src/agents/embedded-agent-subscribe.ts:279, 8d4d02a3cfa4)
  • Bounded fallback gate: The branch requires a configured fallback, an aborted but non-external and non-signal-owned attempt, no timeout state, no visible assistant output, replay-safe metadata, and no potential side effects before allowing the prompt-failure failover path. (src/agents/embedded-agent-runner/run/attempt-recovery.ts:259, ccc9500b014d)
  • Terminal-state preservation: The branch additionally permits the prompt failover condition after an abort only through the new replay-safe override, while retaining replay metadata and potential-side-effect guards in the shared prompt-failure path. (src/agents/embedded-agent-runner/run/prompt-failure.ts:140, ccc9500b014d)
  • Prior review blockers addressed: The PR discussion records fixes for the caller-abort race, visible streamed or block-reply output, and runs with no configured fallback; the final head includes the configured-fallback guard and regression coverage. (src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts:58, ccc9500b014d)
  • Real-environment evidence remains negative: The July 20 contributor update reports 11 real Bedrock attempts across macOS and Linux that completed normally, but none reproduced the provider-owned abort or demonstrated replay-safe classification, configured fallback selection, and successful completion. (ccc9500b014d)
  • Related implementation context: The related Bedrock live-smoke PR was merged on July 6, 2026, providing a real-runtime harness improvement, but its own PR body states it does not resolve the affected abort/fallback behavior. (src/gateway/gateway-models.profiles.live.test.ts, bacb33a90912)

Likely related people:

  • brian-bell: Authored this recovery-path change and the merged Bedrock live-smoke harness work that supplies the available reproduction environment. (role: recent adjacent contributor; confidence: medium; commits: ccc9500b014d, bacb33a90912; files: src/agents/embedded-agent-runner/run/attempt-recovery.ts, src/agents/embedded-agent-subscribe.ts)
  • steipete: Current main history shown for the embedded agent/session area indicates recent core-agent maintenance, but the available review evidence does not establish a more specific ownership trail. (role: recent adjacent owner; confidence: low; files: src/agents/embedded-agent-subscribe.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (5 earlier review cycles)
  • reviewed 2026-07-03T19:49:31.317Z sha 270be19 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T04:48:59.208Z sha 343c3ab :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T05:31:38.429Z sha 343c3ab :: needs real behavior proof before merge. :: [P2] Preserve signal-owned interruptions before fallback
  • reviewed 2026-07-19T16:55:38.645Z sha ccc9500 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T17:38:19.696Z sha ccc9500 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 3, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 3, 2026
@brian-bell

Copy link
Copy Markdown
Contributor Author

Updated the PR body with the live Bedrock evidence I could produce.

Summary: using profile-based AWS auth only (AWS_PROFILE + AWS_SDK_LOAD_CONFIG=1, no exported AWS key/session env), the direct Bedrock model smoke and Gateway Bedrock smoke both passed against amazon-bedrock/global.anthropic.claude-sonnet-4-6. I also ran two longer issue-specific probes with Bedrock primary plus anthropic/claude-haiku-4-5 fallback; both completed normally with status=ok / stopReason=stop, so this AWS setup did not reproduce the provider-originated abort envelope.

I cannot honestly attach the exact after-fix abort-to-fallback log ClawSweeper asked for from this environment. The remaining decision is whether maintainers accept the source-level predicate/tests plus these negative live attempts, or keep this blocked until a reporter/production environment can produce the real Bedrock abort after applying this patch.

@brian-bell

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 18, 2026
@brian-bell
brian-bell force-pushed the fix/bedrock-prompt-abort-fallback-87876 branch from 270be19 to c255da1 Compare July 19, 2026 04:45
@brian-bell
brian-bell force-pushed the fix/bedrock-prompt-abort-fallback-87876 branch from c255da1 to 343c3ab Compare July 19, 2026 04:46
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jul 19, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 343c3ab08d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/embedded-agent-runner/run/attempt-recovery.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05317b8383

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/embedded-agent-runner/run/attempt-recovery.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f9759d10e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/embedded-agent-runner/run/attempt-recovery.ts
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jul 20, 2026
@brian-bell

Copy link
Copy Markdown
Contributor Author

Evidence update: expanded organic Bedrock matrix (no qualifying abort)

This supersedes and expands my earlier negative live probes. I completed the planned macOS + Linux organic reproduction matrix, including a refreshable-auth prerequisite for Linux.

Scope

  • PR head: ccc9500b014d136864eb41f5ad70627045733e1a
  • Tested source: 362f3d48210f8daadbba741d60bb380c241c851d
    • This is the PR head plus one local committed, content-free observability change used to classify replay safety. That extra commit was not pushed.
  • Node / pnpm: v24.15.0 / 11.2.2
  • Frozen prompt: 72,986 bytes; SHA-256 df02349af9bc34b51bb27420bf0484567889e269fe9f5cfa19045ff230bccc10
  • Real Gateway, bedrock-converse-stream, default HTTP/2, tools disabled

Credential-refresh gate

The Linux lane used an isolated AWS login cache through credential_process; static AWS credential environment variables were unset.

  • Initial remote STS and Bedrock control-plane calls passed with credentials expiring at 2026-07-20T04:46:05Z.
  • I waited beyond that expiration without another login, profile rewrite, or credential-cache transfer.
  • Post-expiration STS, Bedrock control-plane, and exact installed-SDK calls passed through the same cache/profile, with refreshed credentials expiring at 2026-07-20T05:02:04Z.
  • A short real-Gateway primary smoke then passed with provider amazon-bedrock, model us.anthropic.claude-sonnet-4-6, API bedrock-converse-stream, zero tools, and no ExpiredToken.

No long Linux attempt began until this gate passed.

Accepted organic matrix

Lane Attempts Durations (seconds) Result
macOS arm64, us.anthropic.claude-sonnet-4-6 4 1088, 1067, 937, 567 Primary completed; no abort/fallback
macOS arm64, global.anthropic.claude-sonnet-4-6 4 1139, 714, 1117, 1139 Primary completed; no abort/fallback
Linux x86_64, us.anthropic.claude-sonnet-4-6 3 1119, 1053, 1101 Primary completed; no abort/fallback/auth expiry

All 11 accepted attempts returned status=ok, finishReason=stop, and a one-entry successful primary execution trace. Tools and side effects were zero. The three Linux trajectories additionally recorded every abort and timeout flag as false and had no terminal error, ExpiredToken, replay-safe abort event, failover decision, or fallback decision.

The earlier two Linux attempts that used static exported credentials are excluded. They ended in credential expiry and are diagnostics, not provider-abort evidence.

Honest conclusion

This is stronger negative evidence only. The organic matrix did not reproduce the reported provider-owned abort, so it does not demonstrate the requested provider-abort → replay-safe classification → configured fallback → successful completion chain. I cannot claim that the requested rank-up proof is complete.

The next useful evidence is a redacted trace from the original reporter or a production environment running the instrumented source. I do not think a synthetic fault proxy should be built until that real envelope is available.

I retained a checksum-verified private bundle containing the raw CLI results, Gateway logs, and exported trajectories. It was not uploaded publicly: the available Crabbox artifact publisher has no configured remote broker in this account and resolved only to local storage. The inline evidence above is content-free and was scanned for AWS keys/tokens, account/ARN data, local paths, prompts, generated responses, and raw session/run identifiers.

@brian-bell

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Bedrock Converse Streaming silently aborts on long-context agent sessions (~6 min timeout, no retry, no fallback)

2 participants