fix: fall back after replay-safe Bedrock prompt aborts#99608
fix: fall back after replay-safe Bedrock prompt aborts#99608brian-bell wants to merge 5 commits into
Conversation
|
Codex review: needs real behavior proof before merge. Reviewed July 20, 2026, 9:19 AM ET / 13:19 UTC. Summary PR surface: Source +68, Tests +275. Total +343 across 5 files. Reproducibility: no. high-confidence live reproduction is currently available: the source-level abort shape is covered by focused tests, but the contributor's 11 real Bedrock attempts did not trigger the provider-owned abort from the report. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Land only after a maintainer either obtains a redacted affected-environment trace proving the provider abort is replay-safe and completes on the configured fallback, or explicitly accepts the focused source guards and regression tests as sufficient for this rare provider envelope. Do we have a high-confidence way to reproduce the issue? No high-confidence live reproduction is currently available: the source-level abort shape is covered by focused tests, but the contributor's 11 real Bedrock attempts did not trigger the provider-owned abort from the report. Is this the best way to solve the issue? Yes, conditionally: reusing the existing model-fallback path with strict no-output, no-side-effect, no-timeout, and no-caller-stop gates is narrower than adding provider-specific retry logic, but the intended live envelope still needs maintainer acceptance or direct proof. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 8d4d02a3cfa4. Label changesLabel justifications:
Evidence reviewedPR surface: Source +68, Tests +275. Total +343 across 5 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (5 earlier review cycles)
|
|
Updated the PR body with the live Bedrock evidence I could produce. Summary: using profile-based AWS auth only ( I cannot honestly attach the exact after-fix abort-to-fallback log ClawSweeper asked for from this environment. The remaining decision is whether maintainers accept the source-level predicate/tests plus these negative live attempts, or keep this blocked until a reporter/production environment can produce the real Bedrock abort after applying this patch. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
This pull request has been automatically marked as stale due to inactivity. |
270be19 to
c255da1
Compare
c255da1 to
343c3ab
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 343c3ab08d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05317b8383
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f9759d10e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Evidence update: expanded organic Bedrock matrix (no qualifying abort)This supersedes and expands my earlier negative live probes. I completed the planned macOS + Linux organic reproduction matrix, including a refreshable-auth prerequisite for Linux. Scope
Credential-refresh gateThe Linux lane used an isolated AWS login cache through
No long Linux attempt began until this gate passed. Accepted organic matrix
All 11 accepted attempts returned The earlier two Linux attempts that used static exported credentials are excluded. They ended in credential expiry and are diagnostics, not provider-abort evidence. Honest conclusionThis is stronger negative evidence only. The organic matrix did not reproduce the reported provider-owned abort, so it does not demonstrate the requested provider-abort → replay-safe classification → configured fallback → successful completion chain. I cannot claim that the requested rank-up proof is complete. The next useful evidence is a redacted trace from the original reporter or a production environment running the instrumented source. I do not think a synthetic fault proxy should be built until that real envelope is available. I retained a checksum-verified private bundle containing the raw CLI results, Gateway logs, and exported trajectories. It was not uploaded publicly: the available Crabbox artifact publisher has no configured remote broker in this account and resolved only to local storage. The inline evidence above is content-free and was scanned for AWS keys/tokens, account/ARN data, local paths, prompts, generated responses, and raw session/run identifiers. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Closes #87876
AI-assisted: yes
What Problem This Solves
Fixes an issue where users running long Bedrock agent sessions could lose the run when the Bedrock provider surfaced a provider-originated prompt abort. In the replay-safe prompt-abort shape from #87876, the run was marked aborted before the existing model fallback path could evaluate it, so configured fallback models were not tried.
Why This Change Was Made
The change lets only replay-safe provider prompt aborts enter the existing prompt failover path. It keeps user/external aborts, session timeouts, compaction timeouts, tool-execution timeouts, and attempts with possible side effects out of fallback so intentional cancellation and unsafe replay behavior stay unchanged.
User Impact
Bedrock users with configured fallback models can recover from this provider-originated prompt abort class instead of ending the whole agent run immediately. Runs that are not safe to replay still fail rather than duplicating side effects.
Evidence
Source-level proof:
node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts src/agents/model-fallback.run-embedded.e2e.test.tsfix/bedrock-prompt-abort-fallback-87876: clean, no accepted/actionable findings.Live Bedrock proof attempts, July 3, 2026:
AWS_PROFILEwithAWS_SDK_LOAD_CONFIG=1. AWS key/session env was not exported to the child runs.node scripts/test-live.mjs --no-quiet -- src/agents/models.profiles.live.test.ts -t "completes across selected models"withOPENCLAW_LIVE_PROVIDERS=amazon-bedrockandOPENCLAW_LIVE_MODELS=amazon-bedrock/global.anthropic.claude-sonnet-4-6.node scripts/test-live.mjs --no-quiet -- src/gateway/gateway-models.profiles.live.test.ts -t "runs meaningful prompts across models with available keys"withOPENCLAW_LIVE_GATEWAY_PROVIDERS=amazon-bedrockand the same model. The prompt and tool-read probes both endedstopReason=stop.anthropic/claude-haiku-4-5fallback, using a 66,188-byte replay-safe prompt:runId=29d923a4-4014-471a-bfa4-a3d6d78a02c6,status=ok,stopReason=stop; noembedded run failover decision, nomodel fallback decision, and noThis operation was aborted.runId=2f21aaee-8341-4aab-a435-77735596e2f8,status=ok,stopReason=stop; again no abort/fallback.Proof gap:
externalAbort=false, replay-safe metadata, and the configured fallback model being attempted. Exact live proof needs either reporter/production-environment logs after this patch, or a maintainer decision to accept the source-level predicate/tests plus these negative live attempts.