Skip to content

fix: reconcile timed-out gateway message actions#104632

Merged
obviyus merged 4 commits into
openclaw:mainfrom
Glucksberg:fix/message-action-timeout-reconcile
Jul 13, 2026
Merged

fix: reconcile timed-out gateway message actions#104632
obviyus merged 4 commits into
openclaw:mainfrom
Glucksberg:fix/message-action-timeout-reconcile

Conversation

@Glucksberg

@Glucksberg Glucksberg commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Related: #80177
Related: #102179

What Problem This Solves

Fixes duplicate channel replies when a Gateway-backed message(action=send) exceeds the client deadline but succeeds later. The model can otherwise see a tool failure, compose a paraphrased retry, and make both deliveries visible. This was observed on Telegram after upgrading from an OpenClaw 2026.6.10 control window to the 2026.7.1 beta line.

Why This Change Was Made

On GatewayTransportError(kind="timeout") only, send reattaches once to the unchanged message.action payload and idempotency key. The Gateway already joins same-key in-flight work and caches its completed result, so reconciliation observes the original action instead of dispatching another one.

For agent calls with an abort signal, the reattachment has its own nine-minute deadline and still respects caller cancellation. The initial send wait is capped at 30 seconds, keeping transport plus reconciliation inside Codex's ten-minute outer watchdog even when a larger timeoutMs is supplied. Calls without a cancellation lifecycle use one bounded 60-second fallback attempt. There is no unbounded retry loop, non-send actions are unchanged, and definitive errors still return immediately.

This is intentionally a process-local mitigation: it relies on the live Gateway's in-flight join/cache and does not claim exactly-once delivery across Gateway restarts.

User Impact

Agents wait for the original delivery result instead of treating an ambiguous client timeout as an immediate send failure. This closes the timeout -> late success -> paraphrased duplicate path without lengthening normal sends or changing channel configuration.

Evidence

  • Sanitized production evidence in the related items shows a first message(action=send) timing out after 30 seconds, the Gateway later recording the Telegram send as successful, and a paraphrased second send following.
  • Focused regression coverage verifies a timed-out send reattaches exactly once with the same idempotency key, caller cancellation prevents reattachment, no-signal calls remain bounded, and a configured 120-second transport timeout is capped to preserve the total deadline.
  • node scripts/run-vitest.mjs src/infra/outbound/message-action-runner.plugin-dispatch.test.ts extensions/codex/src/app-server/dynamic-tool-execution.test.ts -- --reporter=dot — 37/37 outbound and 26/26 Codex tests passed.
  • node scripts/run-vitest.mjs src/gateway/server-methods/send.test.ts — 138/138 passed, including the same-key in-flight join contract.
  • .agents/skills/autoreview/scripts/autoreview --mode uncommitted --stream-engine-output — clean; no accepted/actionable findings (0.89 confidence on the final timeout-cap patch).
  • A redacted current-head Telegram live-proof request has been submitted to Mantis and remains pending.
  • AI-assisted. No private message content, chat identifiers, or credentials are included.

This was referenced Jul 11, 2026
Open
Closed
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels Jul 11, 2026
@clawsweeper

clawsweeper Bot commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 12, 2026, 5:02 PM ET / 21:02 UTC.

Summary
The PR adds one bounded same-idempotency-key reattachment after a Gateway-backed message send times out and extends the Codex message-tool deadline to cover reconciliation.

PR surface: Source +52, Tests +206. Total +258 across 5 files.

Reproducibility: yes. Current source and the linked production incident establish a client timeout while Gateway work continues, later delivery, and unsafe retry pressure; the corrected current-head path still needs live observation.

Review metrics: 1 noteworthy metric.

  • Reconciliation envelope: 30-second initial wait plus up to 9 minutes observation. The timing fits the ten-minute OpenClaw Codex watchdog while leaving roughly 30 seconds for setup and the structured response.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #80177
Summary: This PR is a candidate fix for the canonical ambiguous Gateway timeout and late-delivery report; the linked PRs address separate reply-state behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Attach a redacted current-head Telegram transcript, recording, timing logs, or live output showing exactly one visible delivery.

Proof guidance:

  • [P1] Needs real behavior proof before merge: Focused tests and pre-fix incident evidence are supplemental only; the Mantis comment is a request with no result, so add redacted current-head proof, update the PR body for automatic review, or ask a maintainer to comment @clawsweeper re-review.

Mantis proof suggestion
The supported Telegram live lane can reproduce the delayed send and return the redacted transcript and timing evidence needed for review. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram live: reproduce a delayed gateway message send past the client deadline and verify exactly one visible delivery, returning a redacted transcript and timing logs.

Risk before merge

  • [P2] Merging without exact-head live proof leaves the real Telegram timeout-to-late-success path unobserved after the fix.
  • [P1] The reconciliation is intentionally process-local; a Gateway restart can discard the in-flight promise and completed-result cache before the client observes the outcome.

Maintainer options:

  1. Add exact-head delivery proof (recommended)
    Attach a redacted Telegram transcript, recording, timing logs, or live output showing the initial timeout, successful same-key reconciliation, and exactly one visible message.
  2. Pause until the live path is reproducible
    Keep the PR open without merging if a controlled delayed-send scenario cannot demonstrate the claimed one-delivery result.

Next step before merge

  • [P1] No code repair is indicated; the remaining merge actions are contributor-owned real behavior proof and ordinary branch refresh.

Security
Cleared: The runtime and test patch introduces no dependency, permission, secret, artifact-download, workflow, package-resolution, or other concrete security or supply-chain concern.

Review details

Best possible solution:

Keep the shared same-key reconciliation and bounded cancellation design, rebase the small adjacent current-main change, and merge only after redacted exact-head Telegram evidence shows a delayed send produces exactly one visible delivery.

Do we have a high-confidence way to reproduce the issue?

Yes. Current source and the linked production incident establish a client timeout while Gateway work continues, later delivery, and unsafe retry pressure; the corrected current-head path still needs live observation.

Is this the best way to solve the issue?

Yes. Reattaching the unchanged action and idempotency key at the shared Gateway boundary uses the existing in-flight join and completed-result cache and is narrower than downstream channel-specific suppression.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e3163d1e1f2d.

Label changes

Label justifications:

  • P1: The reported regression can produce duplicate visible channel messages during normal agent use.
  • merge-risk: 🚨 message-delivery: The patch changes ambiguous send handling, and exact-head live proof is needed to exclude duplicate or unresolved delivery.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: Focused tests and pre-fix incident evidence are supplemental only; the Mantis comment is a request with no result, so add redacted current-head proof, update the PR body for automatic review, or ask a maintainer to comment @clawsweeper re-review.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The fix is directly observable as one versus duplicate Telegram messages after the client deadline.
Evidence reviewed

PR surface:

Source +52, Tests +206. Total +258 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 3 57 5 +52
Tests 2 206 0 +206
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 263 5 +258

What I checked:

Likely related people:

  • steipete: Repeated merged work spans Gateway send behavior, outbound messaging, and the Codex dynamic-tool lifecycle, making this the strongest cross-surface routing candidate. (role: recent area contributor; confidence: high; commits: f94a7dc183c6, f53346944db0, a09f6b1b2787; files: src/gateway/server-methods/send.ts, src/infra/outbound/message-action-runner.ts, extensions/codex/src/app-server/dynamic-tool-execution.ts)
  • vincentkoc: History attributes the shared Gateway send in-flight helper refactor and several outbound delivery changes to this contributor. (role: introduced shared behavior; confidence: medium; commits: 12cf34a8eabd, a22a1edc8faa, 6a0fdea90ad0; files: src/gateway/server-methods/send.ts, src/infra/outbound/message-action-runner.ts)
  • cxbAsDev: Introduced recent Codex dynamic-tool timeout budgeting and headroom behavior directly adjacent to the watchdog changed by this PR. (role: recent adjacent contributor; confidence: medium; commits: 68487e4c1e34; files: extensions/codex/src/app-server/dynamic-tool-execution.ts, extensions/codex/src/app-server/dynamic-tool-execution.test.ts)
  • joshavant: Recently changed both the Gateway message-action and shared outbound runner surfaces while preserving channel action contracts. (role: recent adjacent contributor; confidence: medium; commits: fbd330b7aa4a; files: src/gateway/server-methods/send.ts, src/infra/outbound/message-action-runner.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (11 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-12T06:52:48.032Z sha b3e94a4 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T11:38:57.917Z sha 5e12771 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T12:22:33.034Z sha 5b63486 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T12:30:43.189Z sha 5b63486 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T12:45:54.447Z sha bb0d2a2 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T13:05:30.834Z sha 0b36509 :: needs real behavior proof before merge. :: [P1] Keep reconciling while the original action remains in flight
  • reviewed 2026-07-12T17:33:51.034Z sha 2c2cfcb :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-12T20:50:52.999Z sha de23220 :: needs real behavior proof before merge. :: none

@Glucksberg
Glucksberg marked this pull request as ready for review July 11, 2026 19:28
@Glucksberg
Glucksberg force-pushed the fix/message-action-timeout-reconcile branch 3 times, most recently from d9b9fa2 to b3e94a4 Compare July 12, 2026 06:41
Glucksberg added a commit to Glucksberg/OpenClaw that referenced this pull request Jul 12, 2026
@Glucksberg
Merge PR openclaw#104632: reconcile timed-out gateway message actions
ddcc770
@Glucksberg
Glucksberg force-pushed the fix/message-action-timeout-reconcile branch 2 times, most recently from 5e12771 to 5b63486 Compare July 12, 2026 12:16
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 12, 2026
@Glucksberg
Glucksberg force-pushed the fix/message-action-timeout-reconcile branch 2 times, most recently from bb0d2a2 to 0b36509 Compare July 12, 2026 12:56
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. and removed mantis: telegram-visible-proof Mantis should capture Telegram visible proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 12, 2026
@Glucksberg

Glucksberg commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

@openclaw-mantis telegram live: reproduce a delayed gateway message send past the client deadline and verify exactly one visible delivery, returning a redacted transcript and timing logs.

@Glucksberg
Glucksberg force-pushed the fix/message-action-timeout-reconcile branch from 0b36509 to fc20e01 Compare July 12, 2026 14:26
@Glucksberg

Glucksberg commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 12, 2026
@Glucksberg Glucksberg mentioned this pull request Jul 12, 2026
Merged
@Glucksberg

Glucksberg commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@Glucksberg
Glucksberg force-pushed the fix/message-action-timeout-reconcile branch from de23220 to 025f89b Compare July 13, 2026 02:54
Glucksberg added a commit to Glucksberg/OpenClaw that referenced this pull request Jul 13, 2026
@Glucksberg
Merge PR openclaw#104632: reconcile timed-out gateway message actions
7c1f694
@obviyus

obviyus commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Maintainer review + landing proof (pre-merge):

  • Premise verified in source: the Gateway request context is created once at server startup (src/gateway/server.impl.ts:1711), so both the in-flight join map and the completed-result dedupe cache (src/gateway/server-methods/send.ts:169, key message.action:<origin>:<idem>, DEDUPE_TTL_MS 5 min) are server-global — a same-key reattach joins in-flight work or receives the cached result across connections.
  • Transport contract verified: callGatewayLeastPrivilege supports timeoutMs: null (per-request timer removed after hello) and signal (src/gateway/call.ts).
  • Codex envelope verified against the harness source: codex-rs/app-server/src/dynamic_tools.rs on_call_response awaits the client tool response with no deadline, so OpenClaw's 600s dynamic-tool watchdog is the outer bound; the 30s initial wait + 9-min reconciliation fits inside it.
  • Scope judgment: reconciling only kind="timeout" is correct — a closed transport error can mean a restarted Gateway whose dedupe state is gone, where a reattach would re-dispatch and duplicate. No change requested.
  • CI: green on exact head 025f89b7607c069299d37665dc63b0bd8b3424ef (only skipped/neutral non-required contexts remain).
  • Known proof gap: no live Telegram delayed-send run was captured on this exact head (the Mantis request is still pending). Landing on maintainer instruction with code-level and CI proof; if a live regression surfaces, the reconciliation path is bounded and revertible in isolation.

@obviyus
obviyus merged commit 6dfa4c2 into openclaw:main Jul 13, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: codex mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Glucksberg @obviyus