Skip to content

Fix gateway timeout embedded fallback session lock#74543

Merged
steipete merged 2 commits into
openclaw:mainfrom
HemantSudarshan:hemant/62981-agent-gateway-timeout-fallback
Apr 29, 2026
Merged

Fix gateway timeout embedded fallback session lock#74543
steipete merged 2 commits into
openclaw:mainfrom
HemantSudarshan:hemant/62981-agent-gateway-timeout-fallback

Conversation

@HemantSudarshan

Copy link
Copy Markdown
Contributor

Summary

  • Problem: when openclaw agent hit the gateway final-response timeout, embedded fallback reused the same session/run id while the gateway could still hold the session lock.
  • Why it matters: the fallback could immediately fail with session file locked, then cascade through model fallbacks.
  • What changed: gateway timeout fallbacks now run the embedded agent under a fresh fallback session/run id while preserving normal embedded fallback behavior for non-timeout gateway failures.
  • What did NOT change (scope boundary): no gateway protocol, session-store format, or non-timeout fallback behavior changes.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause (if applicable)

  • Root cause: agentCliCommand() reused the original local options after a gateway timeout, so embedded fallback inherited the same session id/run id as the still-running gateway agent path.
  • Missing detection / guardrail: there was no regression test distinguishing gateway timeout fallback from other gateway connection failures.
  • Contributing context (if known): the gateway agent RPC can acknowledge a run and continue working after the CLI-side final response timeout expires.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/commands/agent-via-gateway.test.ts
  • Scenario the test should lock in: gateway timeout fallback invokes embedded agent with a fresh gateway-fallback-* session/run id instead of the locked original values.
  • Why this is the smallest reliable guardrail: the bug is in CLI fallback option construction after callGateway() rejects.
  • Existing test that already covers this (if any): existing fallback tests covered generic gateway failures, not timeout lock isolation.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

Gateway timeout fallback still attempts embedded execution, but the embedded run uses a fresh fallback session/run id to avoid racing the gateway-held session lock.

Diagram (if applicable)

Before:
[gateway timeout] -> [embedded fallback with same session id] -> [session lock failure]

After:
[gateway timeout] -> [embedded fallback with fresh session id] -> [fallback can acquire its own transcript]

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: Windows local checkout
  • Runtime/container: Node 22 via repo scripts
  • Model/provider: mocked tests
  • Integration/channel (if any): Gateway CLI fallback path
  • Relevant config (redacted): default test config

Steps

  1. Mock callGateway() to reject with a typed gateway timeout.
  2. Run agentCliCommand() with an existing sessionId and runId.
  3. Assert embedded fallback receives a fresh gateway-fallback-* session/run id.

Expected

  • Embedded fallback does not reuse the gateway-locked session/run id.

Actual

  • Fixed by this PR.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios:
    • pnpm test src/commands/agent-via-gateway.test.ts -t "uses a fresh embedded session when gateway agent times out" -- --reporter=verbose
    • pnpm test src/commands/agent-via-gateway.test.ts -- --reporter=verbose
    • pnpm test src/commands/agent-via-gateway.test.ts src/gateway/call.test.ts src/agents/run-wait.test.ts -- --reporter=verbose
    • pnpm check:changed
    • pnpm test:changed
  • Edge cases checked: generic gateway failure fallback still uses the existing path; timeout fallback overrides both sessionId and runId.
  • What you did not verify: live gateway/model timeout against a real provider.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: timeout fallback starts a new transcript instead of continuing the original session context.
    • Mitigation: this only applies after gateway timeout, where the original session may still be locked by the gateway run; non-timeout fallback behavior is unchanged.

@openclaw-barnacle openclaw-barnacle Bot added commands Command implementations size: S labels Apr 29, 2026
@clawsweeper

clawsweeper Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

What this changes:

The PR changes agentCliCommand to detect Gateway agent timeouts, run embedded fallback with a generated gateway-fallback-* session/run id, and add a focused CLI regression test.

Maintainer follow-up before merge:

This is an open implementation PR paired with #62981; the remaining action is maintainer review, CI validation, and product judgment on fresh-session fallback semantics, not an automated replacement fix lane.

Security review:

Security review cleared: The diff only changes CLI fallback control flow and unit tests, with no new permissions, dependency sources, secret handling, workflow, package, or downloaded-code execution surface.

Review details

Best possible solution:

Land a reviewed variant that prevents an accepted Gateway run which later times out from contending with embedded fallback on the same transcript, with focused regression coverage and clear fallback metadata; if maintainers prefer wait/abort semantics over session forking, adjust that behavior before merge.

Acceptance criteria:

  • pnpm test src/commands/agent-via-gateway.test.ts -t "uses a fresh embedded session when gateway agent times out" -- --reporter=verbose
  • pnpm test src/commands/agent-via-gateway.test.ts src/gateway/call.test.ts src/agents/run-wait.test.ts -- --reporter=verbose
  • pnpm check:changed

What I checked:

  • Current main fallback path reuses original options: agentCliCommand catches Gateway failures and calls agentCommand with localOpts, which is built by spreading the original CLI options, so current main can reuse the same session selectors after a timeout. (src/commands/agent-via-gateway.ts:196, 68912111cfea)
  • Current main waits for final Gateway response with a timeout: The CLI Gateway request uses expectFinal: true and passes an explicit timeout, matching the accepted-run-then-CLI-timeout shape described by the linked issue. (src/commands/agent-via-gateway.ts:143, 68912111cfea)
  • Gateway accepted runs continue asynchronously: The Gateway agent handler responds with an accepted payload and then schedules dispatchAgentRunFromGateway, so a timed-out CLI caller can fall back while the Gateway-side run is still active. (src/gateway/server-methods/agent.ts:1109, 68912111cfea)
  • Embedded attempts still acquire transcript locks: The embedded runner acquires acquireSessionWriteLock({ sessionFile: params.sessionFile, ... }) before transcript/session mutation, so reuse of the same transcript can still hit the reported lock contention. (src/agents/pi-embedded-runner/run/attempt.ts:1256, 68912111cfea)
  • PR diff targets the narrow timeout fallback: The PR imports isGatewayTransportError, recognizes timeout-shaped Gateway failures, creates a gateway-fallback-* id, and passes it as both sessionId and runId only on that timeout path. (src/commands/agent-via-gateway.ts:223, b1d9a80f2d01)
  • PR adds focused regression coverage: The new test mocks a typed Gateway timeout and asserts embedded fallback receives a fresh gateway-fallback-* session id and matching run id while preserving fallback metadata. (src/commands/agent-via-gateway.test.ts:206, b1d9a80f2d01)

Likely related people:

  • steipete: Prior related review context and public commit metadata connect this maintainer to session write-lock hardening, one-shot agent cleanup, and adjacent CLI/Gateway fallback behavior. (role: recent maintainer and adjacent owner; confidence: medium; commits: 8de02c318ba5, 35ec4a9991fd, e0bee76fb02f; files: src/commands/agent-via-gateway.ts, src/agents/session-write-lock.ts, src/agents/pi-embedded-runner/run/attempt.ts)
  • vincentkoc: Recent public commit metadata connects this maintainer to Gateway agent accepted-ack scheduling and Gateway client seams that shape the timeout handoff path. (role: adjacent Gateway agent/wait maintainer; confidence: medium; commits: 985000026e35, 1d61862adb0d, 0f7d9c957093; files: src/gateway/server-methods/agent.ts, src/gateway/client.ts, src/gateway/call.ts)
  • amknight: Recent public commit metadata shows work on the embedded fallback metadata path in the same CLI command and test surface changed by this PR. (role: adjacent fallback-path contributor; confidence: low; commits: b1e530b20439; files: src/commands/agent-via-gateway.ts, src/commands/agent-via-gateway.test.ts)

Remaining risk / open question:

  • This was a read-only static review; I did not run the PR's tests or CI.
  • The patch chooses an explicit fresh fallback session instead of waiting for or aborting the accepted Gateway run, so maintainers should confirm that UX tradeoff before merging.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 68912111cfea.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation agents Agent runtime and tooling labels Apr 29, 2026
@steipete
steipete force-pushed the hemant/62981-agent-gateway-timeout-fallback branch from 91457fc to 7e88f46 Compare April 29, 2026 18:38
@steipete
steipete merged commit fbae2a6 into openclaw:main Apr 29, 2026
13 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Landed in fbae2a6. Thanks @HemantSudarshan.

Maintainer fixup before landing:

  • kept the timeout-only embedded fallback behavior
  • bound timeout fallback to an explicit gateway-fallback-* session key so it cannot replace the original routed conversation session while the Gateway-owned run is still active
  • exposed fallbackReason, fallbackSessionId, and fallbackSessionKey in fallback metadata and documented the behavior

Proof:

  • OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test src/commands/agent-via-gateway.test.ts src/commands/agent.session.test.ts src/commands/agent/session.test.ts src/gateway/call.test.ts src/agents/run-wait.test.ts src/gateway/server.chat.gateway-server-chat.test.ts src/gateway/server.agent.gateway-server-agent-b.test.ts -- --reporter=verbose
  • pnpm exec oxfmt --check --threads=1 CHANGELOG.md docs/cli/agent.md src/agents/command/types.ts src/agents/command/session.ts src/commands/agent-via-gateway.ts src/commands/agent-via-gateway.test.ts
  • git diff --check

I also attempted the Testbox check:changed path, but the created box stayed queued for several minutes; I stopped that box rather than leaving it idle.

lxe pushed a commit to lxe/openclaw that referenced this pull request May 6, 2026
* Agent: isolate gateway timeout fallback sessions

* fix(cli): isolate gateway timeout fallback sessions

---------

Co-authored-by: Peter Steinberger <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
* Agent: isolate gateway timeout fallback sessions

* fix(cli): isolate gateway timeout fallback sessions

---------

Co-authored-by: Peter Steinberger <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
* Agent: isolate gateway timeout fallback sessions

* fix(cli): isolate gateway timeout fallback sessions

---------

Co-authored-by: Peter Steinberger <[email protected]>
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
* Agent: isolate gateway timeout fallback sessions

* fix(cli): isolate gateway timeout fallback sessions

---------

Co-authored-by: Peter Steinberger <[email protected]>
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
* Agent: isolate gateway timeout fallback sessions

* fix(cli): isolate gateway timeout fallback sessions

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling commands Command implementations docs Improvements or additions to documentation size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

session file locked when gateway times out and falls back to embedded runner

2 participants