Skip to content

fix: route embedded tool calls through in-process dispatch (#40237)#42497

Closed
frankbuild wants to merge 2 commits into
openclaw:mainfrom
frankbuild:fix/ws-self-contention-local-dispatch
Closed

fix: route embedded tool calls through in-process dispatch (#40237)#42497
frankbuild wants to merge 2 commits into
openclaw:mainfrom
frankbuild:fix/ws-self-contention-local-dispatch

Conversation

@frankbuild

Copy link
Copy Markdown
Contributor

Summary

Fixes #40237 (also resolves #5703 and #6508 which were circular-duped shut).

When agent tools like cron.run, cron.list, cron.add are called from within an active LLM session, callGatewayTool opens a new WebSocket connection to the same gateway. Because the gateway's Node.js event loop is busy processing the current LLM turn, the second WS connection sits in queue and times out after 10s.

This PR routes embedded tool calls through the existing in-process dispatch path (dispatchGatewayMethod) when running inside the gateway process, bypassing WS entirely. External CLI tool calls continue to use WS unchanged.

Changes

  • src/gateway/local-dispatch.ts (new): lightweight registry that the gateway server populates at startup. Exposes tryLocalGatewayDispatch which returns a promise when in-process dispatch is available, or undefined when running outside the gateway (e.g. CLI).

  • src/gateway/server-plugins.ts: registers dispatchGatewayMethod as the local dispatcher when setFallbackGatewayContext is called during gateway startup.

  • src/agents/tools/gateway.ts: callGatewayTool now checks tryLocalGatewayDispatch first. If available (inside gateway), dispatches in-process with zero WS overhead. Otherwise falls through to the existing WS path.

How it works

CLI tool call (external):
  callGatewayTool → tryLocalGatewayDispatch returns undefined → WS path (unchanged)

Embedded tool call (inside gateway LLM session):
  callGatewayTool → tryLocalGatewayDispatch returns Promise → dispatchGatewayMethod
    → handleGatewayRequest (in-process, no WS) → immediate response

The dispatchGatewayMethod function already exists and is battle-tested — it's used by plugin subagent runtime for sessions, agent dispatch, etc. This PR simply makes it available to agent tools via the registry pattern.

Test plan

  • Unit tests for local-dispatch.ts (registry behavior, reset, error propagation)
  • Integration tests in gateway.test.ts verifying:
    • Embedded tool calls use in-process dispatch (no WS callGateway mock called)
    • External tool calls fall back to WS when no dispatcher registered
    • undefined params correctly normalized to {}
  • Existing server-plugins.test.ts tests pass (5/5)
  • Type check clean (pnpm tsgo)
  • Lint/format clean (pnpm check)

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime agents Agent runtime and tooling size: S labels Mar 10, 2026
@greptile-apps

greptile-apps Bot commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a real WebSocket self-contention bug (#40237) by routing embedded agent tool calls through in-process dispatch instead of opening a new WS connection. The local-dispatch.ts registry is well-designed and well-tested, and the integration point via setFallbackGatewayContext is correct.

However, the interception in callGatewayTool (lines 147-152 of gateway.ts) is too broad. Two critical issues need to be addressed before merge:

  1. Unconditional opts bypass: The early return skips resolveGatewayOptions, silently ignoring opts.gatewayUrl, opts.gatewayToken, and opts.timeoutMs. This breaks the supported use case of explicitly routing to a remote gateway while inside the gateway process, and breaks bash-tools.exec-approval-followup.ts which passes timeoutMs: 60_000 expecting it to be respected.

  2. expectFinal semantics lost: The extra?.expectFinal parameter is not forwarded to the local dispatcher. dispatchGatewayMethod only captures the first respond call, losing the completion-waiting semantics. This breaks bash-tools.exec-approval-followup.ts which calls with expectFinal: true to wait for the agent run to complete before continuing.

Missing test coverage: No test asserts that explicit gatewayUrl overrides reach the WS path, which would catch the first issue.

Confidence Score: 2/5

  • Not safe to merge as-is. Two logic bugs silently break existing caller semantics and routing overrides.
  • The core design (local-dispatch.ts registry and setFallbackGatewayContext integration) is sound and well-tested. However, the callGatewayTool guard in gateway.ts has two critical flaws: (1) it unconditionally bypasses opts without checking for explicit gatewayUrl overrides, breaking the supported use case of routing to a remote gateway from inside the gateway, and explicitly-passed timeoutMs; (2) it drops the expectFinal parameter, changing completion-waiting semantics for existing callers like bash-tools.exec-approval-followup.ts. Both issues affect real code paths and would need to be fixed before merge.
  • src/agents/tools/gateway.ts — the local dispatch guard at lines 147-152 must check opts.gatewayUrl and forward extra?.expectFinal to preserve existing semantics.

Last reviewed commit: af6f728

Comment thread src/agents/tools/gateway.ts
Comment thread src/agents/tools/gateway.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af6f728b21

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/tools/gateway.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 23746d5191

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +154 to +158
if (!hasExplicitUrl && !needsExpectFinal) {
const localResult = tryLocalGatewayDispatch<T>(
method,
(params ?? {}) as Record<string, unknown>,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve least-privilege scopes for local dispatch

By returning early through tryLocalGatewayDispatch, this path skips resolveLeastPrivilegeOperatorScopesForMethod and executes requests as dispatchGatewayMethod's synthetic admin client, which changes authorization-sensitive behavior for existing flows. A concrete regression is the node exec timeout fallback: bash-tools.exec-host-node.ts retries node.invoke with approved=true/approvalDecision='allow-once' when approval times out, and sanitizeSystemRunParamsForForwarding only allows that override for admin/approvals clients; before this commit callGatewayTool("node.invoke") used operator.write and was rejected, but local dispatch now grants admin and allows the run.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ff85b436a6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +154 to +160
if (!hasExplicitUrl && !needsExpectFinal) {
const localResult = tryLocalGatewayDispatch<T>(
method,
(params ?? {}) as Record<string, unknown>,
);
if (localResult !== undefined) {
return await localResult;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve timeoutMs behavior in local dispatch path

When local dispatch is available, this branch returns tryLocalGatewayDispatch directly and never applies opts.timeoutMs, so embedded tool calls can block forever instead of failing fast like the WS path. This is a regression for callers that rely on transport timeouts to bound execution (for example src/agents/bash-tools.exec-host-node.ts passes invokeTimeoutMs to callGatewayTool("node.invoke", ...) specifically to cap long/stuck runs); in an in-process gateway session, that timeout is now ignored and the turn can hang indefinitely if the handler never responds.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 43433d2b97

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +152 to +154
const hasExplicitUrl = trimToUndefined(opts.gatewayUrl) !== undefined;
const needsExpectFinal = extra?.expectFinal === true;
if (!hasExplicitUrl && !needsExpectFinal) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Respect explicit gatewayToken overrides in local dispatch

callGatewayTool now takes the in-process path whenever gatewayUrl is absent, but that decision ignores opts.gatewayToken. In embedded gateway runs (where a local dispatcher is registered), a call that explicitly supplies a token now skips resolveGatewayOptions/WS authentication and executes under ambient request credentials instead, so invalid or intentionally constrained tokens are silently ignored. This regresses auth override semantics for tools that expose gatewayToken and can make auth-sensitive calls succeed when they should fail.

Useful? React with 👍 / 👎.

@frankbuild

Copy link
Copy Markdown
Contributor Author

Hi @tyler6204 @joshavant — this PR addresses the WS self-contention bug (#40237, #5703) by routing embedded tool calls through in-process dispatch. Greptile flagged two concerns on the initial review (opts bypass and expectFinal semantics) — both have been addressed in subsequent commits. CI is green. Would appreciate a review when you get a chance 🙏

@frankbuild
frankbuild force-pushed the fix/ws-self-contention-local-dispatch branch from 43433d2 to 44b0452 Compare March 14, 2026 09:42

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44b045211e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +155 to +158
const localResult = tryLocalGatewayDispatch<T>(
method,
(params ?? {}) as Record<string, unknown>,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Normalize params before in-process dispatch

The new local path forwards params directly to dispatchGatewayMethod (tryLocalGatewayDispatch(method, (params ?? {})...)), which skips the JSON serialization behavior of the WS path. This changes semantics for existing callers that pass optional fields as undefined (for example exec.approval.request payloads built in src/agents/tools/nodes-tool.ts), because validators like validateExecApprovalRequestParams accept omitted fields but reject fields explicitly set to undefined. As a result, embedded gateway tool calls can now fail with INVALID_REQUEST even though the same call succeeds over WebSocket.

Useful? React with 👍 / 👎.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Apr 27, 2026
@clawsweeper

clawsweeper Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution. I reviewed the branch, and this PR is not a good landing base for OpenClaw.

The underlying gateway self-contention bug is real, but this stale, conflicting branch is not a safe landing candidate because its local-dispatch shortcut bypasses current auth, scope, timeout, and request-shape contracts and lacks real behavior proof.

So I’m closing this PR rather than keeping an unmergeable branch open. A new narrow PR that carries only the useful part is welcome.

Review details

Best possible solution:

Replace this branch with a fresh narrow fix on current main that reuses dispatchGatewayMethodInProcessRaw or the scoped request context and preserves auth, scopes, timeout, expectFinal, and request-normalization semantics with real active-session cron proof.

Do we have a high-confidence way to reproduce the issue?

Do we have a high-confidence way to reproduce the issue? Yes for the PR regressions at source level: the added branch returns before current auth, scope, timeout, and request-shape handling. I did not live-run the original active-session cron timeout in this read-only review.

Is this the best way to solve the issue?

Is this the best way to solve the issue? No. The maintainable path should integrate with current scoped in-process dispatch instead of adding a parallel global registry that skips existing gateway contracts.

Security review:

Security review needs attention: The diff introduces an in-process gateway control-plane shortcut without preserving current credential, approval-runtime, and authorization-scope semantics.

  • [high] Local dispatch bypasses gateway auth and scopes — src/agents/tools/gateway.ts:154
    The early return can skip token resolution, approval-runtime setup, explicit scope overrides, and least-privilege scope selection before dispatching through an in-process control-plane path.
    Confidence: 0.91

What I checked:

  • PR diff introduces early local-dispatch return: The PR adds an early tryLocalGatewayDispatch path in callGatewayTool before normal gateway option resolution, scope selection, approval-runtime handling, and WebSocket request timeout behavior. (src/agents/tools/gateway.ts:154, 44b045211e35)
  • Current main gateway-tool contract: Current main resolves gateway options, least-privilege or caller-supplied scopes, approval-runtime tokens, timeout, expectFinal, and then calls callGateway; those semantics are the contracts this PR must preserve. (src/agents/tools/gateway.ts:175, 45fbf2d81a0e)
  • Existing in-process dispatch contract on main: Current main already has dispatchGatewayMethodInProcessRaw with scoped-client requirements, expectFinal waiting, timeout handling, and synthetic scope options, making the PR's parallel global registry the wrong integration point. (src/gateway/server-plugins.ts:328, 45fbf2d81a0e)
  • PR is stale and conflicting: Live PR metadata reports mergeStateStatus=DIRTY and mergeable=CONFLICTING; repeated real behavior proof checks are failing, while the PR body only lists unit/integration tests and static checks. (44b045211e35)
  • Current main provenance for safer dispatch surface: Recent merged history added the gateway method dispatch contract and scoped local gateway request context that a fresh fix should integrate with instead of bypassing. (src/gateway/server-plugins.ts:328, dfeaf6f7cf30)
  • Related canonical issue was closed in favor of this PR: The prior gateway self-contention issue is closed as superseded by this PR, so closing this branch should be paired with inviting a fresh narrow PR rather than treating the bug as implemented.

Likely related people:

  • steipete: Recent merged commits added the gateway method dispatch contract and local gateway request-scope path that a replacement fix should reuse. (role: recent gateway dispatch contract contributor; confidence: high; commits: dfeaf6f7cf30, d643cd5a9f08, 8ba0bb2a8a9b; files: src/gateway/server-plugins.ts, src/plugin-sdk/gateway-method-runtime.ts, src/gateway/local-request-context.ts)
  • jesse-merhi: Recent merged work changed approval-runtime behavior in callGatewayTool and callGateway, which is one of the semantics the PR's local path bypasses. (role: recent gateway-tool approval-runtime contributor; confidence: high; commits: 198f20fd20cc, 65ad33e03a8a; files: src/agents/tools/gateway.ts, src/agents/tools/gateway.test.ts, src/gateway/call.ts)
  • coygeek: Recent merged security hardening touched gateway HTTP/runtime dispatch context, adjacent to the auth-sensitive in-process control-plane shortcut proposed here. (role: adjacent gateway dispatch security contributor; confidence: medium; commits: 5dba99d0b35b; files: src/gateway/server/plugins-http.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 45fbf2d81a0e.

@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Apr 28, 2026
@clawsweeper clawsweeper Bot added the P1 High-priority user-facing bug, regression, or broken workflow. label May 16, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 16, 2026
@clawsweeper clawsweeper Bot added impact:security Security boundary, credential, authz, sandbox, or sensitive-data risk. impact:auth-provider Auth, provider routing, model choice, or SecretRef resolution may break. labels May 17, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed impact:security Security boundary, credential, authz, sandbox, or sensitive-data risk. impact:auth-provider Auth, provider routing, model choice, or SecretRef resolution may break. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels May 18, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@clawsweeper

clawsweeper Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this May 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling gateway Gateway runtime merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

1 participant