Skip to content

Codex PreToolUse native hook relay spawns CPU-bound openclaw-hooks processes and stalls gateway RPC #91009

Description

@aspalagin

Summary

On OpenClaw 2026.6.1 with the bundled @openclaw/codex integration, Codex app-server tool calls can spawn multiple short-lived openclaw-hooks / openclaw hooks relay --provider codex --event pre_tool_use processes. In the observed case these processes consumed ~100%+ CPU each, the gateway event loop became degraded, and local gateway RPC calls (health / status) timed out after 10s even though openclaw-gateway.service stayed alive and the agent session completed successfully.

This looks like a core Codex native hook relay issue rather than a Telegram/systemd/disk/RAM problem.

Environment

  • OpenClaw: 2026.6.1 (2e08f0f)
  • @openclaw/codex: 2026.6.1
  • Runtime: systemd-managed gateway, Node 22.22.2
  • Gateway command: node --max-old-space-size=12288 .../openclaw/dist/index.js gateway --port 18789
  • Provider/harness: Codex app-server (codex app-server --listen stdio://)
  • Agent config had tools.loopDetection.enabled=true
  • openclaw hooks list --json showed only bundled/internal hooks (boot-md, bootstrap-extra-files, command-logger, compaction-notifier, session-memory)
  • No custom before_tool_call hooks and no trusted tool policies were configured

Observed behavior

During an agent turn with multiple tool calls:

  1. openclaw gateway call health --json and openclaw gateway call status --json failed with:
GatewayTransportError: gateway timeout after 10000ms
Gateway target: ws://127.0.0.1:18789
Source: local loopback
  1. The gateway service did not crash. The active agent session later ended with status=success, timedOut=false.

  2. ps snapshots during the degraded window showed several concurrent hook relay processes:

openclaw-hooks  ~108-116% CPU, elapsed ~4-5s
openclaw-hooks  ~108-116% CPU, elapsed ~4-5s
openclaw-hooks  ~108-116% CPU, elapsed ~4-5s
openclaw-hooks  ~108-116% CPU, elapsed ~4-5s

One parent command was:

node .../openclaw.mjs hooks relay --provider codex --relay-id <redacted> --generation <redacted> --event pre_tool_use --timeout 5000
  1. Gateway liveness warning around the same window:
liveness warning: reasons=event_loop_delay,event_loop_utilization,cpu
eventLoopDelayP99Ms=3613.4
eventLoopUtilization=1
cpuCoreRatio=1.032
last=codex_app_server:notification:item/completed
  1. A later 30s health call succeeded but reported degraded event loop:
{
  "ok": true,
  "durationMs": 32,
  "eventLoop": {
    "degraded": true,
    "reasons": ["event_loop_delay", "event_loop_utilization", "cpu"],
    "delayP99Ms": 3347.1,
    "utilization": 1,
    "cpuCoreRatio": 1.133
  }
}

Code path that looks suspicious

In the installed bundle, @openclaw/codex/dist/harness.js hard-enables native hook relay for both attempts and side questions:

nativeHookRelay: { enabled: true }

The Codex native hook relay config maps pre_tool_use to Codex PreToolUse and registers a synchronous command hook. The installed bundle also appears to special-case pre_tool_use so it can still install a PreToolUse hook even when shouldRelayEvent(event) is false/no-op.

In the gateway bundle, nativeHookRelayEventHasLocalWork(registration, "pre_tool_use") returns true when either a before_tool_call policy exists or loop detection may run. With tools.loopDetection.enabled=true, this makes PreToolUse relay active even without custom hooks or trusted policies.

The resulting command path is a separate CLI process per hook invocation:

openclaw hooks relay --provider codex --event pre_tool_use --timeout 5000

That per-tool-call process fan-out appears to be enough to starve the gateway/event loop under normal Codex tool bursts.

Expected behavior

  • Gateway local RPCs (health, status) should stay responsive during Codex tool-call bursts.
  • A no-op or loop-detection-only PreToolUse path should not spawn a heavyweight CLI process per tool call.
  • PreToolUse should only be registered when it has real local work, or it should be handled in-process/resident rather than through repeated CLI relay processes.
  • If relay is required, it should have bounded per-session concurrency and diagnostics for relay duration/count.

Local workaround tested

As a temporary production hotfix, I changed the active project-scoped @openclaw/codex/dist/harness.js from:

nativeHookRelay: { enabled: true }

to:

nativeHookRelay: { enabled: false }

Then restarted openclaw-gateway.service.

After the workaround:

  • Gateway health: durationMs=31ms, eventLoop.degraded=false, delayP99Ms≈21ms
  • Telegram channels reconnected normally
  • A controlled agent test completed 8 bash tool calls with 0 failures
  • During and after that test, no openclaw-hooks processes appeared
  • Logs for the test window had 0 liveness warnings, 0 gateway timeouts, 0 nativeHook.invoke, and 0 hook relay entries

This workaround is intentionally blunt and likely disables useful Codex native hook functionality. It is not a proper upstream fix.

Suggested fix direction

  • Do not register Codex PreToolUse when shouldRelayEvent("pre_tool_use") is false and no real before-tool-call/trusted policy work exists.
  • Move loop detection for Codex native tool events in-process or into a resident bridge instead of spawning openclaw hooks relay for every tool call.
  • Add a config escape hatch such as codex.nativeHookRelay.enabled / codex.nativeHookRelay.events / codex.nativeHookRelay.preToolUse.
  • Add bounded concurrency/timeout enforcement for native hook relay processes.
  • Add regression coverage: 20-30 rapid Codex tool calls should not produce gateway RPC timeouts or event-loop degradation.

Related issues I found

These look related but not the exact same failure mode:

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions