Skip to content

[Bug]: Possible repeated Codex Apps plugin discovery causing excessive disk I/O during a single request (investigated with opensnoop and atop) #99071

Description

@ldrlinj

Bug type

Crash (process/app exits or hangs)

Beta release blocker

No

Summary

I have been investigating a reproducible disk I/O issue involving OpenClaw and the embedded OpenAI Codex app-server.

During complex investigation tasks, the Codex app-server repeatedly reads nearly all plugin definition files (plugin.json) under the Codex Apps cache, together with repeated accesses to rollout JSONL and related cache files. Under heavier workloads, this behavior can result in extremely high disk I/O and eventually make the server unresponsive.

The collected evidence suggests that this may be caused by repeated plugin discovery or plugin registry rebuilding during a single request. I would appreciate confirmation whether this is expected behavior or a bug.

Steps to reproduce

  1. Start OpenClaw v2026.6.1 on Ubuntu 24.04.4 with the default Codex app-server integration.
  2. Use a persistent main agent.
  3. Send a complex investigation task that requires multiple rounds of reasoning and source inspection (for example, tracing Gateway/Codex runtime behavior across several implementation files).
  4. Observe the Codex app-server process using atop and opensnoop-bpfcc.
  5. In my environment, the Codex app-server repeatedly opens nearly all plugin.json files under:
    ~/.openclaw/agents/main/agent/codex-home/.tmp/plugins/plugins
  6. Under heavier requests, disk reads continue to increase and may eventually lead to very high disk utilization and an unresponsive server.BB
    I can consistently reproduce this behavior and can provide additional opensnoop traces, atop logs, or perform further experiments if needed.

Expected behavior

The Codex app-server should initialize or discover plugins only when necessary and should not repeatedly rescan nearly all plugin definitions during a single request.

Complex reasoning tasks should not cause continuously increasing disk reads or excessive repeated access to the same plugin metadata.

Actual behavior

During a single request, the Codex app-server repeatedly opens approximately 180 plugin.json files multiple times.

opensnoop-bpfcc shows repeated accesses to:

  • plugin.json
  • rollout-*.jsonl
  • models_cache.json
  • state.sqlite

atop reports up to approximately 1.3–1.4 GB of disk reads within 30 seconds by the Codex binary. Under heavier workloads, disk utilization approaches 100%, and the server eventually becomes unresponsive.

OpenClaw version

v2026.6.1

Operating system

Ubuntu 24.04.4

Install method

installed globally via npm, running as a systemd user service (node /usr/lib/node_modules/openclaw/dist/index.js gateway), Codex provider is the bundled OpenAI Codex app-server.

Model

openai/gpt-5.4

Provider / routing chain

Feishu → OpenClaw Gateway → Codex provider → local Codex app-server (vendor binary) → OpenAI

Additional provider/model setup details

No response

Logs

Process observed:

Gateway:
node /usr/lib/node_modules/openclaw/dist/index.js gateway

Codex app-server:
.../vendor/.../bin/codex app-server

atop observation during the problematic request:

RDDSK ≈ 1.3-1.4 GB / 30 seconds
Disk busy ≈ 99%

opensnoop summary:

Approximately 180 unique plugin.json files were repeatedly opened during a single request.

Example:

rollout-2026-07-02T14-57-09-....jsonl        125 opens
figma/.codex-plugin/plugin.json             105 opens
teams/.codex-plugin/plugin.json              77 opens
sharepoint/.codex-plugin/plugin.json         77 opens
slack/.codex-plugin/plugin.json              70 opens
outlook-email/.codex-plugin/plugin.json      70 opens
outlook-calendar/.codex-plugin/plugin.json   70 opens
google-drive/.codex-plugin/plugin.json       63 opens
google-calendar/.codex-plugin/plugin.json    63 opens
models_cache.json                            15 opens

Approximately 180 different plugin.json files were visited repeatedly.

Gateway investigation:

No evidence that Gateway creates a new Codex session for every request.
The existing session is normally reused.

Screenshots, recordings, and evidence

Image

Impact and severity

Affected users/systems/channels

Observed in a persistent main agent using the OpenAI Codex app-server through OpenClaw Gateway. The issue was reproduced while processing complex investigation tasks. NOT_ENOUGH_INFO for other providers or deployment configurations.

Severity

Blocks workflow. During affected requests, disk I/O becomes extremely high, SSH becomes unresponsive, and the server eventually cannot continue processing until the task is terminated or the Gateway is restarted.

Frequency

Intermittent but reproducible in my environment. It consistently occurs with certain complex investigation tasks, but does not occur for every request.

Consequence

The affected request cannot complete normally. The server becomes temporarily unusable, requiring manual intervention (terminating the task or restarting the Gateway) before normal operation can resume. No data loss or corruption was observed.

Additional information

Environment

  • Alibaba Cloud Lightweight Application Server
  • Ubuntu 24.04.4 LTS
  • 2 vCPU /4 GB RAM+3GB SWP
  • OpenClaw v2026.6.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions