Skip to content

fix(codex): cache plugin inventory to prevent repeated disk I/O (#99071)#99177

Closed
jtcole wants to merge 1 commit into
openclaw:mainfrom
jtcole:fix/codex-plugin-discovery-io
Closed

fix(codex): cache plugin inventory to prevent repeated disk I/O (#99071)#99177
jtcole wants to merge 1 commit into
openclaw:mainfrom
jtcole:fix/codex-plugin-discovery-io

Conversation

@jtcole

@jtcole jtcole commented Jul 2, 2026

Copy link
Copy Markdown

Problem

During complex multi-turn tasks, buildCodexPluginThreadConfig calls readCodexPluginInventory up to 4 times per request cycle. Each call issues a plugin/list RPC to the Codex app-server, which scans ~180 plugin.json files from disk. This produces 1.3–1.4 GB of disk reads in 30 seconds and drives disk utilization to 99% on resource-constrained cloud servers (2 vCPU / 4 GB RAM Ubuntu 24.04.4), making the server unresponsive.

Reported in #99071.

Root Cause

The existing CodexAppInventoryCache only caches app/list responses. The plugin/list and plugin/read RPC paths — the ones that trigger the heavy disk scanning — had no cache at all. Every call to readCodexPluginInventory resulted in a fresh plugin/list RPC, and buildCodexPluginThreadConfig calls it up to 4 times:

  1. Initial inventory read (line 126)
  2. After waiting for initial app inventory (line 147)
  3. After post-install refresh (line 197)
  4. After not-ready plugin apps refresh (line 216)

Additionally, ensureCodexPluginActivation in plugin-activation.ts makes its own plugin/list call for each plugin that needs activation.

Fix

Introduces CodexPluginListCache — a process-local in-memory cache for plugin/list responses with:

  • 5-minute TTL (shorter than app inventory's 1-hour TTL since plugin install/enable state changes are more disruptive)
  • Coalesced refreshes per cache key (concurrent callers share the same in-flight request)
  • Refresh generation tokens to prevent older in-flight refreshes from overwriting newer snapshots
  • Invalidation API for post-install state changes

Wiring

File Change
plugin-list-cache.ts New file — CodexPluginListCache class + defaultCodexPluginListCache singleton
plugin-inventory.ts readCodexPluginInventory uses readOrRefresh() for plugin/list when cache is provided
plugin-thread-config.ts All 4 readCodexPluginInventory calls + ensureCodexPluginActivation pass the shared cache
attempt-startup.ts buildCodexPluginThreadConfig callback passes defaultCodexPluginListCache
plugin-activation.ts Cache invalidated on plugin install and during refreshCodexPluginRuntimeState

Design Decision: Activation Path Stays Uncached

ensureCodexPluginActivation intentionally does NOT use the cache for its initial plugin/list read. It needs fresh install/enable state to decide whether plugin/install is needed. Using a cached response could show a plugin as not-yet-installed when it actually was just installed by a concurrent process, leading to a redundant plugin/install call. The cache is invalidated after install so subsequent reads see the new state.

Impact

Before: 4+ plugin/list RPCs per buildCodexPluginThreadConfig call × ~180 plugin.json file reads each = 1.3–1.4 GB disk I/O in 30 seconds.

After: 1 plugin/list RPC per cache TTL window (5 minutes). Subsequent calls within that window return the cached snapshot. Cache is invalidated on plugin install/activation so post-install state changes are always observed.

Tests

  • 10 new tests in plugin-list-cache.test.ts: cache reads, refreshes, coalescing, invalidation, forced refetch, error propagation, revision tracking, clear
  • 2 new tests in plugin-inventory.test.ts: verify repeated readCodexPluginInventory calls use the cached plugin/list response, and forcePluginListRefetch bypasses the cache
  • plugin-thread-config.test.ts: beforeEach clears defaultCodexPluginListCache to prevent cross-test cache pollution
  • All 73 existing + new tests pass across 6 test files

Fixes #99071

@jtcole
fix(codex): cache plugin inventory to prevent repeated disk I/O (#99071)
7eac710 
During complex multi-turn tasks, buildCodexPluginThreadConfig calls
readCodexPluginInventory up to 4 times, and each call issues a
plugin/list RPC to the Codex app-server. The app-server then scans
~180 plugin.json files from disk on every call, producing 1.3–1.4 GB
of disk reads in 30 seconds and driving disk utilization to 99% on
resource-constrained cloud servers.

The existing CodexAppInventoryCache only caches app/list responses.
The plugin/list and plugin/read RPC paths had no cache at all.

This commit introduces CodexPluginListCache — a process-local
in-memory cache for plugin/list responses with a 5-minute TTL and
coalesced refreshes. The cache is wired into:

- readCodexPluginInventory (plugin-inventory.ts): plugin/list calls
  now use readOrRefresh() which returns a cached fresh snapshot
  instead of hitting the server
- buildCodexPluginThreadConfig (plugin-thread-config.ts): all 4
  readCodexPluginInventory calls now pass the shared plugin list cache
- attempt-startup.ts: the build callback passes
  defaultCodexPluginListCache alongside the existing app inventory
  cache
- plugin-activation.ts: the cache is invalidated on plugin install
  and during refreshCodexPluginRuntimeState so post-install state
  changes are always observed

The activation path (ensureCodexPluginActivation) intentionally does
NOT use the cache for its initial plugin/list read — it needs fresh
install/enable state to decide whether plugin/install is needed.

Tests:
- 10 new tests in plugin-list-cache.test.ts covering cache reads,
  refreshes, coalescing, invalidation, forced refetch, error
  propagation, and revision tracking
- 2 new tests in plugin-inventory.test.ts verifying that repeated
  readCodexPluginInventory calls use the cached plugin/list response
  and that forcePluginListRefetch bypasses the cache
- plugin-thread-config.test.ts clears defaultCodexPluginListCache in
  beforeEach to prevent cross-test cache pollution

Fixes #99071
@openclaw-barnacle openclaw-barnacle Bot added extensions: codex size: L triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 2, 2026
@clawsweeper

clawsweeper Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 2, 2026, 1:13 PM ET / 17:13 UTC.

Summary
The PR adds a process-local Codex plugin/list cache, wires it into native plugin inventory/thread-config reads, invalidates it around activation refreshes, and adds unit coverage.

PR surface: Source +258, Tests +335. Total +593 across 8 files.

Reproducibility: no. high-confidence live reproduction was run for the full disk-I/O symptom on current main. Source inspection does confirm the reported repeated OpenClaw plugin/list call path, and the linked issue provides concrete v2026.6.1 opensnoop/atop evidence.

Review metrics: 1 noteworthy metric.

  • Control-plane RPC cache: 1 added. Caching plugin/list adds a freshness window to a runtime state read, so invalidation and upgrade behavior need maintainer attention before merge.

Stored data model
Persistent data-model change detected: persistent cache schema: extensions/codex/src/app-server/attempt-startup.ts, persistent cache schema: extensions/codex/src/app-server/plugin-activation.ts, persistent cache schema: extensions/codex/src/app-server/plugin-inventory.test.ts, persistent cache schema: extensions/codex/src/app-server/plugin-inventory.ts, persistent cache schema: extensions/codex/src/app-server/plugin-list-cache.test.ts, persistent cache schema: extensions/codex/src/app-server/plugin-thread-config.test.ts, and 1 more. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #99071
Summary: This PR is the candidate fix for the linked high Codex plugin discovery disk-I/O issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦪 silver shellfish
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Fix the unused-symbol type/lint failures on the PR head.
  • [P1] Add redacted terminal output, logs, or a recording from a real Gateway/Codex app-server run showing reduced repeated plugin/list or manifest reads after the change.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists unit tests but does not include redacted after-fix terminal output, logs, recording, or linked artifacts from a real Gateway/Codex app-server run; contributors should redact private data before posting proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The PR still needs redacted live proof that a real Gateway/Codex app-server run reduces repeated plugin/list or manifest reads in the reported environment.
  • [P1] The new five-minute plugin/list cache changes how quickly externally made Codex plugin install/enable changes are observed; maintainers should explicitly accept or narrow that freshness window.

Maintainer options:

  1. Prove and bound plugin-list freshness (recommended)
    Before merge, fix CI and show live proof that the cache reduces disk I/O while post-install and externally changed plugin state remain acceptably fresh.
  2. Accept the five-minute state lag
    Maintainers may intentionally accept that external Codex plugin install/enable changes can be delayed for up to the cache TTL.
  3. Pause for current-main profiling
    If the live proof cannot isolate plugin/list as the remaining hot path, pause this branch and profile the linked issue on current main first.

Next step before merge

  • [P1] The contributor needs to fix CI and provide real behavior proof; ClawSweeper should not repair this branch while the external setup proof gate is missing.

Security
Cleared: The diff adds in-memory caching and tests under the Codex plugin app-server path and does not change dependencies, secrets, CI workflows, package resolution, or privilege boundaries.

Review findings

  • [P2] Remove the unused cache type import — extensions/codex/src/app-server/attempt-startup.ts:38
  • [P2] Remove or prefix the unused invalidation timestamp — extensions/codex/src/app-server/plugin-list-cache.ts:118
  • [P2] Prefix the unused test request parameter — extensions/codex/src/app-server/plugin-inventory.test.ts:367
Review details

Best possible solution:

Fix the unused-symbol errors, keep the bounded cache only with an accepted freshness/invalidation story, and add redacted live Gateway/Codex profiling proof before merge.

Do we have a high-confidence way to reproduce the issue?

No high-confidence live reproduction was run for the full disk-I/O symptom on current main. Source inspection does confirm the reported repeated OpenClaw plugin/list call path, and the linked issue provides concrete v2026.6.1 opensnoop/atop evidence.

Is this the best way to solve the issue?

No, not as submitted: the cache is a plausible bounded fix for repeated OpenClaw plugin/list calls, but this head fails type/lint checks and lacks live after-fix disk-I/O proof.

Full review comments:

  • [P2] Remove the unused cache type import — extensions/codex/src/app-server/attempt-startup.ts:38
    CodexPluginListCache is imported but never referenced in this module, so tsgo:prod and lint fail on this PR head. Remove the unused type from the import.
    Confidence: 0.98
  • [P2] Remove or prefix the unused invalidation timestamp — extensions/codex/src/app-server/plugin-list-cache.ts:118
    The new invalidate signature accepts nowMs but never reads it, which fails both type and lint checks. Drop the parameter or rename it with the repository's unused-parameter convention.
    Confidence: 0.98
  • [P2] Prefix the unused test request parameter — extensions/codex/src/app-server/plugin-inventory.test.ts:367
    The added test request callback declares params but does not use it, so extension lint fails. Prefix it as unused or remove it from the callback signature.
    Confidence: 0.96

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against d3d7282a8cf4.

Label changes

Label justifications:

  • P1: The linked bug describes a Codex-backed Gateway workflow that can drive disk utilization near 100% and leave the host unresponsive.
  • merge-risk: 🚨 compatibility: The PR changes the freshness of Codex plugin install/enable state observed by existing running OpenClaw processes.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists unit tests but does not include redacted after-fix terminal output, logs, recording, or linked artifacts from a real Gateway/Codex app-server run; contributors should redact private data before posting proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +258, Tests +335. Total +593 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 5 261 3 +258
Tests 3 336 1 +335
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 8 597 4 +593

What I checked:

Likely related people:

  • kevinslin: Authored the original native Codex plugin app support and multiple recent fixes in plugin-thread-config.ts, plugin-inventory.ts, plugin-activation.ts, and app-inventory-cache.ts. (role: feature owner / recent area contributor; confidence: high; commits: a1ac559ed7e6, bc5081c58730, 58367137eac6; files: extensions/codex/src/app-server/plugin-thread-config.ts, extensions/codex/src/app-server/plugin-inventory.ts, extensions/codex/src/app-server/plugin-activation.ts)
  • steipete: Recently touched Codex app-server cleanup docs and app inventory cache expiry behavior adjacent to this cache/freshness review. (role: recent adjacent contributor; confidence: medium; commits: 5fde637ba88f, 23258c86bec8, d99268ae514a; files: extensions/codex/src/app-server/app-inventory-cache.ts, extensions/codex/src/app-server/plugin-thread-config.ts)
  • Shijie Rao: The inspected sibling Codex checkout blames the current plugin/list processor and loaded-plugin cache path to this author in the dependency code OpenClaw calls. (role: upstream Codex dependency contributor; confidence: medium; commits: cbdd7f0047c1; files: ../codex/codex-rs/app-server/src/request_processors/plugins.rs, ../codex/codex-rs/core-plugins/src/manager.rs)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jul 2, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jul 6, 2026
Closed
@zhangguiping-xydt zhangguiping-xydt mentioned this pull request Jul 9, 2026
Merged
5 tasks
@openclaw-barnacle

openclaw-barnacle Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 17, 2026
@ldrlinj ldrlinj mentioned this pull request Jul 17, 2026
Open
@clawsweeper

clawsweeper Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(codex): cache plugin inventory to prevent repeated disk I/O (#99071) This is item 1/1 in the current shard. Shard 1/22.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@jtcole jtcole closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: codex merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: L stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Possible repeated Codex Apps plugin discovery causing excessive disk I/O during a single request (investigated with opensnoop and atop)

1 participant

@jtcole