Skip to content

fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out#77973

Merged
vincentkoc merged 2 commits into
openclaw:mainfrom
fede-kamel:security/agent-job-cache-bound
Jul 1, 2026
Merged

fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out#77973
vincentkoc merged 2 commits into
openclaw:mainfrom
fede-kamel:security/agent-job-cache-bound

Conversation

@fede-kamel

@fede-kamel fede-kamel commented May 5, 2026

Copy link
Copy Markdown
Contributor

Closes #77976

Summary

src/gateway/server-methods/agent-job.ts keeps an in-process agentRunCache: Map<runId, AgentRunSnapshot> populated by every terminal lifecycle event. There is a 10-minute TTL pruned on every set via pruneAgentRunCache, but no max-size cap. Under sustained subagent fan-out (lots of agent.run lifecycle traffic), the snapshot map grew in lockstep with run rate up to "all runs in the last 10 minutes" before the next TTL sweep could bound it. Same shape as the Discord REST entity cache fixed in #77952.

This change adds:

  • AGENT_RUN_CACHE_MAX_ENTRIES (default 5,000) and an enforceAgentRunCacheMaxEntries pass that runs after each agentRunCache.set, dropping oldest entries by Map insertion order until size ≤ max.
  • __testing.getAgentRunCacheSize(), __testing.resetAgentRunCache(), and __testing.agentRunCacheMaxEntries so the cap is regression-testable.
  • A regression test in server-methods.test.ts that drives MAX + 25 unique end-lifecycle events through the real emitAgentEvent path and asserts size pins at the cap.

The sibling Maps in this file (agentRunStarts, agentRunWaiterCounts, pendingAgentRunErrors, pendingAgentRunTimeouts) are already lifecycle-bound via start/end/wait-register paths; only the terminal-snapshot cache lacked a hard cap.

Verification

  • pnpm test src/gateway/server-methods/server-methods.test.ts → 82/82 pass including new caps agentRunCache at AGENT_RUN_CACHE_MAX_ENTRIES via FIFO drop.
  • pnpm exec oxfmt --check --threads=1 src/gateway/server-methods/agent-job.ts src/gateway/server-methods/server-methods.test.ts CHANGELOG.md clean.
  • Live tsx runtime proof included below in Real behavior proof.

Real behavior proof

  • Behavior addressed: agentRunCache had a TTL but no hard cap; under sustained run fan-out it could hold every run snapshot from the past 10 minutes simultaneously, scaling with run rate × 10 min and capped only by process restart or quiet periods.
  • Real environment tested: local Node v22 runtime against the real patched agent-job.ts module (no mocks, no test framework). Driven via pnpm exec tsx against the actual emitAgentEvent lifecycle path that production code uses.
  • Exact steps or command run after this patch: ran /tmp/agent-job-cache-proof.mts which imports the real patched module and emitAgentEvent, then drives MAX + 50 = 5,050 unique terminal lifecycle events through the real listener and prints __testing.getAgentRunCacheSize() at intervals.
  • Evidence after fix: live console output captured directly from the node runtime:
agentRunCacheMaxEntries: 5000
emitting 5050 unique end-lifecycle events...
  emitted     0  cacheSize=1
  emitted  1000  cacheSize=1001
  emitted  2000  cacheSize=2001
  emitted  3000  cacheSize=3001
  emitted  4000  cacheSize=4001
  emitted  5000  cacheSize=5000
final cacheSize after 5050 unique runs: 5000
expected ≤ 5000: PASS
  • Observed result after fix: cache grows linearly to 5,000, then enforceAgentRunCacheMaxEntries pins it at exactly 5,000 across the remaining 50 inserts. Oldest snapshots evicted by Map insertion order (FIFO over insertion-order; on every set the new key moves to the tail, so this is effectively close-to-LRU for repeat runs and FIFO for unique-runId floods).
  • What was not tested: behavior against a real long-running gateway with thousands of concurrent subagents. The fix is purely a Map cap on a deterministic insert path and the runtime demo exercises the same listener / record path that production runs through.

Notes for reviewer

  • The TTL prune (pruneAgentRunCache) and the new cap (enforceAgentRunCacheMaxEntries) compose: TTL prunes before insert, cap enforces after insert. Either alone is insufficient — TTL doesn't bound under sustained insert rate, cap doesn't free expired-but-still-inside-cap entries.
  • Cap value 5,000 was chosen to mirror the Discord cache cap precedent and gives ~10 minutes of headroom at ~8 runs/sec sustained terminal events, which is generously above any realistic single-gateway run rate.
  • This is part of a small audit sweep following fix(discord): bound REST entity cache to prevent unbounded Map growth #77952; same shape, different module.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 5, 2026
@clawsweeper

clawsweeper Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed June 30, 2026, 3:27 PM ET / 19:27 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +27, Tests +16. Total +43 across 2 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Merge readiness
Not assessed.
Failure reason: timeout.

This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict.
Keep any merge decision on the normal maintainer review path until ClawSweeper can complete a fresh review.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 66e676d29b92.

Label changes

Label changes:

  • remove P2: Current review triage priority is none.
  • remove merge-risk: 🚨 session-state: Current PR review selected no merge-risk labels.
Evidence reviewed

PR surface:

Source +27, Tests +16. Total +43 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 27 0 +27
Tests 1 17 1 +16
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 44 1 +43

What I checked:

  • failure reason: timeout.
  • codex failure detail: Codex review failed for this PR: Codex process timed out after 1200000ms.
  • codex stderr: 2026-06-30T19:11:34.330788Z ERROR codex_core::tools::router: error=write_stdin failed: stdin is closed for this session; rerun exec_command with tty=true to keep stdin open.
  • codex stdout: -delta.mjs | 556 +\n .agents/skills/openclaw-test-performance/SKILL.md | 266 +\n .../openclaw-test-performance/agents/openai.yaml | 6 +\n .agents/skills/openclaw-testing/SKILL.md | 679 +\n .agents/skills/openclaw-testing/agents/openai.yaml | 4 +\n .../skills/parallels-discord-roundtrip/SKILL.md | 63 +\n .../skills/release-openclaw-announcement/SKILL.md | 87 +\n .../agents/openai.yaml | 4 +\n .agents/skills/release-openclaw-ci/SKILL.md | 174 +\n .../skills/release-openclaw-ci/agents/openai.yaml | 4 +\n .../references/release-ci-notes.md | 41 +\n .../scripts/release-ci-summary.mjs | 128 +\n .../scripts/verify-provider-secrets.mjs | 142 +\n .agents/skills/release-openclaw-mac/SKILL.md | 133 +\n .../skills/release-openclaw-maintainer/SKILL.md | 891 ++\n .agents/skills/rel.
  • process error code: ETIMEDOUT.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 5, 2026
@fede-kamel
fede-kamel force-pushed the security/agent-job-cache-bound branch from 5c2c56d to 442f424 Compare May 6, 2026 21:58
@openclaw-barnacle openclaw-barnacle Bot added the proof: supplied External PR includes structured after-fix real behavior proof. label May 6, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 6, 2026
@fede-kamel

Copy link
Copy Markdown
Contributor Author

Rebased onto current origin/main (was 372 commits behind). The previous "Real behavior proof" CI failure was a missing-script artifact — scripts/github/real-behavior-proof-check.mjs was added in #77622 after this branch was first pushed, so the workflow could not load on the old base. The script is on HEAD now and CI is re-running.

@openclaw/openclaw-secops — flagging for review (gateway runtime cache bound under fan-out).

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label May 31, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. labels May 31, 2026
@barnacle-openclaw barnacle-openclaw Bot removed the stale Marked as stale due to inactivity label May 31, 2026
@fede-kamel
fede-kamel force-pushed the security/agent-job-cache-bound branch from 442f424 to 1c9498a Compare June 11, 2026 20:58
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 11, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 14, 2026
… fan-out

Time-based prune only reclaims entries past the 10-minute TTL window; a burst
of run fan-out can add far more entries than the window reclaims, so the cache
could grow without bound between prunes. Add a FIFO entry cap (5000) enforced
on insert, mirroring the existing Discord REST entity-cache bound.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
@fede-kamel
fede-kamel force-pushed the security/agent-job-cache-bound branch from 1c9498a to cf9ec81 Compare June 29, 2026 16:53
@clawsweeper clawsweeper Bot removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 30, 2026
@vincentkoc vincentkoc self-assigned this Jul 1, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Maintainer update: pushed a narrow follow-up to preserve cached terminal snapshots for active fresh agent.wait callers while enforcing the agentRunCache cap.

Validation run on the repaired head:

  • node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts -t "agentRunCache|active fresh waiters"
  • node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@vincentkoc vincentkoc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after maintainer repair.

What I checked:

  • agentRunCache was TTL-only on main; linked issue #77976 is valid.
  • The PR adds a hard cap and regression coverage.
  • I pushed a follow-up that preserves cached terminal snapshots for active fresh agent.wait callers while enforcing the cap.
  • No review threads are open.
  • Exact-head CI is green on 64ae34e9b37101b04f22524e9bf893cc08cf677a: 65 successful, 34 skipped, 0 failing, 0 pending after rerunning one unrelated message-tool timeout shard.

Local proof:

  • node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts -t "agentRunCache|active fresh waiters"
  • node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts

@vincentkoc
vincentkoc merged commit f35fbc8 into openclaw:main Jul 1, 2026
157 of 160 checks passed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 2, 2026
… fan-out (openclaw#77973)

* fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out

Time-based prune only reclaims entries past the 10-minute TTL window; a burst
of run fan-out can add far more entries than the window reclaims, so the cache
could grow without bound between prunes. Add a FIFO entry cap (5000) enforced
on insert, mirroring the existing Discord REST entity-cache bound.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* test(gateway): preserve waited run snapshots under cache cap

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
… fan-out (openclaw#77973)

* fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out

Time-based prune only reclaims entries past the 10-minute TTL window; a burst
of run fan-out can add far more entries than the window reclaims, so the cache
could grow without bound between prunes. Add a FIFO entry cap (5000) enforced
on insert, mirroring the existing Discord REST entity-cache bound.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* test(gateway): preserve waited run snapshots under cache cap

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
sheyanmin pushed a commit to sheyanmin/openclaw that referenced this pull request Jul 8, 2026
… fan-out (openclaw#77973)

* fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out

Time-based prune only reclaims entries past the 10-minute TTL window; a burst
of run fan-out can add far more entries than the window reclaims, so the cache
could grow without bound between prunes. Add a FIFO entry cap (5000) enforced
on insert, mirroring the existing Discord REST entity-cache bound.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* test(gateway): preserve waited run snapshots under cache cap

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
wheakerd pushed a commit to wheakerd/clawdbot that referenced this pull request Jul 15, 2026
… fan-out (openclaw#77973)

* fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out

Time-based prune only reclaims entries past the 10-minute TTL window; a burst
of run fan-out can add far more entries than the window reclaims, so the cache
could grow without bound between prunes. Add a FIFO entry cap (5000) enforced
on insert, mirroring the existing Discord REST entity-cache bound.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* test(gateway): preserve waited run snapshots under cache cap

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
(cherry picked from commit f35fbc8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: agent-job agentRunCache grows unbounded under sustained run fan-out

2 participants