fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out#77973
Conversation
|
ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed June 30, 2026, 3:27 PM ET / 19:27 UTC. Summary PR surface: Source +27, Tests +16. Total +43 across 2 files. Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path. Review metrics: none identified. Merge readiness This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict. Risk before merge
Maintainer options:
Next step before merge
Review detailsBest possible solution: Retry the Codex review after fixing the execution failure. Do we have a high-confidence way to reproduce the issue? Unclear. The review failed before ClawSweeper could establish a reproduction path. Is this the best way to solve the issue? Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction. AGENTS.md: unclear because the file could not be read completely. Codex review notes: model internal, reasoning high; reviewed against 66e676d29b92. Label changesLabel changes:
Evidence reviewedPR surface: Source +27, Tests +16. Total +43 across 2 files. View PR surface stats
What I checked:
Likely related people:
How this review workflow works
|
5c2c56d to
442f424
Compare
|
Rebased onto current @openclaw/openclaw-secops — flagging for review (gateway runtime cache bound under fan-out). |
|
This pull request has been automatically marked as stale due to inactivity. |
442f424 to
1c9498a
Compare
… fan-out Time-based prune only reclaims entries past the 10-minute TTL window; a burst of run fan-out can add far more entries than the window reclaims, so the cache could grow without bound between prunes. Add a FIFO entry cap (5000) enforced on insert, mirroring the existing Discord REST entity-cache bound. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1c9498a to
cf9ec81
Compare
|
Maintainer update: pushed a narrow follow-up to preserve cached terminal snapshots for active fresh Validation run on the repaired head:
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
vincentkoc
left a comment
There was a problem hiding this comment.
Approved after maintainer repair.
What I checked:
agentRunCachewas TTL-only on main; linked issue #77976 is valid.- The PR adds a hard cap and regression coverage.
- I pushed a follow-up that preserves cached terminal snapshots for active fresh
agent.waitcallers while enforcing the cap. - No review threads are open.
- Exact-head CI is green on
64ae34e9b37101b04f22524e9bf893cc08cf677a: 65 successful, 34 skipped, 0 failing, 0 pending after rerunning one unrelated message-tool timeout shard.
Local proof:
node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts -t "agentRunCache|active fresh waiters"node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts
… fan-out (openclaw#77973) * fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out Time-based prune only reclaims entries past the 10-minute TTL window; a burst of run fan-out can add far more entries than the window reclaims, so the cache could grow without bound between prunes. Add a FIFO entry cap (5000) enforced on insert, mirroring the existing Discord REST entity-cache bound. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * test(gateway): preserve waited run snapshots under cache cap --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: Vincent Koc <[email protected]>
… fan-out (openclaw#77973) * fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out Time-based prune only reclaims entries past the 10-minute TTL window; a burst of run fan-out can add far more entries than the window reclaims, so the cache could grow without bound between prunes. Add a FIFO entry cap (5000) enforced on insert, mirroring the existing Discord REST entity-cache bound. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * test(gateway): preserve waited run snapshots under cache cap --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: Vincent Koc <[email protected]>
… fan-out (openclaw#77973) * fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out Time-based prune only reclaims entries past the 10-minute TTL window; a burst of run fan-out can add far more entries than the window reclaims, so the cache could grow without bound between prunes. Add a FIFO entry cap (5000) enforced on insert, mirroring the existing Discord REST entity-cache bound. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * test(gateway): preserve waited run snapshots under cache cap --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: Vincent Koc <[email protected]>
… fan-out (openclaw#77973) * fix(gateway): cap agentRunCache to prevent unbounded growth under run fan-out Time-based prune only reclaims entries past the 10-minute TTL window; a burst of run fan-out can add far more entries than the window reclaims, so the cache could grow without bound between prunes. Add a FIFO entry cap (5000) enforced on insert, mirroring the existing Discord REST entity-cache bound. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * test(gateway): preserve waited run snapshots under cache cap --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit f35fbc8)
Closes #77976
Summary
src/gateway/server-methods/agent-job.tskeeps an in-processagentRunCache: Map<runId, AgentRunSnapshot>populated by every terminal lifecycle event. There is a 10-minute TTL pruned on everysetviapruneAgentRunCache, but no max-size cap. Under sustained subagent fan-out (lots ofagent.runlifecycle traffic), the snapshot map grew in lockstep with run rate up to "all runs in the last 10 minutes" before the next TTL sweep could bound it. Same shape as the Discord REST entity cache fixed in #77952.This change adds:
AGENT_RUN_CACHE_MAX_ENTRIES(default 5,000) and anenforceAgentRunCacheMaxEntriespass that runs after eachagentRunCache.set, dropping oldest entries by Map insertion order until size ≤ max.__testing.getAgentRunCacheSize(),__testing.resetAgentRunCache(), and__testing.agentRunCacheMaxEntriesso the cap is regression-testable.server-methods.test.tsthat drivesMAX + 25unique end-lifecycle events through the realemitAgentEventpath and asserts size pins at the cap.The sibling Maps in this file (
agentRunStarts,agentRunWaiterCounts,pendingAgentRunErrors,pendingAgentRunTimeouts) are already lifecycle-bound viastart/end/wait-register paths; only the terminal-snapshot cache lacked a hard cap.Verification
pnpm test src/gateway/server-methods/server-methods.test.ts→ 82/82 pass including newcaps agentRunCache at AGENT_RUN_CACHE_MAX_ENTRIES via FIFO drop.pnpm exec oxfmt --check --threads=1 src/gateway/server-methods/agent-job.ts src/gateway/server-methods/server-methods.test.ts CHANGELOG.mdclean.Real behavior proof
agentRunCachehad a TTL but no hard cap; under sustained run fan-out it could hold every run snapshot from the past 10 minutes simultaneously, scaling with run rate × 10 min and capped only by process restart or quiet periods.agent-job.tsmodule (no mocks, no test framework). Driven viapnpm exec tsxagainst the actualemitAgentEventlifecycle path that production code uses./tmp/agent-job-cache-proof.mtswhich imports the real patched module andemitAgentEvent, then drivesMAX + 50 = 5,050unique terminal lifecycle events through the real listener and prints__testing.getAgentRunCacheSize()at intervals.enforceAgentRunCacheMaxEntriespins it at exactly 5,000 across the remaining 50 inserts. Oldest snapshots evicted by Map insertion order (FIFO over insertion-order; on every set the new key moves to the tail, so this is effectively close-to-LRU for repeat runs and FIFO for unique-runId floods).Mapcap on a deterministic insert path and the runtime demo exercises the same listener / record path that production runs through.Notes for reviewer
pruneAgentRunCache) and the new cap (enforceAgentRunCacheMaxEntries) compose: TTL prunes before insert, cap enforces after insert. Either alone is insufficient — TTL doesn't bound under sustained insert rate, cap doesn't free expired-but-still-inside-cap entries.