Skip to content

fix(memory-core): bound managed dreaming under pressure#84848

Open
galiniliev wants to merge 4 commits into
openclaw:mainfrom
galiniliev:bug-026-memory-dreaming-pressure
Open

fix(memory-core): bound managed dreaming under pressure#84848
galiniliev wants to merge 4 commits into
openclaw:mainfrom
galiniliev:bug-026-memory-dreaming-pressure

Conversation

@galiniliev

@galiniliev galiniliev commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: managed memory dreaming cron could keep adding workspace and Dream Diary work while the gateway was already under RSS/heap pressure, and the pressure wait path did not observe the enclosing cron timeout/cancellation.
  • Solution: preserve the existing full scheduled sweep, but add abort-aware bounded memory-pressure backoff before each cron workspace attempt and before detached Dream Diary scheduling.
  • Exact behavior now: each cron-triggered managed dreaming run still iterates every configured workspace in the same cron invocation. Before starting each workspace, memory-core checks process RSS and heap usage. Before scheduling the detached deep Dream Diary narrative, it checks the same pressure again because promotion/report work may have raised memory usage. If RSS is at least 1536 MiB or heap used is at least 1024 MiB, it waits and retries with exponential backoff: 30s, 60s, 120s, 240s, then capped at 300s between retries. It keeps retrying at the capped delay until pressure clears or the enclosing cron run aborts/times out.
  • Cron cancellation behavior now: the cron run abort signal is forwarded through before_agent_reply hook contexts for getReply, CLI cron, and embedded Pi cron paths. If that signal fires while memory-core is sleeping in pressure backoff, the sleep resolves immediately, the dreaming hook stops, and the timed-out cron run does not later resume memory mutations after pressure clears.
  • Dream Diary behavior now: Dream Diary generation is not dropped just because pressure is high after promotion. Detached narrative scheduling waits with the same bounded backoff and proceeds after pressure clears; if the cron timeout/cancellation arrives first, the cron run stops cleanly.
  • What did NOT change: direct non-cron dreaming requests are not delayed by this scheduler guard; the dreaming frequency/config switches are unchanged; memory storage formats are unchanged; cron still owns one scheduled managed dreaming job; memory-core does not intentionally defer remaining workspaces to a later cron tick.

Motivation

  • Observed gateway performance logs showed managed memory dreaming processing 10 workspaces in one background tick, taking 69621 ms, while nearby diagnostics reported high RSS and saturated event-loop/CPU conditions.
  • The desired fix is to avoid adding more work while the process is under pressure, without dropping the existing full-sweep semantics for multi-workspace users or dropping Dream Diary output under pressure.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Real behavior proof (required for external PRs)

  • Behavior addressed: Managed memory-core dreaming cron keeps the existing all-workspace sweep, waits under gateway RSS/heap pressure before each workspace and before detached Dream Diary scheduling, and exits pressure backoff promptly when the enclosing cron abort signal fires.
  • Real environment tested: Local OpenClaw worktree with focused static/type proof and autoreview; before evidence comes from redacted runtime diagnostics captured from the affected gateway.
  • Exact steps or command run after this patch: pnpm format:fix extensions/memory-core/src/dreaming.ts extensions/memory-core/src/dreaming.test.ts src/auto-reply/reply/get-reply.ts src/auto-reply/reply/get-reply.before-agent-reply.test.ts src/agents/cli-runner.ts src/agents/cli-runner.before-agent-reply-cron.test.ts src/agents/pi-embedded-runner/run.ts src/agents/pi-embedded-runner/run.before-agent-reply-cron.test.ts src/plugins/hook-types.ts; git diff --check; pnpm run lint:extensions:bundled; node scripts/run-oxlint.mjs src/plugins/hook-types.ts src/auto-reply/reply/get-reply.ts src/auto-reply/reply/get-reply.before-agent-reply.test.ts src/agents/cli-runner.ts src/agents/cli-runner.before-agent-reply-cron.test.ts src/agents/pi-embedded-runner/run.ts src/agents/pi-embedded-runner/run.before-agent-reply-cron.test.ts; pnpm tsgo:extensions; pnpm tsgo:core; AUTOREVIEW_AUTO_TESTS=0 .agents/skills/autoreview/scripts/autoreview --mode local.
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
$ git diff --check
# no output

$ pnpm run lint:extensions:bundled
# passed

$ node scripts/run-oxlint.mjs src/plugins/hook-types.ts src/auto-reply/reply/get-reply.ts src/auto-reply/reply/get-reply.before-agent-reply.test.ts src/agents/cli-runner.ts src/agents/cli-runner.before-agent-reply-cron.test.ts src/agents/pi-embedded-runner/run.ts src/agents/pi-embedded-runner/run.before-agent-reply-cron.test.ts
# passed

$ pnpm tsgo:extensions
# passed

$ pnpm tsgo:core
# passed

$ AUTOREVIEW_AUTO_TESTS=0 .agents/skills/autoreview/scripts/autoreview --mode local
autoreview clean: no accepted/actionable findings reported
overall: patch is correct (0.81)
  • Observed result after fix: The changed tests cover abort-aware pressure backoff, propagation of abort signals through before_agent_reply hook contexts, continued full-workspace cron iteration after pressure clears, capped pressure retry behavior, and Dream Diary scheduling after pressure clears instead of being dropped.
  • What was not tested: Local Vitest execution is currently blocked in this worktree: OPENCLAW_VITEST_MAX_WORKERS=1 timeout 180s node scripts/run-vitest.mjs ... timed out after printing only the Vitest banner. Remote focused Vitest proof was also blocked here: Blacksmith Testbox is unavailable because the blacksmith CLI is not installed, and direct AWS Crabbox is unavailable because AWS credentials/IMDS are not present. A live multi-workspace gateway run with private agent/session data was not rerun.
  • Before evidence (optional but encouraged):
memory-core: dreaming promotion complete (workspaces=10, candidates=10, applied=1, failed=0)
observed managed background task duration: 69621ms
memory pressure warning: rssBytes=1827979264 heapUsedBytes=684072312 thresholdBytes=1610612736
liveness warning: eventLoopUtilization=0.991 cpuCoreRatio=1.107 active=0 waiting=0 queued=0

Root Cause (if applicable)

  • Root cause: Managed cron dreaming collected every configured workspace and processed them in one cron delivery without pausing between workspaces when process memory was already above warning thresholds. The new pressure backoff initially had no cancellation/deadline awareness, so a timed-out cron run could keep sleeping and later mutate memory after pressure cleared.
  • Missing detection / guardrail: Existing tests covered short-term dreaming promotion but did not exercise multi-workspace managed cron sweeps, pressure backoff cancellation, detached Dream Diary pressure handling, or hook-context abort propagation.
  • Contributing context (if known): The managed cron system event can be claimed through before-agent-reply hooks in getReply, CLI cron, and embedded Pi cron paths, so all three hook contexts need the same abort signal.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: extensions/memory-core/src/dreaming.test.ts, src/auto-reply/reply/get-reply.before-agent-reply.test.ts, src/agents/cli-runner.before-agent-reply-cron.test.ts, src/agents/pi-embedded-runner/run.before-agent-reply-cron.test.ts
  • Scenario the test should lock in: managed cron dreaming keeps the full configured workspace sweep, waits under RSS/heap pressure before workspace and detached narrative work, doubles retry delay from 30 seconds to a 5 minute cap, continues in the same cron invocation after pressure clears, and stops without later memory mutation when the cron abort signal fires.
  • Why this is the smallest reliable guardrail: The behavior is owned inside the memory-core dreaming cron seam and before-agent-reply hook contexts, so it can be exercised without private gateway sessions or provider credentials.
  • Existing test that already covers this (if any): Existing dreaming tests covered promotion behavior but not the pressure, cancellation, and detached narrative cases.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

Scheduled memory dreaming still uses dreaming.frequency as the cadence for a full configured-workspace sweep. When gateway RSS or heap usage is above the warning threshold before a workspace starts, or before the detached Dream Diary narrative is scheduled, scheduled dreaming waits before continuing. The wait starts at 30 seconds, doubles, and caps at 5 minutes between retries. No remaining workspaces are intentionally deferred to a later cron tick by memory-core; the run continues when pressure clears unless the enclosing cron run aborts/times out.

Diagram (if applicable)

Before:
cron tick -> workspace A -> workspace B -> workspace C -> detached diary work, without pressure-aware waits or abort-aware pressure sleep

After:
cron tick -> pressure check before workspace A -> workspace A -> pressure check before detached diary A -> pressure wait if needed -> diary A -> pressure check before workspace B -> ... full sweep continues until clear/completed or cron aborts

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: Linux worktree
  • Runtime/container: Node via repo wrappers for static/type checks
  • Model/provider: N/A
  • Integration/channel (if any): memory-core plugin cron seam and before-agent-reply hook contexts
  • Relevant config (redacted): memory-core dreaming enabled with multiple agent workspaces

Steps

  1. Run the focused formatting, lint, and TS checks listed above.
  2. Review the memory-core pressure tests and hook-context tests added for the cron abort path.
  3. Confirm full-sweep semantics are preserved, pressure waits happen before workspace and Dream Diary scheduling, backoff caps at 5 minutes, and abort signals are available to memory-core from all cron before-agent-reply paths.

Expected

  • Cron-triggered managed dreaming preserves the full configured workspace sweep, waits under RSS/heap pressure before adding workspace or detached narrative work, and stops promptly when cron cancellation fires during pressure backoff.

Actual

  • After the patch, static/type proof and autoreview are clean. Focused Vitest could not be executed in this environment because the local Vitest runner hung before test execution and remote providers were unavailable.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios: code-path review for memory-core pressure waits and before-agent-reply abort propagation, focused formatting/lint/type checks, and autoreview closeout.
  • Edge cases checked: queued cron work receives abort signals through all relevant hook paths, pressure backoff exits on abort, detached Dream Diary scheduling waits under pressure rather than being dropped, and full-sweep behavior remains in the same cron invocation.
  • What you did not verify: focused Vitest execution and private live multi-workspace gateway run, for the blockers listed in Real behavior proof.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: If memory pressure remains high for a long time, a scheduled dreaming cron run may take longer because it waits before starting the next workspace or detached Dream Diary work.
    • Mitigation: The backoff is capped at 5 minutes per retry, logs each wait with RSS/heap context, preserves the existing full-sweep behavior instead of skipping workspaces, preserves Dream Diary generation instead of dropping it, and now observes cron abort/cancellation.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation extensions: memory-core Extension: memory-core gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: L status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.