Skip to content

[Bug]: Gateway background work causes high CPU and page fault churn #84849

Description

@galiniliev

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

Gateway background work can keep the process above one full CPU core with high minor page-fault churn even when diagnostic active/waiting/queued counters are zero.

Steps to reproduce

  1. Run a gateway with memory-core dreaming and compaction/checkpoint sidecars enabled.
  2. Observe the gateway process during background work when no foreground request is queued.
  3. Capture CPU, memory, and liveness diagnostics for the gateway process.

Expected behavior

Background work should downshift or defer when gateway RSS, heap, CPU, or event-loop pressure is high so it does not keep the gateway above one core while no foreground work is queued.

Actual behavior

Observed samples showed the gateway using about 1.1-1.2 CPU cores while active/waiting/queued were zero. The same window showed high system CPU, high minor page-fault rates, private anonymous RSS over 2.6 GiB, and liveness warnings for CPU/event-loop pressure.

OpenClaw version

NOT_ENOUGH_INFO

Operating system

NOT_ENOUGH_INFO

Install method

NOT_ENOUGH_INFO

Model

NOT_ENOUGH_INFO

Provider / routing chain

NOT_ENOUGH_INFO

Additional provider/model setup details

NOT_ENOUGH_INFO

Logs, screenshots, and evidence

process sample: RSS=2745172 kB PSS=2691306 kB Private_Dirty=2674492 kB Anonymous=2674464 kB Swap=0 kB
pidstat 5s average: usr=74.80 system=42.00 cpu=116.80
pidstat memory 5s average: minflt/s=154878.64 majflt/s=0.00 rss=2234887 kB
memory pressure: level=warning reason=rss_threshold rssBytes=1827979264 heapUsedBytes=684072312 thresholdBytes=1610612736
liveness warning: reasons=event_loop_delay,event_loop_utilization,cpu eventLoopDelayP99Ms=306.2 eventLoopDelayMaxMs=1163.9 eventLoopUtilization=0.991 cpuCoreRatio=1.107 active=0 waiting=0 queued=0

Impact and severity

Affected: gateways with background memory/checkpoint work enabled.
Severity: high for gateway responsiveness.
Frequency: observed in two live pidstat samples from the same investigation.
Consequence: background sidecars can consume CPU and memory budget that should remain available for chat latency, websocket control traffic, timers, and health checks.

Additional information

Primary fix path: bound and pressure-defer managed memory-core dreaming. Related fix path: cap retained compaction checkpoint bytes so old large checkpoint artifacts do not keep accumulating per session.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.maintainerMaintainer-authored PRmaturity:stableIssue affects a taxonomy feature currently scored M4/M5.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions