-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Slack socket starved by blocking SQLite VACUUM on bloated main.sqlite; interrupted vacuums leak 53GB of orphaned .tmp files #83712
Copy link
Copy link
Closed as not planned
Closed as not planned
Copy link
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Summary
On a long-running gateway, synchronous SQLite work against
~/.openclaw/memory/main.sqliteblocks the Node event loop for 8–55 seconds at a time. This starves the Slack Socket Mode ping/pong, so Slack disconnects and frequently never reconnects — the agent goes silent on Slack while Telegram keeps working. A secondary bug makes it self-perpetuating: interruptedVACUUM/backup operations leave orphanedmain.sqlite.tmp-<uuid>copies that are never cleaned up (53 GB / 1,570 files accumulated here).Environment
Symptoms
channels/slack:slack socket disconnected (disconnect); reconnecting in 2s (attempt 1/12)followed by noslack socket mode connectedfor many minutes. Slack stays dead until a manualopenclaw gateway restart.channel stop exceeded 5000ms after abortand is then permanently wedged (only a gateway restart recovers it).diagnosticliveness warnings:liveness warning: reasons=event_loop_delay,event_loop_utilization eventLoopDelayP99Ms=9940.5 eventLoopDelayMaxMs=9940.5[trace:embedded-run] ... totalMs=17085).Root cause (evidence gathered)
main.sqlitehad grown to 664 MB (689 MB on a prior occasion). Table sizes viadbstat:chunksembedding_cachechunks_vec_vector_chunks00chunks_fts_*A full
VACUUMof this DB took 55 seconds, during which the event loop is blocked the entire time — which itself triggers the very Slack disconnects described above. AfterDELETE FROM embedding_cachethe sameVACUUMcompleted in ~8 s and reclaimed 251 MB (664 MB → 412 MB), and the event-loop stalls / Slack drops stopped.Secondary bug: orphaned temp files (the bigger surprise)
VACUUM/backup writes a temporary copy alongside the DB. When the process is killed or restarted mid-operation (e.g. by the gateway restart that users/watchdogs perform to recover the stuck Slack socket), the temp file is abandoned and never cleaned up. Found in~/.openclaw/memory/:main.sqlite.tmp-<uuid>files totaling 53 GB (many 308 MB each), oldest dating back ~2 weeks.This is a vicious cycle: large DB → long blocking VACUUM → event-loop stall → Slack drops → operator restarts gateway → VACUUM interrupted → another ~300 MB orphan → disk fills, FS scans slow, repeat.
Suggested fixes
VACUUMto a worker thread or a separate process so socket heartbeats keep firing.main.sqlite.tmp-*on startup (and on vacuum failure). A simple sweep of stale temp copies would have prevented 53 GB of waste.embedding_cachegrowth (LRU/TTL or cap), or usePRAGMA incremental_vacuum/auto_vacuum=INCREMENTALinstead of a monolithic blockingVACUUM.embedded-run(observed 17 s) cannot starve Socket Mode pongs.channel stop exceeded 5000ms after abortstate, which currently never self-recovers.Happy to provide full logs or
dbstatdumps if useful.