Skip to content

Commit 7dbb21b

Browse files
steipetesteipete
committed
feat: add pre-compaction memory flush
1 parent cc8a245 commit 7dbb21b

19 files changed

Lines changed: 583 additions & 22 deletions

‎CHANGELOG.md‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
## 2026.1.11
44

5+
### Highlights
6+
- Plugins are now first-class: loader + CLI management, plus the new Voice Call plugin.
7+
- Config: modular `$include` support for split config files. (#731) — thanks @pasogott.
8+
- Agents/Pi: reserve compaction headroom so pre-compaction memory writes can run before auto-compaction.
9+
- Agents: automatic pre-compaction memory flush turn to store durable memories before compaction.
10+
511
### Changes
612
- CLI: simplify configure section selection (single-select with optional add-more).
713
- Onboarding/CLI: group model/auth choice by provider and label Z.AI as GLM 4.7.
@@ -12,43 +18,58 @@
1218
- Docs: add beginner-friendly plugin quick start + expand Voice Call plugin docs.
1319
- Tests: add Docker plugin loader + tgz-install smoke test.
1420
- Tests: extend Docker plugin E2E to cover installing from local folders (`plugins.load.paths`) and `file:` npm specs.
21+
- Tests: add coverage for pre-compaction memory flush settings.
1522
- Agents/Tools: add `apply_patch` tool for multi-file edits (experimental; gated by tools.exec.applyPatch; OpenAI-only).
1623
- Agents/Tools: rename the bash tool to exec (config alias maintained). (#748) — thanks @myfunc.
24+
- Agents: add pre-compaction memory flush config (`agents.defaults.compaction.*`) with a soft threshold + system prompt.
1725
- Config: add `$include` directive for modular config files. (#731) — thanks @pasogott.
1826
- Build: set pnpm minimum release age to 2880 minutes (2 days). (#718) — thanks @dan-dr.
1927
- macOS: prompt to install the global `clawdbot` CLI when missing in local mode; install via `clawd.bot/install-cli.sh` (no onboarding) and use external launchd/CLI instead of the embedded gateway runtime.
2028
- Docs: add gog calendar event color IDs from `gog calendar colors`. (#715) — thanks @mjrussell.
29+
- Cron/CLI: add `--model` flag to cron add/edit commands. (#711) — thanks @mjrussell.
2130
- Cron/CLI: trim model overrides on cron edits and document main-session guidance. (#711) — thanks @mjrussell.
2231
- Skills: bundle `skill-creator` to guide creating and packaging skills.
23-
- Discord: expose channel/category management actions in the message tool. (#730) — thanks @NicholasSpisak
32+
- Providers: add per-DM history limit overrides (`dmHistoryLimit`) with provider-level config. (#728) — thanks @pkrmf.
33+
- Discord: expose channel/category management actions in the message tool. (#730) — thanks @NicholasSpisak.
2434
- Docs: rename README “macOS app” section to “Apps”. (#733) — thanks @AbhisekBasu1.
2535
- Gateway: require `client.id` in WebSocket connect params; use `client.instanceId` for presence de-dupe; update docs/tests.
2636
- macOS: remove the attach-only gateway setting; local mode now always manages launchd while still attaching to an existing gateway if present.
2737

2838
### Installer
2939
- Postinstall: replace `git apply` with builtin JS patcher (works npm/pnpm/bun; no git dependency) plus regression tests.
40+
- Postinstall: skip pnpm patch fallback when the new patcher is active.
3041
- Installer tests: add root+non-root docker smokes, CI workflow to fetch clawd.bot scripts and run install sh/cli with onboarding skipped.
3142
- Installer UX: support `CLAWDBOT_NO_ONBOARD=1` for non-interactive installs; fix npm prefix on Linux and auto-install git.
3243
- Installer UX: add `install.sh --help` with flags/env and git install hint.
3344
- Installer UX: add `--install-method git|npm` and auto-detect source checkouts (prompt to update git checkout vs migrate to npm).
3445

3546
### Fixes
3647
- Models/Onboarding: configure MiniMax (minimax.io) via Anthropic-compatible `/anthropic` endpoint by default (keep `minimax-api` as a legacy alias).
37-
- Gateway/WebChat: include handshake validation details in the WebSocket close reason for easier debugging.
48+
- Gateway/WebChat: include handshake validation details in the WebSocket close reason for easier debugging; preserve close codes.
3849
- Gateway/Auth: send invalid connect responses before closing the handshake; stabilize invalid-connect auth test.
50+
- Gateway: tighten gateway listener detection.
51+
- Control UI: hide onboarding chat when configured and guard the mobile chat sidebar overlay.
52+
- Auth: read Codex keychain credentials and make the lookup platform-aware.
53+
- macOS/Release: avoid bundling dist artifacts in relay builds and generate appcasts from zip-only sources.
3954
- Doctor: surface plugin diagnostics in the report.
40-
- Plugins: treat `plugins.load.paths` directory entries as package roots when they contain `package.json` + `clawdbot.extensions`.
41-
- Config: expand `~` in `CLAWDBOT_CONFIG_PATH` and common path-like config fields (including `plugins.load.paths`).
55+
- Plugins: treat `plugins.load.paths` directory entries as package roots when they contain `package.json` + `clawdbot.extensions`; load plugin packages from config dirs; extract archives without system tar.
56+
- Config: expand `~` in `CLAWDBOT_CONFIG_PATH` and common path-like config fields (including `plugins.load.paths`); guard invalid `$include` paths. (#731) — thanks @pasogott.
4257
- Agents: stop pre-creating session transcripts so first user messages persist in JSONL history.
4358
- Auto-reply: align `/think` default display with model reasoning defaults. (#751) — thanks @gabriel-trigo.
4459
- Auto-reply: flush block reply buffers on tool boundaries. (#750) — thanks @sebslight.
60+
- Heartbeat: refresh prompt text for updated defaults.
61+
- Agents/Tools: use PowerShell on Windows to capture system utility output. (#748) — thanks @myfunc.
4562
- Docker: tolerate unset optional env vars in docker-setup.sh under strict mode. (#725) — thanks @petradonka.
4663
- CLI/Update: preserve base environment when passing overrides to update subprocesses. (#713) — thanks @danielz1z.
4764
- Agents: treat message tool errors as failures so fallback replies still send; require `to` + `message` for `action=send`. (#717) — thanks @theglove44.
65+
- Agents: preserve reasoning items on tool-only turns.
66+
- Agents/Subagents: wait for completion before announcing, align wait timeout with run timeout, and make announce prompts more emphatic.
4867
- Agents: route subagent transcripts to the target agent sessions directory and add regression coverage. (#708) — thanks @xMikeMickelson.
4968
- Agents/Tools: preserve action enums when flattening tool schemas. (#708) — thanks @xMikeMickelson.
5069
- Gateway/Agents: canonicalize main session aliases for store writes and add regression coverage. (#709) — thanks @xMikeMickelson.
5170
- Agents: reset sessions and retry when auto-compaction overflows instead of crashing the gateway.
71+
- Providers/Telegram: normalize command mentions for consistent parsing. (#729) — thanks @obviyus.
72+
- Providers: skip DM history limit handling for non-DM sessions. (#728) — thanks @pkrmf.
5273
- Sandbox: fix non-main mode incorrectly sandboxing the main DM session and align `/status` runtime reporting with effective sandbox state.
5374
- Sandbox/Gateway: treat `agent:<id>:main` as a main-session alias when `session.mainKey` is customized (backwards compatible).
5475

@@ -74,7 +95,6 @@
7495
- Docker: allow optional home volume + extra bind mounts in `docker-setup.sh`. (#679) — thanks @gabriel-trigo.
7596

7697
### Fixes
77-
- Agents/Pi: raise compaction `reserveTokens` floor to 20k to leave enough headroom for pre-compaction “memory write” turns.
7898
- Auto-reply: suppress draft/typing streaming for `NO_REPLY` (silent system ops) so it doesn’t leak partial output.
7999
- CLI/Status: expand tables to full terminal width; clarify provider setup vs runtime warnings; richer per-provider detail; token previews in `status` while keeping `status --all` redacted; add troubleshooting link footer; keep log tails pasteable; show gateway auth used when reachable; surface provider runtime errors (Signal/iMessage/Slack); harden `tailscale status --json` parsing; make `status --all` scan progress determinate; and replace the footer with a 3-line “Next steps” recommendation (share/debug/probe).
80100
- CLI/Gateway: clarify that `clawdbot gateway status` reports RPC health (connect + RPC) and shows RPC failures separately from connect failures.

‎docs/concepts/agent-workspace.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,8 @@ These are the standard files Clawdbot expects inside the workspace:
9999
- Curated long-term memory.
100100
- Only load in the main, private session (not shared/group contexts).
101101

102+
See [Memory](/concepts/memory) for the workflow and automatic memory flush.
103+
102104
- `skills/` (optional)
103105
- Workspace-specific skills.
104106
- Overrides managed/bundled skills when names collide.

‎docs/concepts/compaction.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ You’ll see:
2222
- `🧹 Auto-compaction complete` in verbose mode
2323
- `/status` showing `🧹 Compactions: <count>`
2424

25+
Before compaction, Clawdbot can run a **silent memory flush** turn to store
26+
durable notes to disk. See [Memory](/concepts/memory) for details and config.
27+
2528
## Manual compaction
2629
Use `/compact` (optionally with instructions) to force a compaction pass:
2730
```

‎docs/concepts/memory.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
summary: "How Clawdbot memory works (workspace files + automatic memory flush)"
3+
read_when:
4+
- You want the memory file layout and workflow
5+
- You want to tune the automatic pre-compaction memory flush
6+
---
7+
# Memory
8+
9+
Clawdbot memory is **plain Markdown in the agent workspace**. The files are the
10+
source of truth; the model only “remembers” what gets written to disk.
11+
12+
## Memory files (Markdown)
13+
14+
The default workspace layout uses two memory layers:
15+
16+
- `memory/YYYY-MM-DD.md`
17+
- Daily log (append-only).
18+
- Read today + yesterday at session start.
19+
- `MEMORY.md` (optional)
20+
- Curated long-term memory.
21+
- **Only load in the main, private session** (never in group contexts).
22+
23+
These files live under the workspace (`agents.defaults.workspace`, default
24+
`~/clawd`). See [Agent workspace](/concepts/agent-workspace) for the full layout.
25+
26+
## When to write memory
27+
28+
- Decisions, preferences, and durable facts go to `MEMORY.md`.
29+
- Day-to-day notes and running context go to `memory/YYYY-MM-DD.md`.
30+
- If someone says “remember this,” write it down (don’t keep it in RAM).
31+
32+
## Automatic memory flush (pre-compaction ping)
33+
34+
When a session is **close to auto-compaction**, Clawdbot triggers a **silent
35+
agentic turn** that reminds the model to write durable memory **before** the
36+
context is compacted. The default prompt encourages the model to respond with
37+
`NO_REPLY` when there’s nothing to store, so the user never sees this turn.
38+
39+
This is controlled by `agents.defaults.compaction.memoryFlush`:
40+
41+
```json5
42+
{
43+
agents: {
44+
defaults: {
45+
compaction: {
46+
reserveTokensFloor: 20000,
47+
memoryFlush: {
48+
enabled: true,
49+
softThresholdTokens: 4000,
50+
systemPrompt: "Session nearing compaction. Store durable memories now.",
51+
prompt: "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."
52+
}
53+
}
54+
}
55+
}
56+
}
57+
```
58+
59+
Details:
60+
- **Soft threshold**: flush triggers when the session token estimate crosses
61+
`contextWindow - reserveTokensFloor - softThresholdTokens`.
62+
- **Silent** by default: prompts include `NO_REPLY` so nothing is delivered.
63+
- **One flush per compaction cycle** (tracked in `sessions.json`).
64+
65+
For the full compaction lifecycle, see
66+
[Session management + compaction](/reference/session-management-compaction).

‎docs/docs.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -608,6 +608,7 @@
608608
"token-use",
609609
"concepts/oauth",
610610
"concepts/agent-workspace",
611+
"concepts/memory",
611612
"concepts/multi-agent",
612613
"concepts/compaction",
613614
"concepts/session",

‎docs/gateway/configuration.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1368,6 +1368,40 @@ Example (adaptive tuned):
13681368

13691369
See [/concepts/session-pruning](/concepts/session-pruning) for behavior details.
13701370

1371+
#### `agents.defaults.compaction` (reserve headroom + memory flush)
1372+
1373+
`agents.defaults.compaction.reserveTokensFloor` enforces a minimum `reserveTokens`
1374+
value for Pi compaction (default: `20000`). Set it to `0` to disable the floor.
1375+
1376+
`agents.defaults.compaction.memoryFlush` runs a **silent** agentic turn before
1377+
auto-compaction, instructing the model to store durable memories on disk (e.g.
1378+
`memory/YYYY-MM-DD.md`). It triggers when the session token estimate crosses a
1379+
soft threshold below the compaction limit.
1380+
1381+
Defaults:
1382+
- `memoryFlush.enabled`: `true`
1383+
- `memoryFlush.softThresholdTokens`: `4000`
1384+
- `memoryFlush.prompt` / `memoryFlush.systemPrompt`: built-in defaults with `NO_REPLY`
1385+
1386+
Example (tuned):
1387+
```json5
1388+
{
1389+
agents: {
1390+
defaults: {
1391+
compaction: {
1392+
reserveTokensFloor: 24000,
1393+
memoryFlush: {
1394+
enabled: true,
1395+
softThresholdTokens: 6000,
1396+
systemPrompt: "Session nearing compaction. Store durable memories now.",
1397+
prompt: "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."
1398+
}
1399+
}
1400+
}
1401+
}
1402+
}
1403+
```
1404+
13711405
Block streaming:
13721406
- `agents.defaults.blockStreamingDefault`: `"on"`/`"off"` (default off).
13731407
- Provider overrides: `*.blockStreaming` (and per-account variants) to force block streaming on/off.

‎docs/reference/session-management-compaction.md‎

Lines changed: 29 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,8 @@ Key fields (not exhaustive):
107107
- Token counters (best-effort / provider-dependent):
108108
- `inputTokens`, `outputTokens`, `totalTokens`, `contextTokens`
109109
- `compactionCount`: how often auto-compaction completed for this session key
110+
- `memoryFlushAt`: timestamp for the last pre-compaction memory flush
111+
- `memoryFlushCompactionCount`: compaction count when the last flush ran
110112

111113
The store is safe to edit, but the Gateway is the authority: it may rewrite or rehydrate entries as sessions run.
112114

@@ -191,12 +193,15 @@ Pi’s compaction settings live in Pi settings:
191193

192194
Clawdbot also enforces a safety floor for embedded runs:
193195

194-
- If `compaction.reserveTokens < 20000`, Clawdbot bumps it to 20000.
196+
- If `compaction.reserveTokens < reserveTokensFloor`, Clawdbot bumps it.
197+
- Default floor is `20000` tokens.
198+
- Set `agents.defaults.compaction.reserveTokensFloor: 0` to disable the floor.
195199
- If it’s already higher, Clawdbot leaves it alone.
196200

197201
Why: leave enough headroom for multi-turn “housekeeping” (like memory writes) before compaction becomes unavoidable.
198202

199-
Implementation: `ensurePiCompactionReserveTokens()` in `src/agents/pi-settings.ts` (called from `src/agents/pi-embedded-runner.ts`).
203+
Implementation: `ensurePiCompactionReserveTokens()` in `src/agents/pi-settings.ts`
204+
(called from `src/agents/pi-embedded-runner.ts`).
200205

201206
---
202207

@@ -223,22 +228,33 @@ As of `2026.1.10`, Clawdbot also suppresses **draft/typing streaming** when a pa
223228

224229
---
225230

226-
## Pre-compaction “memory flush” (design)
231+
## Pre-compaction “memory flush” (implemented)
227232

228-
Goal: before auto-compaction happens, run a short sequence of turns that writes durable state to disk (e.g. `memory/YYYY-MM-DD.md` in the agent workspace) so compaction can’t erase critical context.
233+
Goal: before auto-compaction happens, run a silent agentic turn that writes durable
234+
state to disk (e.g. `memory/YYYY-MM-DD.md` in the agent workspace) so compaction can’t
235+
erase critical context.
229236

230-
Two viable hooks:
237+
Clawdbot uses the **pre-threshold flush** approach:
231238

232-
1) **Pre-threshold flush (Clawdbot-side)**
233-
- Monitor session context usage.
234-
- When it crosses a “soft threshold” (below Pi’s real compaction threshold), enqueue a silent “write memory now” directive to the agent.
235-
- Use `NO_REPLY` so the user sees nothing.
239+
1) Monitor session context usage.
240+
2) When it crosses a “soft threshold” (below Pi’s compaction threshold), run a silent
241+
“write memory now” directive to the agent.
242+
3) Use `NO_REPLY` so the user sees nothing.
236243

237-
2) **Pi extension hook (`session_before_compact`)**
238-
- Pi’s extension API exposes a `session_before_compact` event that receives compaction preparation details and can cancel or replace compaction.
239-
- Clawdbot can ship an extension that reacts here and performs housekeeping (and/or produces a custom compaction result).
244+
Config (`agents.defaults.compaction.memoryFlush`):
245+
- `enabled` (default: `true`)
246+
- `softThresholdTokens` (default: `4000`)
247+
- `prompt` (user message for the flush turn)
248+
- `systemPrompt` (extra system prompt appended for the flush turn)
240249

241-
Clawdbot currently documents the *concept* of daily memory in the workspace template (see [/concepts/agent-workspace](/concepts/agent-workspace)) but does not yet ship an automated pre-compaction flush loop.
250+
Notes:
251+
- The default prompt/system prompt include a `NO_REPLY` hint to suppress delivery.
252+
- The flush runs once per compaction cycle (tracked in `sessions.json`).
253+
- The flush runs only for embedded Pi sessions (CLI backends skip it).
254+
- See [Memory](/concepts/memory) for the workspace file layout and write patterns.
255+
256+
Pi also exposes a `session_before_compact` hook in the extension API, but Clawdbot’s
257+
flush logic lives on the Gateway side today.
242258

243259
---
244260

@@ -251,4 +267,3 @@ Clawdbot currently documents the *concept* of daily memory in the workspace temp
251267
- compaction settings (`reserveTokens` too high for the model window can cause earlier compaction)
252268
- tool-result bloat: enable/tune session pruning
253269
- Silent turns leaking? Confirm the reply starts with `NO_REPLY` (exact token) and you’re on a build that includes the streaming suppression fix.
254-

‎docs/start/clawd.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ clawdbot setup
102102
```
103103

104104
Full workspace layout + backup guide: [Agent workspace](/concepts/agent-workspace)
105+
Memory workflow: [Memory](/concepts/memory)
105106

106107
Optional: choose a different workspace with `agents.defaults.workspace` (supports `~`).
107108

‎docs/start/faq.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,15 @@ ClawdHub installs into `./skills` under your current directory; Clawdbot treats
197197

198198
Yes. See [Sandboxing](/gateway/sandboxing). For Docker-specific setup (full gateway in Docker or sandbox images), see [Docker](/install/docker).
199199

200+
### How does memory work?
201+
202+
Clawdbot memory is just Markdown files in the agent workspace:
203+
- Daily notes in `memory/YYYY-MM-DD.md`
204+
- Curated long-term notes in `MEMORY.md` (main/private sessions only)
205+
206+
Clawdbot also runs a **silent pre-compaction memory flush** to remind the model
207+
to write durable notes before auto-compaction. See [Memory](/concepts/memory).
208+
200209
## Where things live on disk
201210

202211
### Where does Clawdbot store its data?

‎docs/start/hubs.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Use these hubs to discover every page, including deep dives and reference docs t
3434
- [Architecture](/concepts/architecture)
3535
- [Agent runtime](/concepts/agent)
3636
- [Agent workspace](/concepts/agent-workspace)
37+
- [Memory](/concepts/memory)
3738
- [Agent loop](/concepts/agent-loop)
3839
- [Streaming + chunking](/concepts/streaming)
3940
- [Multi-agent routing](/concepts/multi-agent)

0 commit comments

Comments
 (0)