docs(memory): add memory system deep dive guide by neyric · Pull Request #60048 · openclaw/openclaw

neyric · 2026-04-03T03:48:32Z

Summary

Add a comprehensive memory system deep dive guide, addressing #43032.

Problem

docs/concepts/memory.md covers basics (~600 words) but lacks depth on memory best practices, search optimization, lifecycle management, and advanced patterns.

New document: `docs/concepts/memory-deep-dive.md` (401 lines)

Covers all 6 sections requested in the issue:

Memory Architecture — Two-layer model (MEMORY.md + memory/YYYY-MM-DD.md), indexing scope, what is searchable vs not
Writing Effective Memory — What belongs in MEMORY.md vs daily logs, promotion workflow, formatting tips for better search indexing
Search Optimization — Hybrid retrieval pipeline (keyword + vector + MMR), query strategies (semantic vs exact), memory_search → memory_get workflow, tuning parameters (maxResults, minScore)
Memory Maintenance — Heartbeat-based periodic review, cleanup strategies, heartbeat-state.json tracking
Advanced Patterns — Entity tracking with stable headings, opinion evolution with timestamps, cross-session recall via sessionMemory
Troubleshooting — No results, keyword-only matches, memory_get file restrictions, MEMORY.md bloat, missing transcript search

Navigation

Added to docs.json under Sessions and memory → Memory group, right after concepts/memory.

Source references

packages/memory-host-sdk/src/host/backend-config.ts — default settings (maxResults: 6, maxSnippetChars: 700)
packages/memory-host-sdk/src/host/read-file.ts — memory_get file boundary enforcement
src/agents/memory-search.ts — search config resolution and sources
docs/concepts/memory-builtin.md — existing builtin engine docs

Closes #43032

Create docs/concepts/memory-deep-dive.md covering: 1. Memory Architecture — two-layer model (MEMORY.md + daily logs), indexing scope, and what is searchable 2. Writing Effective Memory — what belongs in MEMORY.md vs daily logs, promotion workflow, formatting for searchability 3. Search Optimization — hybrid retrieval pipeline, query strategies, memory_search + memory_get workflow, tuning parameters 4. Memory Maintenance — heartbeat-based review, cleanup strategies, heartbeat-state.json tracking 5. Advanced Patterns — entity tracking, opinion evolution, cross-session recall via session transcripts 6. Troubleshooting — common issues with search, memory_get, file size Register the new page in docs.json Mintlify navigation under Sessions and memory > Memory group. Closes openclaw#43032

greptile-apps · 2026-04-03T03:50:25Z

Greptile Summary

This PR adds docs/concepts/memory-deep-dive.md, a 401-line guide covering memory architecture, writing strategies, search optimization, maintenance, advanced patterns, and troubleshooting. The source references cited in the PR description were verified: defaults (maxResults: 6, maxSnippetChars: 700) match backend-config.ts/memory-search.ts, and the memory_get file boundary description accurately reflects read-file.ts.

The guide states (line 47) that daily logs are "not injected automatically" and are "indexed and surfaced on demand," but docs/concepts/memory.md line 22 directly contradicts this: "Today and yesterday's notes are loaded automatically." One of these is wrong, and users reading both docs will get conflicting accounts of a core behavior that affects how they should structure their workflows.

Confidence Score: 4/5

Safe to merge after resolving the factual contradiction between the new guide and the existing memory overview on daily log auto-injection.

One P1 factual discrepancy exists between the new doc and the existing memory.md overview: the deep dive claims daily logs are never auto-injected, while memory.md says today and yesterday's notes load automatically. This needs resolution before the guide ships, otherwise users get directly conflicting mental models about how the system works.

docs/concepts/memory-deep-dive.md line 47 — needs to be reconciled with docs/concepts/memory.md line 22.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: docs/concepts/memory-deep-dive.md
Line: 46-48

Comment:
**Contradicts existing memory overview on auto-injection**

The existing `docs/concepts/memory.md` (line 22) explicitly states: *"Today and yesterday's notes are loaded automatically."* This new guide says the opposite — that daily logs are *not* injected automatically and are only surfaced on demand. A user who reads both docs will get directly conflicting information about a fundamental behavior. One of these needs to be corrected or reconciled before shipping.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "docs(memory): add memory system deep div..." | Re-trigger Greptile}

greptile-apps · 2026-04-03T03:50:28Z

+Daily logs are not injected automatically. They are indexed and surfaced on
+demand through `memory_search`, and `memory_get` can read the exact file or line
+range you need.


Contradicts existing memory overview on auto-injection

The existing docs/concepts/memory.md (line 22) explicitly states: "Today and yesterday's notes are loaded automatically." This new guide says the opposite — that daily logs are not injected automatically and are only surfaced on demand. A user who reads both docs will get directly conflicting information about a fundamental behavior. One of these needs to be corrected or reconciled before shipping.

Prompt To Fix With AI

This is a comment left during a code review. Path: docs/concepts/memory-deep-dive.md Line: 46-48 Comment: **Contradicts existing memory overview on auto-injection** The existing `docs/concepts/memory.md` (line 22) explicitly states: *"Today and yesterday's notes are loaded automatically."* This new guide says the opposite — that daily logs are *not* injected automatically and are only surfaced on demand. A user who reads both docs will get directly conflicting information about a fundamental behavior. One of these needs to be corrected or reconciled before shipping. How can I resolve this? If you propose a fix, please make it concise.

clawsweeper · 2026-05-01T01:26:27Z

Codex review: needs changes before merge. Reviewed July 3, 2026, 4:59 PM ET / 20:59 UTC.

Summary
The PR adds docs/concepts/memory-deep-dive.md and registers that page in the docs navigation under the Memory group.

PR surface: Docs +402. Total +402 across 2 files.

Reproducibility: not applicable. this is a documentation-only PR. The blockers are source-verifiable against current docs, runtime code, tests, and upstream Codex source rather than a failing runtime reproduction path.

Review metrics: none identified.

Root-cause cluster
Relationship: canonical
Canonical: #60048
Summary: This PR is the live canonical implementation path for the memory deep-dive guide request; related items either feed into it or overlap only on narrower memory wording.

Members:

fixed_by_candidate: Docs: Memory system deep dive guide #43032 - The issue requested this guide, was closed as superseded by this open PR, and the PR body uses closing syntax for it.
superseded: docs: add Memory Deep Dive guide #43071 - Search found an earlier closed-unmerged attempt to add the same Memory Deep Dive guide, leaving this PR as the newer live path.
partial_overlap: docs(memory): align descriptors and docs with recursive memory/**/*.md #85899 - That open PR aligns recursive memory/**/*.md wording across docs/descriptors, but it does not add or replace this deep-dive guide.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🌊 off-meta tidepool
Patch quality: 🦐 gold shrimp
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Reword daily-log loading to distinguish ordinary on-demand memory from startup/reset daily-memory preload.
Qualify MEMORY.md token-cost guidance for native Codex memory-tool routing versus fallback and non-Codex behavior.
Refresh against current main and rerun narrow docs validation.

Risk before merge

[P1] Merging as submitted would publish a public guide that says daily logs are not injected automatically without explaining the one-shot startup/reset daily-memory path.
[P1] Merging as submitted would overstate MEMORY.md token cost for native Codex users when memory tools route workspace MEMORY.md instead of pasting it into every turn.
[P1] A maintainer comment says this branch needs an update against current main before the required Dependency Guard check can pass, so docs validation should be rerun after the wording repair and refresh.

Maintainer options:

Repair source-backed memory wording (recommended)
Update the guide to distinguish ordinary on-demand memory from startup/reset daily-memory preload and qualify token-cost language against current native Codex memory routing.
Pause for docs shape review
Keep the PR paused if maintainers prefer splitting the accurate deep-dive material across existing memory, memory-search, system-prompt, and token-use pages instead of adding one long guide.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Reconcile docs/concepts/memory-deep-dive.md with current memory behavior: ordinary turns use memory_search/memory_get on demand; /new and /reset startup model runs can prepend recent daily memory once; qualify MEMORY.md token-cost guidance for native Codex memory-tool routing versus fallback/non-Codex behavior; keep the change docs-only and do not edit CHANGELOG.md; rerun docs validation after rebasing.

Next step before merge

The remaining blockers are narrow docs-only wording repairs with clear source evidence and validation commands.

Security
Cleared: The diff changes only docs content and docs navigation; no executable, dependency, workflow, credential, package, or supply-chain surface is touched.

Review findings

[P2] Reconcile daily-log startup loading — docs/concepts/memory-deep-dive.md:46-48
[P3] Qualify MEMORY.md token-cost guidance — docs/concepts/memory-deep-dive.md:34-35

Review details

Best possible solution:

Merge the guide after correcting those source-backed behavior claims, rebasing or refreshing against current main, and rerunning the narrow docs validation path.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a documentation-only PR. The blockers are source-verifiable against current docs, runtime code, tests, and upstream Codex source rather than a failing runtime reproduction path.

Is this the best way to solve the issue?

No, not as submitted. A deep-dive guide is an appropriate docs surface, but it must first distinguish ordinary on-demand memory from startup preload behavior and native Codex tool routing from fallback injection.

Full review comments:

[P2] Reconcile daily-log startup loading — docs/concepts/memory-deep-dive.md:46-48
The guide says daily logs are not injected automatically, but current source and docs distinguish ordinary on-demand memory from /new and /reset startup model runs that can prepend recent daily memory once. Please describe both paths so readers do not learn that daily notes are never auto-loaded.
Confidence: 0.94
[P3] Qualify MEMORY.md token-cost guidance — docs/concepts/memory-deep-dive.md:34-35
This says MEMORY.md costs tokens on every turn. Current native Codex routing omits the workspace root MEMORY.md from prompt injection when memory tools are available and reports injectedChars: 0; narrow the warning to fallback and non-Codex paths.
Confidence: 0.9

Overall correctness: patch is incorrect
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 010b61746379.

Label changes

Label justifications:

P3: This is a docs-only improvement with limited runtime blast radius, though factual wording must be repaired before merge.
merge-risk: 🚨 other: Green docs checks will not prove that the new public memory guide matches current startup-memory and native Codex routing behavior.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: Real behavior proof is not required because this PR only changes files under docs/.

Evidence reviewed

PR surface:

Docs +402. Total +402 across 2 files.

View PR surface stats

Area	Files	Added	Net
Source	0	0	0
Tests	0	0	0
Docs	2	402	+402
Config	0	0	0
Generated	0	0	0
Other	0	0	0
Total	2	402	+402

Acceptance criteria:

[P1] pnpm docs:list.
[P1] pnpm docs:check-mdx.
[P1] pnpm docs:check-links.
[P1] git diff --check.

What I checked:

Repository policy read: Root policy and docs-scoped policy require source-backed docs review, scoped docs instructions, and current behavior checks before a verdict. (AGENTS.md:1, 010b61746379)
Docs policy read: The docs guide confirms Mintlify root-relative links, generic public docs content, and docs validation expectations for this change. Public docs: docs/AGENTS.md. (docs/AGENTS.md:1, 010b61746379)
Live PR diff: The live diff adds a new memory deep-dive docs page and one docs/docs.json navigation entry; no runtime, workflow, dependency, or package files are touched. Public docs: docs/concepts/memory-deep-dive.md. (docs/concepts/memory-deep-dive.md:1, 5a0b3908297c)
Current memory overview: Current docs distinguish daily memory being loaded at session start from daily files not being injected into the normal bootstrap prompt on every turn. Public docs: docs/concepts/memory.md. (docs/concepts/memory.md:17, 010b61746379)
Startup memory source: buildSessionStartupContextPrelude selects recent daily memory paths and returns a runtime-loaded startup context block for new/reset session starts. (src/auto-reply/reply/startup-context.ts:292, 010b61746379)
Startup memory tests: Tests assert that today's and yesterday's daily memory files are loaded for the first turn and that slugged startup-memory artifacts are included. (src/auto-reply/reply/startup-context.test.ts:25, 010b61746379)

Likely related people:

Takhoffman: Recent history shows this person introduced and refined runtime startup daily-memory preload behavior that constrains the guide's daily-log wording. (role: startup-memory behavior contributor; confidence: high; commits: 94340b959830, 06ff594a3e99, 4f00b769251d; files: src/auto-reply/reply/startup-context.ts, src/auto-reply/reply/startup-context.test.ts)
steipete: Recent history shows this person routed workspace MEMORY.md through native Codex memory tools and maintained adjacent token-use/startup-memory docs. (role: native Codex memory-routing contributor; confidence: high; commits: d93524d1ccb1, 0b5298d24e6f, a66462b58368; files: extensions/codex/src/app-server/attempt-context.ts, extensions/codex/src/app-server/run-attempt.test.ts, docs/reference/token-use.md)
vincentkoc: Recent docs history includes broad memory overview/search/system-prompt maintenance near this PR's public docs surface. (role: recent memory docs contributor; confidence: medium; commits: 91e835ebe0ab, 367b9721b63b, c5baf63fa5e5; files: docs/concepts/memory.md, docs/concepts/memory-search.md, docs/concepts/system-prompt.md)
lastguru-net: Recent history moved the routed workspace-memory pointer into turn-scoped developer instructions, directly adjacent to the MEMORY.md token-cost wording reviewed here. (role: adjacent Codex context contributor; confidence: medium; commits: 3fbd2432b696; files: extensions/codex/src/app-server/attempt-context.ts, docs/reference/token-use.md)
rubencu: Recent Codex history preserved memory prompt registration, which is adjacent to the native Codex memory recall guidance path. (role: adjacent memory prompt contributor; confidence: medium; commits: 7387083a95b6; files: extensions/codex/src/app-server/attempt-context.ts, extensions/codex/src/app-server/run-attempt.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

clawsweeper · 2026-05-20T14:00:42Z

ClawSweeper PR egg

🔥 Warming up: real-behavior proof passed; findings, security review, or rank-up moves are still in progress.

Hatch command

Comment @clawsweeper hatch when this PR is hatchable.

Hatchability rules:

Merged PRs are hatchable.
Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

What is this egg doing here?

Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
Hatchability usually comes from sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness. A merged PR is already final, so merge makes the egg hatchable independently.
The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

RomneyDa · 2026-05-29T06:36:55Z

Heads up: this PR needs to be updated against current main before the new required Dependency Guard check can pass.

openclaw-barnacle · 2026-06-13T04:57:02Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

openclaw-barnacle Bot added docs Improvements or additions to documentation size: XS labels Apr 3, 2026

greptile-apps Bot reviewed Apr 3, 2026

View reviewed changes

clawsweeper Bot mentioned this pull request Apr 26, 2026

Docs: Memory system deep dive guide #43032

Closed

clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels May 19, 2026

openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 19, 2026

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 13, 2026

clawsweeper Bot added the merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. label Jun 14, 2026

openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 15, 2026

clawsweeper Bot mentioned this pull request Jun 19, 2026

docs(memory): align descriptors and docs with recursive memory/**/*.md #85899

Closed

4 tasks

clawsweeper Bot removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 22, 2026

openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

docs(memory): add memory system deep dive guide#60048

docs(memory): add memory system deep dive guide#60048
neyric wants to merge 1 commit into
openclaw:mainfrom
neyric:docs/memory-deep-dive

neyric commented Apr 3, 2026

Uh oh!

greptile-apps Bot commented Apr 3, 2026

Uh oh!

greptile-apps Bot Apr 3, 2026

Uh oh!

clawsweeper Bot commented May 1, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 20, 2026

Uh oh!

RomneyDa commented May 29, 2026

Uh oh!

openclaw-barnacle Bot commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

neyric commented Apr 3, 2026

Summary

Problem

New document: docs/concepts/memory-deep-dive.md (401 lines)

Navigation

Source references

Uh oh!

greptile-apps Bot commented Apr 3, 2026

Greptile Summary

Confidence Score: 4/5

Uh oh!

greptile-apps Bot Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

clawsweeper Bot commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented May 20, 2026

Hatch command

Uh oh!

RomneyDa commented May 29, 2026

Uh oh!

openclaw-barnacle Bot commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

New document: `docs/concepts/memory-deep-dive.md` (401 lines)

clawsweeper Bot commented May 1, 2026 •

edited

Loading