docs: event-log frame — palace-daemon as materialized-view coordinator#11
docs: event-log frame — palace-daemon as materialized-view coordinator#11jphein wants to merge 1 commit intorboarescu:mainfrom
Conversation
…u#12 (palace-mode security)
There was a problem hiding this comment.
Pull request overview
Adds an architectural-frame document explaining palace-daemon’s role in a “log + materialized views” system (à la Kleppmann), to clarify how current behavior and upcoming backend changes (e.g., Postgres/pgvector) fit into that model.
Changes:
- Added a new architecture doc (
docs/event-log-frame.md) describing the event-log vs. materialized-views framing. - Mapped current palace-daemon behaviors/endpoints onto “append/query/rebuild/coordinate” responsibilities.
- Discussed implications for a Postgres backend and knowledge-graph evolution.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | `GET /stats` | Aggregate query across multiple views (KG, graph stats, status). | | ||
| | `GET /graph` | Compose a snapshot across multiple views (wings, rooms, KG, tunnels) for SME-style consumers. | | ||
| | `GET /viz` | Render the snapshot as an HTML page — multi-view inspector for humans (D3 force-graph + Mermaid hierarchy + bar charts) over the same `/graph` data. | |
There was a problem hiding this comment.
This endpoint table lists GET /graph and GET /viz, but those routes do not exist in the current daemon (main.py has /mcp, /health, /search, /context, /memory, /stats, /flush, /reload, /backup, /mine, /silent-save, /repair, /repair/status). Please correct the table (and any downstream references) so the doc matches the implemented API, or explicitly label these as proposed endpoints.
| | `GET /stats` | Aggregate query across multiple views (KG, graph stats, status). | | |
| | `GET /graph` | Compose a snapshot across multiple views (wings, rooms, KG, tunnels) for SME-style consumers. | | |
| | `GET /viz` | Render the snapshot as an HTML page — multi-view inspector for humans (D3 force-graph + Mermaid hierarchy + bar charts) over the same `/graph` data. | | |
| | `GET /stats` | Aggregate query across implemented views and coordinator state. | |
There was a problem hiding this comment.
Stale finding — the current doc no longer lists /graph or /viz in the endpoint table. Line 49 says wings/rooms/halls are surfaced by MCP tools (mempalace_list_wings / mempalace_list_rooms / mempalace_list_tunnels), and the endpoint table at lines 63-76 covers only routes that actually exist. The aspirational refs were dropped in a later commit; sorry for not closing the loop sooner.
| | `POST /repair mode=rebuild` | Drop and recompute the view from log; coordinate writes via queue-and-drain. | | ||
| | `GET /repair/status` | Surface view-coordinator state to clients. | | ||
| | `POST /flush` | Force checkpoint of pending log entries. | | ||
| | `POST /reload` | Invalidate cached view handles (alias for `/repair mode=light` from a different vocabulary). | |
There was a problem hiding this comment.
POST /reload is described as an alias for /repair mode=light, but in code it only clears the client/collection caches and does not participate in the /repair state machine (e.g., no repair locking/state, no quarantine_stale_hnsw note/warning behavior). Consider rewording this to "similar to" / "overlaps with" mode=light, or have /reload delegate to the same implementation if true aliasing is intended.
| | `POST /reload` | Invalidate cached view handles (alias for `/repair mode=light` from a different vocabulary). | | |
| | `POST /reload` | Invalidate cached view handles; overlaps with `/repair mode=light`, but from a different vocabulary. | |
There was a problem hiding this comment.
Fair — "alias" oversimplifies. Both routes DO clear the client/collection cache (the only thing /reload does, and what /repair?mode=light boils down to via flush_palace), but the alias framing implies they share state-machine semantics, which they don't. Will soften to: "Invalidate cached view handles. Same effect as /repair?mode=light from the perspective of cache state, though it doesn't enter the repair state machine."
| - The view-coordinator role. `/silent-save`, `/search`, `/graph`, | ||
| `/stats` — same shape, same semantics, same client contract. |
There was a problem hiding this comment.
This section claims the view-coordinator role will keep the same client contract for /graph, but /graph is not currently an implemented endpoint in this repo. Please update the list of "stays the same" endpoints to the ones that actually exist (e.g. /mcp, /silent-save, /search, /stats, /repair, etc.) or annotate /graph as future work.
| - The view-coordinator role. `/silent-save`, `/search`, `/graph`, | |
| `/stats` — same shape, same semantics, same client contract. | |
| - The view-coordinator role. `/mcp`, `/silent-save`, `/search`, | |
| `/stats`, `/repair` — same shape, same semantics, same client | |
| contract. |
There was a problem hiding this comment.
Resolved by the doc reorganization — the current line 100 is in the "Stays the same" bullet about queue-and-drain. The "client contract" sentence in the current text talks about the role itself (HTTP/MCP API surface), not a specific endpoint. No /graph claim in the current doc.
| - View-rebuild coordination (only as needed by the backend) | ||
| - Structural fast paths for consumers (the `/graph` direct-sqlite read | ||
| is one example; under postgres it'd be a direct-postgres read) |
There was a problem hiding this comment.
The "structural fast paths" bullet cites a /graph direct-sqlite read as an existing example, but /graph is not present in the daemon codebase. Please adjust this example to a real endpoint/fast-path, or rephrase it as a potential future optimization under postgres.
There was a problem hiding this comment.
Current line 125-127 phrases this generically: "Optional structural fast paths for consumers (e.g. composing a single snapshot across wings + KG + tunnels in one HTTP roundtrip rather than forcing each adapter to fan out across multiple MCP calls). Under chroma the fast path can read the underlying sqlite directly; under postgres it'd be a direct-postgres aggregation query." No longer cites /graph by name. Worth flagging that #13 has since landed an actual GET /graph endpoint that materializes this exact pattern — could be a one-line cross-ref in a follow-up.
| 2. **Different storage technologies serve different query patterns.** | ||
| For 1-hop lookups against a few facts, plain SQL is right. If the | ||
| KG ever grew to support multi-hop traversal at scale, Apache AGE | ||
| (cypher on postgres) would be the natural view technology. |
There was a problem hiding this comment.
Minor terminology/casing: the graph query language is typically capitalized as "Cypher" (not "cypher"). Consider updating for correctness/consistency, especially since it's used as a proper name alongside "Apache AGE" and "Postgres".
| (cypher on postgres) would be the natural view technology. | |
| (Cypher on postgres) would be the natural view technology. |
There was a problem hiding this comment.
Already "Cypher" in the current doc (line 143: "(Cypher on postgres) would be the natural view technology"). Closed via earlier commit; sorry for not closing the comment loop.
| | Immutable event log | Raw conversation transcripts + verbatim drawers in the chroma collection. The 96.6% LongMemEval result is _from this log_, not from any reformatted view. | | ||
| | Materialized view: semantic search | The **chroma vector index** (HNSW). | | ||
| | Materialized view: structured facts | The **knowledge graph** (`knowledge_graph.sqlite3` — entities + triples). | | ||
| | Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*` and palace-daemon's `/graph`). | |
There was a problem hiding this comment.
The doc references palace-daemon's /graph endpoint as the way wings/rooms/halls are surfaced, but this repo's daemon currently does not implement a /graph route (endpoints are defined in main.py and include e.g. /stats, /repair, /mcp, etc.). Please update this row to reflect the actual API surface or clearly mark /graph as a future/planned endpoint.
| | Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*` and palace-daemon's `/graph`). | | |
| | Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*`; a palace-daemon `/graph` endpoint would be future/planned, not currently implemented). | |
There was a problem hiding this comment.
Same as the line-49 finding above — the current line says wings/rooms/halls are surfaced by mempalace_list_* MCP tools, not /graph. The earlier draft DID reference /graph as aspirational; removed during revision. With #13 in flight, a one-line cross-ref to the actual endpoint would be reasonable in a follow-up commit.
Adds `docs/event-log-frame.md` — an architectural-frame doc that
articulates how palace-daemon fits Kleppmann's "log + materialized
views" pattern from _Designing Data-Intensive Applications_ /
"Turning the database inside-out."
## Why this doc
The thesis: mempalace is event-streaming-shaped even though it hasn't
been articulated that way. The conversation transcripts are the
immutable log; the chroma vectors, knowledge graph, wing/room
categorization, and AAAK compressed text are all *materialized views*
computed from that log.
Under that frame, palace-daemon's role is the **view coordinator**:
accept writes to the log, serve queries against views, coordinate view
rebuilds (which is what `/repair` does), and handle write fairness
during rebuilds. That role is *durable across any backend*: ChromaDB,
pgvector, AGE, anything else — the backend just determines how a
particular view is stored.
## What it maps
A table mapping mempalace components onto Kleppmann concepts:
- Immutable event log → conversation transcripts + verbatim drawers
- Materialized view: semantic search → chroma vector index
- Materialized view: structured facts → knowledge graph (sqlite)
- Materialized view: categorical structure → wings / rooms / halls
- View rebuild operation → `/repair {mode: rebuild}`
- View consistency window → write semaphore + rebuild coordination
Plus what palace-daemon *already* implements implicitly under this
frame, what's chroma-specific vs. role-level, and what changes (and
doesn't) under the upcoming postgres backend (#665) and other
storage transitions.
## Why now
A reference doc ahead of the multi-backend transition that Igor's
work makes possible — articulating *what role* the daemon plays
makes the question "should this code be daemon-side or backend-side?"
tractable when the next storage adapter lands.
No code changes; pure reference doc, 191 lines. Linked from upstream
Discussion #5 (Postgres backend) and rboarescu#6 (TS rewrite) via comments
once it lands so contributors can reach for it without digging through
the README.
Originally fork commit `432b3a6`; sending standalone per the
"small separate PRs" preference.
d3b6532 to
62ff62d
Compare
…nding queue; trim patch Filed four upstream PRs on 2026-04-30: - rboarescu#15 feat: GET /viz status dashboard (stacks on rboarescu#13) - rboarescu#16 feat: GET /list — query-free metadata browse - rboarescu#17 feat: DELETE /memory + PATCH /memory - rboarescu#18 feat(lifespan): auto-migrate Stop-hook checkpoints on startup Also rebased PR rboarescu#13 onto upstream/main to clear a CHANGELOG conflict left by upstream's b4aee82 (patch sync) — state went CONFLICTING -> MERGEABLE / CLEAN. README: - Open upstream PRs table: four new rows (rboarescu#15-rboarescu#18) plus a 2026-04-30 note covering today's rebase + new PRs in one breath. - Pending PRs queue: now empty. Replaced the four stale rows (event-log-frame and graph-endpoint were already in flight via rboarescu#11 and rboarescu#13; mempal-fast.py was already upstream via the merged PR #4 omnibus; /viz is now PR rboarescu#15) with a brief empty-state note. CLAUDE.md: - Patch description trimmed to reflect that the hnsw:num_threads enforcement landed upstream via _pin_hnsw_threads(); only the log + retry-once slice remains. patches/mcp_server_get_collection.patch: - Regenerated against current mempalace develop. The patch is now just the "log exception + retry once on cache failure" change. Filed upstream as MemPalace/mempalace#1286; once that merges this patch retires. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Summary
Adds
docs/event-log-frame.md— an architectural-frame doc articulating how palace-daemon fits Kleppmann's "log + materialized views" pattern from Designing Data-Intensive Applications / "Turning the database inside-out".Thesis
mempalace is event-streaming-shaped even though it hasn't been articulated that way. Conversation transcripts + verbatim drawers are the immutable log; chroma vectors, the knowledge graph, wing/room categorization, and AAAK-compressed text are all materialized views computed from the log.
Under that frame, palace-daemon's role is the view coordinator: accept writes to the log, serve queries against views, coordinate view rebuilds (
/repair), handle write fairness during rebuilds. That role is durable across any backend — ChromaDB today, pgvector / AGE / SQLiteVec tomorrow. The backend determines how a view is stored; the daemon defines what role views play.What the doc covers
/repairmodes as view operations, write semaphore as view-consistency boundary, the pending-writes queue during rebuild)Why now
A reference doc ahead of the multi-backend transition Igor's work enables — articulating what role the daemon plays makes the question "should this code live daemon-side or backend-side?" tractable when the next storage adapter lands.
Scope
+191 / -0. Single new docs file. No code changes, no new dependencies.I'd intend to link this from the existing Discussion #5 (Postgres backend) and #6 (TS rewrite) via comments once it lands, so contributors thinking about either transition can reach for the frame without digging through the README. Happy to wait until you've had a chance to react to the framing before doing that.
Originally fork commit
432b3a6; sending standalone per the "small separate PRs" preference.