Skip to content

docs: event-log frame — palace-daemon as materialized-view coordinator#11

Open
jphein wants to merge 1 commit intorboarescu:mainfrom
jphein:docs/event-log-frame
Open

docs: event-log frame — palace-daemon as materialized-view coordinator#11
jphein wants to merge 1 commit intorboarescu:mainfrom
jphein:docs/event-log-frame

Conversation

@jphein
Copy link
Copy Markdown
Contributor

@jphein jphein commented Apr 27, 2026

Summary

Adds docs/event-log-frame.md — an architectural-frame doc articulating how palace-daemon fits Kleppmann's "log + materialized views" pattern from Designing Data-Intensive Applications / "Turning the database inside-out".

Thesis

mempalace is event-streaming-shaped even though it hasn't been articulated that way. Conversation transcripts + verbatim drawers are the immutable log; chroma vectors, the knowledge graph, wing/room categorization, and AAAK-compressed text are all materialized views computed from the log.

Under that frame, palace-daemon's role is the view coordinator: accept writes to the log, serve queries against views, coordinate view rebuilds (/repair), handle write fairness during rebuilds. That role is durable across any backend — ChromaDB today, pgvector / AGE / SQLiteVec tomorrow. The backend determines how a view is stored; the daemon defines what role views play.

What the doc covers

  • The Kleppmann frame (background)
  • Mapping mempalace components onto log-vs-view roles
  • What palace-daemon already implements implicitly under this frame (/repair modes as view operations, write semaphore as view-consistency boundary, the pending-writes queue during rebuild)
  • What changes and doesn't under the upcoming postgres backend (#665)
  • What this implies for the knowledge-graph view evolution
  • Open questions
  • References

Why now

A reference doc ahead of the multi-backend transition Igor's work enables — articulating what role the daemon plays makes the question "should this code live daemon-side or backend-side?" tractable when the next storage adapter lands.

Scope

+191 / -0. Single new docs file. No code changes, no new dependencies.

I'd intend to link this from the existing Discussion #5 (Postgres backend) and #6 (TS rewrite) via comments once it lands, so contributors thinking about either transition can reach for the frame without digging through the README. Happy to wait until you've had a chance to react to the framing before doing that.

Originally fork commit 432b3a6; sending standalone per the "small separate PRs" preference.

Copilot AI review requested due to automatic review settings April 27, 2026 17:05
jphein added a commit to jphein/palace-daemon that referenced this pull request Apr 27, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an architectural-frame document explaining palace-daemon’s role in a “log + materialized views” system (à la Kleppmann), to clarify how current behavior and upcoming backend changes (e.g., Postgres/pgvector) fit into that model.

Changes:

  • Added a new architecture doc (docs/event-log-frame.md) describing the event-log vs. materialized-views framing.
  • Mapped current palace-daemon behaviors/endpoints onto “append/query/rebuild/coordinate” responsibilities.
  • Discussed implications for a Postgres backend and knowledge-graph evolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/event-log-frame.md Outdated
Comment on lines +68 to +70
| `GET /stats` | Aggregate query across multiple views (KG, graph stats, status). |
| `GET /graph` | Compose a snapshot across multiple views (wings, rooms, KG, tunnels) for SME-style consumers. |
| `GET /viz` | Render the snapshot as an HTML page — multi-view inspector for humans (D3 force-graph + Mermaid hierarchy + bar charts) over the same `/graph` data. |
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This endpoint table lists GET /graph and GET /viz, but those routes do not exist in the current daemon (main.py has /mcp, /health, /search, /context, /memory, /stats, /flush, /reload, /backup, /mine, /silent-save, /repair, /repair/status). Please correct the table (and any downstream references) so the doc matches the implemented API, or explicitly label these as proposed endpoints.

Suggested change
| `GET /stats` | Aggregate query across multiple views (KG, graph stats, status). |
| `GET /graph` | Compose a snapshot across multiple views (wings, rooms, KG, tunnels) for SME-style consumers. |
| `GET /viz` | Render the snapshot as an HTML page — multi-view inspector for humans (D3 force-graph + Mermaid hierarchy + bar charts) over the same `/graph` data. |
| `GET /stats` | Aggregate query across implemented views and coordinator state. |

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale finding — the current doc no longer lists /graph or /viz in the endpoint table. Line 49 says wings/rooms/halls are surfaced by MCP tools (mempalace_list_wings / mempalace_list_rooms / mempalace_list_tunnels), and the endpoint table at lines 63-76 covers only routes that actually exist. The aspirational refs were dropped in a later commit; sorry for not closing the loop sooner.

Comment thread docs/event-log-frame.md
| `POST /repair mode=rebuild` | Drop and recompute the view from log; coordinate writes via queue-and-drain. |
| `GET /repair/status` | Surface view-coordinator state to clients. |
| `POST /flush` | Force checkpoint of pending log entries. |
| `POST /reload` | Invalidate cached view handles (alias for `/repair mode=light` from a different vocabulary). |
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POST /reload is described as an alias for /repair mode=light, but in code it only clears the client/collection caches and does not participate in the /repair state machine (e.g., no repair locking/state, no quarantine_stale_hnsw note/warning behavior). Consider rewording this to "similar to" / "overlaps with" mode=light, or have /reload delegate to the same implementation if true aliasing is intended.

Suggested change
| `POST /reload` | Invalidate cached view handles (alias for `/repair mode=light` from a different vocabulary). |
| `POST /reload` | Invalidate cached view handles; overlaps with `/repair mode=light`, but from a different vocabulary. |

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair — "alias" oversimplifies. Both routes DO clear the client/collection cache (the only thing /reload does, and what /repair?mode=light boils down to via flush_palace), but the alias framing implies they share state-machine semantics, which they don't. Will soften to: "Invalidate cached view handles. Same effect as /repair?mode=light from the perspective of cache state, though it doesn't enter the repair state machine."

Comment thread docs/event-log-frame.md Outdated
Comment on lines +99 to +100
- The view-coordinator role. `/silent-save`, `/search`, `/graph`,
`/stats` — same shape, same semantics, same client contract.
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section claims the view-coordinator role will keep the same client contract for /graph, but /graph is not currently an implemented endpoint in this repo. Please update the list of "stays the same" endpoints to the ones that actually exist (e.g. /mcp, /silent-save, /search, /stats, /repair, etc.) or annotate /graph as future work.

Suggested change
- The view-coordinator role. `/silent-save`, `/search`, `/graph`,
`/stats` — same shape, same semantics, same client contract.
- The view-coordinator role. `/mcp`, `/silent-save`, `/search`,
`/stats`, `/repair` — same shape, same semantics, same client
contract.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by the doc reorganization — the current line 100 is in the "Stays the same" bullet about queue-and-drain. The "client contract" sentence in the current text talks about the role itself (HTTP/MCP API surface), not a specific endpoint. No /graph claim in the current doc.

Comment thread docs/event-log-frame.md Outdated
Comment on lines +125 to +127
- View-rebuild coordination (only as needed by the backend)
- Structural fast paths for consumers (the `/graph` direct-sqlite read
is one example; under postgres it'd be a direct-postgres read)
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "structural fast paths" bullet cites a /graph direct-sqlite read as an existing example, but /graph is not present in the daemon codebase. Please adjust this example to a real endpoint/fast-path, or rephrase it as a potential future optimization under postgres.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current line 125-127 phrases this generically: "Optional structural fast paths for consumers (e.g. composing a single snapshot across wings + KG + tunnels in one HTTP roundtrip rather than forcing each adapter to fan out across multiple MCP calls). Under chroma the fast path can read the underlying sqlite directly; under postgres it'd be a direct-postgres aggregation query." No longer cites /graph by name. Worth flagging that #13 has since landed an actual GET /graph endpoint that materializes this exact pattern — could be a one-line cross-ref in a follow-up.

Comment thread docs/event-log-frame.md Outdated
2. **Different storage technologies serve different query patterns.**
For 1-hop lookups against a few facts, plain SQL is right. If the
KG ever grew to support multi-hop traversal at scale, Apache AGE
(cypher on postgres) would be the natural view technology.
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor terminology/casing: the graph query language is typically capitalized as "Cypher" (not "cypher"). Consider updating for correctness/consistency, especially since it's used as a proper name alongside "Apache AGE" and "Postgres".

Suggested change
(cypher on postgres) would be the natural view technology.
(Cypher on postgres) would be the natural view technology.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already "Cypher" in the current doc (line 143: "(Cypher on postgres) would be the natural view technology"). Closed via earlier commit; sorry for not closing the comment loop.

Comment thread docs/event-log-frame.md Outdated
| Immutable event log | Raw conversation transcripts + verbatim drawers in the chroma collection. The 96.6% LongMemEval result is _from this log_, not from any reformatted view. |
| Materialized view: semantic search | The **chroma vector index** (HNSW). |
| Materialized view: structured facts | The **knowledge graph** (`knowledge_graph.sqlite3` — entities + triples). |
| Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*` and palace-daemon's `/graph`). |
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc references palace-daemon's /graph endpoint as the way wings/rooms/halls are surfaced, but this repo's daemon currently does not implement a /graph route (endpoints are defined in main.py and include e.g. /stats, /repair, /mcp, etc.). Please update this row to reflect the actual API surface or clearly mark /graph as a future/planned endpoint.

Suggested change
| Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*` and palace-daemon's `/graph`). |
| Materialized view: categorical structure | **Wings / rooms / halls** (encoded in metadata, surfaced by `mempalace_list_*`; a palace-daemon `/graph` endpoint would be future/planned, not currently implemented). |

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the line-49 finding above — the current line says wings/rooms/halls are surfaced by mempalace_list_* MCP tools, not /graph. The earlier draft DID reference /graph as aspirational; removed during revision. With #13 in flight, a one-line cross-ref to the actual endpoint would be reasonable in a follow-up commit.

Adds `docs/event-log-frame.md` — an architectural-frame doc that
articulates how palace-daemon fits Kleppmann's "log + materialized
views" pattern from _Designing Data-Intensive Applications_ /
"Turning the database inside-out."

## Why this doc

The thesis: mempalace is event-streaming-shaped even though it hasn't
been articulated that way. The conversation transcripts are the
immutable log; the chroma vectors, knowledge graph, wing/room
categorization, and AAAK compressed text are all *materialized views*
computed from that log.

Under that frame, palace-daemon's role is the **view coordinator**:
accept writes to the log, serve queries against views, coordinate view
rebuilds (which is what `/repair` does), and handle write fairness
during rebuilds. That role is *durable across any backend*: ChromaDB,
pgvector, AGE, anything else — the backend just determines how a
particular view is stored.

## What it maps

A table mapping mempalace components onto Kleppmann concepts:

- Immutable event log → conversation transcripts + verbatim drawers
- Materialized view: semantic search → chroma vector index
- Materialized view: structured facts → knowledge graph (sqlite)
- Materialized view: categorical structure → wings / rooms / halls
- View rebuild operation → `/repair {mode: rebuild}`
- View consistency window → write semaphore + rebuild coordination

Plus what palace-daemon *already* implements implicitly under this
frame, what's chroma-specific vs. role-level, and what changes (and
doesn't) under the upcoming postgres backend (#665) and other
storage transitions.

## Why now

A reference doc ahead of the multi-backend transition that Igor's
work makes possible — articulating *what role* the daemon plays
makes the question "should this code be daemon-side or backend-side?"
tractable when the next storage adapter lands.

No code changes; pure reference doc, 191 lines. Linked from upstream
Discussion #5 (Postgres backend) and rboarescu#6 (TS rewrite) via comments
once it lands so contributors can reach for it without digging through
the README.

Originally fork commit `432b3a6`; sending standalone per the
"small separate PRs" preference.
@jphein jphein force-pushed the docs/event-log-frame branch from d3b6532 to 62ff62d Compare April 27, 2026 17:52
jphein added a commit to jphein/palace-daemon that referenced this pull request Apr 30, 2026
…nding queue; trim patch

Filed four upstream PRs on 2026-04-30:
- rboarescu#15  feat: GET /viz status dashboard (stacks on rboarescu#13)
- rboarescu#16  feat: GET /list — query-free metadata browse
- rboarescu#17  feat: DELETE /memory + PATCH /memory
- rboarescu#18  feat(lifespan): auto-migrate Stop-hook checkpoints on startup

Also rebased PR rboarescu#13 onto upstream/main to clear a CHANGELOG conflict
left by upstream's b4aee82 (patch sync) — state went CONFLICTING ->
MERGEABLE / CLEAN.

README:
- Open upstream PRs table: four new rows (rboarescu#15-rboarescu#18) plus a 2026-04-30
  note covering today's rebase + new PRs in one breath.
- Pending PRs queue: now empty. Replaced the four stale rows
  (event-log-frame and graph-endpoint were already in flight via
  rboarescu#11 and rboarescu#13; mempal-fast.py was already upstream via the merged
  PR #4 omnibus; /viz is now PR rboarescu#15) with a brief empty-state note.

CLAUDE.md:
- Patch description trimmed to reflect that the hnsw:num_threads
  enforcement landed upstream via _pin_hnsw_threads(); only the
  log + retry-once slice remains.

patches/mcp_server_get_collection.patch:
- Regenerated against current mempalace develop. The patch is now
  just the "log exception + retry once on cache failure" change.
  Filed upstream as MemPalace/mempalace#1286; once that merges this
  patch retires.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants