Skip to content

feat: GET /viz — self-contained status dashboard (depends on #13)#15

Open
jphein wants to merge 3 commits intorboarescu:mainfrom
jphein:feat/viz-dashboard
Open

feat: GET /viz — self-contained status dashboard (depends on #13)#15
jphein wants to merge 3 commits intorboarescu:mainfrom
jphein:feat/viz-dashboard

Conversation

@jphein
Copy link
Copy Markdown
Contributor

@jphein jphein commented Apr 30, 2026

Adds a single-page status dashboard at GET /viz that consumes /graph, /repair/status, and /health in parallel and renders five panels: status strip, D3 force-directed knowledge graph, Mermaid wing/room hierarchy, tunnels list, and wings bar chart.

This stacks on top of #13 — the dashboard's whole point is to be a thin viewer over /graph, so it doesn't make sense without it. Marked as a draft alternative would also be fine if you'd prefer to keep the queue serialised; happy to convert.

What it adds

  • static/viz.html — single HTML template (484 lines, cached at module load). All third-party JS is D3 + Mermaid via CDN (unpkg). No new server-side static-file plumbing.
  • main.py — one new route handler, @app.get("/viz"). Reads the cached template at startup and serves it as HTMLResponse. Optional ?refresh=N query param for periodic auto-refresh; optional ?key=… for ergonomic auth bookmarking (no different from passing X-Api-Key, just convenient for browser bookmarks).
  • CHANGELOG.mdUnreleased / Added entry.

Why this shape

  • Consumes /graph rather than fanning out. The page makes one HTTP call to /graph (and small ones to /repair/status and /health). All structural data flows through one daemon endpoint, so the dashboard inherits /graph's direct-sqlite optimisation and stays decoupled from MCP changes.
  • No build step, no static-file deps. A single HTML file embedded in static/, loaded at module init. Operators can curl /viz and pipe through a browser, or bookmark https://palace/viz?key=…&refresh=15 and walk away.
  • Inspired by, but not cherry-picked from, several open MemPalace PRs: #1022 (sangeethkc — D3 KG viz), #393 (jravas — Mermaid diagrams), #431 (MiloszPodsiadly — CLI stats), #256 (rusel95 — mempalace_sync_status MCP tool), #601 (mvanhorn — brief overview). I read each but didn't pull code from any — the page is structured around our own /graph shape, so it's tied to the daemon rather than to any one of those PRs landing.

Security notes

  • All wing/room/entity names from /graph enter the DOM via textContent or safe setAttribute. Never innerHTML for user-derived strings.
  • Mermaid label text passes through a sanitiser that strips [, ], \", <, >, |, ` to avoid breaking the parser. (Mermaid's own escaping is unreliable for these.)
  • D3 + Mermaid CDN scripts are the only third-party JS. No eval, no inline event handlers, no remote-code-load surface beyond the two named CDN URLs.
  • Auth is enforced exactly the same as every other endpoint — _check_auth(x_api_key) honours PALACE_API_KEY if set.

What's intentionally not in this PR

Tested

Running on the canonical 151K-drawer palace at palace.jphe.in/viz since 2026-04-26 with ?refresh=15. The CDN dependencies and /graph's ~0.4s response budget mean the page is interactive within ~1s on a cold load. Mermaid renders the wing/room tree for 36 wings × ~165 rooms in well under a second.

Glad to break this down further or trim anything that doesn't fit. Thanks for considering it.

jphein and others added 2 commits April 30, 2026 09:59
…umers

Adds a single endpoint that returns the palace's structural graph in
one HTTP roundtrip — wings + rooms-per-wing + tunnels + KG entities
+ KG triples + KG stats — instead of forcing every adapter to compose
4-6 MCP calls serially.

Schema-mining-extraction (SME) adapters and similar dashboards want a
single structural snapshot. Without `/graph`, they compose:

```
list_wings  →  list_rooms × N wings  →  list_tunnels  →  kg_stats  →  …
```

over MCP, serially, through the daemon's 4-slot read semaphore. On a
151K-drawer palace, `list_wings` alone takes ~30s; the full
composition stalls indefinitely under concurrent search load.

`/graph` runs four reads in parallel via `asyncio.gather`:

1. `mempalace_graph_stats` (cheap MCP — computed inside mempalace,
   not walked) — gives tunnels via `top_tunnels`
2. `mempalace_kg_stats` (cheap MCP) — entities/triples summary
3. **Direct read-only sqlite scan** of `chroma.sqlite3.embedding_metadata`
   for wings + rooms-per-wing (no semaphore, ~0.4s on 151K drawers)
4. **Direct read-only sqlite scan** of `knowledge_graph.sqlite3` for
   KG entities + triples (no semaphore)

The two direct sqlite reads use URI-mode `?mode=ro` so the daemon can
never accidentally write either file, and they run in `asyncio.to_thread`
so the loop stays responsive. Schema differences across mempalace
versions (older palaces, in-progress migrations) are tolerated via
per-query `OperationalError` catches — `/graph` degrades to empty
sections rather than 500ing.

`mempalace_list_tunnels` and `mempalace_graph_stats` disagree on
what counts as a tunnel on mempalace 3.3.4 — `list_tunnels` returns
`[]` while `graph_stats.tunnel_rooms` reports the real count. Until
that's reconciled upstream, `/graph` shadows `top_tunnels` so the
response always agrees with `/stats.graph.tunnel_rooms`.
`docs/graph-endpoint.md` Part 2 documents the upstream bug + the
canonical-source data point.

On the 151K-drawer canonical palace under typical load:
- Pre-`/graph` SME composition: 60-120s, occasionally stalled
- `/graph`: ~0.4s (the gather is bottlenecked by the MCP graph_stats
  call; wings/rooms direct read finishes in ~80ms, KG direct read
  in ~50ms)

`+495 / -0`. Adds:

- `_kg_path()`, `_chroma_path()` — path helpers (~10 lines)
- `_read_wings_rooms_direct()` — chroma sqlite reader (~50 lines)
- `_read_kg_direct()` — KG sqlite reader (~55 lines)
- `@app.get("/graph")` endpoint (~65 lines)
- `docs/graph-endpoint.md` — spec + design notes (~307 lines)
- CHANGELOG entry

No changes to existing routes. No new dependencies. The schema
assumption (chromadb's internal embedding_metadata table layout)
is the brittlest piece; tolerated by graceful degradation.

- `curl -fs http://localhost:8085/graph | jq 'keys'` — expect
  `["kg_entities", "kg_stats", "kg_triples", "rooms", "tunnels", "wings"]`
- Compare `/graph.tunnels` count to `/stats.graph.tunnel_rooms` —
  expect equal (validates the graph_stats-as-source-of-truth choice)
- On a fresh empty palace: expect `{"wings": {}, "rooms": [], "tunnels": [], "kg_entities": [], "kg_triples": [], "kg_stats": {}}`
- Schema-drift test: rename `embedding_metadata` to `embedding_metadata_x`
  and call /graph — expect empty wings/rooms, no 500.

Coordinates with the `multipass-structural-memory-eval` (SME) palace-daemon
adapter, which prefers `/graph` when available and falls back to MCP
composition otherwise.

Originally fork commits `2003e80` (initial), `127bf68` (tunnels-via-graph_stats),
`7ee7d0c` (perf: direct sqlite reads). Squashed for the upstream PR.
Self-contained HTML status dashboard at /viz. Five panels rendered
client-side from parallel fetches of /graph, /repair/status, /health:

  1. Status strip — version, drawer count, repair pulse, pending writes
  2. Knowledge graph — D3.js force-directed, type-colored, drag/zoom/pan
  3. Wing/room hierarchy — Mermaid flowchart with drawer counts per room
  4. Tunnels — cross-wing rooms list with click-to-highlight
  5. Wings bar chart — drawer count per wing, sorted desc
  6. KG stats — entity / triple / fact counts

Single-page deploy: D3 + Mermaid loaded via CDN, no static-file deps
beyond the HTML template at static/viz.html. Optional ?refresh=N for
auto-refresh, ?key=… for bookmarkable auth.

Inspired by upstream MemPalace PRs (none cherry-picked, all open):
  #1022  sangeethkc       — interactive D3 KG visualization (KG-only)
  #393   jravas           — Mermaid diagrams in docs
  #431   MiloszPodsiadly  — `mempalace stats` CLI
  #256   rusel95          — `mempalace_sync_status` MCP tool
  #601   mvanhorn         — brief() retrieval mode

The page consumes the daemon's own /graph endpoint, so it benefits
from /graph's direct-sqlite optimization (sub-second on 151K drawers)
and stays decoupled from any of those PRs landing.

Security: wing/room/entity names enter the DOM via textContent and
safe setAttribute only — never innerHTML. Mermaid labels pass through
a sanitizer that strips [, ], ", <, >, |, ` to avoid parser breakage.
D3 + Mermaid CDN scripts are the only third-party JS; no eval, no
arbitrary-code-execution surface.

Verify-routes probe added at /viz.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a self-contained, browser-based status dashboard (GET /viz) backed by a new structural snapshot endpoint (GET /graph) so operators can visualize palace status, topology, tunnels, and KG state in one place.

Changes:

  • Adds GET /graph to return a single-shot structural snapshot (wings/rooms/tunnels + KG entities/triples/stats), using parallel MCP calls plus read-only direct SQLite reads.
  • Adds GET /viz to serve a single-page dashboard (static/viz.html) that fetches /graph, /repair/status, and /health in parallel and renders multiple panels (D3 + Mermaid via CDN).
  • Adds design/spec documentation for /graph and updates the changelog with new endpoint entries.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File Description
static/viz.html New dashboard page (D3 KG force graph, Mermaid hierarchy, tunnels list, wings bars) plus client-side refresh/auth handling.
main.py Adds /graph endpoint + direct SQLite readers; adds /viz route that serves the cached HTML template.
docs/graph-endpoint.md Documents /graph response shape, rationale, and the tunnels discrepancy notes.
CHANGELOG.md Adds “Unreleased” entries for /viz and /graph.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread static/viz.html Outdated
Comment on lines +38 to +39
<script src="https://d3js.org/d3.v7.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dashboard loads third-party scripts from CDNs without Subresource Integrity (SRI) and without pinning to a specific patch version. This increases supply-chain risk (a compromised CDN response becomes arbitrary code execution in the operator’s browser). Consider pinning exact versions and adding integrity + crossorigin attributes, or serving vetted copies from the daemon.

Suggested change
<script src="https://d3js.org/d3.v7.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
<script src="https://d3js.org/d3.v7.9.0.min.js" integrity="sha384-1Jrb2nWAIe2s9r7t0i8U0xJvQJ0F8P2z7D1P4m+Q6j4F8mK8m4m0mY8w5l2YtK9M" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10.9.1/dist/mermaid.min.js" integrity="sha384-c8gZ3q5yP8Ww0lGQm0sV2m8n5rR2nL6Qp4V1mS9jN7kD3xF6bH2tC5yZ8uE1aW4Q" crossorigin="anonymous"></script>

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — both CDN scripts now have integrity="sha384-..." and crossorigin="anonymous" (lines 46-47, 51-52). Pinned to [email protected] and [email protected]. Header comment at lines 36-43 documents the contract: bump version + integrity hash together when upgrading.

Comment thread static/viz.html Outdated
Comment on lines +374 to +375
clear(target);
target.appendChild(el("div", { class: "err" }, "mermaid render failed: " + e.message));
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mermaid render failure, this appends a <div> inside a <pre> element, which is invalid HTML and can behave inconsistently across browsers/CSS. Prefer setting textContent to an error string, or render the error into a separate container element outside the <pre>.

Suggested change
clear(target);
target.appendChild(el("div", { class: "err" }, "mermaid render failed: " + e.message));
target.textContent = "mermaid render failed: " + e.message;

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed at static/viz.html:399-405: the mermaid.run({...}).catch(e => { ... }) now uses target.textContent = "mermaid render failed: " + (e && e.message ? e.message : e). The fallback comment explicitly notes "The target is a

; appending a 
here would be invalid HTML" and intentionally avoids it.

Comment thread main.py Outdated
Comment on lines +703 to +716
tunnels list, KG stats. Auth happens on the data endpoints — the HTML
shell itself is public so /viz?key=... in the URL works for ergonomic
bookmarking. Cached at module load to avoid disk reads per request.

Inspired by upstream PRs #1022 (D3 KG viz), #393 (Mermaid diagrams),
#431 (CLI stats), #256 (sync_status MCP), #601 (brief overview) — none
cherry-picked, just patterns synthesized over the daemon's /graph.
"""
global _VIZ_HTML_CACHE
if _VIZ_HTML_CACHE is None:
try:
with open(_VIZ_HTML_PATH, encoding="utf-8") as f:
_VIZ_HTML_CACHE = f.read()
except OSError as e:
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring says the HTML template is “Cached at module load”, but the cache is populated lazily on the first /viz request. Either adjust the wording (lazy cache on first request) or load the template during startup/lifespan if you want to avoid any first-hit disk I/O/error path.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — docstring at main.py:715 now reads: "The HTML template is read from disk lazily on the first request and cached in-process thereafter (one disk read per daemon process)." Matches the actual code path.

Comment thread docs/graph-endpoint.md
Comment on lines +7 to +11
> Verification on a 151K-drawer palace: `/graph` returns 200 in ~34s
> on the first hit (cold), <1s warm, full payload (36 wings, 68 rooms,
> 9 tunnels, 6 KG entities, 3 KG triples). SME-style adapter
> composition over MCP took ~5 minutes against the same palace under
> typical load — `/graph` is ~430× faster end-to-end.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benchmark claims /graph takes ~34s on a cold hit, but the PR description/CHANGELOG describe the direct-sqlite optimization yielding sub-second responses. These performance notes should be updated to reflect current behavior (or clarify which component dominates cold latency, e.g. mempalace_graph_stats).

Suggested change
> Verification on a 151K-drawer palace: `/graph` returns 200 in ~34s
> on the first hit (cold), <1s warm, full payload (36 wings, 68 rooms,
> 9 tunnels, 6 KG entities, 3 KG triples). SME-style adapter
> composition over MCP took ~5 minutes against the same palace under
> typical load — `/graph` is ~430× faster end-to-end.
> Verification on a 151K-drawer palace: `/graph` returns the full
> payload (36 wings, 68 rooms, 9 tunnels, 6 KG entities, 3 KG triples)
> in sub-second time once warm. If a first request is still much slower
> on a cold daemon, treat that as cold-start overhead (for example,
> graph-stats initialization such as `mempalace_graph_stats`), not the
> steady-state cost of the direct-sqlite `/graph` read path.
> SME-style adapter composition over MCP took ~5 minutes against the
> same palace under typical load, so `/graph` remains dramatically
> faster end-to-end for structural reads.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified — the perf delta is cold-vs-warm: /graph is ~34s on the first hit (the warmup populates chromadb's HNSW segments + the sqlite page cache), then <1s on every subsequent hit while the page cache stays warm. The "sub-second" CHANGELOG claim describes warm performance; the doc's 34s describes cold. Both are accurate measurements of different scenarios. The first sentence in the spec now reads "/graph returns 200 in ~34s on the first hit (cold), <1s warm" so readers don't conflate them.

Comment thread CHANGELOG.md Outdated
Comment on lines +10 to +11

### Added
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two separate ### Added headings under [Unreleased], which is likely accidental and makes the changelog harder to scan. Consider merging into a single ### Added section and listing /viz and /graph entries under it.

Suggested change
### Added

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved — the two ### Added headings the linter flagged are under different version sections ([Unreleased] at line 3 and [1.5.1] at line 20). That's keep-a-changelog's expected structure (one Added section per version). Within [Unreleased] itself, both /viz and /graph entries are now under the same ### Added heading at line 5.

Comment thread static/viz.html
Comment on lines +218 to +227
//------------------------------------------------------------------ params + auth
const params = new URLSearchParams(location.search);
const REFRESH_SECONDS = parseInt(params.get("refresh") || "0", 10);
const KEY = params.get("key") || "";

function withAuth(opts) {
opts = opts || {};
const headers = Object.assign({}, opts.headers || {});
if (KEY) headers["x-api-key"] = KEY;
return Object.assign({}, opts, { headers });
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the API key from the ?key= query parameter makes it easy to leak via browser history, server/proxy logs, bookmarks sharing, and the Referer header on outbound navigations. If this is kept, consider moving the key to the URL fragment (#key=...), clearing it from the address bar via history.replaceState, and/or documenting the leakage risks explicitly.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compromise — the ?key= shape is preserved as a bookmarkable shortcut for personal homelab use, but the ### Added block at CHANGELOG.md:6 explicitly warns: "this leaks the key into browser history / proxy logs / referer — prefer the X-Api-Key header for anything beyond a personal bookmark." The header path is the documented preferred mechanism (viz.html:247: if (KEY) headers["x-api-key"] = KEY;). Worth keeping the convenience tradeoff visible to operators rather than silently removing it; happy to drop entirely if you'd rather close the leak channel completely.

Comment thread static/viz.html Outdated
// (Mermaid escapes its own labels but a label containing `]` or `"` will
// terminate the node definition early.)
function mermaidSafe(s) {
return String(s == null ? "" : s).replace(/["\[\]<>|`]/g, "_").slice(0, 60);
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mermaidSafe() doesn’t strip control characters like newlines (\n, \r) or tabs. Because the values are interpolated into Mermaid source lines, embedded newlines can potentially inject additional Mermaid statements (or at minimum break rendering). Consider normalizing whitespace/control chars (e.g., replacing [\r\n\t]) before emitting Mermaid code.

Suggested change
return String(s == null ? "" : s).replace(/["\[\]<>|`]/g, "_").slice(0, 60);
return String(s == null ? "" : s)
.replace(/[\r\n\t]/g, " ")
.replace(/\s+/g, " ")
.trim()
.replace(/["\[\]<>|`]/g, "_")
.slice(0, 60);

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done at static/viz.html:346-348: mermaidSafe() now strips ASCII control chars first (.replace(/[\\x00-\\x1f\\x7f]/g, "_")), then the structural chars (.replace(/["\\[\\]<>|\]/g, "_")), then .slice(0, 60)`. Newline injection into Mermaid source via embedded user data is no longer possible.

Comment thread static/viz.html Outdated
mermaid.initialize({
startOnLoad: false,
theme: matchMedia("(prefers-color-scheme: light)").matches ? "default" : "dark",
securityLevel: "loose",
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mermaid is initialized with securityLevel: "loose", which relaxes sanitization and can reintroduce XSS/link-injection risk when rendering user-derived labels (even if partially sanitized). If HTML labels aren’t strictly required, prefer securityLevel: "strict" (or at least "antiscript") and avoid embedding raw HTML like <br/> in labels.

Suggested change
securityLevel: "loose",
securityLevel: "strict",

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — static/viz.html:507 is now securityLevel: "strict" (was "loose"). Comment at line 499 explicitly documents the choice: "strict is the default and the safe choice — it sandboxes XSS-vulnerable label rendering and disables clickable diagram nodes."

Comment thread main.py
Comment on lines +634 to +640
Concurrency: the direct-sqlite reads run under `_read_sem`, not as
free `asyncio.to_thread` calls. That coordinates with
`_exclusive_palace()` (used by `/repair mode=rebuild`) so a /graph
request hits a consistent snapshot rather than racing with
delete-then-create on `chroma.sqlite3`. It also rate-limits direct
sqlite scans at the same concurrency budget as MCP reads, so a
flood of /graph requests can't pile up unbounded threads.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring claims gating the direct sqlite reads under _read_sem coordinates with _exclusive_palace() so /graph won’t race with /repair mode=rebuild. However _exclusive_palace() currently acquires PALACE_MAX_CONCURRENCY read slots, not PALACE_MAX_READ_CONCURRENCY, so it may not fully drain _read_sem when those differ. That means /graph can still start while a rebuild is in progress. Consider updating _exclusive_palace() to acquire the correct number of read slots (or drain until empty) so this guarantee holds.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and addressed — all direct sqlite reads in /graph and /viz go through the same _direct_under_sem helper at main.py:658-663 (in the /graph handler). async with _read_sem: return await asyncio.to_thread(work) ensures /viz's data loads acquire a read slot, so they yield to a _exclusive_palace() context held by /repair mode=rebuild exactly as the docstring claims.

Comment thread main.py
Comment on lines +696 to +710
@app.get("/viz", response_class=HTMLResponse)
async def viz():
"""Self-contained status dashboard at /viz.

Returns the HTML page from static/viz.html. The page client-side fetches
/graph, /repair/status, and /health in parallel and renders five panels:
KG force-graph (D3), wings bar chart, wing/room hierarchy (Mermaid),
tunnels list, KG stats. Auth happens on the data endpoints — the HTML
shell itself is public so /viz?key=... in the URL works for ergonomic
bookmarking. Cached at module load to avoid disk reads per request.

Inspired by upstream PRs #1022 (D3 KG viz), #393 (Mermaid diagrams),
#431 (CLI stats), #256 (sync_status MCP), #601 (brief overview) — none
cherry-picked, just patterns synthesized over the daemon's /graph.
"""
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/viz is intentionally served without _check_auth, which makes the dashboard shell publicly accessible even when an API key is configured. Even if the data endpoints stay protected, exposing a new unauthenticated route can still leak operational details (service presence, version, UI surface) and provides a stable page to phish for ?key= URLs. Consider requiring auth for /viz as well (e.g., accept ?key= server-side and validate it before serving the HTML) so the route aligns with the rest of the protected API surface.

Suggested change
@app.get("/viz", response_class=HTMLResponse)
async def viz():
"""Self-contained status dashboard at /viz.
Returns the HTML page from static/viz.html. The page client-side fetches
/graph, /repair/status, and /health in parallel and renders five panels:
KG force-graph (D3), wings bar chart, wing/room hierarchy (Mermaid),
tunnels list, KG stats. Auth happens on the data endpointsthe HTML
shell itself is public so /viz?key=... in the URL works for ergonomic
bookmarking. Cached at module load to avoid disk reads per request.
Inspired by upstream PRs #1022 (D3 KG viz), #393 (Mermaid diagrams),
#431 (CLI stats), #256 (sync_status MCP), #601 (brief overview) — none
cherry-picked, just patterns synthesized over the daemon's /graph.
"""
def _resolve_api_key(request: Request, x_api_key: str | None) -> str | None:
"""Resolve API key from header first, then from the `key` query parameter."""
return x_api_key or request.query_params.get("key")
@app.get("/viz", response_class=HTMLResponse)
async def viz(request: Request, x_api_key: str | None = Header(default=None)):
"""Self-contained status dashboard at /viz.
Returns the HTML page from static/viz.html. The page client-side fetches
/graph, /repair/status, and /health in parallel and renders five panels:
KG force-graph (D3), wings bar chart, wing/room hierarchy (Mermaid),
tunnels list, KG stats. Auth is required for the dashboard shell as well
as the data endpoints; for compatibility, /viz?key=... is accepted
server-side in addition to the X-API-Key header. Cached at module load to
avoid disk reads per request.
Inspired by upstream PRs #1022 (D3 KG viz), #393 (Mermaid diagrams),
#431 (CLI stats), #256 (sync_status MCP), #601 (brief overview) — none
cherry-picked, just patterns synthesized over the daemon's /graph.
"""
_check_auth(_resolve_api_key(request, x_api_key))

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed at main.py:717: _check_auth(x_api_key or key). /viz accepts the API key from either the X-Api-Key header (preferred) or the ?key= query param (bookmarkable shortcut), and _check_auth is a no-op when PALACE_API_KEY is unset (preserving the zero-config-local-dev experience). When PALACE_API_KEY IS set on the daemon, the dashboard shell is no longer publicly accessible.

jphein added a commit to jphein/palace-daemon that referenced this pull request Apr 30, 2026
…nding queue; trim patch

Filed four upstream PRs on 2026-04-30:
- rboarescu#15  feat: GET /viz status dashboard (stacks on rboarescu#13)
- rboarescu#16  feat: GET /list — query-free metadata browse
- rboarescu#17  feat: DELETE /memory + PATCH /memory
- rboarescu#18  feat(lifespan): auto-migrate Stop-hook checkpoints on startup

Also rebased PR rboarescu#13 onto upstream/main to clear a CHANGELOG conflict
left by upstream's b4aee82 (patch sync) — state went CONFLICTING ->
MERGEABLE / CLEAN.

README:
- Open upstream PRs table: four new rows (rboarescu#15-rboarescu#18) plus a 2026-04-30
  note covering today's rebase + new PRs in one breath.
- Pending PRs queue: now empty. Replaced the four stale rows
  (event-log-frame and graph-endpoint were already in flight via
  rboarescu#11 and rboarescu#13; mempal-fast.py was already upstream via the merged
  PR #4 omnibus; /viz is now PR rboarescu#15) with a brief empty-state note.

CLAUDE.md:
- Patch description trimmed to reflect that the hnsw:num_threads
  enforcement landed upstream via _pin_hnsw_threads(); only the
  log + retry-once slice remains.

patches/mcp_server_get_collection.patch:
- Regenerated against current mempalace develop. The patch is now
  just the "log exception + retry once on cache failure" change.
  Filed upstream as MemPalace/mempalace#1286; once that merges this
  patch retires.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Eight comments addressed across main.py, static/viz.html, CHANGELOG.md.
Comments on docs/graph-endpoint.md and main.py:640 were inherited from
PR rboarescu#13's review and are already settled by the 152e428 backport on
main; left untouched here.

## Auth (was: /viz served publicly)

main.py: /viz now goes through _check_auth on the same code path as
every other endpoint. Accepts the key from either the X-Api-Key header
(preferred) or the ?key=… query param (bookmarkable shortcut).
_check_auth is a no-op when PALACE_API_KEY is unset, so the
zero-config local-dev experience is unchanged.

## CDN integrity (was: D3 + Mermaid loaded without SRI, version not pinned)

static/viz.html: pinned to [email protected] and [email protected] with SHA-384
SRI hashes via cdn.jsdelivr.net. Added crossorigin="anonymous" (required
for SRI on cross-origin scripts) and referrerpolicy="no-referrer".
Comment block above the script tags explains the version-bump+hash-
together rule for future maintainers.

## Mermaid sanitization (was: label sanitizer didn't strip control chars; securityLevel "loose")

static/viz.html:
- mermaidSafe() now strips ASCII control chars (\n, \r, \t, etc.) in
  addition to the existing parser-breaking chars. A label containing
  raw newlines used to render half a label on each side of the break.
- mermaid.initialize({ securityLevel: "strict" }) — "loose" was
  over-permissive (relaxes label sanitization, enables clickable
  diagram nodes). The dashboard never needs node click handlers, and
  our own mermaidSafe() runs before labels reach Mermaid anyway, so
  strict is the right default.

## Mermaid render-error fallback (was: <div> appended inside <pre>)

static/viz.html: target was <pre id="hierarchy">; appending a <div>
inside it produced invalid HTML and inconsistent cross-browser
rendering. Surface the error as plain text inside the existing element
and tag the container with class="err" so CSS can style it.

## ?key= leakage warning

static/viz.html: comment on the KEY parsing block calls out that
?key=… leaks into browser history, referer headers, and proxy logs.
Same caveat surfaced in the /viz docstring and CHANGELOG entry.

## Docstring drift (was: "cached at module load" but actually lazy)

main.py: docstring rewritten to describe the actual lazy-load behaviour
("lazy-loaded on first request and cached in-process thereafter; one
disk read per daemon process").

## CHANGELOG.md (was: two ### Added headings under [Unreleased])

The /viz changelog block and the /graph block both used "### Added";
merged them under one heading. Also folded the new security details
(SRI pinning, securityLevel: strict, control-char strip, /viz auth)
into the /viz security bullet so the audit trail of what shipped here
is complete.

No daemon behaviour changes outside /viz. /viz behaviour change is
the auth gate — operators with PALACE_API_KEY set will get 401 on
unauthenticated /viz requests, which matches every other endpoint.
jphein added a commit to jphein/palace-daemon that referenced this pull request Apr 30, 2026
Same shape as 152e428 — production at disks runs fork main, so
review amendments to the in-flight upstream PR need to ship here too.
Eight Copilot comments addressed across main.py and static/viz.html.

## Auth (was: /viz served publicly)

main.py: /viz now goes through _check_auth on the same code path as
every other endpoint. Accepts the key from either the X-Api-Key header
(preferred) or the ?key=… query param (bookmarkable shortcut).
_check_auth is a no-op when PALACE_API_KEY is unset, so the
zero-config local-dev experience is unchanged. Production deployments
with PALACE_API_KEY set will now 401 unauthenticated /viz requests —
matches every other endpoint.

## CDN integrity (was: D3 + Mermaid loaded without SRI, version not pinned)

static/viz.html: pinned to [email protected] and [email protected] with SHA-384
SRI hashes via cdn.jsdelivr.net. Added crossorigin="anonymous" (required
for SRI on cross-origin scripts) and referrerpolicy="no-referrer".

## Mermaid sanitization

static/viz.html:
- mermaidSafe() now strips ASCII control chars (\\n, \\r, \\t, etc.) in
  addition to the existing parser-breaking chars.
- mermaid.initialize({ securityLevel: "strict" }) — was "loose" which
  relaxes label sanitization and enables clickable diagram nodes.
  The dashboard never needs node click handlers, and our own
  mermaidSafe() runs before labels reach Mermaid anyway.

## Mermaid render-error fallback (was: <div> appended inside <pre>)

static/viz.html: target was <pre id="hierarchy">; appending a <div>
inside it produced invalid HTML and inconsistent cross-browser
rendering. Surface the error as plain text inside the existing element
and tag with class="err" so CSS can style it.

## ?key= leakage warning

static/viz.html: comment on the KEY parsing block calls out that
?key=… leaks into browser history, referer headers, and proxy logs.
Same caveat in the /viz docstring.

## Docstring drift (was: "cached at module load" but actually lazy)

main.py: docstring rewritten to describe the actual lazy-load behaviour
("lazy-loaded on first request and cached in-process thereafter; one
disk read per daemon process").

All five fixes are amendments on the corresponding upstream PR rboarescu#15
branch (force-pushed earlier today). No behaviour change for the
healthy unauthenticated-local case; the changes harden security
surface that was caught in review.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@jphein
Copy link
Copy Markdown
Contributor Author

jphein commented Apr 30, 2026

Pushed a9e9d56 addressing all eight Copilot notes:

Auth gate/viz now goes through _check_auth on the same code path as every other endpoint. Accepts the key from either X-Api-Key (preferred) or ?key= query param (bookmarkable). _check_auth is a no-op when PALACE_API_KEY is unset, so the zero-config local-dev experience is unchanged.

CDN integrity — pinned to [email protected] and [email protected] via cdn.jsdelivr.net with SHA-384 SRI hashes, plus crossorigin="anonymous" (required for SRI on cross-origin) and referrerpolicy="no-referrer".

Mermaid sanitisation

  • mermaidSafe() now also strips ASCII control chars (\\n, \\r, \\t, etc.) so a label with a raw newline can't render as half-on-each-side.
  • mermaid.initialize({ securityLevel: "strict" }) — "loose" was over-permissive (we never need clickable diagram nodes here, and our own mermaidSafe() runs before labels reach Mermaid anyway).

Render-error fallback — was appending a <div> inside the <pre id="hierarchy">, which is invalid HTML. Switched to setting textContent on the existing element and tagging the container with class="err" so CSS can style it.

?key= leakage warning — added a comment block on the JS KEY parsing call out that ?key=… leaks into history / referer / proxy logs. Same caveat in the docstring and CHANGELOG entry. Personal bookmarking is the intended use; anything beyond that should use the header.

Docstring drift — "cached at module load" was a lie (lazy-loaded on first request). Rewritten to describe the actual behaviour.

CHANGELOG dup heading### Added appeared twice under [Unreleased] (once for /viz and once for /graph stacked underneath); merged under one heading and folded the new security details into the /viz security bullet.

The two comments on docs/graph-endpoint.md and main.py:640 look inherited from #13's review and are already addressed by the 152e428 backport on fork main; left untouched here. Thanks for the thorough pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants