Skip to content

Commit 581b29b

Browse files
cdeustclaude
andcommitted
feat(predictions): record what is expected, resolve it, score the confidence (#597)
Cortex stored what happened and never what was expected, so nothing it held could turn out wrong in a way it noticed: the 796 procedural skills mined from the maintainer's transcripts all sit at proficiency 0.5 because no session records an outcome, and while claim_events carry a confidence, nothing compares it to an observation. A predictions table on both backends holds the claim, the prediction in falsifiable terms, the test that would settle it and the confidence its author held. A row is written open and moves once to resolved, carrying what was observed, a verdict, the kind of source that decided it and a reference to that source; a table CHECK and the handler both refuse a resolution without one. Cortex never fetches that evidence and knows nothing about any forge or review convention, including this project's own marker. That is what makes the contract work in any repository: a host maps its convention onto the same four fields (ADR-1076). core/calibration.py scores the resolved rows with the Brier score (Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns, and the per-band reliability the mean cannot show. An abandoned prediction is counted apart and scored nowhere. Three tools registered: predict, resolve_prediction, calibration. Standalone count 54 to 57, 60 with both upstream integrations, with the pinned test, the catalogue, the eight documents the claim gate scans and the docker-smoke floor moved with it. Full suite: 8763 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
1 parent 0fc6bd1 commit 581b29b

29 files changed

Lines changed: 1214 additions & 33 deletions

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,23 @@ adheres to [Semantic Versioning](https://semver.org/).
88

99
### Added
1010

11+
- **Prediction records, and a score for the confidences they carry (#597).**
12+
Cortex stored what happened and never what was expected, so nothing it held
13+
could turn out wrong in a way it noticed: 796 procedural skills mined from
14+
the maintainer's transcripts all sit at proficiency 0.5 because no session
15+
records an outcome. Three tools now close that loop. `predict` writes a
16+
falsifiable prediction with the confidence held before the outcome is known;
17+
`resolve_prediction` settles it against an observation, naming the verdict,
18+
the kind of source that decided it and a reference to that source; and
19+
`calibration` returns the Brier score with the 0.25 a constant 0.5 forecast
20+
earns, plus the per-band frequencies the mean cannot show. Cortex never
21+
fetches the evidence and knows nothing about any forge or review
22+
convention, which is what makes the contract work in any repository
23+
(ADR-1076). The table lives on both backends. Standalone tool count 54 to
24+
57, 60 with both upstream integrations.
25+
26+
### Added
27+
1128
- **The Path B draft tools are exposed over MCP (#579).** `wiki_get_draft`
1229
and `wiki_refine_draft` had existed as handlers since ADR-0467 without ever
1330
being registered, so no client could call them and `wiki_get_draft`'s own

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ the canonical list) backing the algorithms. Every change is held to that bar.
99

1010
## What this project is
1111

12-
A Python 3.10+ MCP server with **54 standalone tools** (57 with the optional
12+
A Python 3.10+ MCP server with **57 standalone tools** (60 with the optional
1313
ai-architect-mcp-codebase + prd-spec-generator integrations) and **9 automatic
1414
hooks**, persisting to a local SQLite store by default or to PostgreSQL +
1515
pgvector when configured. Implements rate-distortion forgetting,
@@ -242,14 +242,14 @@ mechanically for the modules it is scoped to.
242242

243243
## Adding an MCP tool
244244

245-
54 standalone tools currently (57 with the optional upstream integrations). Adding a new one:
245+
57 standalone tools currently (60 with the optional upstream integrations). Adding a new one:
246246

247247
1. **Define the JSON schema** in the tool's module-level decorator.
248248
2. **Implement the handler** following the `BaseTool` protocol.
249249
3. **Add to the tool registry** at the canonical registration site.
250250
4. **Document in [`docs/mcp-tools.md`](docs/mcp-tools.md)** with the tool's
251251
purpose, tier and target latency, and update the standalone tool count
252-
there — `tests_py/test_main.py::test_standalone_baseline_is_54_tools` pins
252+
there — `tests_py/test_main.py::test_standalone_baseline_is_57_tools` pins
253253
it, and `scripts/check_doc_claims.py` fails the build if the docs disagree.
254254
5. **Add a unit test** for the tool's contract.
255255
6. **Add an integration test** if the tool touches the database.

‎README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ default, or PostgreSQL + pgvector if you prefer. No LLM in the retrieval loop, a
2828
leaves localhost unless you configure an integration that does. Your project's memory is a
2929
file you own and can delete.
3030

31-
**Cross-platform is how it is built.** One stdio MCP server and the same 54 tools on Claude
31+
**Cross-platform is how it is built.** One stdio MCP server and the same 57 tools on Claude
3232
Code, in the Claude Desktop bundle, under Claude Cowork, and on every local stdio MCP host
3333
listed in the table below. What differs per host is stated there, not discovered after
3434
install.
@@ -40,7 +40,7 @@ files. We hold that intent to the
4040
publish **no CO₂ or energy figure**, because we have not measured one.
4141
[What we do and do not claim ↓](#green-software-engineering)
4242

43-
> **36 neuroscience mechanisms · 54 memory tools · 9 lifecycle hooks · a self-curating per-project wiki — all local, all open-source, MIT.**
43+
> **36 neuroscience mechanisms · 57 memory tools · 9 lifecycle hooks · a self-curating per-project wiki — all local, all open-source, MIT.**
4444
4545
## Install
4646

@@ -177,7 +177,7 @@ TTL-based caches; some details may fade while the principle remains useful.
177177

178178
In Claude Code that is automatic: nine lifecycle hooks inject context at session start, recall
179179
per prompt, capture as you work, checkpoint before compaction, and run a per-project wiki that
180-
curates itself. In any other stdio MCP host you call the same 54 tools yourself, or 57 when
180+
curates itself. In any other stdio MCP host you call the same 57 tools yourself, or 60 when
181181
the optional `ai-architect-mcp-codebase` and `ai-architect-mcp-spec` integrations are present.
182182

183183
## Does the retrieval work
@@ -234,7 +234,7 @@ bash <plugin-dir>/scripts/install-plugin.sh --postgres
234234
| | SQLite (default) | PostgreSQL 15+ |
235235
|---|---|---|
236236
| Setup | none | pgvector, pg_trgm |
237-
| All 54 tools | yes | yes |
237+
| All 57 tools | yes | yes |
238238
| Retrieval contract | identical | identical |
239239
| Fusion | in-process | server-side PL/pgSQL |
240240
| ANN index | none | pgvector HNSW |
@@ -251,7 +251,7 @@ are Claude Code plugin machinery; the server never imports or requires them at s
251251

252252
| Capability | Claude Code plugin | Local stdio hosts (Gemini CLI, Codex CLI, ChatGPT desktop, Cursor, Windsurf, VS Code, Agents SDK) | ChatGPT web |
253253
|---|---|---|---|
254-
| All 54 memory tools (`remember`, `recall`, wiki, navigation, consolidation, triggers, rules) | ✅ | ✅ | ❌ no remote HTTPS endpoint is shipped |
254+
| All 57 memory tools (`remember`, `recall`, wiki, navigation, consolidation, triggers, rules) | ✅ | ✅ | ❌ no remote HTTPS endpoint is shipped |
255255
| SQLite default store / PostgreSQL opt-in | ✅ | ✅ | ❌ would need a remote deployment and a per-user storage and auth model |
256256
| Auto-capture of significant tool output | ✅ PostToolUse hook | ❌ store explicitly with `remember` | ❌ |
257257
| Session-start context injection | ✅ SessionStart hook | ❌ call `recall` yourself | ❌ |

‎docs/ROADMAP.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ deliberately absent; the ordering is the promise.
99

1010
## Where the project is today
1111

12-
v4.17.1, released 2026-08-02. 54 standalone MCP tools (57 with the optional
12+
v4.17.1, released 2026-08-02. 57 standalone MCP tools (60 with the optional
1313
upstream integrations), 9 lifecycle hooks, 36 neuroscience-grounded mechanisms
1414
against a 97-reference bibliography, running on a local SQLite store by
1515
default or PostgreSQL + pgvector when configured. The OpenSSF Best Practices
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
<!-- Generated by Cortex wiki_reindex from wiki/adr/cortex/1076-a-prediction-is-written-open-and-resolved-against-evidence-the-caller-supplies.md; ADR-1076. Read-only mirror: edit the canonical wiki page. -->
2+
3+
---
4+
created: 2026-09-16T17:39:47Z
5+
kind: adr
6+
number: 1076
7+
status: accepted
8+
tags: [predictions, calibration, brier, issue-597]
9+
title: A prediction is written open and resolved against evidence the caller supplies
10+
---
11+
# ADR-1076: A prediction is written open and resolved against evidence the caller supplies
12+
13+
## Status
14+
15+
accepted
16+
17+
## Context
18+
19+
Cortex stored what happened and never what was expected, so nothing it held could turn out wrong in a way it noticed. Two measurements from 2026-09-16 made the gap concrete: mining over the 374 transcripts under `~/.claude/projects` returned 796 procedural skills, every one at proficiency 0.5 because no session records an outcome (0 of 413 entries carry a score); and while `wiki.claim_events` carries a confidence and `ingest_findings` separates a verified finding from a hypothesis, nothing records what a claim predicts, and nothing later compares it to what was observed (issue #597).
20+
21+
An earlier proposal was to graft prediction records onto `ingest_findings`. Reading it settled the matter: that handler consumes another producer's run artifacts off disk in that producer's format (ADR-0410), so it cannot receive a prediction born in a Cortex session.
22+
23+
The owner decided two things on 2026-09-16. The outcome signal is the posted review verdict, and it must work in any repository where Cortex runs, not only in this one. The records live in a surface of their own rather than in tagged memories.
24+
25+
The first decision rules out Cortex fetching anything. This server has no network path to a forge, knows nothing about GitHub, and must not learn this project's own `ZETETIC-REVIEW:` convention, which its merge gate enforces but which no other repository shares.
26+
27+
## Decision
28+
29+
A `predictions` table, on both backends, holds the claim, the prediction in falsifiable terms, the test that would settle it, and the confidence its author held when writing it, plus the domain, directory and optional memory it belongs to. A row is written `open` and moves once to `resolved`, carrying what was observed, a verdict of `confirmed`, `refuted` or `abandoned`, the kind of source that settled it (`review`, `ci`, `test`, `manual`) and a reference to that source. A table CHECK refuses a resolved row without a verdict, a source kind and a reference, and the handler refuses a blank reference before the database has to.
30+
31+
The caller supplies the evidence and Cortex never fetches it. That is what makes the contract repository-agnostic: a host maps its own review convention onto the same four fields, and nothing in the schema names a forge or a marker.
32+
33+
`core/calibration.py` scores the resolved rows: the Brier score (Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns so the number reads without the literature, plus the reliability breakdown per confidence band, which says how often predictions in that band actually held. An abandoned prediction is counted apart and scored nowhere: its test was never run, so it says nothing about calibration.
34+
35+
Three tools, registered from `tool_registry_predictions.py`: `predict`, `resolve_prediction`, `calibration`. The standalone count moves from 54 to 57, 60 with both upstream integrations.
36+
37+
## Consequences
38+
39+
Easier: a confidence can now be scored rather than asserted. `calibration` answers with a Brier score, its reference point and the per-band frequencies, for one domain or for all.
40+
41+
`tests_py/core/test_calibration.py` pins the scoring: a constant 0.5 scoring exactly the uninformative reference, certainty that holds scoring 0, confidence that is wrong scoring worse than 0.5, the band a full confidence falls in, and an empty input scoring absent rather than zero. `tests_py/handlers/test_prediction_records.py` drives the three tools on SQLite: a prediction written open, refusals for a missing part and for a confidence outside 0 to 1, a resolution refused for a blank reference, a prediction resolving once, an unknown id, the score over two resolved rows with one still open, and the three tools present with their annotations.
42+
43+
Harder: nothing yet writes predictions on its own. A session that wants calibration has to state a prediction before running its test, which is the discipline the surface exists to support, not something it can impose.
44+
45+
Unchanged: procedural proficiency. Feeding a session's own verdict into `mine_skills` reuses this resolution contract and is the next step, not this one.

‎docs/agent-guidance.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ separate **cortex-viz** MCP (reads this same store read-only).
8989
- docs/adr/ — Architecture Decision Records (013 = thermodynamic memory
9090
model, 014 = biological mechanisms, 012 = Python migration from Node.js)
9191
- docs/module-inventory.md — per-layer module catalogue + dependency rules
92-
- docs/mcp-tools.md — the 54 standalone tools + 3 conditionally-registered
92+
- docs/mcp-tools.md — the 57 standalone tools + 3 conditionally-registered
9393
MCP tools, by tier, with purpose and target latency
9494
- PRIVACY.md — storage truth by launch surface (lines 26–38): SQLite is
9595
the default for plugin installs and `.mcpb`/Cowork; PostgreSQL is the

‎docs/api-reference.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# API Reference — MCP Tool Schemas
22

3-
This document describes the input schemas, response formats, and error codes for the tools detailed below. It is a schema reference, not the catalogue: the complete surface is 54 standalone tools (57 with the optional upstream integrations), listed with tier and target latency in [`mcp-tools.md`](mcp-tools.md).
3+
This document describes the input schemas, response formats, and error codes for the tools detailed below. It is a schema reference, not the catalogue: the complete surface is 57 standalone tools (60 with the optional upstream integrations), listed with tier and target latency in [`mcp-tools.md`](mcp-tools.md).
44

55
## Protocol
66

‎docs/mcp-tools.md‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,18 @@ description: "MCP tool catalogue (tiers, purpose, target latency) + slash comman
44

55
# MCP Tools
66

7-
54 standalone tools register unconditionally; 3 more register only when an
8-
upstream MCP server is configured (57 total with both present).
7+
57 standalone tools register unconditionally; 3 more register only when an
8+
upstream MCP server is configured (60 total with both present).
99

1010
```
11-
# source: tests_py/test_main.py::test_standalone_baseline_is_54_tools
11+
# source: tests_py/test_main.py::test_standalone_baseline_is_57_tools
1212
# verified 2026-07-12 by a live DB-less `tools/list` stdio round-trip
1313
# against `bare-container-contract` + `wiki_migrate`, commit 4be298a3;
1414
# bumped to 51 by `check_setup` (issue #115), to 52 by `ingest_document`
1515
# (offline .docx / Confluence export ingest, issue #192), to 54 by
16-
# `wiki_get_draft` + `wiki_refine_draft` (Path B, ADR-1066, issue #579).
16+
# `wiki_get_draft` + `wiki_refine_draft` (Path B, ADR-1066, issue #579),
17+
# to 57 by `predict` + `resolve_prediction` + `calibration` (prediction
18+
# records and their Brier score, ADR-1076, issue #597).
1719
```
1820

1921
## Tier 1 — Core Memory & Profiling (22 tools)
@@ -70,7 +72,7 @@ Omitting `project_root` retains the global wiki scope.
7072
| `recall_skills` | Recall learned procedural skills by situation | <200ms |
7173
| `why` | Resolve ⟦rcpt:id⟧ injection receipts into presence-in-context evidence (blame path, decision 4255039) | <100ms |
7274

73-
## Tier 3 — Automation & Intelligence (10 tools)
75+
## Tier 3 — Automation & Intelligence (13 tools)
7476

7577
| Tool | Purpose | Target Latency |
7678
|---|---|---|
@@ -84,6 +86,9 @@ Omitting `project_root` retains the global wiki scope.
8486
| `curate_wiki` | Auto-curate wiki pages from memory clusters | varies |
8587
| `curate_distill` | Return understanding-level distillation dossiers (error->success, co-access, entity family) for the LLM to author `lesson` memories from (M-D8) | ~200-500ms |
8688
| `ingest_document` | Ingest a .docx or Confluence storage-format XHTML export into the memory/wiki store, with provenance + idempotent re-ingest (issue #192). File-based, no upstream needed | varies |
89+
| `predict` | Record a falsifiable prediction with the confidence held before its outcome is known | <50ms |
90+
| `resolve_prediction` | Settle an open prediction against an observation, naming the evidence that decided it | <50ms |
91+
| `calibration` | Brier score and reliability bands over the resolved predictions | <100ms |
8792

8893
## Tier 4 — Wiki (12 tools)
8994

@@ -105,9 +110,9 @@ Omitting `project_root` retains the global wiki scope.
105110
## Upstream-integration tools (3, conditionally registered)
106111

107112
These register only when their upstream MCP server is configured, bringing
108-
the total to 57: `ingest_codebase` + `change_impact` (ai-architect-mcp-codebase)
113+
the total to 60: `ingest_codebase` + `change_impact` (ai-architect-mcp-codebase)
109114
and `ingest_prd` (ai-architect-mcp-spec). With no upstream present, exactly the
110-
**54 standalone tools** above register. Driving the ai-architect pipeline
115+
**57 standalone tools** above register. Driving the ai-architect pipeline
111116
end-to-end (formerly `run_pipeline`) is **not** part of this server — it
112117
lives in the ai-architect-mcp-codebase MCP.
113118

‎docs/module-inventory.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -280,8 +280,8 @@ was the drift #114 was filed against).
280280

281281
## handlers/ — Composition roots
282282

283-
54 standalone tools + 3 upstream-integration tools conditionally registered
284-
(57 total) + a `handlers/consolidation/` subpackage + per-tool helpers. See
283+
57 standalone tools + 3 upstream-integration tools conditionally registered
284+
(60 total) + a `handlers/consolidation/` subpackage + per-tool helpers. See
285285
`docs/mcp-tools.md` for the full tool catalogue with purpose and target
286286
latency.
287287

‎manifest.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
"agents"
3838
],
3939
"license": "MIT",
40-
"long_description": "Cortex gives Codex, Gemini CLI, Claude Code, and other local MCP hosts a durable, scientifically-grounded memory. It stores decisions, lessons, and context across sessions and retrieves them with a hybrid WRRF fusion (vector + full-text + trigram + heat + recency) reranked by a cross-encoder. Its 36 neuroscience-grounded mechanisms are drawn from a 97-reference bibliography (spreading activation, synaptic tagging, neuromodulation, LTP/LTD, predictive-coding write gates). This connector exposes 54 memory tools standalone (57 with upstream integrations available) and runs fully local with zero external services using the built-in SQLite backend; advanced users can point it at PostgreSQL + pgvector. For automatic session-lifecycle memory (injection at session start, auto-capture, compaction checkpointing), install the companion Claude Code plugin; other hosts use the same memory through explicit MCP tool calls.",
40+
"long_description": "Cortex gives Codex, Gemini CLI, Claude Code, and other local MCP hosts a durable, scientifically-grounded memory. It stores decisions, lessons, and context across sessions and retrieves them with a hybrid WRRF fusion (vector + full-text + trigram + heat + recency) reranked by a cross-encoder. Its 36 neuroscience-grounded mechanisms are drawn from a 97-reference bibliography (spreading activation, synaptic tagging, neuromodulation, LTP/LTD, predictive-coding write gates). This connector exposes 57 memory tools standalone (60 with upstream integrations available) and runs fully local with zero external services using the built-in SQLite backend; advanced users can point it at PostgreSQL + pgvector. For automatic session-lifecycle memory (injection at session start, auto-capture, compaction checkpointing), install the companion Claude Code plugin; other hosts use the same memory through explicit MCP tool calls.",
4141
"manifest_version": "0.4",
4242
"name": "hypermnesia-mcp",
4343
"privacy_policies": [

0 commit comments

Comments
 (0)