You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 581b29b
Browse filesBrowse the repository at this point in the historyBrowse files
feat(predictions): record what is expected, resolve it, score the confidence (#597)
Cortex stored what happened and never what was expected, so nothing it
held could turn out wrong in a way it noticed: the 796 procedural skills
mined from the maintainer's transcripts all sit at proficiency 0.5
because no session records an outcome, and while claim_events carry a
confidence, nothing compares it to an observation.
A predictions table on both backends holds the claim, the prediction in
falsifiable terms, the test that would settle it and the confidence its
author held. A row is written open and moves once to resolved, carrying
what was observed, a verdict, the kind of source that decided it and a
reference to that source; a table CHECK and the handler both refuse a
resolution without one.
Cortex never fetches that evidence and knows nothing about any forge or
review convention, including this project's own marker. That is what
makes the contract work in any repository: a host maps its convention
onto the same four fields (ADR-1076).
core/calibration.py scores the resolved rows with the Brier score
(Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns, and
the per-band reliability the mean cannot show. An abandoned prediction is
counted apart and scored nowhere.
Three tools registered: predict, resolve_prediction, calibration.
Standalone count 54 to 57, 60 with both upstream integrations, with the
pinned test, the catalogue, the eight documents the claim gate scans and
the docker-smoke floor moved with it.
Full suite: 8763 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
<!-- Generated by Cortex wiki_reindex from wiki/adr/cortex/1076-a-prediction-is-written-open-and-resolved-against-evidence-the-caller-supplies.md; ADR-1076. Read-only mirror: edit the canonical wiki page. -->
title: A prediction is written open and resolved against evidence the caller supplies
10
+
---
11
+
# ADR-1076: A prediction is written open and resolved against evidence the caller supplies
12
+
13
+
## Status
14
+
15
+
accepted
16
+
17
+
## Context
18
+
19
+
Cortex stored what happened and never what was expected, so nothing it held could turn out wrong in a way it noticed. Two measurements from 2026-09-16 made the gap concrete: mining over the 374 transcripts under `~/.claude/projects` returned 796 procedural skills, every one at proficiency 0.5 because no session records an outcome (0 of 413 entries carry a score); and while `wiki.claim_events` carries a confidence and `ingest_findings` separates a verified finding from a hypothesis, nothing records what a claim predicts, and nothing later compares it to what was observed (issue #597).
20
+
21
+
An earlier proposal was to graft prediction records onto `ingest_findings`. Reading it settled the matter: that handler consumes another producer's run artifacts off disk in that producer's format (ADR-0410), so it cannot receive a prediction born in a Cortex session.
22
+
23
+
The owner decided two things on 2026-09-16. The outcome signal is the posted review verdict, and it must work in any repository where Cortex runs, not only in this one. The records live in a surface of their own rather than in tagged memories.
24
+
25
+
The first decision rules out Cortex fetching anything. This server has no network path to a forge, knows nothing about GitHub, and must not learn this project's own `ZETETIC-REVIEW:` convention, which its merge gate enforces but which no other repository shares.
26
+
27
+
## Decision
28
+
29
+
A `predictions` table, on both backends, holds the claim, the prediction in falsifiable terms, the test that would settle it, and the confidence its author held when writing it, plus the domain, directory and optional memory it belongs to. A row is written `open` and moves once to `resolved`, carrying what was observed, a verdict of `confirmed`, `refuted` or `abandoned`, the kind of source that settled it (`review`, `ci`, `test`, `manual`) and a reference to that source. A table CHECK refuses a resolved row without a verdict, a source kind and a reference, and the handler refuses a blank reference before the database has to.
30
+
31
+
The caller supplies the evidence and Cortex never fetches it. That is what makes the contract repository-agnostic: a host maps its own review convention onto the same four fields, and nothing in the schema names a forge or a marker.
32
+
33
+
`core/calibration.py` scores the resolved rows: the Brier score (Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns so the number reads without the literature, plus the reliability breakdown per confidence band, which says how often predictions in that band actually held. An abandoned prediction is counted apart and scored nowhere: its test was never run, so it says nothing about calibration.
34
+
35
+
Three tools, registered from `tool_registry_predictions.py`: `predict`, `resolve_prediction`, `calibration`. The standalone count moves from 54 to 57, 60 with both upstream integrations.
36
+
37
+
## Consequences
38
+
39
+
Easier: a confidence can now be scored rather than asserted. `calibration` answers with a Brier score, its reference point and the per-band frequencies, for one domain or for all.
40
+
41
+
`tests_py/core/test_calibration.py` pins the scoring: a constant 0.5 scoring exactly the uninformative reference, certainty that holds scoring 0, confidence that is wrong scoring worse than 0.5, the band a full confidence falls in, and an empty input scoring absent rather than zero. `tests_py/handlers/test_prediction_records.py` drives the three tools on SQLite: a prediction written open, refusals for a missing part and for a confidence outside 0 to 1, a resolution refused for a blank reference, a prediction resolving once, an unknown id, the score over two resolved rows with one still open, and the three tools present with their annotations.
42
+
43
+
Harder: nothing yet writes predictions on its own. A session that wants calibration has to state a prediction before running its test, which is the discipline the surface exists to support, not something it can impose.
44
+
45
+
Unchanged: procedural proficiency. Feeding a session's own verdict into `mine_skills` reuses this resolution contract and is the next step, not this one.
Copy file name to clipboardExpand all lines: docs/api-reference.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# API Reference — MCP Tool Schemas
2
2
3
-
This document describes the input schemas, response formats, and error codes for the tools detailed below. It is a schema reference, not the catalogue: the complete surface is 54 standalone tools (57 with the optional upstream integrations), listed with tier and target latency in [`mcp-tools.md`](mcp-tools.md).
3
+
This document describes the input schemas, response formats, and error codes for the tools detailed below. It is a schema reference, not the catalogue: the complete surface is 57 standalone tools (60 with the optional upstream integrations), listed with tier and target latency in [`mcp-tools.md`](mcp-tools.md).
@@ -84,6 +86,9 @@ Omitting `project_root` retains the global wiki scope.
84
86
|`curate_wiki`| Auto-curate wiki pages from memory clusters | varies |
85
87
|`curate_distill`| Return understanding-level distillation dossiers (error->success, co-access, entity family) for the LLM to author `lesson` memories from (M-D8) |~200-500ms |
86
88
|`ingest_document`| Ingest a .docx or Confluence storage-format XHTML export into the memory/wiki store, with provenance + idempotent re-ingest (issue #192). File-based, no upstream needed | varies |
89
+
|`predict`| Record a falsifiable prediction with the confidence held before its outcome is known | <50ms |
90
+
|`resolve_prediction`| Settle an open prediction against an observation, naming the evidence that decided it | <50ms |
91
+
|`calibration`| Brier score and reliability bands over the resolved predictions | <100ms |
87
92
88
93
## Tier 4 — Wiki (12 tools)
89
94
@@ -105,9 +110,9 @@ Omitting `project_root` retains the global wiki scope.
Copy file name to clipboardExpand all lines: manifest.json
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,7 @@
37
37
"agents"
38
38
],
39
39
"license": "MIT",
40
-
"long_description": "Cortex gives Codex, Gemini CLI, Claude Code, and other local MCP hosts a durable, scientifically-grounded memory. It stores decisions, lessons, and context across sessions and retrieves them with a hybrid WRRF fusion (vector + full-text + trigram + heat + recency) reranked by a cross-encoder. Its 36 neuroscience-grounded mechanisms are drawn from a 97-reference bibliography (spreading activation, synaptic tagging, neuromodulation, LTP/LTD, predictive-coding write gates). This connector exposes 54 memory tools standalone (57 with upstream integrations available) and runs fully local with zero external services using the built-in SQLite backend; advanced users can point it at PostgreSQL + pgvector. For automatic session-lifecycle memory (injection at session start, auto-capture, compaction checkpointing), install the companion Claude Code plugin; other hosts use the same memory through explicit MCP tool calls.",
40
+
"long_description": "Cortex gives Codex, Gemini CLI, Claude Code, and other local MCP hosts a durable, scientifically-grounded memory. It stores decisions, lessons, and context across sessions and retrieves them with a hybrid WRRF fusion (vector + full-text + trigram + heat + recency) reranked by a cross-encoder. Its 36 neuroscience-grounded mechanisms are drawn from a 97-reference bibliography (spreading activation, synaptic tagging, neuromodulation, LTP/LTD, predictive-coding write gates). This connector exposes 57 memory tools standalone (60 with upstream integrations available) and runs fully local with zero external services using the built-in SQLite backend; advanced users can point it at PostgreSQL + pgvector. For automatic session-lifecycle memory (injection at session start, auto-capture, compaction checkpointing), install the companion Claude Code plugin; other hosts use the same memory through explicit MCP tool calls.",
0 commit comments