Skip to content

Commit dd29c89

Browse files
cdeustclaude
andauthored
feat(predictions): record what is expected, resolve it, score the confidence (#597) (#599)
* feat(predictions): record what is expected, resolve it, score the confidence (#597) Cortex stored what happened and never what was expected, so nothing it held could turn out wrong in a way it noticed: the 796 procedural skills mined from the maintainer's transcripts all sit at proficiency 0.5 because no session records an outcome, and while claim_events carry a confidence, nothing compares it to an observation. A predictions table on both backends holds the claim, the prediction in falsifiable terms, the test that would settle it and the confidence its author held. A row is written open and moves once to resolved, carrying what was observed, a verdict, the kind of source that decided it and a reference to that source; a table CHECK and the handler both refuse a resolution without one. Cortex never fetches that evidence and knows nothing about any forge or review convention, including this project's own marker. That is what makes the contract work in any repository: a host maps its convention onto the same four fields (ADR-1076). core/calibration.py scores the resolved rows with the Brier score (Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns, and the per-band reliability the mean cannot show. An abandoned prediction is counted apart and scored nowhere. Three tools registered: predict, resolve_prediction, calibration. Standalone count 54 to 57, 60 with both upstream integrations, with the pinned test, the catalogue, the eight documents the claim gate scans and the docker-smoke floor moved with it. Full suite: 8763 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH * fix(predictions): close the five findings of the #599 review Brier convention: the module computes mean((p - o)^2), the single-probability scale scikit-learn and most contemporary writing call the Brier score; Brier's own 1950 paper sums over both categories and reports twice it. The docstring said the paper's formula. It now states the scale it computes and why the ranking is the same, and the ADR and the CHANGELOG say the same. _band_of folded an out-of-range confidence into an edge band and left a dead branch behind it. It now raises ValueError outside [0, 1]: a silent bucket would make the reliability diagram lie about what it measured. The zip is strict, and the final return is reachable for exactly 1.0. The table CHECK demanded a non-null source_ref, not a non-empty one, so only the handler stood between an empty reference and a resolved row. Both DDLs now require source_ref and observed to be non-empty. docs/mcp-tools.md advertised 57 tools while its four tier tables listed 54: ingest_findings, lesson_promotion and get_grooming_health were registered and in no table. That predates this branch (51 against 54 before it). The three rows are added and the arithmetic closes. Four new cases: three out-of-range confidences refused, and the schema refusing an empty reference. Full suite: 8767 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH * fix(ci): derive the host-contract tool bounds from the registries (#597) The verifier carried MIN_FULL_TOOL_COUNT = 52 with a ceiling three above, so it stayed right when the surface moved to 54 (ADR-1066) and only broke when the prediction tools took it to 57: CI failed on "discovered 57 full tools; expected 52..55", taking down Test (SQLite backend) and Release dependency set. mcp_server/tool_surface.py reads the surface from the registry modules themselves and the verifier derives both bounds from it (ADR-1077). Three tests pin the derivation against what register_all actually registers, including that the verifier's bounds follow it. A literal that is right for a while, then wrong, and only caught once it drifts far enough is a second source of truth, not a check. Full suite: 8770 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH * fix(doc-claims): gate the three tool-count copies the review found (#597) The audit asked for the class, not the number, and found three older copies. `.bestpractices.json` cited the pinned test as test_standalone_baseline_is_52_tools, two count moves stale, and the TOOL_CLAIM regex could not see it because the digits sit inside an identifier with no preceding space: a new PINNED_TEST_NAME_CLAIM family checks that form. `.claude-plugin/marketplace.json` advertised 50 tools (53 with upstreams) to every install and was scanned by nothing; it now has a targeted check rather than joining SCANNED_FILES, because the same description field recounts the plugin's version history, where past counts are quotations and the generic scan would fail them. The docker-smoke floor stays a literal, now pinned to the derived surface by a test. Both new checks were driven to failure before being trusted: 55/58 in the marketplace and 52 in the pinned name each produce a named failure. The gate's own fixtures gained the two subjects, since a family that matches nothing fails as vacuous by design. Full suite: 8771 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH * fix(ci): import the tool surface late in the host verifier (#597) The derivation at module scope pulled the MCP SDK through the registries, and the mcp-host-config job installs the three host CLIs with no Python dependencies: "ModuleNotFoundError: No module named 'mcp'" took the job down, and CI Green with it. That job verifies lean profiles only, which never need the bounds. full_tool_bounds() moves to mcp_server/tool_surface.py, where the surface knowledge already lives, and the verifier keeps a thin wrapper importing it inside the function. The wrapper also keeps that script at 292 lines, under the 300-line cap it would otherwise have crossed. A new test imports the script with `mcp` blocked in sys.meta_path and asserts it still imports; I reproduced the failing condition that way before fixing it. ADR-1077 says why the late import is load-bearing. Full suite: 8772 passed, 8 skipped. The verifier passes its six cases. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH * test(tool-surface): make the SDK-free import test able to fail (#597) The probe installed a meta-path finder written to the pre-PEP 451 find_module/load_module protocol. CPython ignores a finder without find_spec, so the blocker blocked nothing: the test passed against c710439, the very commit whose module-scope import broke the mcp-host-config job. A test that cannot fail is not a pin. The probe now implements find_spec and asserts the block works, by importing mcp first and requiring ImportError before it touches the script at all. Its source sits in a module constant, which also keeps the test method under the craftsmanship gate's 40-line cap. Measured both directions with the corrected test: against c710439 it fails with the CI symptom, "from mcp.server.mcpserver import MCPServer" raising ImportError through the blocker; against this head it passes. The review of PR #599 found this; the vacuity is mine. Full suite: 8772 passed, 8 skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent e565d12 commit dd29c89

39 files changed

Lines changed: 1608 additions & 42 deletions

‎.bestpractices.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,7 @@
231231
"documentation_quick_start_justification": "https://github.com/cdeust/Cortex#getting-started \u2014 download the .mcpb bundle and open it in Claude Desktop, or install the Claude Code plugin; it runs immediately on the built-in SQLite backend with no database to provision. The Examples section then shows a full store/recall round trip.",
232232

233233
"documentation_current_status": "Met",
234-
"documentation_current_justification": "Documentation defects are treated as defects and fixed in the same PR, and the advertised numbers are now machine-checked rather than hand-maintained: https://github.com/cdeust/Cortex/blob/main/scripts/check_doc_claims.py fails the build when the tool count, reference count, mechanism count, version or test count in README, CONTRIBUTING, CLAUDE.md, GOVERNANCE.md, the MCPB manifest or the docs tree disagrees with the repository (the tool count is itself pinned to the live registry by tests_py/test_main.py::test_standalone_baseline_is_52_tools). It runs in the Lint job on every push and pull request, and with the live collected test count in the test job. Introducing it surfaced and fixed eight stale claims in this same PR, including a `mypy --strict` gate the project has never run and two MCP tools that moved to cortex-viz in v3.21.0 but were still documented here.",
234+
"documentation_current_justification": "Documentation defects are treated as defects and fixed in the same PR, and the advertised numbers are now machine-checked rather than hand-maintained: https://github.com/cdeust/Cortex/blob/main/scripts/check_doc_claims.py fails the build when the tool count, reference count, mechanism count, version or test count in README, CONTRIBUTING, CLAUDE.md, GOVERNANCE.md, the MCPB manifest or the docs tree disagrees with the repository (the tool count is itself pinned to the live registry by tests_py/test_main.py::test_standalone_baseline_is_57_tools). It runs in the Lint job on every push and pull request, and with the live collected test count in the test job. Introducing it surfaced and fixed eight stale claims in this same PR, including a `mypy --strict` gate the project has never run and two MCP tools that moved to cortex-viz in v3.21.0 but were still documented here.",
235235

236236
"documentation_achievements_status": "Met",
237237
"documentation_achievements_justification": "The OpenSSF Best Practices badge is displayed in the README badge row, hyperlinked to the project page, within hours of the passing level being awarded on 2026-07-27: https://github.com/cdeust/Cortex#readme (badge image https://www.bestpractices.dev/projects/13836/badge).",

‎.claude-plugin/marketplace.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
{
1313
"name": "hypermnesia-mcp",
1414
"source": "./",
15-
"description": "Cortex — persistent memory and cognitive profiling for Claude Code (renamed from the 'cortex' plugin in v4.15.0 to match the PyPI/MCP-registry identity hypermnesia-mcp) — thermodynamic memory with heat/decay, intent-aware retrieval, biological plasticity, codebase intelligence, and cognitive profiling. 50 MCP tools (53 with the optional ai-architect-mcp-codebase + ai-architect-mcp-spec integrations) with enriched schemas (visualization extracted to the standalone Hypermnesia MCP Viz server). PostgreSQL + pgvector in CLI mode; automatic SQLite fallback in Cowork/sandboxed mode. v3.17.0 — autonomous per-project wiki: SessionStart auto-spawns a 6-hour consolidate cycle; a headless `claude -p` worker drains the curation-gap queue, calls codebase-intelligence MCP tools to ground each section in the real call graph, and authors missing anchor pages (architecture / services / api / data-flow / operations / decisions / PRD) per project from the source tree. 15 canonical scopes × 13 file sections; per-project dashboards under `wiki/_dashboards/`. Mermaid diagrams have a 🔍 lens with zoom + pan. Workflow graph with caller-qualified CALLS chains rendering full method-to-method dependencies (native tree-sitter, no AP required). Side panel humanized for non-technical users. Ingests codebase analysis (ai-architect-mcp-codebase) and PRDs (ai-architect-mcp-spec) into wiki + memory + knowledge graph. Docker image available.",
15+
"description": "Cortex — persistent memory and cognitive profiling for Claude Code (renamed from the 'cortex' plugin in v4.15.0 to match the PyPI/MCP-registry identity hypermnesia-mcp) — thermodynamic memory with heat/decay, intent-aware retrieval, biological plasticity, codebase intelligence, and cognitive profiling. 57 MCP tools (60 with the optional ai-architect-mcp-codebase + ai-architect-mcp-spec integrations) with enriched schemas (visualization extracted to the standalone Hypermnesia MCP Viz server). PostgreSQL + pgvector in CLI mode; automatic SQLite fallback in Cowork/sandboxed mode. v3.17.0 — autonomous per-project wiki: SessionStart auto-spawns a 6-hour consolidate cycle; a headless `claude -p` worker drains the curation-gap queue, calls codebase-intelligence MCP tools to ground each section in the real call graph, and authors missing anchor pages (architecture / services / api / data-flow / operations / decisions / PRD) per project from the source tree. 15 canonical scopes × 13 file sections; per-project dashboards under `wiki/_dashboards/`. Mermaid diagrams have a 🔍 lens with zoom + pan. Workflow graph with caller-qualified CALLS chains rendering full method-to-method dependencies (native tree-sitter, no AP required). Side panel humanized for non-technical users. Ingests codebase analysis (ai-architect-mcp-codebase) and PRDs (ai-architect-mcp-spec) into wiki + memory + knowledge graph. Docker image available.",
1616
"version": "4.22.0",
1717
"author": {
1818
"name": "Clement Deust",

‎CHANGELOG.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,41 @@ adheres to [Semantic Versioning](https://semver.org/).
88

99
### Added
1010

11+
- **The host-contract verifier counts the tools the server registers
12+
(ADR-1077).** `scripts/verify_mcp_hosts.py` held its own floor of 52 with a
13+
ceiling three above, so it stayed right through the move to 54 (ADR-1066)
14+
and only failed when the prediction tools took the count to 57, blocking two
15+
CI jobs on a number nothing had updated. `mcp_server/tool_surface.py` now
16+
reads the surface from the registry modules and the verifier derives its
17+
bounds from it, with a test pinning both against what `register_all`
18+
actually registers. The review of that fix found three more copies, all
19+
older than it: `.bestpractices.json` named the pinned test at 52 inside an
20+
identifier, where the claim regex could not see it; `.claude-plugin/
21+
marketplace.json` advertised 50 tools to every install and was scanned by
22+
nothing; and the docker-smoke floor was a bare literal. The first two are
23+
corrected and now gate-checked, the marketplace one by a targeted check
24+
because that same field also recounts the plugin's version history, and the
25+
third is pinned by a test.
26+
27+
- **Prediction records, and a score for the confidences they carry (#597).**
28+
Cortex stored what happened and never what was expected, so nothing it held
29+
could turn out wrong in a way it noticed: 796 procedural skills mined from
30+
the maintainer's transcripts all sit at proficiency 0.5 because no session
31+
records an outcome. Three tools now close that loop. `predict` writes a
32+
falsifiable prediction with the confidence held before the outcome is known;
33+
`resolve_prediction` settles it against an observation, naming the verdict,
34+
the kind of source that decided it and a reference to that source; and
35+
`calibration` returns the mean squared distance between confidence and
36+
outcome, the scale most contemporary writing calls the Brier score (Brier's
37+
1950 paper reports twice it), with the 0.25 a constant 0.5 forecast earns
38+
and the per-band frequencies the mean cannot show. Cortex never
39+
fetches the evidence and knows nothing about any forge or review
40+
convention, which is what makes the contract work in any repository
41+
(ADR-1076). The table lives on both backends. Standalone tool count 54 to
42+
57, 60 with both upstream integrations.
43+
44+
### Added
45+
1146
- **The Path B draft tools are exposed over MCP (#579).** `wiki_get_draft`
1247
and `wiki_refine_draft` had existed as handlers since ADR-0467 without ever
1348
being registered, so no client could call them and `wiki_get_draft`'s own

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ the canonical list) backing the algorithms. Every change is held to that bar.
99

1010
## What this project is
1111

12-
A Python 3.10+ MCP server with **54 standalone tools** (57 with the optional
12+
A Python 3.10+ MCP server with **57 standalone tools** (60 with the optional
1313
ai-architect-mcp-codebase + prd-spec-generator integrations) and **9 automatic
1414
hooks**, persisting to a local SQLite store by default or to PostgreSQL +
1515
pgvector when configured. Implements rate-distortion forgetting,
@@ -242,14 +242,14 @@ mechanically for the modules it is scoped to.
242242

243243
## Adding an MCP tool
244244

245-
54 standalone tools currently (57 with the optional upstream integrations). Adding a new one:
245+
57 standalone tools currently (60 with the optional upstream integrations). Adding a new one:
246246

247247
1. **Define the JSON schema** in the tool's module-level decorator.
248248
2. **Implement the handler** following the `BaseTool` protocol.
249249
3. **Add to the tool registry** at the canonical registration site.
250250
4. **Document in [`docs/mcp-tools.md`](docs/mcp-tools.md)** with the tool's
251251
purpose, tier and target latency, and update the standalone tool count
252-
there — `tests_py/test_main.py::test_standalone_baseline_is_54_tools` pins
252+
there — `tests_py/test_main.py::test_standalone_baseline_is_57_tools` pins
253253
it, and `scripts/check_doc_claims.py` fails the build if the docs disagree.
254254
5. **Add a unit test** for the tool's contract.
255255
6. **Add an integration test** if the tool touches the database.

‎README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ default, or PostgreSQL + pgvector if you prefer. No LLM in the retrieval loop, a
2828
leaves localhost unless you configure an integration that does. Your project's memory is a
2929
file you own and can delete.
3030

31-
**Cross-platform is how it is built.** One stdio MCP server and the same 54 tools on Claude
31+
**Cross-platform is how it is built.** One stdio MCP server and the same 57 tools on Claude
3232
Code, in the Claude Desktop bundle, under Claude Cowork, and on every local stdio MCP host
3333
listed in the table below. What differs per host is stated there, not discovered after
3434
install.
@@ -40,7 +40,7 @@ files. We hold that intent to the
4040
publish **no CO₂ or energy figure**, because we have not measured one.
4141
[What we do and do not claim ↓](#green-software-engineering)
4242

43-
> **36 neuroscience mechanisms · 54 memory tools · 9 lifecycle hooks · a self-curating per-project wiki — all local, all open-source, MIT.**
43+
> **36 neuroscience mechanisms · 57 memory tools · 9 lifecycle hooks · a self-curating per-project wiki — all local, all open-source, MIT.**
4444

4545
## Install
4646

@@ -177,7 +177,7 @@ TTL-based caches; some details may fade while the principle remains useful.
177177

178178
In Claude Code that is automatic: nine lifecycle hooks inject context at session start, recall
179179
per prompt, capture as you work, checkpoint before compaction, and run a per-project wiki that
180-
curates itself. In any other stdio MCP host you call the same 54 tools yourself, or 57 when
180+
curates itself. In any other stdio MCP host you call the same 57 tools yourself, or 60 when
181181
the optional `ai-architect-mcp-codebase` and `ai-architect-mcp-spec` integrations are present.
182182

183183
## Does the retrieval work
@@ -234,7 +234,7 @@ bash <plugin-dir>/scripts/install-plugin.sh --postgres
234234
| | SQLite (default) | PostgreSQL 15+ |
235235
|---|---|---|
236236
| Setup | none | pgvector, pg_trgm |
237-
| All 54 tools | yes | yes |
237+
| All 57 tools | yes | yes |
238238
| Retrieval contract | identical | identical |
239239
| Fusion | in-process | server-side PL/pgSQL |
240240
| ANN index | none | pgvector HNSW |
@@ -251,7 +251,7 @@ are Claude Code plugin machinery; the server never imports or requires them at s
251251

252252
| Capability | Claude Code plugin | Local stdio hosts (Gemini CLI, Codex CLI, ChatGPT desktop, Cursor, Windsurf, VS Code, Agents SDK) | ChatGPT web |
253253
|---|---|---|---|
254-
| All 54 memory tools (`remember`, `recall`, wiki, navigation, consolidation, triggers, rules) | ✅ | ✅ | ❌ no remote HTTPS endpoint is shipped |
254+
| All 57 memory tools (`remember`, `recall`, wiki, navigation, consolidation, triggers, rules) | ✅ | ✅ | ❌ no remote HTTPS endpoint is shipped |
255255
| SQLite default store / PostgreSQL opt-in | ✅ | ✅ | ❌ would need a remote deployment and a per-user storage and auth model |
256256
| Auto-capture of significant tool output | ✅ PostToolUse hook | ❌ store explicitly with `remember` | ❌ |
257257
| Session-start context injection | ✅ SessionStart hook | ❌ call `recall` yourself | ❌ |

‎docs/ROADMAP.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ deliberately absent; the ordering is the promise.
99

1010
## Where the project is today
1111

12-
v4.17.1, released 2026-08-02. 54 standalone MCP tools (57 with the optional
12+
v4.17.1, released 2026-08-02. 57 standalone MCP tools (60 with the optional
1313
upstream integrations), 9 lifecycle hooks, 36 neuroscience-grounded mechanisms
1414
against a 97-reference bibliography, running on a local SQLite store by
1515
default or PostgreSQL + pgvector when configured. The OpenSSF Best Practices
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
<!-- Generated by Cortex wiki_reindex from wiki/adr/cortex/1076-a-prediction-is-written-open-and-resolved-against-evidence-the-caller-supplies.md; ADR-1076. Read-only mirror: edit the canonical wiki page. -->
2+
3+
---
4+
created: 2026-09-16T17:39:47Z
5+
kind: adr
6+
number: 1076
7+
status: accepted
8+
tags: [predictions, calibration, brier, issue-597]
9+
title: A prediction is written open and resolved against evidence the caller supplies
10+
---
11+
# ADR-1076: A prediction is written open and resolved against evidence the caller supplies
12+
13+
## Status
14+
15+
accepted
16+
17+
## Context
18+
19+
Cortex stored what happened and never what was expected, so nothing it held could turn out wrong in a way it noticed. Two measurements from 2026-09-16 made the gap concrete: mining over the 374 transcripts under `~/.claude/projects` returned 796 procedural skills, every one at proficiency 0.5 because no session records an outcome (0 of 413 entries carry a score); and while `wiki.claim_events` carries a confidence and `ingest_findings` separates a verified finding from a hypothesis, nothing records what a claim predicts, and nothing later compares it to what was observed (issue #597).
20+
21+
An earlier proposal was to graft prediction records onto `ingest_findings`. Reading it settled the matter: that handler consumes another producer's run artifacts off disk in that producer's format (ADR-0410), so it cannot receive a prediction born in a Cortex session.
22+
23+
The owner decided two things on 2026-09-16. The outcome signal is the posted review verdict, and it must work in any repository where Cortex runs, not only in this one. The records live in a surface of their own rather than in tagged memories.
24+
25+
The first decision rules out Cortex fetching anything. This server has no network path to a forge, knows nothing about GitHub, and must not learn this project's own `ZETETIC-REVIEW:` convention, which its merge gate enforces but which no other repository shares.
26+
27+
## Decision
28+
29+
A `predictions` table, on both backends, holds the claim, the prediction in falsifiable terms, the test that would settle it, and the confidence its author held when writing it, plus the domain, directory and optional memory it belongs to. A row is written `open` and moves once to `resolved`, carrying what was observed, a verdict of `confirmed`, `refuted` or `abandoned`, the kind of source that settled it (`review`, `ci`, `test`, `manual`) and a reference to that source. A table CHECK refuses a resolved row without a verdict, a source kind and a reference, and the handler refuses a blank reference before the database has to.
30+
31+
The caller supplies the evidence and Cortex never fetches it. That is what makes the contract repository-agnostic: a host maps its own review convention onto the same four fields, and nothing in the schema names a forge or a marker.
32+
33+
`core/calibration.py` scores the resolved rows with the mean squared distance between confidence and outcome, the single-probability scale `scikit-learn` and most contemporary writing call the Brier score; Brier's own 1950 paper sums over both categories of a binary event and reports exactly twice this number, which ranks forecasters identically. It is reported beside the 0.25 a constant 0.5 forecast earns on this scale so the number reads without the literature, plus the reliability breakdown per confidence band, which says how often predictions in that band actually held. An abandoned prediction is counted apart and scored nowhere: its test was never run, so it says nothing about calibration.
34+
35+
Three tools, registered from `tool_registry_predictions.py`: `predict`, `resolve_prediction`, `calibration`. The standalone count moves from 54 to 57, 60 with both upstream integrations.
36+
37+
## Consequences
38+
39+
Easier: a confidence can now be scored rather than asserted. `calibration` answers with a Brier score, its reference point and the per-band frequencies, for one domain or for all.
40+
41+
`tests_py/core/test_calibration.py` pins the scoring: a constant 0.5 scoring exactly the uninformative reference, certainty that holds scoring 0, confidence that is wrong scoring worse than 0.5, the band a full confidence falls in, and an empty input scoring absent rather than zero. `tests_py/handlers/test_prediction_records.py` drives the three tools on SQLite: a prediction written open, refusals for a missing part and for a confidence outside 0 to 1, a resolution refused for a blank reference, a prediction resolving once, an unknown id, the score over two resolved rows with one still open, and the three tools present with their annotations.
42+
43+
Harder: nothing yet writes predictions on its own. A session that wants calibration has to state a prediction before running its test, which is the discipline the surface exists to support, not something it can impose.
44+
45+
Unchanged: procedural proficiency. Feeding a session's own verdict into `mine_skills` reuses this resolution contract and is the next step, not this one.

0 commit comments

Comments
 (0)