Repository navigation
Commit dd29c89
* feat(predictions): record what is expected, resolve it, score the confidence (#597)
Cortex stored what happened and never what was expected, so nothing it
held could turn out wrong in a way it noticed: the 796 procedural skills
mined from the maintainer's transcripts all sit at proficiency 0.5
because no session records an outcome, and while claim_events carry a
confidence, nothing compares it to an observation.
A predictions table on both backends holds the claim, the prediction in
falsifiable terms, the test that would settle it and the confidence its
author held. A row is written open and moves once to resolved, carrying
what was observed, a verdict, the kind of source that decided it and a
reference to that source; a table CHECK and the handler both refuse a
resolution without one.
Cortex never fetches that evidence and knows nothing about any forge or
review convention, including this project's own marker. That is what
makes the contract work in any repository: a host maps its convention
onto the same four fields (ADR-1076).
core/calibration.py scores the resolved rows with the Brier score
(Brier 1950), reported beside the 0.25 a constant 0.5 forecast earns, and
the per-band reliability the mean cannot show. An abandoned prediction is
counted apart and scored nowhere.
Three tools registered: predict, resolve_prediction, calibration.
Standalone count 54 to 57, 60 with both upstream integrations, with the
pinned test, the catalogue, the eight documents the claim gate scans and
the docker-smoke floor moved with it.
Full suite: 8763 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
* fix(predictions): close the five findings of the #599 review
Brier convention: the module computes mean((p - o)^2), the
single-probability scale scikit-learn and most contemporary writing call
the Brier score; Brier's own 1950 paper sums over both categories and
reports twice it. The docstring said the paper's formula. It now states
the scale it computes and why the ranking is the same, and the ADR and
the CHANGELOG say the same.
_band_of folded an out-of-range confidence into an edge band and left a
dead branch behind it. It now raises ValueError outside [0, 1]: a silent
bucket would make the reliability diagram lie about what it measured. The
zip is strict, and the final return is reachable for exactly 1.0.
The table CHECK demanded a non-null source_ref, not a non-empty one, so
only the handler stood between an empty reference and a resolved row.
Both DDLs now require source_ref and observed to be non-empty.
docs/mcp-tools.md advertised 57 tools while its four tier tables listed
54: ingest_findings, lesson_promotion and get_grooming_health were
registered and in no table. That predates this branch (51 against 54
before it). The three rows are added and the arithmetic closes.
Four new cases: three out-of-range confidences refused, and the schema
refusing an empty reference. Full suite: 8767 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
* fix(ci): derive the host-contract tool bounds from the registries (#597)
The verifier carried MIN_FULL_TOOL_COUNT = 52 with a ceiling three above,
so it stayed right when the surface moved to 54 (ADR-1066) and only broke
when the prediction tools took it to 57: CI failed on "discovered 57 full
tools; expected 52..55", taking down Test (SQLite backend) and Release
dependency set.
mcp_server/tool_surface.py reads the surface from the registry modules
themselves and the verifier derives both bounds from it (ADR-1077). Three
tests pin the derivation against what register_all actually registers,
including that the verifier's bounds follow it.
A literal that is right for a while, then wrong, and only caught once it
drifts far enough is a second source of truth, not a check.
Full suite: 8770 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
* fix(doc-claims): gate the three tool-count copies the review found (#597)
The audit asked for the class, not the number, and found three older
copies. `.bestpractices.json` cited the pinned test as
test_standalone_baseline_is_52_tools, two count moves stale, and the
TOOL_CLAIM regex could not see it because the digits sit inside an
identifier with no preceding space: a new PINNED_TEST_NAME_CLAIM family
checks that form. `.claude-plugin/marketplace.json` advertised 50 tools
(53 with upstreams) to every install and was scanned by nothing; it now
has a targeted check rather than joining SCANNED_FILES, because the same
description field recounts the plugin's version history, where past
counts are quotations and the generic scan would fail them. The
docker-smoke floor stays a literal, now pinned to the derived surface by
a test.
Both new checks were driven to failure before being trusted: 55/58 in
the marketplace and 52 in the pinned name each produce a named failure.
The gate's own fixtures gained the two subjects, since a family that
matches nothing fails as vacuous by design.
Full suite: 8771 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
* fix(ci): import the tool surface late in the host verifier (#597)
The derivation at module scope pulled the MCP SDK through the registries,
and the mcp-host-config job installs the three host CLIs with no Python
dependencies: "ModuleNotFoundError: No module named 'mcp'" took the job
down, and CI Green with it. That job verifies lean profiles only, which
never need the bounds.
full_tool_bounds() moves to mcp_server/tool_surface.py, where the surface
knowledge already lives, and the verifier keeps a thin wrapper importing
it inside the function. The wrapper also keeps that script at 292 lines,
under the 300-line cap it would otherwise have crossed.
A new test imports the script with `mcp` blocked in sys.meta_path and
asserts it still imports; I reproduced the failing condition that way
before fixing it. ADR-1077 says why the late import is load-bearing.
Full suite: 8772 passed, 8 skipped. The verifier passes its six cases.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
* test(tool-surface): make the SDK-free import test able to fail (#597)
The probe installed a meta-path finder written to the pre-PEP 451
find_module/load_module protocol. CPython ignores a finder without
find_spec, so the blocker blocked nothing: the test passed against
c710439, the very commit whose module-scope import broke the
mcp-host-config job. A test that cannot fail is not a pin.
The probe now implements find_spec and asserts the block works, by
importing mcp first and requiring ImportError before it touches the
script at all. Its source sits in a module constant, which also keeps
the test method under the craftsmanship gate's 40-line cap.
Measured both directions with the corrected test: against c710439 it
fails with the CI symptom, "from mcp.server.mcpserver import MCPServer"
raising ImportError through the blocker; against this head it passes.
The review of PR #599 found this; the vacuity is mine.
Full suite: 8772 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent e565d12 commit dd29c89
39 files changed
Lines changed: 1608 additions & 42 deletions
File tree
- .claude-plugin
- docs
- adr
- mcp_server
- core
- handlers
- infrastructure
- scripts
- tests_py
- core
- handlers
- scripts
- wiki
- adr/cortex
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
234 | | - | |
| 234 | + | |
235 | 235 | | |
236 | 236 | | |
237 | 237 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
11 | 46 | | |
12 | 47 | | |
13 | 48 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
242 | 242 | | |
243 | 243 | | |
244 | 244 | | |
245 | | - | |
| 245 | + | |
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
252 | | - | |
| 252 | + | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
| 180 | + | |
181 | 181 | | |
182 | 182 | | |
183 | 183 | | |
| |||
234 | 234 | | |
235 | 235 | | |
236 | 236 | | |
237 | | - | |
| 237 | + | |
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
| |||
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
254 | | - | |
| 254 | + | |
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
0 commit comments