Skip to content

Commit 64ae942

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/722-recall-stage-timing
2 parents da29523 + f9f8074 commit 64ae942

15 files changed

Lines changed: 758 additions & 13 deletions

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ adheres to [Semantic Versioning](https://semver.org/).
2424

2525
### Changed
2626

27+
- Each LongMemEval line of `<stem>.queries.jsonl` carries the question's gold
28+
`answer_session_ids` and its own `mrr` and `recall10`, so the published Recall@10
29+
and MRR are recomputable from the journal alone (#708).
30+
- The mutation gate runs only the mutants of the changed functions (ADR-1122).
31+
`mutation_check.sh` passes one `<prefix>*` name per changed function to
32+
`mutmut run`; it used to run every mutant of the file and judge the changed
33+
functions afterwards, which held #865 for six hours. On #877's diff the gate
34+
now runs 36 mutants instead of 1,164 and reaches its verdict in 21 min.
2735
- Pull requests must kill every mutant in the functions they change (ADR-1122).
2836
`scripts/ci_unit.sh` ends with `scripts/ci_mutation_diff.py`, which mutates the
2937
changed functions of `mcp_server` with their unit tests and accepts only
@@ -198,6 +206,11 @@ adheres to [Semantic Versioning](https://semver.org/).
198206

199207
### Added
200208

209+
- Visualization companion audited against Cortex's default stores (#724,
210+
slice 1): `docs/audits/visualization-companion-backends-2026-11-05.md`. The
211+
companion reads PostgreSQL at `127.0.0.1:5432/cortex` only, so a SQLite store
212+
or the plugin-managed server (kernel-chosen port, SCRAM password) leaves it in
213+
Trace-only mode with a message that says to install Cortex.
201214
- Transcript capture producers audited (#716, slice 1):
202215
`docs/audits/transcript-capture-producers-2026-11-05.md` lists the five
203216
producers, their session keys, dedups and replay behaviour. No producer keeps

‎benchmarks/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,10 @@ next to `<stem>.json`:
123123
- `<stem>.queries.jsonl` — one line per scored query: the retrieved items'
124124
ids, sources, scores, UTF-8 bytes, code points and sha256 (and the text,
125125
with `--query-log-content`). LongMemEval lines also carry the bytes of the
126-
question's whole haystack.
126+
question's whole haystack, its gold `answer_session_ids` and the question's
127+
own `mrr` and `recall10`: the mean of those columns over the lines of one
128+
`run` is the run's MRR and Recall@10, so the headline is recomputable from
129+
the journal alone (`tests_py/benchmarks/test_longmemeval_journal_rows.py`).
127130
- `<stem>.phases.jsonl` — wall-clock start and end of every `ingest` and
128131
`recall` phase.
129132

‎benchmarks/longmemeval/run_benchmark.py‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,14 @@ def run_benchmark(
293293
# Run production retrieval
294294
with log.phase("recall", qid):
295295
results = db.recall(question, top_k=10, domain="longmemeval")
296+
retrieved_sids = [source_map.get(r["memory_id"], "") for r in results]
297+
298+
# Compute metrics
299+
mrr = compute_mrr(retrieved_sids, answer_sids)
300+
r10 = recall_at_k_binary(retrieved_sids, answer_sids)
301+
302+
# The row carries the gold sessions and its own scores, so
303+
# the headline is recomputable from the journal alone (#708).
296304
log.record(
297305
qid,
298306
results,
@@ -303,13 +311,11 @@ def run_benchmark(
303311
"haystack_bytes": sum(
304312
len(m["content"].encode("utf-8")) for m in memories
305313
),
314+
"answer_session_ids": list(answer_sids),
315+
"mrr": mrr,
316+
"recall10": r10,
306317
},
307318
)
308-
retrieved_sids = [source_map.get(r["memory_id"], "") for r in results]
309-
310-
# Compute metrics
311-
mrr = compute_mrr(retrieved_sids, answer_sids)
312-
r10 = recall_at_k_binary(retrieved_sids, answer_sids)
313319

314320
all_mrr.append(mrr)
315321
all_recall10.append(r10)

‎docs/README.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Cortex documentation
2+
3+
This page sorts the existing documentation by the task you came to do. It holds
4+
no facts of its own: every entry points at the one document that owns the
5+
subject, so a correction is made once, where it belongs. The links and their
6+
section anchors are checked by `tests_py/scripts/test_docs_index_links.py`.
7+
8+
## 1. First local memory round trip
9+
10+
1. Install the plugin: [README § Install](../README.md#install).
11+
2. Check the environment before the first session:
12+
`python -m mcp_server.doctor` (one line per check, with the fix for each).
13+
3. Store and retrieve one memory with the `remember` and `recall` tools:
14+
their schemas ([`remember`](api-reference.md#remember),
15+
[`recall`](api-reference.md#recall)), and what each
16+
call does inside ([write path](data-flow.md#memory-write-path-remember),
17+
[read path](data-flow.md#memory-read-path-recall)).
18+
4. See where the memory now lives and how to delete it:
19+
[PRIVACY § Where your data is stored](../PRIVACY.md#where-your-data-is-stored) and
20+
[§ Your controls](../PRIVACY.md#your-controls).
21+
22+
Known problems on this path are tracked in the open: the host ignores the plugin's
23+
`postInstall` step ([#703](https://github.com/cdeust/Cortex/issues/703)), a profile
24+
with no configuration can reach for a local PostgreSQL
25+
([#704](https://github.com/cdeust/Cortex/issues/704)), and the first SessionStart
26+
downloads both model files inside the hook
27+
([#705](https://github.com/cdeust/Cortex/issues/705)). Read
28+
[Limits worth knowing before you install](../README.md#limits-worth-knowing-before-you-install)
29+
first.
30+
31+
## 2. Concepts
32+
33+
- What Cortex keeps, checks and lets fade: [README § Keep context useful](../README.md#keep-context-useful).
34+
- Storage backends, SQLite by default and PostgreSQL as an opt-in:
35+
[README § Storage](../README.md#storage).
36+
- Layers and module boundaries: [architecture.md](architecture.md),
37+
[module-inventory.md](module-inventory.md).
38+
- Every pipeline, step by step: [data-flow.md](data-flow.md).
39+
- What the retrieval numbers do and do not establish:
40+
[README § Does the retrieval work](../README.md#does-the-retrieval-work),
41+
[README § Verification](../README.md#verification),
42+
[benchmarks-detail.md](benchmarks-detail.md).
43+
44+
## 3. Set up a host
45+
46+
- Claude Code and Codex: [README § Cortex in Codex and Claude Code](../README.md#cortex-in-codex-and-claude-code),
47+
[codex-plugin.md](codex-plugin.md).
48+
- Claude and Codex on the same memory: [shared-host-memory.md](shared-host-memory.md).
49+
- Claude Desktop, Cowork and other MCP hosts: [README § Every other MCP host](../README.md#every-other-mcp-host).
50+
- The upstream MCP servers that `ingest_codebase` and `ingest_prd` consume:
51+
[mcp-connections.example.json](mcp-connections.example.json).
52+
- Windows, WSL, dev containers, remote PostgreSQL, corporate proxies and headless CI:
53+
[deployment-scenarios.md](deployment-scenarios.md).
54+
- Scheduled background grooming: [groomer-scheduling.md](groomer-scheduling.md).
55+
56+
## 4. API contracts
57+
58+
- Tool catalogue, by tier, with purpose and target latency: [mcp-tools.md](mcp-tools.md).
59+
- Input schemas, responses and errors: [api-reference.md](api-reference.md),
60+
[error codes](api-reference.md#error-codes).
61+
- Design decisions behind the contracts: [adr/](adr/).
62+
63+
## 5. Troubleshooting
64+
65+
- A codebase ingest that looks hung: [troubleshooting.md](troubleshooting.md), starting at
66+
[Is it hung or just slow?](troubleshooting.md#1-is-it-hung-or-just-slow).
67+
- Setup errors: [troubleshooting.md § Common setup errors](troubleshooting.md#5-common-setup-errors).
68+
- A model download that stalls behind a proxy:
69+
[troubleshooting.md § Stalled model download](troubleshooting.md#4-stalled-model-download),
70+
[deployment-scenarios.md § Corporate proxy](deployment-scenarios.md#corporate-proxy--custom-ca-model-downloads).
71+
- Local log files and their rotation: [runbooks/local-log-rotation.md](runbooks/local-log-rotation.md).
72+
73+
## 6. Privacy, security and costs
74+
75+
- What is processed, where it is stored and what leaves the machine: [PRIVACY.md](../PRIVACY.md),
76+
in particular [What leaves your machine](../PRIVACY.md#what-leaves-your-machine).
77+
- What Cortex can reach and how to report a vulnerability: [SECURITY.md](../SECURITY.md),
78+
and the argument with its evidence: [ASSURANCE-CASE.md](ASSURANCE-CASE.md).
79+
- Energy and carbon: no figure is published, and the page says why:
80+
[README § Green software engineering](../README.md#green-software-engineering).
81+
82+
## 7. Contributing
83+
84+
- The gates a change has to clear: [CONTRIBUTING.md](../CONTRIBUTING.md),
85+
[testing-guide.md](testing-guide.md).
86+
- Who decides and where the project is going: [GOVERNANCE.md](../GOVERNANCE.md),
87+
[ROADMAP.md](ROADMAP.md).

‎docs/adr/ADR-1122-a-pull-request-must-kill-every-mutant-in-the-functions-it-changes.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,12 @@ Prior art: Google mutates only the lines a change touches and reports the result
2727
1. `scripts/ci_unit.sh` ends with `scripts/ci_mutation_diff.py`, so the `unit` job (and its coverage run) carries the mutation gate. `ci.json` keeps its five keys. Outside a pull request (`GITHUB_BASE_REF` empty) the step does nothing.
2828
2. The script fetches the base branch, reads `git diff -U0 FETCH_HEAD...HEAD` (the merge base, so a base branch that moves on adds nothing) for `mcp_server/**/*.py`, and keeps the top-level functions and methods whose lines the PR changed, decorators and trailing blank lines included. A deletion of a whole top-level definition selects nothing; a deletion inside a function selects it. mutmut cannot mutate lines, so the granularity is the function, the finest unit it names. The deletion of a whole method selects nothing in its class, the deletion of a nested function selects its enclosing function. A changed function whose decorators mutmut 3.8.1 cannot mutate (anything but `staticmethod` and `classmethod`) produces no mutant at all, so it is reported as a `::warning::` and not gated.
2929
3. Per changed file, the tests are the unit test files that import the module (`tests_py/scripts`, `integration`, `benchmarks` excluded). Imports are read with `ast`, so a parenthesized multiline import counts. A changed function whose module no test imports fails the job.
30-
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES`; `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard, no per-file suppression.
30+
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES` and passes one `<prefix>*` name per changed function to `mutmut run`, so only the mutants of the changed functions run (`scripts/ci_mutation_scope.py`). mutmut 3.8.1 asserts when no mutant matches those names; after a failed run the gate passes only if mutmut's stats file exists (the stats run succeeded) and the `.meta` file of every mutated file exists and holds no mutant in scope; any other failure fails the job, and so does a failure of the helper that builds the names (it never falls back to running every mutant). `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard in the equivalents registry, no per-file suppression.
3131
5. Survivors in functions the PR did not change are not this PR's failure; a PR that rewrites such a function owns its mutants.
3232

3333
## Consequences
3434

3535
- Metric: surviving unregistered mutants in changed functions at merge: 0 (not measured today).
3636
- Measured end to end on a real diff: touching `_callee_basename` passes in 44 s (its 5 survivors are registered equivalents); touching `_walk_for_calls` fails with 4 unregistered survivors.
37-
- The `unit` job grows by about one minute per changed module. A PR that changes many modules lengthens it linearly; no cap is applied, so no change goes unchecked.
37+
- The `unit` job grows with the mutants of the changed functions, not with the size of their files. Before the names were passed, mutmut ran every mutant of the file and the gate judged the changed functions afterwards: #865 (`backend_resolution.py`, 262 mutants) was cancelled at the six-hour job limit after 141 of them, and #877 ran past two hours. Measured on #877's diff (2026-11-06, local macOS arm64, PostgreSQL 18.6 from `launcher_pg_assets`): 36 mutants run instead of 1,164 (`remember.py` 26 of 741, `domain_mapping.py` about 10 of 423), and the gate reaches its verdict in 21 min 15 s, including the stats and clean-test runs. At the measured 0.07 mutants/s of `remember.py`, its 741 mutants would take about 2 h 56 min (extrapolated, not run). No mutant hit mutmut's CPU timeout (`⏰ 0`), so no wall-clock cap is added; no cap is applied, so no change goes unchecked.
3838
- Module-level code (constants, imports) is not mutated by mutmut and is not gated.

0 commit comments

Comments
 (0)