Skip to content

Commit e9b0077

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/721-explicit-domain-identity
2 parents 5c6cc80 + f9f8074 commit e9b0077

82 files changed

Lines changed: 3914 additions & 164 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,34 @@ adheres to [Semantic Versioning](https://semver.org/).
66

77
## [Unreleased]
88

9+
### Added
10+
11+
- A claim keeps the time of the event it describes apart from the time it was
12+
extracted (#717, slice 1). `ClaimEvent.event_time` is `unknown`, a `point`
13+
or an `interval`, each bound carrying its value, its ISO 8601 precision and,
14+
when the source states one, its offset. A claim with no stated date stays
15+
`unknown`; it never inherits `extracted_at`. Persistence and extraction of
16+
the field are later slices.
17+
918
### Changed
1019

20+
- Each LongMemEval line of `<stem>.queries.jsonl` carries the question's gold
21+
`answer_session_ids` and its own `mrr` and `recall10`, so the published Recall@10
22+
and MRR are recomputable from the journal alone (#708).
23+
- The mutation gate runs only the mutants of the changed functions (ADR-1122).
24+
`mutation_check.sh` passes one `<prefix>*` name per changed function to
25+
`mutmut run`; it used to run every mutant of the file and judge the changed
26+
functions afterwards, which held #865 for six hours. On #877's diff the gate
27+
now runs 36 mutants instead of 1,164 and reaches its verdict in 21 min.
28+
- Pull requests must kill every mutant in the functions they change (ADR-1122).
29+
`scripts/ci_unit.sh` ends with `scripts/ci_mutation_diff.py`, which mutates the
30+
changed functions of `mcp_server` with their unit tests and accepts only
31+
registered equivalents; outside a PR it does nothing.
32+
A removed method no longer selects its neighbours, the mutation sandbox copies
33+
`benchmarks`, and a changed function mutmut cannot mutate (a decorator other than
34+
`staticmethod`/`classmethod`) is reported as a warning instead of passing silently.
35+
A comment above a removed method no longer selects its neighbours, and the next
36+
file's header no longer hides a trailing decorator in the last hunk of a file.
1137
- The Claude review verdict decides the merge (ADR-1121). The post-merge
1238
resolver merges a PR only when the latest `claude[bot]` verdict on its head
1339
is `Verdict: approve` and links the `claude-review.yml` run of that head whose
@@ -35,6 +61,41 @@ adheres to [Semantic Versioning](https://semver.org/).
3561

3662
### Fixed
3763

64+
- A document section memory is labelled by its full heading path
65+
(`[Install > Notes]`), not its own heading alone, so the same text under two
66+
parents is recalled as two distinguishable segments (#713, slice 1).
67+
Top-level sections keep their label; a re-ingest of an already-ingested
68+
nested document retires the old flat-labelled receipts as for any other
69+
normalizer change (#714).
70+
- Transcript discovery honours the host's config directory (#710, slice 1).
71+
Profile rebuilds, `import_sessions`, `backfill_memories`, explore features and
72+
the SessionStart session count read `$CLAUDE_CONFIG_DIR/projects` when the
73+
variable is set, instead of always `~/.claude/projects`. `CORTEX_CLAUDE_DIR`
74+
still wins, and Cortex's own data (store, profiles, wiki) does not move.
75+
- A user's own tags no longer hide a memory from default recall (#706,
76+
ADR-1123). The low-signal filter of `recall` and of trigger injection now
77+
drops only capture markers (`auto-captured`, `tool:<name>`, `_backfill`,
78+
`imported`, `session-summary`, emitted by Cortex writers, and `tool-output`,
79+
listed in #706);
80+
`audit`, `automated`, `wip`, `progress`, `code-review` and `stage-1` to
81+
`stage-11` no longer drop the memory. The wiki admission list is unchanged.
82+
- The post-commit reindex hook reads its cooldown file as UTF-8, the encoding
83+
`gated_files.write_json` writes it in, instead of the locale code page (cp1252 on
84+
Windows). The test files that still wrote or read text in the locale code page
85+
now name UTF-8 (#692, class 3).
86+
- The mutation sandbox copies `.claude-plugin` and `plugins`: the hook tests
87+
selected for `post_commit_reindex.py` read both, so mutmut's clean run failed
88+
before any mutant ran. New tests pin the cooldown window, the unknown-root
89+
default and the pruning of a full cooldown file, which the diff-scoped gate
90+
(ADR-1122) mutates whole.
91+
- The benchmark machine-load snapshot records no load average on Windows
92+
instead of raising: `os.getloadavg` does not exist there, and the
93+
`AttributeError` escaped the `except OSError` (#692 class 4, 13 tests).
94+
- The README now states that both LongMemEval headline numbers (v4.14.1,
95+
v4.20.0) describe the PostgreSQL store with the reranker, and the Storage
96+
table no longer claims an "identical" retrieval contract on SQLite: the tool
97+
interface is identical, and SQLite has only a bounded n=50 run without the
98+
reranker, no run comparable to the n=500 headline (#708).
3899
- The groomer coordinator no longer starts a consolidate cycle outside its
39100
lock in three failure cases (#683, ADR-1118). An existing `groomer.pid` that
40101
cannot be read (permission denied, I/O error) now raises
@@ -138,6 +199,15 @@ adheres to [Semantic Versioning](https://semver.org/).
138199

139200
### Added
140201

202+
- Visualization companion audited against Cortex's default stores (#724,
203+
slice 1): `docs/audits/visualization-companion-backends-2026-11-05.md`. The
204+
companion reads PostgreSQL at `127.0.0.1:5432/cortex` only, so a SQLite store
205+
or the plugin-managed server (kernel-chosen port, SCRAM password) leaves it in
206+
Trace-only mode with a message that says to install Cortex.
207+
- Transcript capture producers audited (#716, slice 1):
208+
`docs/audits/transcript-capture-producers-2026-11-05.md` lists the five
209+
producers, their session keys, dedups and replay behaviour. No producer keeps
210+
a cursor into a transcript and none discovers Codex rollouts.
141211
- `benchmarks/outcome/paired_delta.py`: scoring core of the longitudinal
142212
outcome benchmark (first slice of #343). Pairs `recall` and `no_recall`
143213
trials strictly by task and repeat, and reports the mean delta of

‎PRIVACY.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,10 +135,13 @@ The only outbound network activity is:
135135
`gh attestation verify <archive> --repo cdeust/Cortex --source-ref refs/heads/main`
136136
(the workflow publishes only from main).
137137
2. **One-time model download.** On first use, the open-source embedding model
138-
(`sentence-transformers`, all-MiniLM-L6-v2) and the reranking model
139-
(`flashrank`, ms-marco-MiniLM) are downloaded from the public Hugging Face
140-
model hub. These transfers fetch **model files only** — no user content is
141-
sent. After the first download the models run fully offline. To forbid the
138+
(`sentence-transformers`, all-MiniLM-L6-v2, about 87 MB, from the
139+
`sentence-transformers/all-MiniLM-L6-v2` Hugging Face repository) and the
140+
reranking model (`flashrank`, ms-marco-MiniLM-L-12-v2, about 33 MB, the
141+
`ms-marco-MiniLM-L-12-v2.zip` archive of the `prithivida/flashrank` Hugging
142+
Face repository) are downloaded from the public Hugging Face model hub
143+
(sizes measured by the 2026-10-10 audit in issue #705). These transfers
144+
fetch **model files only** — no user content is sent. After the first download the models run fully offline. To forbid the
142145
reranker fetch outright (air-gapped installs, or any environment where an
143146
unexpected outbound connection is unacceptable), set
144147
`CORTEX_RERANKER_OFFLINE=1`: Cortex then refuses the download when the model

‎README.md‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -292,7 +292,18 @@ from it.
292292
| Recall@10 | **98.2%** | **97.8%** |
293293
| MRR | **0.9167** | **0.905** |
294294

295-
Both are single runs: n=500, clean database, consolidation disabled, retrieval only.
295+
Both are single runs: n=500, consolidation disabled, retrieval only. Both describe the
296+
**PostgreSQL** store with the cross-encoder reranker: the harness (`benchmarks/lib/bench_db.py`)
297+
always builds a `PgMemoryStore` and calls `core.pg_recall`, and both artifacts record
298+
`"reranker_state": "loaded"`. The v4.20.0 run used a clean database in the ephemeral
299+
`pgvector/pgvector:pg16` container of `benchmarks/reproduce.sh` (its manifest records that
300+
`pg_image`); the v4.14.1 run did not use that container and ran against the live PostgreSQL
301+
store (CHANGELOG 4.14.1, "Verified").
302+
The default SQLite store has no comparable run yet (n=500, reranker loaded;
303+
[#708](https://github.com/cdeust/Cortex/issues/708)), so these figures do not describe it.
304+
The best SQLite LongMemEval-S result is a bounded run without the reranker (n=50,
305+
sentence-transformers embeddings: MRR 0.609, Recall@10 94.0%,
306+
[benchmarks/results/semantic-fallback-169](benchmarks/results/semantic-fallback-169/MANIFEST.md)).
296307

297308
v4.14.1, 2026-07-14: [artifact JSON](benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json);
298309
[code SHA](https://github.com/cdeust/Cortex/commit/28145f0b7a113fc06e22568de6feea7f8444eaf5).
@@ -333,7 +344,8 @@ bash <plugin-dir>/scripts/install-plugin.sh --postgres
333344
|---|---|---|
334345
| Setup | none | pgvector >= 0.8.0, pg_trgm |
335346
| All 57 tools | yes | yes |
336-
| Retrieval contract | identical | identical |
347+
| Retrieval tool interface | identical | identical |
348+
| Published retrieval quality | n=50 without reranker only ([#708](https://github.com/cdeust/Cortex/issues/708)) | LongMemEval-S n=500 above |
337349
| Fusion | in-process | server-side PL/pgSQL |
338350
| ANN index | none | pgvector HNSW |
339351
| Cross-agent team decisions, preemptive context, pipeline heat bumps | no-op | active |

‎benchmarks/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,10 @@ next to `<stem>.json`:
123123
- `<stem>.queries.jsonl` — one line per scored query: the retrieved items'
124124
ids, sources, scores, UTF-8 bytes, code points and sha256 (and the text,
125125
with `--query-log-content`). LongMemEval lines also carry the bytes of the
126-
question's whole haystack.
126+
question's whole haystack, its gold `answer_session_ids` and the question's
127+
own `mrr` and `recall10`: the mean of those columns over the lines of one
128+
`run` is the run's MRR and Recall@10, so the headline is recomputable from
129+
the journal alone (`tests_py/benchmarks/test_longmemeval_journal_rows.py`).
127130
- `<stem>.phases.jsonl` — wall-clock start and end of every `ingest` and
128131
`recall` phase.
129132

‎benchmarks/lib/machine_load_snapshot.py‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,15 +45,28 @@ def count_docker_containers() -> int | None:
4545
return len([line for line in docker_out.splitlines() if line.strip()])
4646

4747

48+
def _load_averages() -> tuple[float | None, float | None, float | None]:
49+
"""1/5/15-minute load averages, or three None where the OS reports none.
50+
51+
``os.getloadavg`` is Unix-only: on Windows the attribute does not exist,
52+
so the lookup raises AttributeError, not OSError (issue #692 class 4).
53+
On Unix it raises OSError when the value is unobtainable.
54+
source: CPython 3.12 docs, os.getloadavg (Availability: Unix)"""
55+
getloadavg = getattr(os, "getloadavg", None)
56+
if getloadavg is None:
57+
return None, None, None
58+
try:
59+
return getloadavg()
60+
except OSError:
61+
return None, None, None
62+
63+
4864
def machine_load_snapshot() -> dict:
4965
"""Load average + concurrent pytest/container counts, as this run saw
5066
them.
5167
5268
source: ADR-0080"""
53-
try:
54-
load1, load5, load15 = os.getloadavg()
55-
except OSError: # not available on this platform (e.g. Windows)
56-
load1 = load5 = load15 = None
69+
load1, load5, load15 = _load_averages()
5770
return {
5871
"load_average_1m": load1,
5972
"load_average_5m": load5,

‎benchmarks/longmemeval/run_benchmark.py‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,14 @@ def run_benchmark(
293293
# Run production retrieval
294294
with log.phase("recall", qid):
295295
results = db.recall(question, top_k=10, domain="longmemeval")
296+
retrieved_sids = [source_map.get(r["memory_id"], "") for r in results]
297+
298+
# Compute metrics
299+
mrr = compute_mrr(retrieved_sids, answer_sids)
300+
r10 = recall_at_k_binary(retrieved_sids, answer_sids)
301+
302+
# The row carries the gold sessions and its own scores, so
303+
# the headline is recomputable from the journal alone (#708).
296304
log.record(
297305
qid,
298306
results,
@@ -303,13 +311,11 @@ def run_benchmark(
303311
"haystack_bytes": sum(
304312
len(m["content"].encode("utf-8")) for m in memories
305313
),
314+
"answer_session_ids": list(answer_sids),
315+
"mrr": mrr,
316+
"recall10": r10,
306317
},
307318
)
308-
retrieved_sids = [source_map.get(r["memory_id"], "") for r in results]
309-
310-
# Compute metrics
311-
mrr = compute_mrr(retrieved_sids, answer_sids)
312-
r10 = recall_at_k_binary(retrieved_sids, answer_sids)
313319

314320
all_mrr.append(mrr)
315321
all_recall10.append(r10)

‎docs/README.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Cortex documentation
2+
3+
This page sorts the existing documentation by the task you came to do. It holds
4+
no facts of its own: every entry points at the one document that owns the
5+
subject, so a correction is made once, where it belongs. The links and their
6+
section anchors are checked by `tests_py/scripts/test_docs_index_links.py`.
7+
8+
## 1. First local memory round trip
9+
10+
1. Install the plugin: [README § Install](../README.md#install).
11+
2. Check the environment before the first session:
12+
`python -m mcp_server.doctor` (one line per check, with the fix for each).
13+
3. Store and retrieve one memory with the `remember` and `recall` tools:
14+
their schemas ([`remember`](api-reference.md#remember),
15+
[`recall`](api-reference.md#recall)), and what each
16+
call does inside ([write path](data-flow.md#memory-write-path-remember),
17+
[read path](data-flow.md#memory-read-path-recall)).
18+
4. See where the memory now lives and how to delete it:
19+
[PRIVACY § Where your data is stored](../PRIVACY.md#where-your-data-is-stored) and
20+
[§ Your controls](../PRIVACY.md#your-controls).
21+
22+
Known problems on this path are tracked in the open: the host ignores the plugin's
23+
`postInstall` step ([#703](https://github.com/cdeust/Cortex/issues/703)), a profile
24+
with no configuration can reach for a local PostgreSQL
25+
([#704](https://github.com/cdeust/Cortex/issues/704)), and the first SessionStart
26+
downloads both model files inside the hook
27+
([#705](https://github.com/cdeust/Cortex/issues/705)). Read
28+
[Limits worth knowing before you install](../README.md#limits-worth-knowing-before-you-install)
29+
first.
30+
31+
## 2. Concepts
32+
33+
- What Cortex keeps, checks and lets fade: [README § Keep context useful](../README.md#keep-context-useful).
34+
- Storage backends, SQLite by default and PostgreSQL as an opt-in:
35+
[README § Storage](../README.md#storage).
36+
- Layers and module boundaries: [architecture.md](architecture.md),
37+
[module-inventory.md](module-inventory.md).
38+
- Every pipeline, step by step: [data-flow.md](data-flow.md).
39+
- What the retrieval numbers do and do not establish:
40+
[README § Does the retrieval work](../README.md#does-the-retrieval-work),
41+
[README § Verification](../README.md#verification),
42+
[benchmarks-detail.md](benchmarks-detail.md).
43+
44+
## 3. Set up a host
45+
46+
- Claude Code and Codex: [README § Cortex in Codex and Claude Code](../README.md#cortex-in-codex-and-claude-code),
47+
[codex-plugin.md](codex-plugin.md).
48+
- Claude and Codex on the same memory: [shared-host-memory.md](shared-host-memory.md).
49+
- Claude Desktop, Cowork and other MCP hosts: [README § Every other MCP host](../README.md#every-other-mcp-host).
50+
- The upstream MCP servers that `ingest_codebase` and `ingest_prd` consume:
51+
[mcp-connections.example.json](mcp-connections.example.json).
52+
- Windows, WSL, dev containers, remote PostgreSQL, corporate proxies and headless CI:
53+
[deployment-scenarios.md](deployment-scenarios.md).
54+
- Scheduled background grooming: [groomer-scheduling.md](groomer-scheduling.md).
55+
56+
## 4. API contracts
57+
58+
- Tool catalogue, by tier, with purpose and target latency: [mcp-tools.md](mcp-tools.md).
59+
- Input schemas, responses and errors: [api-reference.md](api-reference.md),
60+
[error codes](api-reference.md#error-codes).
61+
- Design decisions behind the contracts: [adr/](adr/).
62+
63+
## 5. Troubleshooting
64+
65+
- A codebase ingest that looks hung: [troubleshooting.md](troubleshooting.md), starting at
66+
[Is it hung or just slow?](troubleshooting.md#1-is-it-hung-or-just-slow).
67+
- Setup errors: [troubleshooting.md § Common setup errors](troubleshooting.md#5-common-setup-errors).
68+
- A model download that stalls behind a proxy:
69+
[troubleshooting.md § Stalled model download](troubleshooting.md#4-stalled-model-download),
70+
[deployment-scenarios.md § Corporate proxy](deployment-scenarios.md#corporate-proxy--custom-ca-model-downloads).
71+
- Local log files and their rotation: [runbooks/local-log-rotation.md](runbooks/local-log-rotation.md).
72+
73+
## 6. Privacy, security and costs
74+
75+
- What is processed, where it is stored and what leaves the machine: [PRIVACY.md](../PRIVACY.md),
76+
in particular [What leaves your machine](../PRIVACY.md#what-leaves-your-machine).
77+
- What Cortex can reach and how to report a vulnerability: [SECURITY.md](../SECURITY.md),
78+
and the argument with its evidence: [ASSURANCE-CASE.md](ASSURANCE-CASE.md).
79+
- Energy and carbon: no figure is published, and the page says why:
80+
[README § Green software engineering](../README.md#green-software-engineering).
81+
82+
## 7. Contributing
83+
84+
- The gates a change has to clear: [CONTRIBUTING.md](../CONTRIBUTING.md),
85+
[testing-guide.md](testing-guide.md).
86+
- Who decides and where the project is going: [GOVERNANCE.md](../GOVERNANCE.md),
87+
[ROADMAP.md](ROADMAP.md).

0 commit comments

Comments
 (0)