Frozen-corpus registry, drift-refusing reuse, and the AUC on the path runs actually take - #183

4.2's AUC was wired into the plain `longmemeval` verb only. That verb cold-builds
the graph: ~684 extraction calls and 7-9 hours wall clock for 50 questions. Every
real measurement therefore goes through --prepared-pair, which builds once and
then evaluates arms against a frozen volume in about five minutes.

So the instrument reported only on the path nobody runs. That is the same
dead-option defect this branch already found twice -- IncludeQuestionTypes with
no caller, and a read-only filter that removed nothing -- arriving in my own work.

The prepared-pair path already sets EvidenceIndex and GraphProbe, so both halves
the AUC needs are collected there: the per-section diagnostics and the
answer-presence verdict. Only the reporting line was missing.

4158 unit + 348 LongMemEval green.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE

A cold build is ~684 extraction calls and 7-9 hours for 50 questions, so every
real measurement reuses a frozen one. Reuse read the sealed manifest, printed its
fingerprint and proceeded. VerifyIntegrity only checks the manifest against
ITSELF -- that its recorded fingerprint matches its recorded contents. Nothing
compared it to the run adopting it.

That allowed the worst kind of wrong number. A corpus on this machine was
ingested with AssistantContent=Utterance under AgentEval 0.16; today's default is
Ignore. Evaluating it now would produce a report whose fingerprint block
describes THIS run while every fact in the graph came from the other
configuration -- internally consistent, reproducible, and wrong.

Three changes:

Manifest schema 5 -> 6. It now records what it was ingested with: assistant
content mode, predicate-vocabulary flag, the extraction-vocabulary and
query-lexicon hashes, extraction provenance mode, question seed, memory types,
preparation date and a description. The first five join the fingerprint, so two
materially different corpora stop hashing identically. The description does not:
two corpora differing only in a note are the same corpus, and making a note
change the identity would force needless rebuilds.

Drift comparison on reuse. It refuses unless --allow-stale-prepared, naming every
drifted field and the override. What counts as drift is deliberately narrow: only
what changed the stored graph. The answer model, judge, recall budget and
evidence mode are applied at evaluation time, so a corpus is legitimately
reusable across them -- that reusability is the entire point, and flagging them
would make every reuse "stale" and train the operator to pass the override. A
field an older manifest never recorded reports as drift, not agreement: "we do
not know how this was built" must never read as "it matches".

A machine-local registry at artifacts/evaluation/prepared-corpora.json, written
on retained preparations, listed by --list-prepared-corpora with age and a
schema-too-old note. It is a convenience, never the guarantee -- the authority is
the manifest sealed inside each corpus, which travels with the data and cannot
drift from it. Deleting the registry makes no reuse unsafe.

Also: --memory-types now reaches PREPARATION, not only evaluation. An
episodic-only sample selects different questions, whose histories have to be
ingested for the corpus to answer them, which is precisely why 8.3b needs its own
cold build and cannot reuse the stratified one. The drift check asserts that.

4158 unit + 362 LongMemEval green.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE