Repository navigation
Record the local memory-embedder evidence (benchmark, hybrid re-measurement, live deployment) - #127
Conversation
…, hybrid re-measurement, live deployment Adds evidence/artifacts/local-model-alignment-20260923/ (hash-listed, no selection labels): - memory-embedder-dense-benchmark.json: current embedders set up per their model cards on a sealed 130-page memory corpus with a two-annotator query set; Nemotron-3-Embed-1B highest (nDCG@10 0.858 vs the all-MiniLM-L6-v2 reference 0.713; its lead over Qwen3-Embedding and granite-r2 is within noise); - memory-embedder-hybrid-retrieval.json: through ai-memory's own memory_query, Nemotron with its query/passage prefixes at 8192 context beat the MiniLM configuration by nDCG@10 +0.078 (95% CI 0.037-0.128) against a preregistered 0.03 bar; 4096 context loses nine pages; - ai-memory-embedder-deployment.json: the host's live switch, the patched 2.3.2 build (upstream PR akitaonrails/ai-memory#859, binary sha256), configuration, observed results and rollback. Each carries the independent verifier's findings, including the weaker lock and log provenance. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…it, observed-vs-logged labels, measurement basis, rollback not exercised Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d1384bbbd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "date": "2026-09-23", | ||
| "evidence_class": "local_integration on one WSL2 x86_64 host (RTX 5090 Laptop 24 GB); one run per arm; dense cosine retrieval only", | ||
| "question": "Which current local embedding model best retrieves durable-memory pages when each model is used exactly as its publisher specifies?", | ||
| "corpus": "a sealed snapshot of the host's ai-memory store: 130 latest pages (decisions, sessions, logs), sha256-locked", |
There was a problem hiding this comment.
When this artifact is consumed by the intended blind re-record, the claim that the corpus is “sha256-locked” cannot be verified because no digest or locator is provided; the receipt likewise omits hashes or locators for the query/gold sets, per-query results, harness, protocol, and command output. Consequently, the aggregate scores and confidence intervals cannot be independently reproduced or recounted, so retain sanitized artifacts or at least their exact hashes and run records before treating these values as integration evidence.
AGENTS.md reference: AGENTS.md:L10-L15
Useful? React with 👍 / 👎.
| "arms": { | ||
| "A": "ai-memory 2.3.2, local all-MiniLM-L6-v2 (the configuration running before the change)", | ||
| "A2": "ai-memory 2.4.0 + prefix patch, local all-MiniLM-L6-v2 (isolates the version change)", | ||
| "B": "ai-memory 2.4.0 + prefix patch, openai-compat Nemotron-3-Embed-1B-BF16 with query:/passage: prefixes, vLLM max-model-len 8192", |
There was a problem hiding this comment.
The passing candidate arm uses ai-memory 2.4.0 plus the patch, while the deployment receipt records a 2.3.2-based cherry-pick with three hunks removed. A2 only shows that upgrading to 2.4.0 leaves MiniLM results unchanged; it does not establish that the different 2.3.2 patch build produces the same prefixed vectors or rankings, and the live smoke query does not measure that equivalence. Run the decision arm with the deployed binary SHA (or retain a vector/result-equivalence comparison) before using this rule outcome to justify the live switch.
AGENTS.md reference: AGENTS.md:L10-L15
Useful? React with 👍 / 👎.
| "The metric named recall@5 is 'any relevant page in the top 5'.", | ||
| "Only the final lock of three survived; file times and the command log support the order; two protocol amendments (CPU thread cap, GPU for sentence-transformers arms) were relocked before their runs.", | ||
| "Prefixes and input_type are shown by the harness code; the vLLM servers ran without request logging.", | ||
| "q29 and q30 score 0 because one annotator's gold page is outside the 130-page corpus." |
There was a problem hiding this comment.
When the reported metrics, bootstrap intervals, and subsequent hybrid gate are calculated, q29 and q30 are counted among the 62 answerable queries even though their gold pages are outside the sealed 130-page corpus. Those forced zeroes are not retrieval failures and distort the absolute metrics and paired distribution used by both experiments; reclassify them as no-answer or exclude them, then recalculate the results and decision gate.
Useful? React with 👍 / 👎.
Evidence for the durable-memory and semantic-rag layers, landed before the 20260923 lane re-record so the blind checkout can read it. Adds three hash-listed files under
evidence/artifacts/local-model-alignment-20260923/, with no selection or decision labels:memory_query. Nemotron with its prefixes at 8192 context beat MiniLM by nDCG@10 +0.078 (95% CI 0.037-0.128), clearing the preregistered 0.03 bar. At 4096 context, nine pages fail.Evidence class: local integration on one WSL2 host, one run per arm. Each file carries the independent verifier's findings, including the weaker lock and log provenance. Checks: validate.py, evidence_manifest, validate_catalogs, component_matrix and new_host_grand_list pass; guarded Gitleaks found no leaks.
🤖 Generated with Claude Code