What we found after auditing 10,134 mem0 entries: 97.8% were junk
We've been running mem0 in production for 32 days. One AI agent, one human, daily conversations, Qdrant backend. Two extraction models over that period: gemma2:2b (local, via Ollama) for the first 20 days, then Claude Sonnet 4.6 for the last 12. After noticing the agent kept "remembering" things it had never been told, we decided to stop guessing and look.
We pulled the entire collection. 10,134 entries.
The first pass was triage. We ran keyword searches, hash-duplicate detection, and targeted sweeps for obvious junk clusters. That removed 2,468 entries: exact-hash duplicates, hallucinated category clusters like "formal communication style" and "software developer at Google," and 668 copies of a single feedback-loop hallucination (more on that later). Then a cosine similarity script flagged another 2,943 entries (37.6% of what remained) as near-duplicates. We reviewed the 829 cluster survivors and kept 7.
That got us from 10,134 down to 6,264. Then we read what was left. Every entry, one by one, across eight batches. 96.9% of those were still junk.
224 entries survived the full process. Out of 10,134. That's 97.8% junk across the entire collection. And of those 224 survivors, 186 had to be deleted and rewritten from scratch because the originals were partial or malformed. Only 38 entries in the entire collection were clean enough to keep as-is.
This isn't a bug report about one failure mode. It's a dataset showing where the extraction pipeline breaks in production, measured across two extraction models and 32 days of continuous operation.
How we got here
The collection spans 32 days (Feb 23 to Mar 26, 2026). For the first 20 days, we ran gemma2:2b (a 2B parameter local model via Ollama) for extraction with default prompts. On day 21 we switched to Claude Sonnet 4.6. On day 29 we merged PR #4302 into our fork, adding filtering.ts, isolation.ts, and DEFAULT_CUSTOM_INSTRUCTIONS.
We expected the model upgrade to fix things. It didn't.
What we found
| Batch |
Date Range |
Entries |
Keeps |
Junk Rate |
Extraction Model |
| Feb P1 |
Feb 23-26 |
839 |
~20 |
97.7% |
gemma2:2b |
| Feb P2 |
Feb 27-28 |
620 |
12 |
98.1% |
gemma2:2b |
| Mar P1 |
Mar 1-4 |
880 |
13 |
98.5% |
gemma2:2b |
| Mar P2 |
Mar 5-7 |
966 |
25 |
97.4% |
gemma2:2b |
| Mar P3 |
Mar 8-10 |
808 |
37 |
95.4% |
gemma2:2b |
| Mar P4 |
Mar 11-13 |
908 |
22 |
97.6% |
gemma2:2b |
| Mar P5 |
Mar 14-15 |
820 |
22 |
97.3% |
gemma + Sonnet |
| Mar P6 |
Mar 16-26 |
423 |
44 |
89.6% |
Sonnet |
| Total |
|
6,264 |
195 |
96.9% |
|
| Phase |
Entries Reviewed |
Deleted |
Rewritten |
| Phase 0 (targeted) |
2,468 |
2,468 |
0 |
| Phase 1 (dedup) |
1,572 |
1,572 |
0 |
| Phase 2 (manual audit) |
6,264 |
6,070 |
186 |
| Total |
10,304 |
10,110 |
186 |
(Total exceeds 10,134 because some entries were flagged in Phase 1 dedup and reviewed again individually in Phase 2.)
Final result: 224 clean memories in the live collection. Every original entry was either deleted outright or deleted and rewritten from scratch.
Where the junk comes from
| Category |
~Count |
% of Junk |
What it looks like |
| Boot file / system prompt restating |
3,200 |
52.7% |
The same facts re-extracted every session. "Agent uses she/her pronouns" appeared 50+ times. "Operator prefers Telegram" appeared 200+ times. |
| Heartbeat / cron / system noise |
700 |
11.5% |
Cron outputs, heartbeat responses, NO_REPLY markers, boot sequences |
| System architecture dumps |
500 |
8.2% |
Complete system state stored as "memories." Tool configs, deployment pipelines, agent hierarchies. One entry: 4,000+ words. |
| Transient task state |
450 |
7.4% |
"Complete proposal by Friday." "Deploy blog post." Tasks that go stale within days. |
| Hallucinated user profiles |
315 |
5.2% |
Fabricated demographics, occupations, employers for people who don't exist. |
| Identity confusion |
200 |
3.3% |
Model confused agent with operator, hostnames with user names |
| Security / privacy leaks |
130 |
2.1% |
IP addresses, chat IDs, file paths, and in 2 cases sensitive configuration values that should never have reached the vector store |
| Wrong locations |
60 |
1.0% |
Fabricated cities and countries |
| Fabricated lifestyle |
50 |
0.8% |
Physical activities, daily routines fabricated for an entity with no physical body |
| Other |
165 |
2.8% |
Frozen timestamps, template placeholders, miscellaneous |
Boot-file restating dominates everything. Including Phase 0's hash duplicates and Phase 1's cosine clusters, the true total for this category is over 5,500 entries. More than half the entire collection.
The model upgrade didn't fix it
The gemma results are what you'd expect from a 2B model doing structured extraction: hallucinated user profiles, fabricated employers, a fictional character it invented independently across 6+ days. We'll come back to that.
But when we switched to Sonnet on day 21, the junk rate barely moved. The hallucinations stopped. Sonnet didn't invent fictional people. Instead, it faithfully followed the permissive extraction prompt and stored everything it could see: complete system architectures, every tool configuration, every transient task status. All accurate. None of it memory.
A better model follows the extraction prompt more faithfully, which means it extracts more indiscriminately. The extraction prompt is the bottleneck, not the model.
What we saw from #4302
We merged PR #4302 on day 29 and ran it alongside our existing setup for the remaining 4 days of the audit.
filtering.ts catches what it targets: heartbeats, cron noise, generic acks, embedded metadata. It's the right first layer.
But the junk profile in those last 4 days was the same as the preceding week. The dominant categories are extraction-layer problems that message-level filtering can't reach:
- Boot-file restating (52.7% of junk): The agent's system prompt is legitimate message content.
isNoiseMessage() can't know it's already stored in identity files.
- System architecture dumps (8.2%): Category 1 in
DEFAULT_CUSTOM_INSTRUCTIONS actively encourages these. Sonnet stored complete tool configs and deployment pipelines as "memories."
- Hallucinated profiles (5.2%): The extraction LLM fabricates demographics from conversation fragments. No message filter can catch facts the model invents.
- Feedback loop amplification: Recalled memories get re-extracted and stored as new entries. The pipeline can't distinguish recalled context from new conversation.
Exhibit A: The fictional character
The 2B model fabricated a user called "John Doe" across 6+ separate days with no shared context. Age 30-32, London or Seattle, Google engineer, mobile app developer. In one session the character was female and lived in San Francisco. None of it is true of anyone in the system. The "user" is an AI agent.
A few entries: "Software Engineer at Google in the Cloud team" (day 6). "User's name is John, 32 years old, London UK" (day 12). "Name is John Doe, Seattle, Washington, PST" (day 20). The pipeline stored each with the same confidence as a real fact.
This is a gemma-era problem, solved by upgrading the model. We include it because it reveals the missing validation: there's nothing between extraction and storage that checks whether a fact is grounded in the actual conversation.
Exhibit B: 808 copies of a hallucination
This one matters more, because it's architectural.
808 entries asserting "User prefers Vim." 191 exact copies of one sentence. Nobody in the system uses Vim. The 2B model hallucinated it once. It got stored. Next session, it appeared in recall context. The extraction model treated it as ground truth and extracted it again. The next session amplified it further.
The model caused the initial hallucination, but the pipeline caused it to multiply. There's no mechanism to distinguish recalled memories from new conversation content during extraction. Any hallucination that gets stored once will be re-extracted indefinitely. A better model prevents the initial seed, but the amplification architecture remains.
What we think is missing
Five things, in order of impact:
-
Feedback loop prevention. Mark recalled memories so the extraction step doesn't re-extract them. Would have prevented the Vim infestation and most boot-file restating.
-
A quality gate between extraction and storage. Every extracted fact goes straight to the vector store. Other frameworks (Stanford Generative Agents, LangMem, Letta) score candidates before storage.
-
Negative few-shot examples in the extraction prompt. The prompt teaches what to extract but never what to skip. High-value negatives from our data: inferred demographics, system prompt content, transient deadlines, fabricated physical attributes.
-
A REJECT action in the update-decision prompt. Currently the pipeline can ADD, UPDATE, DELETE, or NONE. There's no way to say "this fact is not worth storing." A fifth action would let the update step serve as a quality gate.
-
Identity-aware extraction. The prompt doesn't know whether the "user" is a human or an AI agent. That's why the 2B model fabricated physical demographics for a software process.
Academic context
Harvard D3's research on selective recall in LLM agents found that "indiscriminate memory storage performs worse than using no memory at all" and that "strict evaluation criteria and filtering before storage led to an average 10% performance boost." (Source)
Our experience confirms this directly. After removing the junk, recall quality improved immediately.
Where we are now
We still use mem0 because the 224 entries that survived are genuinely valuable. When the signal gets through, it's exactly what a long-running agent needs.
But we had to read 10,134 entries to find 38 clean ones. That's not a realistic path for most deployments. If you're running mem0 in production, we'd recommend pulling your vector store and looking at what's actually in it. The patterns we found are probably not unique to our setup.
We have the full audit dataset (6 correction files totaling ~100KB, per-entry categorization, research reports) and are happy to share or collaborate on the extraction pipeline.
What we found after auditing 10,134 mem0 entries: 97.8% were junk
We've been running mem0 in production for 32 days. One AI agent, one human, daily conversations, Qdrant backend. Two extraction models over that period: gemma2:2b (local, via Ollama) for the first 20 days, then Claude Sonnet 4.6 for the last 12. After noticing the agent kept "remembering" things it had never been told, we decided to stop guessing and look.
We pulled the entire collection. 10,134 entries.
The first pass was triage. We ran keyword searches, hash-duplicate detection, and targeted sweeps for obvious junk clusters. That removed 2,468 entries: exact-hash duplicates, hallucinated category clusters like "formal communication style" and "software developer at Google," and 668 copies of a single feedback-loop hallucination (more on that later). Then a cosine similarity script flagged another 2,943 entries (37.6% of what remained) as near-duplicates. We reviewed the 829 cluster survivors and kept 7.
That got us from 10,134 down to 6,264. Then we read what was left. Every entry, one by one, across eight batches. 96.9% of those were still junk.
224 entries survived the full process. Out of 10,134. That's 97.8% junk across the entire collection. And of those 224 survivors, 186 had to be deleted and rewritten from scratch because the originals were partial or malformed. Only 38 entries in the entire collection were clean enough to keep as-is.
This isn't a bug report about one failure mode. It's a dataset showing where the extraction pipeline breaks in production, measured across two extraction models and 32 days of continuous operation.
How we got here
The collection spans 32 days (Feb 23 to Mar 26, 2026). For the first 20 days, we ran gemma2:2b (a 2B parameter local model via Ollama) for extraction with default prompts. On day 21 we switched to Claude Sonnet 4.6. On day 29 we merged PR #4302 into our fork, adding
filtering.ts,isolation.ts, andDEFAULT_CUSTOM_INSTRUCTIONS.We expected the model upgrade to fix things. It didn't.
What we found
(Total exceeds 10,134 because some entries were flagged in Phase 1 dedup and reviewed again individually in Phase 2.)
Final result: 224 clean memories in the live collection. Every original entry was either deleted outright or deleted and rewritten from scratch.
Where the junk comes from
NO_REPLYmarkers, boot sequencesBoot-file restating dominates everything. Including Phase 0's hash duplicates and Phase 1's cosine clusters, the true total for this category is over 5,500 entries. More than half the entire collection.
The model upgrade didn't fix it
The gemma results are what you'd expect from a 2B model doing structured extraction: hallucinated user profiles, fabricated employers, a fictional character it invented independently across 6+ days. We'll come back to that.
But when we switched to Sonnet on day 21, the junk rate barely moved. The hallucinations stopped. Sonnet didn't invent fictional people. Instead, it faithfully followed the permissive extraction prompt and stored everything it could see: complete system architectures, every tool configuration, every transient task status. All accurate. None of it memory.
A better model follows the extraction prompt more faithfully, which means it extracts more indiscriminately. The extraction prompt is the bottleneck, not the model.
What we saw from #4302
We merged PR #4302 on day 29 and ran it alongside our existing setup for the remaining 4 days of the audit.
filtering.tscatches what it targets: heartbeats, cron noise, generic acks, embedded metadata. It's the right first layer.But the junk profile in those last 4 days was the same as the preceding week. The dominant categories are extraction-layer problems that message-level filtering can't reach:
isNoiseMessage()can't know it's already stored in identity files.DEFAULT_CUSTOM_INSTRUCTIONSactively encourages these. Sonnet stored complete tool configs and deployment pipelines as "memories."Exhibit A: The fictional character
The 2B model fabricated a user called "John Doe" across 6+ separate days with no shared context. Age 30-32, London or Seattle, Google engineer, mobile app developer. In one session the character was female and lived in San Francisco. None of it is true of anyone in the system. The "user" is an AI agent.
A few entries: "Software Engineer at Google in the Cloud team" (day 6). "User's name is John, 32 years old, London UK" (day 12). "Name is John Doe, Seattle, Washington, PST" (day 20). The pipeline stored each with the same confidence as a real fact.
This is a gemma-era problem, solved by upgrading the model. We include it because it reveals the missing validation: there's nothing between extraction and storage that checks whether a fact is grounded in the actual conversation.
Exhibit B: 808 copies of a hallucination
This one matters more, because it's architectural.
808 entries asserting "User prefers Vim." 191 exact copies of one sentence. Nobody in the system uses Vim. The 2B model hallucinated it once. It got stored. Next session, it appeared in recall context. The extraction model treated it as ground truth and extracted it again. The next session amplified it further.
The model caused the initial hallucination, but the pipeline caused it to multiply. There's no mechanism to distinguish recalled memories from new conversation content during extraction. Any hallucination that gets stored once will be re-extracted indefinitely. A better model prevents the initial seed, but the amplification architecture remains.
What we think is missing
Five things, in order of impact:
Feedback loop prevention. Mark recalled memories so the extraction step doesn't re-extract them. Would have prevented the Vim infestation and most boot-file restating.
A quality gate between extraction and storage. Every extracted fact goes straight to the vector store. Other frameworks (Stanford Generative Agents, LangMem, Letta) score candidates before storage.
Negative few-shot examples in the extraction prompt. The prompt teaches what to extract but never what to skip. High-value negatives from our data: inferred demographics, system prompt content, transient deadlines, fabricated physical attributes.
A REJECT action in the update-decision prompt. Currently the pipeline can ADD, UPDATE, DELETE, or NONE. There's no way to say "this fact is not worth storing." A fifth action would let the update step serve as a quality gate.
Identity-aware extraction. The prompt doesn't know whether the "user" is a human or an AI agent. That's why the 2B model fabricated physical demographics for a software process.
Academic context
Harvard D3's research on selective recall in LLM agents found that "indiscriminate memory storage performs worse than using no memory at all" and that "strict evaluation criteria and filtering before storage led to an average 10% performance boost." (Source)
Our experience confirms this directly. After removing the junk, recall quality improved immediately.
Where we are now
We still use mem0 because the 224 entries that survived are genuinely valuable. When the signal gets through, it's exactly what a long-running agent needs.
But we had to read 10,134 entries to find 38 clean ones. That's not a realistic path for most deployments. If you're running mem0 in production, we'd recommend pulling your vector store and looking at what's actually in it. The patterns we found are probably not unique to our setup.
We have the full audit dataset (6 correction files totaling ~100KB, per-entry categorization, research reports) and are happy to share or collaborate on the extraction pipeline.