Measured on public benchmarks. Every number published.
LLMSlim against Mem0, Zep, LLMLingua-2, LongLLMLingua, RAG and the full context, on held-out test questions. Every system is read by the same model with the same prompt, and every table shows what 1,000 questions cost end to end.
Agent memory on Core 0.12 · LLMSlim Core 0.9.0 · long documents and agents on Core 0.8.0 · LLMSlim Vault 0.9.1 · round-3 report, harness and raw results in the public repository
Chat memory
2.8×
lower cost than Mem0 on LoCoMo, with more questions right
Mem0$2.66
LLMSlim$0.95
LoCoMo, 93.0% vs 91.0% for Mem0 at about 2,500 tokens
Long histories
42×
lower cost than the whole chat history, at the same accuracy
Whole history$90.21
LLMSlim$2.13
LongMemEval, 77.8% for both at 10,000 tokens
Memory systems
212×
lower cost than Mem0 on its own LongMemEval questions, as accurate
Mem0$220.88
LLMSlim$1.04
Mem0's 60-question sample, 78.3% vs 76.7%
Compressors
4×
lower cost than LongLLMLingua on retrieval, with more questions right
LongLLMLingua$1.32
LLMSlim$0.33
NaturalQuestions, 72.7% vs 70.5% on its 1,000-question sample
Round 6 · BEAM-1M · Core 0.12 · 27 Oct 2026
Million-token agent memory. Held out, run once.
LLMSlim against Mem0 and Zep on BEAM's 1M tier. Every system is read by Claude Haiku 5.5 with BEAM's answer prompt, three times per question, and scored by BEAM's rubric judge. Costs are per 1,000 questions at list prices with BEAM's 20 questions per conversation, including Mem0's and Zep's memory building and every system's compute.
700 held-out questions, 2,500-token budget · BEAM's rubric score out of 100
BEAM-1M, against Mem0 and Zep
35 conversations of about 946,000 tokens each between a user and an assistant, and 700 questions of 10 types about them. Held out: none of it was opened while 0.12 was tuned, and every system was read and judged once. No history fits in the reader's 1M-token window, so there is no full-context line.
On par with Mem0, and 60x and 28.6x cheaper per 1,000 questions at list prices: +1.2 points at 2,500 tokens (95% CI −0.7 to +3.2) and +0.6 at 10,000 (−1.6 to +2.7). Neither difference is significant, so we do not claim to beat Mem0. Mem0 makes an LLM call for every exchange it stores; LLMSlim makes none. Ahead of Zep by 7.5 to 9.9 points, on Zep's 30 conversations, where LLMSlim scored 45.9 and 48.3.
Where it losesAgainst Mem0, summarization trails by 8.9 points at 2,500 tokens (95% CI −13.1 to −4.7) and 4.0 at 10,000 (−7.8 to −0.2), and multi-session reasoning by 6.0 at 10,000 (−10.1 to −1.8). Information extraction is 4.6 and 4.9 points behind, within noise. With the reranker on a laptop CPU, a search took 3.6 s at the median at 2,500 tokens and 4.9 s at 10,000. The ship rule was loosened before any LLMSlim result on this data was read; the original, stricter per-type bar would have failed on summarization (both budgets) and instruction following (10,000 tokens).
BEAM's rubric score out of 100 by configuration
LLMSlim 0.12 + reranker (LLMSlim)44.6
2,436 tokens$1.22 per 1k questions
Mem0, top 50 memories (other system)43.4
3,020 tokens$73.12 per 1k questions
Zep, on 30 of the 35 conversations (other system)38.4
3,397 tokens$330.07 per 1k questions
LLMSlim 0.11.1 + reranker (earlier LLMSlim)38.0
2,493 tokens$0.91 per 1k questions
10,000-token budgetLLMSlim $2.61 · Mem0 $74.53 per 1k questions
BEAM's rubric score out of 100 by configuration
LLMSlim 0.12 + reranker (LLMSlim)47.3
Mem0, top 200 memories (other system)46.7
LLMSlim 0.11.1 + reranker (earlier LLMSlim)41.6
This round is published as results only: the report has the setup, every table, the paired differences, cost, latency and limitations, but the protocol, harness and per-question answers stay private. “LLMSlim 0.12” is commit 16a2708, which became 0.12.0 after hardening and returns the same bytes on English parity checks; the round-6 contexts were not rebuilt with 0.12.0.
The question is not only how many answers a context gets right, but what each right answer costs. Costs are per 1,000 questions at list prices, including building Mem0's and Zep's memory and every system's compute.
1,307 test questions, about 2,500 tokens · accuracy (Mem0's judge)
LoCoMo
Questions about 8 long conversations between two people, each about 27k tokens over up to 35 sessions. The memory systems' own benchmark.
With its reranker LLMSlim answers 93.0%, against 91.0% for Mem0's top 50 memories, at about a third of the cost per question: most of Mem0's and Zep's cost is the model calls that build their memory. LLMSlim is on the cost frontier at every budget; Mem0, Zep, LongLLMLingua and RAG are not.
NoteBecause a conversation's questions were asked back to back, the full history was read from the prompt cache: 95.1% for $0.83 per 1,000 questions. Without the cache it costs $4.48. At 10,000 tokens LLMSlim scores 94.3% for $2.02.
accuracy (Mem0's judge) by configuration
Full history, read from the prompt cache (full context)95.1
180 test questions, about 10,000 tokens · accuracy (official judge)
LongMemEval-S
Answer a question from about 50 earlier chat sessions (111k tokens of history per question).
At 10,000 tokens LLMSlim answers as many questions as the whole history, 77.8%, for $2.13 per 1,000 instead of $90.21 (prompts over 100k tokens are billed at the higher rate). LLMLingua-2 answers 16%.
NoteDense retrieval over turns is ahead: 80.6% at 10,000 tokens and 76.1% at 2,500, at about twice the cost. Neither gap is significant on 180 questions, but both point the same way. A fix that narrowed it on development questions did not hold up on test, so it is not in 0.9.0.
accuracy (official judge) by configuration
RAG over turns (Qwen3-Embedding-0.6B) (baseline)80.6
9,912 tokens$4.45 per 1k questions
Full history (full context)77.8
111k tokens$90.21 per 1k questions
LLMSlim 0.9 (LLMSlim)77.8
10k tokens$2.13 per 1k questions
LLMSlim 0.9 + reranker (LLMSlim)74.4
10k tokens$2.21 per 1k questions
LLMLingua-2 (other system)15.6
7,809 tokens$8.43 per 1k questions
Most recent sessions (baseline)11.1
9,948 tokens$2.09 per 1k questions
About 2,500 tokens180 test questions
accuracy (official judge) by configuration
RAG over turns (baseline)76.1
LLMSlim 0.9 + reranker (LLMSlim)74.4
LLMSlim 0.9 (LLMSlim)70.6
LLMLingua-2 (other system)7.2
Mem0's 60-question sample, about 2,500 tokens · accuracy (official judge)
LongMemEval-S, against Mem0 and Zep
The same task on the pre-registered questions Mem0 and Zep were run on. Each question has its own 500-message history, so each one needs its own memory.
LLMSlim with its reranker matches the whole history and Mem0's best setting for $1.04 per 1,000 questions. Mem0 costs $220.88, almost all of it the extraction calls that build a memory of each history (about $0.22 a question). On Zep's sample, Zep answers 58.3% for $697.56 per 1,000 questions; LLMSlim answers 87.5% for $0.94.
NoteSmall samples: the 95% intervals are about ±10 points on 60 questions and ±17 on 24. Mem0's and Zep's embedding and database compute is not counted, which favours them.
accuracy (official judge) by configuration
RAG over turns (baseline)80.0
2,473 tokens$3.36 per 1k questions
Full history (full context)78.3
111k tokens$90.20 per 1k questions
LLMSlim 0.9 + reranker (LLMSlim)78.3
2,501 tokens$1.04 per 1k questions
Mem0, top 200 memories (other system)76.7
11k tokens$220.88 per 1k questions
Mem0, top 50 memories (other system)75.0
2,848 tokens$219.50 per 1k questions
LLMSlim 0.9 (LLMSlim)71.7
2,501 tokens$0.96 per 1k questions
Zep's 24-question sample24 questions
accuracy (official judge) by configuration
Full history (full context)91.7
LLMSlim 0.9 (LLMSlim)87.5
Mem0, top 50 memories (other system)87.5
Zep (other system)58.3
2,255 test questions · accuracy (answer in response)
NaturalQuestions, 20 documents
Open-domain questions with 20 retrieved passages, the answer in one of them, cut to 25% of the tokens.
With its reranker LLMSlim matches a plain reranker (73.5% against 73.9%) and is 21 points ahead of LLMLingua-2 at the same budget. On LongLLMLingua's 1,000-question sample it is level with LongLLMLingua (72.7% against 70.5%, within noise) at a quarter of its cost.
NoteThe default lexical mode scores 60.6%: paraphrased questions need semantic matching, so use the reranker for open-domain retrieval.
Multi-hop questions over 10 Wikipedia paragraphs, 2 of them relevant, cut to 40% of the tokens.
The default LLMSlim scores 52.4%, against 50.5% for the plain reranker and 46.7% for LLMLingua-2, and on LongLLMLingua's sample it is level with LongLLMLingua (53.3% against 50.4%, within noise). It has the lowest cost per correct answer of any compressor here, level with the plain reranker.
NoteFull context is still 5.6 points ahead: on a 1.3k-token context there is little to remove.
The LLMLingua family, same reader. Same questions, same budget.
We ran the best-known open-source prompt compressors ourselves, with their published settings, on the questions they were pre-registered for.
Compressor
Model it runs
NaturalQuestions (1,000)
HotpotQA (1,000)
LoCoMo (300)
Cost per 1k
Time per question
LLMSlim 0.9 + reranker
22M cross-encoder
72.7
52.8
93.3
$0.33
1.3 s on a CPU
LLMSlim 0.9
none
59.6
53.3
91.7
$0.30
18 ms on a CPU
LongLLMLingua
LLaMA-2-7B, float16
70.5
50.4
73.3
$1.32
6.9 s on a T4 GPU
LLMLingua-2
XLM-RoBERTa-large
52.0
48.9
33.0
$0.86
3.5 s on a T4 GPU
Full context
none
77.2
58.5
94.7
$0.65
LongLLMLingua runs a 7B model for about 7 seconds a question, so it ran on pre-registered samples: 1,000 NaturalQuestions and HotpotQA questions and 300 LoCoMo questions, at about 2,500 tokens. Every system is scored here on those same questions. Against LongLLMLingua, LLMSlim with its reranker is level on NaturalQuestions (+2.2 points, 95% interval −0.8 to +5.3) and ahead on LoCoMo (+20.0, 15.0 to 25.0); the default LLMSlim is level on HotpotQA (+2.9, −0.1 to +5.7). Scores are accuracy, exact match on HotpotQA and Mem0's judge on LoCoMo. Cost is per 1,000 NaturalQuestions questions, end to end. Time is the median to compress one NaturalQuestions question.
Summarization and multi-session reasoning on BEAM-1M. Against Mem0, LLMSlim 0.12 trails on summarization by 8.9 points at 2,500 tokens (95% CI −13.1 to −4.7) and 4.0 at 10,000 (−7.8 to −0.2), and on multi-session reasoning at 10,000 tokens by 6.0 (−10.1 to −1.8). Information extraction is also behind (4.6 and 4.9 points), within noise.
02
Search time with the reranker on BEAM-1M. With the reranker on a laptop CPU, a search over a 946,000-token history took 3.6 s at the median at 2,500 tokens and 4.9 s at 10,000. The default lexical mode takes about 45 ms on a 100,000-token history, but it was not scored in that round.
03
A relaxed ship rule on BEAM-1M. The ship rule was loosened before any LLMSlim result on this data was read. The original, stricter per-type bar, which also required the lower bounds for instruction following and summarization to stay within 5 points of Mem0, would have failed on summarization (both budgets) and instruction following (10,000 tokens). The report gives both verdicts.
04
Open-domain retrieval without the reranker. On NaturalQuestions the default lexical mode scores 60.6%, against 73.9% for a plain reranker. Paraphrased questions need semantic matching. With its optional reranker LLMSlim scores 73.5%.
05
Dense retrieval on LongMemEval. RAG over turns scores 80.6% to LLMSlim's 77.8% at 10,000 tokens, and 76.1% to 74.4% at 2,500, at about twice the cost. Neither gap is significant on 180 questions, but both point the same way. A fix that narrowed it on development questions did not hold up on test, so it was not shipped.
06
Multi-hop and temporal questions at 1,000 tokens. Overall, Mem0's top 20 memories and LLMSlim are within noise (88.7% and 87.5%), but Mem0 is clearly better on LoCoMo's multi-hop (88.7% against 80.3%) and temporal (93.4% against 85.7%) questions, where its extracted facts pack evidence from many sessions into few tokens. With the reranker LLMSlim is ahead overall (90.5%).
07
A cached full history. When a conversation's questions come back to back, the cached full history (95.1% for $0.83 per 1,000 questions) is cheaper than LLMSlim at 10,000 tokens and more accurate than any compressed context. LLMSlim is the cheaper choice when the context changes between questions, when questions arrive too far apart for the cache, or when a history is long enough to be billed at the long-prompt rate.
Long documents · Core 0.8.0 · 12 Oct 2026
Long documents. Against full context and truncation.
From the previous round, with fresh questions run once after the code was frozen. 0.9.0 changes only how chat transcripts are selected. Costs here are the answer call alone.
210 samples, 30 per task · macro average of official metrics
LongBench v1
Seven long-context tasks: document QA, multi-hop QA, summarisation, few-shot QA and code completion (15.5k tokens on average).
The same macro score as full context (95% CI −4.5 to +4.6) on a third of the tokens, and +9.8 over truncation. On the original sample RepoBench-P (cross-file code, 30 samples) scored above full context, 57.2 against 44.3; it did not on the fresh sample.
NoteMuSiQue multi-hop QA is still below full context (58.8 against 64.9; 61.2 with the reranker), though most of the 0.8 pre-release drop (48.3) is recovered.
macro average of official metrics by configuration
Full context (full context)54.9
16k tokens$1.81 per 1k questions
LLMSlim Core 0.8 (LLMSlim)54.9
5,232 tokens$0.78 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)54.2
5,229 tokens$0.78 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)52.4
5,164 tokens$0.78 per 1k questions
Middle truncation, same budget (baseline)45.2
5,147 tokens$0.79 per 1k questions
Fresh questions, never used in development210 new samples
macro average of official metrics by configuration
Full context (full context)55.1
LLMSlim Core 0.8 + reranker (LLMSlim)55.4
LLMSlim Core 0.8 (LLMSlim)55.0
Middle truncation, same budget (baseline)46.8
180 questions · accuracy
LongBench v2 (short split)
Hard multiple-choice questions over long documents, codebases and structured data (30k tokens on average).
72% fewer tokens and 78% lower cost. With the reranker, 4.4 points below full context (95% CI −11.7 to +2.8), and the best compressed arm.
NoteQuestions that need the whole document lose accuracy: hard multiple choice over 30k tokens is where compression costs the most.
accuracy by configuration
Full context (full context)60.0
30k tokens$7.62 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)55.6
8,469 tokens$1.69 per 1k questions
LLMSlim Core 0.8 (LLMSlim)53.9
8,470 tokens$1.69 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)51.1
8,566 tokens$1.90 per 1k questions
Middle truncation, same budget (baseline)48.3
8,566 tokens$1.94 per 1k questions
Agents · Core 0.8.0
Where compression does not help. Yet.
SWE-bench Verified Mini
50 real GitHub issues (Django, Sphinx); a bash-tool agent edits the repository; graded by the official test harness.
Core 0.8 matches the best uncompressed run with a cost per task between the two: no harm, little saving. A careful agent keeps its context near 8k tokens, so there is not much to remove. Published work found LLMLingua-2 on tool output cost an agent 8 points.
Agent, no compression (two runs)
36 and 38 of 5058k and 48k prompt tokens per task
LLMSlim Core 0.8 on outputs over 800 tokens
38 of 5028 outputs compressed; 51k prompt tokens per task
LLMSlim Core 0.7.2, same setting
34 of 5030 outputs compressed; 59k prompt tokens per task
InjecAgent (enhanced attacks)
1,054 cases where a tool result carries an attacker instruction; success means the agent calls the attacker's tool.
The reader model resisted every attack on its own, so this benchmark shows no defence effect with this model. It did expose an agent-loop gap, now fixed: before agent turns, Vault put the user's request after the tool results and the model repeated its call 50 times. With continue_turn, the turn is shown in the order it happened and no call is repeated.
Native tool-use history
0 of 1,054attack success 0.0%; 1 repeated call
Vault 0.9.0 flow (request after the tool results)
0 of 1,054attack success 0.0%; 50 repeated calls
Vault 0.9.1 agent turn (continue_turn)
0 of 1,054attack success 0.0%; 0 repeated calls
Other systems
Their own published results. With their readers.
These numbers come from each project's paper or a third-party re-run, with different reader models. Compare each system's gap to its own full-context baseline, not raw scores across models.
Same model. Same questions. Only the context changes.
01
Reader: Claude Haiku 5.5 at effort low for every system, with the same answer prompt (Mem0's own, vendored verbatim) and the same judges. Only the context changes.
02
Cost first. Every table prices 1,000 questions end to end at list prices: the reader's answer call, Mem0's and Zep's ingestion calls, and each system's build time as cloud machine time (a T4 GPU at $0.526 an hour, or 2 vCPUs at $0.0892). Judge calls are not counted for anyone, and neither is Mem0's and Zep's embedding and database compute, which favours them.
03
Held-out test questions. The splits were committed before any result. LLMSlim was tuned on development questions only, and the test questions were read once per version. A change that helped on development questions but not on test was not shipped.
04
Judges: LoCoMo uses Mem0's judge, from Mem0's own harness; LongMemEval uses the official LongMemEval judge prompts. The full report also gives Mem0's more lenient judge on LongMemEval, and LoCoMo's token F1.
05
Competitors ran their own way: Mem0 OSS 2.2.1 through Mem0's benchmark harness, Zep (Graphiti 0.30.2 on Neo4j) through Zep's, and LLMLingua-2 and LongLLMLingua (LLaMA-2-7B in float16, the paper's settings) at the same token budget. Every Mem0 and Zep model call went to Haiku 5.5 through one metering proxy.
06
Every compressor's context was built in public Kaggle notebooks pinned to a commit, whose manifests hash every input and output, so anyone can rerun them and compare byte for byte.
07
Confidence intervals are paired bootstrap 95% over the same questions.
08
The optional reranker is cross-encoder/ms-marco-MiniLM-L-6-v2 (22M parameters). Install it with pip install "llmslim[semantic]" and pass relevance="semantic". Without it, LLMSlim runs no model at all.
09
LongBench v1 and v2 and the agent results were measured on Core 0.8.0 (12 Oct 2026). 0.9.0 changes only how plain-text chat transcripts are selected.
10
Round 6 (BEAM-1M, agent memory) is scored differently: BEAM's own answer prompt and rubric judge, both on Haiku 5.5, three reads per question, the mean rubric score out of 100, and paired t intervals over conversations. Its costs count Mem0's and Zep's embedding compute too, priced on the device each store was built on. It measured commit 16a2708, which became 0.12.0 after hardening; the round-6 contexts were not rebuilt with 0.12.0. It is published as results only: the protocol, harness and per-question answers stay private.
Mem0
Zep
LLMLingua-2