{"version":"https:\/\/jsonfeed.org\/version\/1","title":"Mixedbread Blog","home_page_url":"https:\/\/www.mixedbread.com\/blog","feed_url":"https:\/\/www.mixedbread.com\/blog\/feed.json","description":"Latest news, updates, and insights from Mixedbread","icon":"https:\/\/www.mixedbread.com\/images\/og\/root.jpg","author":{"name":"Mixedbread Team","url":"https:\/\/www.mixedbread.com"},"items":[{"id":"https:\/\/www.mixedbread.com\/blog\/latent-terms","content_html":"

When we think of neural retrieval, the first assumption that comes to mind is that the role of a model is to represent information in a way that allows it to be retrieved by a related query. But this representation is overly simplistic: the role of a retrieval model is to represent information in a way that can be used by a scoring mechanism<\/strong>. All retrieval work, in essence, is constrained by the operator we choose to rank documents, and all model expressivity is bottlenecked by what this operator can capture.<\/p>\n

This raises one, important question: do retrieval models know more than they are able to express<\/strong>? In other words, despite scoring limitations, does the retrieval training process allow a model to learn richer representations than we assume, waiting to be extracted? The answer is yes<\/strong>, and even more than that: these representations are trivial to (at least partially) extract, and their distribution approaches natural language itself.<\/p>\n

Retrieval: A Song of Representations and Scoring<\/h2>\n

Single-vector embedding models are not a good approach to retrieval, because their inherent<\/a> limitations<\/a> make them unsuitable for a lot of situations that are all too common in the real world.<\/p>\n

This is a conclusion that is increasingly apparent in all domains, with multi-vector models vastly outperforming their single-vector equivalent with an order of magnitude fewer parameters in both agentic and multimodal settings, two areas which are rapidly growing in importance.<\/p>\n

However, while it is tempting to write another article denouncing the evils of single-vector retrieval, it is much more interesting to think about why<\/em> this is the case. At surface-level, it might appear obvious: well, it's a single vector representing multi-faceted information, so the meaning is diluted!<\/p>\n

This assumption is not incorrect: representing nuanced information in one vector is bound to create strong dilution. But it is incomplete: what is diluted is not necessarily the representation itself<\/strong>: after all, LLMs are perfectly capable to create complex task vectors encompassing nuances. What single-vector limits, and what makes it so harmful to generalisation, is the expressiveness of scoring operators<\/strong>.<\/p>\n

We can think about it this way: all (first stage) retrieval operations are effectively about enabling the use of an efficient scoring mechanism<\/strong> that can produce useful relevance rankings, to be further enhanced by a reranker or directly used by the final consumer, whether human or agentic. The role of the embedding model is not to perform this scoring by itself: it would be far too expensive to use a neural model, even if it had just a handful of parameters, to score millions of document at query time.<\/p>\n

Instead, as we discussed in our introduction, the role of the embedding model is to convert the information in the documents into a format that can be readily consumed by the efficient scoring mechanism<\/strong> we discussed above. Thus, training embedding is a pure "representation learning" task, where we are provided with a constraint, the input format of the downstream former, and must learn representations that best enable it. In fact, the three great family of retrievers are shaped by their scoring operators: single-vector dense retrievers and their cosine similarity operator, sparse retrievers and various forms of weighted dot products, and multi-vector models and their MaxSim operator.<\/p>\n

The reason late interaction models, such as Wholembedv3<\/a>, are so powerful is because of this operator: MaxSim allows for a level of fine-grained expressiveness in scoring that is simply not possible with single-vector cosine similarity. Late interaction is about preserving this expressivity<\/strong> by allowing information from both the documents and the query to interact as late in the scoring process as possible (hence the "late" interaction naming).<\/p>\n

In fact, a few people in the ColBERT community sometimes rant that they don't like the term \"multi-vector retrieval\" very much<\/a>. The reason for this is this simple fact: with our current understanding of retrieval, multi-vector approaches are currently required to enable MaxSim, which itself is the scoring operator that is currently required to enable late interaction retrieval. By no means is it the "perfect" operator, but it is a necessary evil to allow expressive scoring, until we develop better ways to do so.<\/p>\n

And just like ColBERT is powerful because its scoring operator maximises expressiveness at the cost of additional engineering constraints, single-vector embeddings are brittle because their scoring operator favours engineering flexibility, with the opposite tradeoffs. But as we discussed: this is not an inherent feature of the models themselves<\/strong>, but of the way their final representations are cast to be made compatible with this operator.<\/p>\n

As such, we find ourselves asking the same question that we did at the start of this article: do dense models contain more information than they are able to express, and, more importantly, can we extract this knowledge? <\/p>\n

If you have read this far, you are probably assuming that the reason we are asking this question is because we found out that, Yes<\/strong>, you can. And even more than that: it's trivial to do so.<\/p>\n

Extracting a model's vocabulary<\/h2>\n\n\n

To an extent, we know that it is very simple to train dense embedding models to become strong multi-vector retrievers, and this makes sense, as MaxSim scoring can be thought of as "cosine-similarity++" and would leverage a great deal of the same information.<\/p>\n

But circling back to our previous point, this is an imperfect solution: yes, MaxSim and late interaction is very powerful. But it's also heavy, requires solid engineering chops, and more importantly, while it is technically a first-stage retrieval approach, it still needs its own "pre-first-stage" candidate generation stage in order for the MaxSim compute cost to be bearable.<\/p>\n

But even more importantly in this context: this conversion is performed with retrieval-specific supervised training<\/strong>, making it less of an extraction and more of an adaptation: after all, you could make the case that ModernBERT itself contains retrieval information, because if you train it with retrieval supervision, it becomes a capable retriever. Not untrue, but missing the point.<\/p>\n

No, instead, we want to see what information a model contains<\/strong>, without further training.<\/p>\n

Sparse AutoEncoders<\/h3>\n

And to do so, we turn towards the field of interpretability. Specifically, we look into Sparse AutoEncoders, or SAEs. These models have become commonplace in studies trying to understand the "black box" of language models, coming to prominence spearheaded by Anthropic<\/a> and now found all over both research and industry.<\/p>\n

At their core, SAEs could not be simpler, as they're built on concepts that will be familiar to the vast majority of machine learning enthusiasts: they are shallow networks composed of single encoder-decoder block. Training follows a simple reconstruction objective: the decoder has to reconstruct the original input after it has been projected through the encoder.<\/p>\n\n\n

What makes SAEs interesting is the constraint that is added to the encoder: a sparsity penalty<\/strong> is added to ensure that each input feature is only able to activate a limited number of features in the encoder's latent space<\/strong>. The intermediate representation between the encoder and the decoder is effectively a large sparse vector, where the vast majority of dimensions have a value of 0.<\/p>\n

The theory, that empirical results support to good extent, is that doing so yields some sort of "latent vocabulary", in which we can analyse which tokens activate which features and better understand how information is represented within the large, dense, and otherwise uninterpretable internal activations of LLMs. This field of research is what led to the (in)famous Golden Gate Claude experiment<\/a>, that we all dearly miss.<\/p>\n

Being a retrieval-focused company, this made us think: if SAEs are capable of extracting a latent vocabulary that can more-or-less clearly mapped to concepts, could this vocabulary also be useful for retrieval? And so, we designed a simple set of experiments: train SAEs on top of common retrievers, both internal and external, and explore the makeup of the resulting features.<\/p>\n

The Latent Space is Zipfian<\/h2>\n

Before we say more about the nature of the representations we extracted, it's important to take a step back, and discuss Zipf's Law. Indeed, one of the core underlying aspects of most classical lexical approaches to NLP is the fact that natural language tends to follow a Zipfian distribution.<\/p>\n

A picture is worth a thousand words, so let's first look at what a Zipfian looks like before we delve (ChatGPT did not write this, but we will not let it claim a good word) further:<\/p>\n\n\n

Zipf's law is a simple empirical observation: when you gather a set of observations and sort them in decreasing order, the distribution is often such that the value of the n<\/em>th entry is inversely proportional to n<\/em>. In practice, what this means is that the third most common element will occur about half as often as the second most common one, which itself will be about half as common as the first.<\/p>\n

Most human languages naturally tend towards a quasi-Zipfian distribution: while they don't follow a perfect Zipf's curve, they all roughly espouse its shape. And as we know by now, humans are very good at optimisation: if a distribution is known and has well-defined properties, we will figure out how to take advantage of this.<\/p>\n

And we did: for a long time, tools were specifically designed around this. The most famous of such approaches in retrieval is BM25, designed around TF-IDF features (Term Frequency - Inverse Document Frequency, essentially a way to give more weight to more discriminative terms) with a few additional tweaks and parameters to tune. In fact, BM25 is a fantastic example of our drive to optimise, in two ways: first, despite having been introduced in 1995, it remains today the pareto-optimal way to do retrieval with lexical features. Secondly, the name itself is a throwback to the necessity of iterations to optimise: BM25 stands for B<\/strong>est M<\/strong>atch 25<\/strong>, and the 25<\/strong> simply refers to the fact that it was the 25th method the team had designed.<\/p>\n

Mainstream neural sparse methods, such as SPLADE, have largely done away with these assumptions: their training methods leads to a smoother curve, with fewer saturated features. But as it turns out, Latent Terms<\/em>, the activated features we extracted through an SAE applied over retrievers, are distributed in a Zipfian way over large corpuses<\/strong>:<\/p>\n\n\n

In practice, this means that the vocabulary we extract via SAEs, which we call Latent Terms<\/em>, follow a distribution that is broadly similar to that of human language<\/strong>. This is unlike current sparse retrieval methods, which are explicitly trained with methods to induce sparsity that result in the absence of the characteristic saturated top of the curve that is present in natural words. And as you've probably guessed it, having a discrete vocabulary with distribution similar to that of lexical terms means that it this vocabulary is readily usable with methods designed for lexical terms<\/strong>.<\/p>\n

What does that vocabulary look like?<\/h3>\n

But before we get into its suitability for retrieval methods like BM25, let us first take a look at the vocabulary itself: what exactly does it capture? What do its features look like?<\/p>\n

Thankfully, the vocabulary is rather small, with most of our experiments targeting a vocabulary size in the {8192, 65536} range. This enables qualitative analysis, that is then easy to scale up with the use of LLMs. <\/p>\n

What this activation analysis revealed is that three broad, unevenly-distributed categories are present in the identified features: Lexical Features<\/strong>, which fire on a single term, Narrow Semantic Features<\/strong>, which capture multiple ways of referring to the same concept, and Broad Topical Features<\/strong>, activated by a wide range of terms around a similar topic.<\/p>\n\n\n

The distribution between these 3 types differs somewhat between models, but the general trend holds: roughly 10% of the features are narrow semantic, around a third is purely lexical, and the rest, comprising over half the features, are broad topical ones. <\/p>\n

Not only does it overcome failure cases, it makes for very strong retrievers<\/h2>\n

While the shape of the features is encouraging, it's time to put them to the real test: do Latent Terms<\/em> simply have a Zipfian distribution, but ultimately do not carry meaningful enough signal, or are they effective retrieval features? <\/p>\n

A (truncated) table is worth a thousand words:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
Method<\/th>\nSciFact<\/th>\nNFC<\/th>\nFiQA<\/th>\nTREC-Covid<\/th>\nDBPedia<\/th>\nNQ<\/th>\nHotpotQA<\/th>\nFEVER<\/th>\n<\/tr>\n<\/thead>\n
Lexical BM25<\/td>\n0.686<\/td>\n0.319<\/td>\n0.249<\/td>\n0.680<\/td>\n0.300<\/td>\n0.285<\/td>\n0.569<\/td>\n0.481<\/td>\n<\/tr>\n
SPLADE-v3<\/td>\n0.710<\/td>\n0.357<\/td>\n0.374<\/td>\n0.748<\/td>\n0.450<\/td>\n0.586<\/td>\n0.692<\/td>\n0.796<\/td>\n<\/tr>\n
Contriever<\/td>\n0.655<\/td>\n0.313<\/td>\n0.274<\/td>\n0.448<\/td>\n0.377<\/td>\n0.419<\/td>\n0.542<\/td>\n0.581<\/td>\n<\/tr>\n
Nomic<\/td>\n0.703<\/td>\n0.346<\/td>\n0.377<\/td>\n0.822<\/td>\n0.431<\/td>\n0.598<\/td>\n0.672<\/td>\n0.813<\/td>\n<\/tr>\n
GTE-MC<\/td>\n0.756<\/td>\n0.381<\/td>\n0.456<\/td>\n0.849<\/td>\n0.475<\/td>\n0.617<\/td>\n0.773<\/td>\n0.875<\/td>\n<\/tr>\n
Latent Terms + Contriever<\/strong><\/td>\n0.713<\/td>\n0.340<\/td>\n0.317<\/td>\n0.709<\/td>\n0.409<\/td>\n0.468<\/td>\n0.627<\/td>\n0.751<\/td>\n<\/tr>\n
Latent Terms + Nomic<\/strong><\/td>\n0.749<\/td>\n0.372<\/td>\n0.382<\/td>\n0.783<\/td>\n0.436<\/td>\n0.577<\/td>\n0.732<\/td>\n0.885<\/td>\n<\/tr>\n
Latent Terms + GTE-ModernColBERT<\/strong><\/td>\n0.730<\/td>\n0.374<\/td>\n0.399<\/td>\n0.759<\/td>\n0.387<\/td>\n0.509<\/td>\n0.653<\/td>\n0.814<\/td>\n<\/tr>\n<\/tbody><\/table>\n

In summary: not only are Latent Terms<\/em> extracted with SAEs compatible with BM25, but they are able to achieve very competitive retrieval results, matching or outperforming their single-vector backbone, whether the backbone is an older, weaker one (Contriever) or a more modern model (in this case, nomic-embed-text-v1.5). The overall performance does appear pretty strongly correlated with retrieval training, as the better text embedding also produces the superior Latent Terms<\/em> results.<\/p>\n

Perhaps even more interestingly, the approach is competitive with SPLADE models from a similar era: SAE+BM25 over Nomic outperforms SPLADEv3, despite the latter's heavy use of knowledge distillation from much more powerful model.<\/p>\n

Finally, the final insight immediately apparent from the table is, again, that scoring operators do matter immensely: while single-vector models lose to their Latent Terms<\/em> cousin, GTE-ModernColBERT, a late interaction model using the powerful MaxSim operator, comfortably outperforms its counterpart, despite it remaining strong.<\/p>\n

But there is another benchmark we are interested in here: LIMIT. LIMIT, which we talked about previously, is a toy task: queries are very straightforward, and documents are essentially a long list of a person's attributes. It is purposefully designed to be trivial for approaches capturing fine-grained information in their scoring: "normal" BM25's Recall@20 is in the high 90s, and so is GTE-ModernColBERT's. However, its very simple formulation is antagonistic to the limitations of single-vector scoring, and even large, 8 billion parameter single-vector models fail to reach double-digits recall numbers.\nAs such, the question is pretty clear: can Latent Terms<\/em>, despite being built upon the same single-vector model, recover LIMIT performance far beyond the single-vector setting scoring limitation?<\/p>\n\n\n\n\n\n\n\n\n\n
Method<\/th>\nRecall (R)@10<\/th>\nR@20<\/th>\nR@100<\/th>\nR@1000<\/th>\n<\/tr>\n<\/thead>\n
Lexical BM25<\/td>\n0.9440<\/td>\n0.9490<\/td>\n0.9645<\/td>\n0.9945<\/td>\n<\/tr>\n
SPLADE-v3<\/td>\n0.5760<\/td>\n0.6650<\/td>\n0.8095<\/td>\n0.9440<\/td>\n<\/tr>\n
GTE-ModernColBERT<\/td>\n0.8430<\/td>\n0.8565<\/td>\n0.8720<\/td>\n0.8795<\/td>\n<\/tr>\n
Contriever<\/td>\n0.0210<\/td>\n0.0265<\/td>\n0.0530<\/td>\n0.1250<\/td>\n<\/tr>\n
Latent Terms + GTE-ModernColBERT<\/td>\n0.7985<\/td>\n0.8315<\/td>\n0.8915<\/td>\n0.9775<\/td>\n<\/tr>\n
Latent Terms + Contriever<\/td>\n0.4140<\/td>\n0.5100<\/td>\n0.7295<\/td>\n0.9290<\/td>\n<\/tr>\n<\/tbody><\/table>\n

The answer is yes<\/strong>, although it is not perfect: while Contriever as a single-vector model reaches a Recall@100 of just 0.053, its Latent Terms variant hits 0.729. This confirms that even though it has been trained only in a single-vector setting the model itself has learned information allowing to avoid collapse on LIMIT beyond that that can be expressed in this same setting<\/strong>. As this avenue of research is still very young and primitive, this suggests that our current training methods teach models significant, meaningful signal that is just waiting for us to develop better ways to extract.<\/p>\n

Is this inherent to retrievers or do all language models contain a sparse vocab eagerly waiting to meet BM25?<\/h2>\n

But first, we need to think about where<\/strong> this information comes from. Earlier, we talked about the core goal of sparse auto-encoders in existing research: given a set of neural activations from a complex, highly-uninterpretable language model, cast them into a sparse set of activations that can be studied to understand how these activations related to language, thus better understanding models.<\/p>\n

Then, we demonstrated that in the case of retrievers, these sparse activations approximate a natural language distribution, and that we identify three "families" of features capturing different levels of lexical and semantic information. The combination of these factors enable methods designed for natural language, in this case BM25, to work extremely well and result in strong retrieval performance.<\/p>\n

With these two factors in mind, there's one question that naturally follows: does this retrieval-friendly Latent Terms<\/em> structure naturally emerge in the representations of encoder models, or are these extractable, meaningful features learned as a byproduct of retrieval-focused contrastive training?<\/p>\n\n\n\n\n\n
Method<\/th>\nSciFact<\/th>\nNFCorpus<\/th>\nFiQA<\/th>\nTREC-Covid<\/th>\nDBPedia<\/th>\nNQ<\/th>\nHotpotQA<\/th>\n<\/tr>\n<\/thead>\n
Latent Terms + BERT<\/td>\n0.585<\/td>\n0.216<\/td>\n0.131<\/td>\n0.212<\/td>\n0.134<\/td>\n0.165<\/td>\n0.345<\/td>\n<\/tr>\n
Latent terms + Contriever<\/td>\n0.713<\/em><\/td>\n0.340<\/em><\/td>\n0.317<\/em><\/td>\n0.709<\/em><\/td>\n0.409<\/em><\/td>\n0.468<\/em><\/td>\n0.627<\/em><\/td>\n<\/tr>\n<\/tbody><\/table>\n

The answer, quite clearly, is the latter: features extracted by SAEs over pre-trained language models do not contain the structure that enables Latent Terms<\/em> to act as strong retrieval discriminators. This confirms one of our intuitions: the SAE process is not, by itself, creating structure or information that is useful for retrieval. However, it is a straightforward way to expose information that the model learns about what makes a given term (or token, in our case) in a document impact its relevance to a query, or vice-versa, in ways that the model can fail to express in a single-vector representation. This finding is also supported by the clear jump in quality between a weaker backbone (Contriever) and a stronger one (Nomic).<\/p>\n

This opens up quite a lot of interesting questions: is this the best way to extract this information? Should we be figuring out training methods that are not so post-hoc, so that this information is extracted in an ever better way? Is the future of retrieval sparser than we have been led to believe?<\/strong><\/p>\n

Where can I learn more, and What's Next?<\/h2>\n

If you want to dig more onto the scientific aspect of this approach, our new preprint is now up on arXiv<\/a>.<\/p>\n

As for what's next, this paper is the first of a series of findings that we intend to publish about this line of work. Characteristically, we expect these to be sparse, but you should stay tuned to find out more about what the vector space of retrievers is hiding. If this kind of research resonates with you, you should definitely get in touch<\/a> and share your excitement.<\/p>\n

Citation<\/h3>\n

If you'd like to formally reference this work, please cite the associated paper:<\/p>\n

@misc{latentterms,\n      title={Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies}, \n      author={Benjamin Clavi\u00e9 and Sean Lee and Aamir Shakir and Makoto P. Kato},\n      year={2026},\n      eprint={2605.29384},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https:\/\/arxiv.org\/abs\/2605.29384}, \n}\n<\/code><\/pre>\n","url":"https:\/\/www.mixedbread.com\/blog\/latent-terms","title":"Dense Retrievers Know More Than They Can Express","summary":"Demonstrating that dense retrieval models learn much more information than they can express through their usual scoring mechanism: they also contain an indexable, natural-language-like sparse vocabulary, which is plug-and-play with BM25.","image":"https:\/\/www.mixedbread.com\/images\/blog\/latent-terms\/latent-terms.jpg","date_modified":"2026-06-02T00:00:00.000Z","author":{"name":"Mixedbread Team","url":"https:\/\/www.mixedbread.com"},"tags":["research"]},{"id":"https:\/\/www.mixedbread.com\/blog\/listwise-rerank","content_html":"
When we released Wholembed v3<\/a>, the retrieval quality bar moved up sharply enough that most rerankers stopped helping. Pointwise rerankers, which score documents independently and sort by score, struggled to add value on top of strong first-stage results. On harder corpora, they actively hurt them.<\/p>\n
Today we are releasing mxbai-rerank-v3-listwise<\/strong>, a listwise reranking model co-designed with Wholembed v3. It is the first reranker we have shipped that improves results on every domain, language, and benchmark we tested, lifting all 56 Vidore v3 runs, +11% NDCG@10<\/strong> on average. It also brings strong instruction-following capabilities. You can steer ranking with natural-language directives, like prioritizing recent documents, preferring internal sources, or resolving conflicts between knowledge bases.<\/p>\n
mxbai-rerank-v3-listwise is available in preview today as part of Mixedbread Search.<\/p>\n
Better Reranking for Stronger Retrieval<\/h2>\n
On ViDoRe v3, a benchmark for real-world document retrieval, Wholembed v3 alone averaged 0.603 NDCG@10.[^1] Adding mxbai-rerank-v3-listwise lifted every one of the 56 runs, with an average gain of 11%.<\/p>\n
\n  \n  \n<\/div>\n\n\n\n\n\n\n
Setting<\/th>\nNDCG@10 (avg, 56 runs)<\/th>\n\u0394 vs. retrieval-only<\/th>\n<\/tr>\n<\/thead>\n
Wholembed v3 only<\/td>\n0.603<\/td>\n-<\/td>\n<\/tr>\n
+ mxbai-rerank-v3-listwise<\/td>\n0.669<\/td>\n+10.92%<\/td>\n<\/tr>\n<\/tbody><\/table>\n
[^1]: We use ViDoRe v3 because it is a widely used benchmark for real-world document retrieval over complex corpora, with multilingual and domain-specific subsets. The benchmark covers seven domains (computer science, energy, finance, HR, industrial, pharmaceuticals, physics) across seven languages, for 56 paired runs in total.<\/p>\n
The biggest gains came on the harder, lower-baseline subsets: industrial documents in German went up 18.8%, HR in French 16.3%. These are the kinds of corpora where small ranking errors compound into worse downstream answers.<\/p>\n
Comparative Ranking with Instruction Following<\/h2>\n
mxbai-rerank-v3-listwise is a listwise reranker. Unlike pointwise rerankers, which focus on the relevance of individual documents, it reads the candidate set as a whole and can resolve conflicts between candidates.<\/p>\n
With strong instruction-following capabilities, you can tell it to prefer newer documents, prioritize internal sources over external summaries, or favor primary sources over commentary.<\/p>\n
A booking confirmation may be superseded by a later cancellation. A product spec may override a launch note. A financial filing may be more authoritative than a same-day article.<\/p>\n
We benchmarked mxbai-rerank-v3-listwise against the strongest pointwise rerankers available on a 900-example instruction-following evaluation covering recency-aware ranking, source-priority resolution, and multi-step composite instructions.[^2]<\/p>\n\n\n\n\n\n\n\n
Reranker<\/th>\nMRR<\/th>\nAccuracy@1<\/th>\n<\/tr>\n<\/thead>\n
mxbai-rerank-v3-listwise<\/strong><\/td>\n0.93<\/strong><\/td>\n88.6%<\/strong><\/td>\n<\/tr>\n
Voyage rerank-2.5<\/td>\n0.84<\/td>\n77.4%<\/td>\n<\/tr>\n
Cohere Rerank 4 Pro<\/td>\n0.77<\/td>\n68.4%<\/td>\n<\/tr>\n
ZeroEntropy zerank-2<\/td>\n0.71<\/td>\n60.3%<\/td>\n<\/tr>\n<\/tbody><\/table>\n
[^2]: The evaluation is derived from real user corpora and search patterns, then converted into controlled ranking tasks. Examples are designed to test whether the reranker can resolve conflicts inside a candidate set, not just match query-document relevance.<\/p>\n
Much of the gap comes from recency-aware ranking, where the model has to understand that a March schedule change supersedes a January booking confirmation, or that Q1 FY27 guidance supersedes Q4 FY26 guidance.<\/p>\n
How Listwise Changes the Answer<\/h2>\n
To make the difference concrete, here are five cases where pointwise and listwise reranking diverge under an explicit instruction.<\/p>\n
Email:<\/strong>\nPointwise scoring rewards the original BA confirmation: it has the highest "BA flight to London" keyword density and looks unsuperseded on its own. Listwise reads the inbox as a chain (confirmation, reschedule, cancellation, then a new booking on a different carrier) and surfaces the United flight.<\/p>\n
\n  
\n    
\n      Query<\/span>\n      \"What's the current status of my flight to London?\"<\/span>\n    <\/div>\n    
\n      Instruction<\/span>\n      Resolve booking changes chronologically; the most recent valid booking wins.<\/span>\n    <\/div>\n  <\/div>\n\n  
\n    
\n      
Pointwise (before)<\/div>\n      
\n        
\n          1<\/span>\n          BA286 SFO\u2192LHR booking confirmation<\/span>\n          Jan 12<\/span>\n        <\/div>\n        
\n          2<\/span>\n          BA286 schedule change \u2192 21:30<\/span>\n          Feb 3<\/span>\n        <\/div>\n        
\n          3<\/span>\n          UA901 SFO\u2192LHR booking confirmation<\/span>\n          Feb 22<\/span>\n        <\/div>\n        
\n          4<\/span>\n          BA286 cancelled, refund processed<\/span>\n          Feb 20<\/span>\n        <\/div>\n        
\n          5<\/span>\n          Hopper: cheap flights to London<\/span>\n          Mar 4<\/span>\n        <\/div>\n      <\/div>\n    <\/div>\n\n
<div>\n  <div class="text-[10px] uppercase tracking-wider text-zinc-500 mb-3">Listwise (after)<\/div>\n  <div class="space-y-2">\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">1<\/span>\n      <span class="text-sm flex-1 font-medium text-white">UA901 SFO\u2192LHR booking confirmation<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">Feb 22<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">2<\/span>\n      <span class="text-sm flex-1">BA286 cancelled, refund processed<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Feb 20<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">3<\/span>\n      <span class="text-sm flex-1">BA286 schedule change \u2192 21:30<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Feb 3<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">4<\/span>\n      <span class="text-sm flex-1">BA286 SFO\u2192LHR booking confirmation<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Jan 12<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">5<\/span>\n      <span class="text-sm flex-1">Hopper: cheap flights to London<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Mar 4<\/span>\n    <\/div>\n  <\/div>\n<\/div>\n<\/code><\/pre>\n  <\/div>\n<\/div>\n\n
Finance:<\/strong>\nThe Q3 FY26 call is the same speaker, same format, and same topic match, a strong pointwise signal, but the guidance it issues has been superseded. The Morgan Stanley note is more recent but is analyst commentary, not company guidance. Listwise routes around both and lands on the Q4 FY26 call as the current primary source.<\/p>\n
\n  
\n    
\n      Query<\/span>\n      \"What is NVDA's most recent forward revenue guidance?\"<\/span>\n    <\/div>\n    
\n      Instruction<\/span>\n      Prefer primary company guidance; later filings supersede earlier; demote analyst commentary.<\/span>\n    <\/div>\n  <\/div>\n\n  
\n    
\n      
Pointwise (before)<\/div>\n      
\n        
\n          1<\/span>\n          Q3 FY26 call \u2014 Q4 FY26 guidance ($37.5B)<\/span>\n          Nov 20 '25<\/span>\n        <\/div>\n        
\n          2<\/span>\n          Morgan Stanley NVDA note (Q1 model)<\/span>\n          Mar 15 '26<\/span>\n        <\/div>\n        
\n          3<\/span>\n          Q4 FY26 call \u2014 Q1 FY27 guidance ($43B)<\/span>\n          Feb 26 '26<\/span>\n        <\/div>\n        
\n          4<\/span>\n          CNBC: \"Nvidia guides Q1 above expectations\"<\/span>\n          Feb 26 '26<\/span>\n        <\/div>\n        
\n          5<\/span>\n          NVDA 8-K \u2014 data center supply chain<\/span>\n          Apr 2 '26<\/span>\n        <\/div>\n      <\/div>\n    <\/div>\n\n
<div>\n  <div class="text-[10px] uppercase tracking-wider text-zinc-500 mb-3">Listwise (after)<\/div>\n  <div class="space-y-2">\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">1<\/span>\n      <span class="text-sm flex-1 font-medium text-white">Q4 FY26 call \u2014 Q1 FY27 guidance ($43B)<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">Feb 26 '26<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">2<\/span>\n      <span class="text-sm flex-1">CNBC: "Nvidia guides Q1 above expectations"<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Feb 26 '26<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">3<\/span>\n      <span class="text-sm flex-1">Morgan Stanley NVDA note (Q1 model)<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Mar 15 '26<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">4<\/span>\n      <span class="text-sm flex-1">Q3 FY26 call \u2014 Q4 FY26 guidance ($37.5B)<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Nov 20 '25<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">5<\/span>\n      <span class="text-sm flex-1">NVDA 8-K \u2014 data center supply chain<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Apr 2 '26<\/span>\n    <\/div>\n  <\/div>\n<\/div>\n<\/code><\/pre>\n  <\/div>\n<\/div>\n\n
Legal:<\/strong>\nThe Code of Conduct is keyword-dense for "indemnification" but belongs to a different agreement; Amendment 4 is the most recent document but doesn't touch \u00a712. Listwise traces the \u00a712 chain (base \u2192 scope rewrite \u2192 cap reset) to identify Amendment 3 as the operative cap, sitting on top of Amendment 2's scope.<\/p>\n
\n  
\n    
\n      Query<\/span>\n      \"Find the operative indemnification provision.\"<\/span>\n    <\/div>\n    
\n      Instruction<\/span>\n      Trace Section 12 amendments; ignore amendments and docs that don't touch this section.<\/span>\n    <\/div>\n  <\/div>\n\n  
\n    
\n      
Pointwise (before)<\/div>\n      
\n        
\n          1<\/span>\n          Code of Conduct (boilerplate indemn.)<\/span>\n          2024-02<\/span>\n        <\/div>\n        
\n          2<\/span>\n          MSA \u00a712 \u2014 base indemnification<\/span>\n          2022-06<\/span>\n        <\/div>\n        
\n          3<\/span>\n          Amendment 2 \u2014 \u00a712 scope rewrite<\/span>\n          2023-11<\/span>\n        <\/div>\n        
\n          4<\/span>\n          Amendment 3 \u2014 \u00a712 cap \u2192 $5M<\/span>\n          2025-07<\/span>\n        <\/div>\n        
\n          5<\/span>\n          Amendment 4 \u2014 \u00a78 only (term)<\/span>\n          2026-01<\/span>\n        <\/div>\n      <\/div>\n    <\/div>\n\n
<div>\n  <div class="text-[10px] uppercase tracking-wider text-zinc-500 mb-3">Listwise (after)<\/div>\n  <div class="space-y-2">\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">1<\/span>\n      <span class="text-sm flex-1 font-medium text-white">Amendment 3 \u2014 \u00a712 cap \u2192 $5M<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">2025-07<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">2<\/span>\n      <span class="text-sm flex-1">Amendment 2 \u2014 \u00a712 scope rewrite<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">2023-11<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">3<\/span>\n      <span class="text-sm flex-1">MSA \u00a712 \u2014 base indemnification<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">2022-06<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">4<\/span>\n      <span class="text-sm flex-1">Amendment 4 \u2014 \u00a78 only (term)<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">2026-01<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">5<\/span>\n      <span class="text-sm flex-1">Code of Conduct (boilerplate indemn.)<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">2024-02<\/span>\n    <\/div>\n  <\/div>\n<\/div>\n<\/code><\/pre>\n  <\/div>\n<\/div>\n\n
Memory:<\/strong>\nPointwise ranks dietary snippets by topic match, mixing current and superseded statements together. Listwise reads the chain (vegetarian \u2192 pescatarian \u2192 plant-based \u2192 pescatarian) and treats the peanut allergy as an orthogonal additive constraint, surfacing both the current pescatarian statement and the additive allergy statement at the top.<\/p>\n
\n  
\n    
\n      Query<\/span>\n      \"What are the user's current dietary restrictions?\"<\/span>\n    <\/div>\n    
\n      Instruction<\/span>\n      Synthesize current state; later statements supersede earlier ones; allergies are additive.<\/span>\n    <\/div>\n  <\/div>\n\n  
\n    
\n      
Pointwise (before)<\/div>\n      
\n        
\n          1<\/span>\n          \"Vegetarian for about five years.\"<\/span>\n          Aug 2025<\/span>\n        <\/div>\n        
\n          2<\/span>\n          \"Going plant-based for a 90-day reset.\"<\/span>\n          Feb 2026<\/span>\n        <\/div>\n        
\n          3<\/span>\n          \"Started eating fish again.\"<\/span>\n          Oct 2025<\/span>\n        <\/div>\n        
\n          4<\/span>\n          \"Plant-based didn't stick \u2014 back to pescatarian.\"<\/span>\n          Apr 2026<\/span>\n        <\/div>\n        
\n          5<\/span>\n          \"Diagnosed mild peanut allergy.\"<\/span>\n          Jan 2026<\/span>\n        <\/div>\n      <\/div>\n    <\/div>\n\n
<div>\n  <div class="text-[10px] uppercase tracking-wider text-zinc-500 mb-3">Listwise (after)<\/div>\n  <div class="space-y-2">\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">1<\/span>\n      <span class="text-sm flex-1 font-medium text-white">"Plant-based didn't stick \u2014 back to pescatarian."<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">Apr 2026<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">2<\/span>\n      <span class="text-sm flex-1 font-medium text-white">"Diagnosed mild peanut allergy."<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">Jan 2026<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">3<\/span>\n      <span class="text-sm flex-1">"Going plant-based for a 90-day reset."<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Feb 2026<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">4<\/span>\n      <span class="text-sm flex-1">"Started eating fish again."<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Oct 2025<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">5<\/span>\n      <span class="text-sm flex-1">"Vegetarian for about five years."<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">Aug 2025<\/span>\n    <\/div>\n  <\/div>\n<\/div>\n<\/code><\/pre>\n  <\/div>\n<\/div>\n\n
Models:<\/strong>\nThe benchmark explainer is the most "MRR + reranker" keyword-dense candidate, so the pointwise model promotes it, even though it is not a measured reranker row. Listwise applies the instruction to compare results from the same evaluation and ranks the model with the highest reported MRR first.<\/p>\n
\n  
\n    
\n      Query<\/span>\n      \"Which reranker has the highest MRR?\"<\/span>\n    <\/div>\n    
\n      Instruction<\/span>\n      Compare only rows from the same instruction-following evaluation; sort by MRR descending.<\/span>\n    <\/div>\n  <\/div>\n\n  
\n    
\n      
Pointwise (before)<\/div>\n      
\n        
\n          1<\/span>\n          Blog: \"how MRR evaluates rerankers\"<\/span>\n          n\/a<\/span>\n        <\/div>\n        
\n          2<\/span>\n          Voyage rerank-2.5 benchmark row<\/span>\n          0.84 MRR<\/span>\n        <\/div>\n        
\n          3<\/span>\n          Cohere Rerank 4 Pro benchmark row<\/span>\n          0.77 MRR<\/span>\n        <\/div>\n        
\n          4<\/span>\n          mxbai-rerank-v3-listwise benchmark row<\/span>\n          0.93 MRR<\/span>\n        <\/div>\n        
\n          5<\/span>\n          ZeroEntropy zerank-2 benchmark row<\/span>\n          0.71 MRR<\/span>\n        <\/div>\n      <\/div>\n    <\/div>\n\n
<div>\n  <div class="text-[10px] uppercase tracking-wider text-zinc-500 mb-3">Listwise (after)<\/div>\n  <div class="space-y-2">\n    <div class="flex items-center gap-3 rounded-md bg-orange-600 px-3 py-2">\n      <span class="text-xs text-orange-100 w-4 tabular-nums">1<\/span>\n      <span class="text-sm flex-1 font-medium text-white">mxbai-rerank-v3-listwise benchmark row<\/span>\n      <span class="text-xs text-orange-100 tabular-nums">0.93 MRR<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">2<\/span>\n      <span class="text-sm flex-1">Voyage rerank-2.5 benchmark row<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">0.84 MRR<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">3<\/span>\n      <span class="text-sm flex-1">Cohere Rerank 4 Pro benchmark row<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">0.77 MRR<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">4<\/span>\n      <span class="text-sm flex-1">ZeroEntropy zerank-2 benchmark row<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">0.71 MRR<\/span>\n    <\/div>\n    <div class="flex items-center gap-3 rounded-md bg-white px-3 py-2 dark:bg-white\/[0.04]">\n      <span class="text-xs text-zinc-500 w-4 tabular-nums">5<\/span>\n      <span class="text-sm flex-1">Blog: "how MRR evaluates rerankers"<\/span>\n      <span class="text-xs text-zinc-500 tabular-nums">n\/a<\/span>\n    <\/div>\n  <\/div>\n<\/div>\n<\/code><\/pre>\n  <\/div>\n<\/div>\n\n
One API<\/h2>\n
mxbai-rerank-v3-listwise is available in preview today through Mixedbread Search:<\/p>\n
Python:<\/strong><\/p>\n
from mixedbread import Mixedbread\n\n    client = Mixedbread()\n\n    results = client.stores.search(\n        store_identifiers=["my-store"],\n        query="when is my flight to London? The most recent valid booking wins",\n        search_options={\n          "rerank": {\n              "model": "mixedbread-ai\/mxbai-rerank-v3-listwise",\n          }\n        },\n    )\n<\/code><\/pre>\n
TypeScript:<\/strong><\/p>\n
import { Mixedbread } from "@mixedbread\/sdk";\n\n    const client = new Mixedbread();\n\n    const results = await client.stores.search({\n        store_identifiers: ["my-store"],\n        query: "when is my flight to London? The most recent valid booking wins",\n        search_options: {\n          rerank: {\n              model: "mixedbread-ai\/mxbai-rerank-v3-listwise",\n          }\n        },\n    });\n<\/code><\/pre>\n
New users get $5 free credits to try it<\/a>.<\/p>\n
If this is a problem you want to work on, we are hiring<\/a>.<\/p>\n","url":"https:\/\/www.mixedbread.com\/blog\/listwise-rerank","title":"Ranking Beyond Binary Relevance: mxbai-rerank-v3-listwise","summary":"Announcing mxbai-rerank-v3-listwise, our new listwise reranker codesigned with Wholembed v3. It improves results on every benchmark we ran, with state-of-the-art instruction following.","image":"https:\/\/www.mixedbread.com\/images\/blog\/listwise-rerank\/intro-listwise.jpg","date_modified":"2026-05-08T00:00:00.000Z","author":{"name":"Mixedbread Team","url":"https:\/\/www.mixedbread.com"},"tags":["research","product"]},{"id":"https:\/\/www.mixedbread.com\/blog\/closing-gap","content_html":"
In retrieval for agentic workflows, the most important number is not the raw score, but the gap to oracle.<\/p>\n
Oracle retrieval is the ceiling: the score you get when the system has access to the correct evidence for every question, without retrieval misses. The smaller the gap, the less retrieval is holding the rest of your stack back.<\/p>\n
Our goal with Mixedbread Search is that you should not have to think about that gap at all. Building agentic systems is already complex enough without also having to debug retrieval quality. Search should be one less thing your team has to worry about.<\/p>\n
Across three agentic benchmarks, Mixedbread Search v3 consistently narrows that gap: on broad general-domain tasks like BrowseComp-Plus, on newer multimodal benchmarks like MADQA, and on enterprise knowledge-work benchmarks like OfficeQA-Pro. We chose these benchmarks because they stress the workflows our customers actually care about.<\/p>\n
\"Oracle<\/p>\n
BrowseComp-Plus<\/h3>\n
BrowseComp-Plus<\/a> measures deep-research agents on multi-hop questions over a corpus of roughly 100,000 web documents. It is designed specifically to separate retrieval quality from model capability.<\/p>\n\n\n\n\n\n\n\n\n
Retriever<\/th>\nScaffold<\/th>\nAccuracy<\/th>\nGap to Oracle (93.5%)[^1]<\/th>\n<\/tr>\n<\/thead>\n
Mixedbread Search<\/strong><\/td>\nget_document<\/strong><\/td>\n90.48%<\/strong><\/td>\n3.02<\/strong><\/td>\n<\/tr>\n
Reason-ModernColBERT<\/td>\nget_document<\/td>\n87.59%<\/td>\n5.9<\/td>\n<\/tr>\n
Mixedbread Search<\/td>\nstandard<\/td>\n80.00<\/td>\n13.5<\/td>\n<\/tr>\n
Reason-ModernColBERT<\/td>\nstandard<\/td>\n79.52<\/td>\n13.98<\/td>\n<\/tr>\n
Qwen3-Embed-8B<\/td>\nstandard<\/td>\n71.69%<\/td>\n21.81<\/td>\n<\/tr>\n<\/tbody><\/table>\n
[^1]: As reported in the BrowseComp-Plus paper.<\/p>\n
Mixedbread Search v3 ranks #1<\/strong> on the leaderboard<\/a> in both settings: with the default standardized benchmarking scaffold, and with the stronger agentic scaffold where the model can read the full document behind each retrieved snippet.<\/p>\n
Notably, the gap between Mixedbread Search and other retrieval approaches widens under the better harness. That suggests retrieval is less often the limiting factor, and that more of the remaining headroom lies in the overall agent setup. In practice, that means teams can spend less time debugging retrieval and more time improving the rest of their system.<\/p>\n
MADQA<\/h3>\n
MADQA<\/a>, released just last week, tests whether agents can navigate more than 18,000 pages from 800 heterogeneous PDFs to answer 500 human-authored questions. The benchmark is inherently multimodal: models are given screenshots of PDF pages, and the dataset is designed to reflect complex in-domain knowledge across areas like financial reports and legal documents.<\/p>\n
It can be run in two settings. In one-turn mode, the model gets a single search call before it must answer. In agentic mode, the model is given up to ten turns, with metrics that explicitly measure the tradeoff between answer quality and effort, a proxy for both token cost and system speed.<\/p>\n\n\n\n\n\n\n\n\n\n
Model<\/th>\nRetriever<\/th>\nAccuracy<\/th>\nGap to Oracle (99.4%)<\/th>\nPage F1<\/th>\n<\/tr>\n<\/thead>\n
Human w\/ Oracle Retriever<\/td>\nOracle<\/td>\n99.4%<\/td>\n0.0<\/td>\n-<\/td>\n<\/tr>\n
Button, an agentic document-QA system built on Gemini 3.1 Pro (Agentic)<\/td>\nHybrid w\/ Mixedbread<\/strong><\/td>\n91.7%<\/strong><\/td>\n7.7<\/strong><\/td>\n86.9<\/strong><\/td>\n<\/tr>\n
Gemini 3 Pro (One-shot RAG)<\/td>\nMixedbread<\/strong><\/td>\n88.2%<\/strong><\/td>\n11.2<\/strong><\/td>\n82.2<\/strong><\/td>\n<\/tr>\n
Human w\/ BM25<\/td>\nBM25<\/td>\n82.2%<\/td>\n17.2<\/td>\n79.3<\/td>\n<\/tr>\n
Claude Sonnet 4.5 (Agentic)<\/td>\nBM25<\/td>\n80.6%<\/td>\n18.8<\/td>\n79.1<\/td>\n<\/tr>\n
Gemini 3 Pro (Preview) with File Search<\/td>\nGoogle Files Search<\/td>\n78.6<\/td>\n20.8<\/td>\n70.1<\/td>\n<\/tr>\n<\/tbody><\/table>\n
On the current MADQA leaderboard<\/a>, Mixedbread-powered systems occupy the top spots in each category, with only a human using oracle documents outperforming them. In the single-search setting, Gemini 3 Pro with Mixedbread outperforms human experts who are allowed up to 10 BM25 searches.<\/p>\n
Gemini 3 Pro also gains nearly 10 points of accuracy with Mixedbread compared to Google\u2019s File Search API. Notably, the top-performing system, Button, is not ours: Distyl AI<\/a> paired Mixedbread Search with its own harness and reached state-of-the-art results without further tuning.<\/p>\n
Taken together, these results suggest that Mixedbread Search is effective at surfacing the evidence agentic workflows need, including on heterogeneous, multimodal corpora.<\/p>\n
OfficeQA-Pro<\/h3>\n
Finally, OfficeQA-Pro<\/a> is a specialized enterprise knowledge-work benchmark designed by Databricks. It contains 89,000 pages of complex financial documents, including U.S. Treasury Bulletins, dense tables, and scanned PDFs, along with questions that require multi-document reasoning to answer satisfactorily.<\/p>\n
We wanted to measure the effect of retrieval quality inside a widely used tool: OpenAI's Codex. So we ran the benchmark in the same Codex-based setup while varying only the retrieval tooling: a corpus baseline, where the raw corpus is stored on disk as OCR plus PDF images and Codex uses its usual tools; an oracle setting, where the model is given all correct documents; and Mixedbread Search-based retrieval.<\/p>\n
This setup does not isolate retrieval as cleanly as the other benchmarks, which were designed specifically for that purpose. Still, we think it is informative because OfficeQA documents are unusually difficult, and because many teams rely on agents\u2019 native \u201cno-RAG\u201d context tools in similar settings.<\/p>\n\n\n\n\n\n\n\n
Method<\/th>\nCorrectness<\/th>\nGap to Oracle<\/th>\nLatency (min)<\/th>\nTool Calls<\/th>\n<\/tr>\n<\/thead>\n
Codex (Oracle, thinking high)<\/td>\n65.41<\/td>\n-<\/td>\n2.2*<\/td>\n20.7*<\/td>\n<\/tr>\n
Codex (Mixedbread, thinking high)<\/strong><\/td>\n64.42<\/strong><\/td>\n0.99<\/strong><\/td>\n2.36<\/strong><\/td>\n17.35<\/strong><\/td>\n<\/tr>\n
Codex (Corpus, thinking high)<\/td>\n56.39<\/td>\n9.02<\/td>\n3.6<\/td>\n34.5<\/td>\n<\/tr>\n
GPT 5.4 Agent + Semantic Search **<\/td>\n51.90<\/td>\n13.51<\/td>\n8.93<\/td>\n86.4<\/td>\n<\/tr>\n<\/tbody><\/table>\n
With all other settings held equal, Mixedbread comes close to the oracle setting while materially reducing latency and tool use. Compared with Codex\u2019s built-in retrieval over the corpus, Mixedbread recovers 89% of the gap between the corpus baseline and oracle (8.03 of 9.02 points) while cutting latency by 34% and reducing tool calls by roughly half.<\/p>\n
Within the same harness, Mixedbread Search moves the agent substantially closer to oracle with less search effort.<\/p>\n
*: performance reported from paper
**: using Databricks agent (from paper)<\/p>\n
More Limits to Overcome<\/h3>\n
Even with these results, knowledge work is far from solved. Hard cases remain: ambiguous queries, genuinely missing evidence, messy enterprise corpora, and tasks where the core difficulty is not finding documents but reasoning correctly over them once found.<\/p>\n
Closing the oracle gap does not eliminate system failures, and it does not fully eliminate retrieval failures either. But as the gap shrinks, the source of failure shifts. When retrieval is less often the bottleneck, more of the remaining headroom moves to reasoning, prompting, and domain-specific system design, which is where teams are often best served spending their effort.<\/p>\n
One API<\/strong><\/p>\n
To make retrieval one less thing teams have to manage, we built Mixedbread Search behind a simple API:<\/p>\n
Python:<\/strong><\/p>\n
from mixedbread import Mixedbread\n    from pathlib import Path\n\n    client = Mixedbread()\n\n    # Index documents (any modality)\n    client.stores.files.upload(\n        store_identifier="my-store",\n        file=Path("doc.pdf")\n    )\n\n    # Search\n    results = client.stores.search(\n        store_identifiers=["my-store"],\n        query="quarterly revenue growth",\n    )\n<\/code><\/pre>\n
TypeScript:<\/strong><\/p>\n
import { Mixedbread } from "@mixedbread\/sdk";\n    import * as fs from "node:fs";\n\n    const client = new Mixedbread();\n\n    \/\/ Index documents (any modality)\n    await client.stores.files.upload({\n        storeIdentifier: "my-store",\n        file: fs.createReadStream("doc.pdf")\n    });\n\n    \/\/ Search\n    const results = await client.stores.search({\n        store_identifiers: ["my-store"],\n        query: "quarterly revenue growth",\n    });\n<\/code><\/pre>\n
No chunking decisions, embedding model choices, image preprocessing, vector database configuration, or reranker threshold tuning.<\/p>\n
We handle multimodal ingestion, late-interaction encoding, indexing, and retrieval. You upload documents and start searching. To get started just sign up<\/a>.<\/p>\n
We are building Mixedbread to close the gap between the search that is possible today and what users and agents of tomorrow will demand. If that sounds like a problem you want to work on, we are hiring<\/a>.<\/p>\n","url":"https:\/\/www.mixedbread.com\/blog\/closing-gap","title":"Closing the Oracle Gap for Your Agents","summary":"Mixedbread Search v3 narrows the oracle gap for agentic retrieval, topping BrowseComp-Plus and setting leading results on MADQA and OfficeQA-Pro.","image":"https:\/\/www.mixedbread.com\/images\/blog\/oracle-gap\/orace-gap.jpg","date_modified":"2026-03-24T00:00:00.000Z","author":{"name":"Mixedbread Team","url":"https:\/\/www.mixedbread.com"},"tags":["engineering","research","product"]},{"id":"https:\/\/www.mixedbread.com\/blog\/wholembed-v3","content_html":"
Today we're releasing Wholembed v3, our new unified, omnimodal, multilingual late-interaction retrieval model.<\/p>\n
Wholembed v3 sets a new state-of-the-art for search, delivering best-in-class performance across languages and modalities in both academic benchmarks and real-world industrial test cases from partners across multiple business domains.<\/p>\n
It is now publicly available on our platform<\/a> and will power all new stores on Mixedbread by default going forward. Co-designed with our custom-built retrieval infrastructure<\/a>, Wholembed v3 enables Mixedbread Search to deliver state-of-the-art retrieval with high throughput, low latency, and a frictionless developer experience.<\/p>\n
A Foundation Model for Modern Retrieval<\/h2>\n
Retrieval is at the core of most agentic applications. For AI systems to be truly powerful, they need to be grounded in relevant knowledge. And AI applications are already moving far beyond simple question answering: they are operating computers, controlling robots, and performing increasingly complex knowledge work.<\/p>\n
Yet search remains brittle, and, for many applications, still surprisingly hard to implement well. In large part, this comes down to the models and algorithms we rely on today. Much of retrieval still relies on older paradigms, which have garnered steady incremental performance gains but have not been able to keep pace with vast capability increases in other areas of AI. Inspired by this and the modeling and algorithmic breakthroughs that have transformed language models, we built Wholembed v3 to tackle current challenges, rather than simply improve on largely solved ones.<\/p>\n
Wholembed v3 was designed from the ground up for complex, real-world retrieval across hundreds of languages and all relevant modalities, including text, audio, and vision. With it, we set out to move beyond the limits and common failure modes of traditional semantic search.<\/p>\n
Modern Problems Require Modern Evaluations<\/h3>\n
Two benchmarks that we found particularly insightful to help understand the limits of semantic search in the current age are LIMIT and BrowseComp-Plus, both of which map to two opposite, but very common, real-world applications.<\/p>\n
LIMIT<\/strong> is a benchmark specifically designed to stress-test the limits of semantic retrieval under situations where the documents contain a lot of fine-grained, "structured-like" information expressed as natural text. In our experience working with customers across domains, we have found this kind of document to be extremely common across many industries. Yet, they are rarely captured by common retrieval benchmarks. LIMIT has thus represented an important frontier for semantic retrieval, with older, lexical-based methods vastly outperforming even the best billion-parameter semantic retrieval models.<\/p>\n
BrowseComp-Plus<\/strong>, on the other hand, aims to measure how well-suited search systems are to the agent-as-a-first-class-citizen paradigm. Rather than evaluating retrievers based on retrieval metrics, which are not always well-correlated to system usefulness, they evaluate retrievers based on their ability to let an agent reach the correct answer to 830 complex deep research queries. Many of them require dozens of sequential searches to accurately answer: miss a single nugget of information and the original question remains unanswerable.<\/p>\n
We considered besting these two benchmarks as our North Star throughout development, while also aiming to avoid any contamination: we never evaluated any in-development model checkpoints on them until preparing for this release.<\/p>\n\n\n\n\n\n\n\n\n\n\n
LIMIT (Benchmark to test limits of semantic search, a lot of real-world large-scale data is similar to this benchmark)<\/th>\nRecall@5<\/th>\nRecall@10<\/th>\nRecall@100<\/th>\n<\/tr>\n<\/thead>\n
Model<\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Cohere Embed 4<\/td>\n1.8<\/td>\n2.55<\/td>\n5.7<\/td>\n<\/tr>\n
OpenAI Text Embedding 3 Large<\/td>\n1.75<\/td>\n2.4<\/td>\n4.95<\/td>\n<\/tr>\n
Voyage 4 Large<\/td>\n1.85<\/td>\n2.9<\/td>\n8.95<\/td>\n<\/tr>\n
Gemini Embedding 2 (03\/26)<\/td>\n1.85<\/td>\n2.25<\/td>\n6.9<\/td>\n<\/tr>\n
BM25<\/td>\n85.7<\/td>\n90.4<\/td>\n93.6<\/td>\n<\/tr>\n
Mixedbread Wholembed V3<\/td>\n92.45<\/strong><\/td>\n94.4<\/strong><\/td>\n98.0<\/strong><\/td>\n<\/tr>\n<\/tbody><\/table>\n
\"BrowseComp<\/p>\n
On LIMIT, Wholembed v3 pushes semantic retrieval far beyond what was previously thought possible for semantic search. It not only outperformed previous semantic methods by a margin, but also became the first model to outperform lexical-based retrieval. On BrowseComp-Plus, we observe how important the choice of retriever, such as Mixedbread Search, is for deep research agents: without a good retrieval system, hundreds more questions are answered incorrectly.<\/p>\n
We believe this strong performance on difficult, real-world domains stems from design choices that align well with what agents need: precise retrieval, strong discrimination between similar candidates, and robustness to noisy, heterogeneous data.<\/p>\n
Evaluating Retrieval In A Multi-Modal Multi-Lingual World<\/h2>\n
Wholembed v3 is not solely built around serving agents and their novel needs, but to improve retrieval for the real world: a world where information does not live in neatly pre-packaged paragraphs with clear relevance boundaries. In practice, that means a single retrieval system that can index and search text, images, audio, and video.<\/p>\n
We strongly believe that to be useful, search needs to be able to find information where it lives: in good quality text, yes, but also in artifacts-ridden OCRised documents, scanned PDFs and screenshots with confusing names, or even in instructional videos.<\/p>\n
\"Benchmark<\/p>\n
Wholembed v3 was designed with the real world in mind. Integrated with Mixedbread Search, it provides you with a seamless experience, ensuring that your service can retrieve exactly the information it needs without requiring you to think about how every single data processing detail will affect the quality of your application.<\/p>\n
In our testing, we found that Wholembed v3 outperforms existing methods in almost all situations, no matter the document type, the kind of information you need to find or the language it\u2019s in.<\/p>\n
As part of our pre-release phase, we have heard from our partners that this held true across a variety of domains, with Wholembed v3 preview-powered Mixedbread Search replacing convoluted pipelines encompassing many brittle parts.<\/p>\n
We are now very excited to finally release Wholembed v3 as our default and only model, and are looking forward to new kinds of applications it will enable. Search doesn\u2019t need to be complicated, it just needs to be good.<\/p>\n
Try it Now<\/h2>\n
Wholembed v3 is now available to all Mixedbread Search users. All new stores use v3 by default, with built-in support for audio and video.<\/p>\n
New users get 2M free tokens to get started<\/a>. Startups can also explore our accelerator programs with Vercel<\/a> and TinyFish<\/a>.<\/p>\n
Create a store in the platform<\/a> or get started directly with the API<\/a> in just a few lines:<\/p>\n
Python:<\/strong><\/p>\n
from mixedbread import Mixedbread\n    from pathlib import Path\n\n    client = Mixedbread()\n\n    client.stores.create(name="example")\n\n    client.stores.files.upload(\n        store_identifier="example",\n        file=Path("example.mp4")\n    )\n\n    client.stores.search(\n        store_identifiers=["example"],\n        query="how we bake search for the future?"\n    )\n<\/code><\/pre>\n
TypeScript:<\/strong><\/p>\n
import { Mixedbread } from "@mixedbread\/sdk";\n    import * as fs from "node:fs";\n\n    const client = new Mixedbread();\n\n    await client.stores.create({ name: "example" });\n\n    await client.stores.files.upload({\n        storeIdentifier: "example",\n        file: fs.createReadStream("example.mp4")\n    });\n\n    await client.stores.search({\n        store_identifiers: ["example"],\n        query: "how we bake search for the future?"\n    });\n<\/code><\/pre>\n
We are building Mixedbread to close the gap between the search that is possible today and what users and agents of tomorrow will demand. If that sounds like a problem you want to work on, we are hiring<\/a>. <\/p>\n","url":"https:\/\/www.mixedbread.com\/blog\/wholembed-v3","title":"Beyond the Limit: Introduce Mixedbread Wholembed v3","summary":"Announcing Mixedbread Wholembed v3, our new unified omnimodal multilingual retrieval model, setting a new state of the art for search across languages, modalities, and real-world retrieval tasks.","image":"https:\/\/www.mixedbread.com\/images\/blog\/wholembed-v3\/wholembed-v3.jpg","date_modified":"2026-03-12T00:00:00.000Z","author":{"name":"Mixedbread Team","url":"https:\/\/www.mixedbread.com"},"tags":["engineering","research","product"]},{"id":"https:\/\/www.mixedbread.com\/blog\/multimodal-late-interaction-billion-scale","content_html":"
Most semantic search issues don't show up as obvious failures. They show up as results that look reasonable, read well, and are still wrong. In our experience, this is a structural limitation of single-vector retrieval on dense and unfamiliar inputs: the representation collapses detail, and the retriever confidently returns "close enough" content that doesn't actually answer the query.<\/p>\n
Building a reliable retriever is also harder than it looks. You're stitching together parsing, chunking, embedding, metadata extraction, and ANN search, and each stage introduces its own brittleness. When quality drops, it's rarely clear whether the problem is upstream ingestion, representation, indexing, or scoring.<\/p>\n
We built a multimodal late-interaction retrieval system to make those failure modes rarer and easier to reason about. The system uses multi-vector representations across text, images, audio, and video, and it's deployed at billion-document scale: 1B+ documents indexed, 500+ QPS per store, and ~50ms search latency end-to-end.<\/strong> The rest of this post walks through the three pieces we had to build and jointly tune to get there: ingestion, encoding, and retrieval.<\/p>\n
Get started for free<\/a><\/p>\n
Multimodal Ingestion<\/h2>\n
For the vast majority of our embedding purposes, our goal is to create a true end-to-end representation pipeline, where we retrieve information from exactly where it lives in the embedding space, surrounded by all the valuable context.<\/p>\n
This means that audio files are first pre-processed to maximize quality before being passed to the model, which dynamically splits it into meaningful units on its own. For textual inputs, we have a series of pre-processing steps which ensures the data is broken down into manageable blocks ("chunks") while maintaining all necessary context. Code is treated as its own separate input, with the AST parsed to determine logical cutoff points. For images, the model natively processes pixels.<\/p>\n
As for document formats, like PDFs and PowerPoint, every single page is individually exported to screenshots of the pages, ensuring that all visual and layout information, such as tables and graphs, are preserved and represented as individual semantic units, with context such as headings preserved across pages.<\/p>\n
\"Multimodal<\/p>\n
Unlike most other systems where the model's training data is largely decoupled from expected real-world inputs, our model is trained specifically on the output of these pre-processing steps. This ensures that it is fully optimized to retrieve documents exactly as they will be in production and yield more accurate search results.<\/p>\n
Even though our model can process the format of every type of document natively, that does not mean that downstream applications can use this information as effectively. Up until recently, the vast majority of the world's applications were built with text as the first-class, and often only, citizen. As such, parallel to preparing inputs for representation, our ingestion pipeline also runs a full document analysis step to extract texts from the input. As a result, audio and video files are transcribed, and PDFs are converted into fully human and LLM-readable markdown format using our custom OCR pipeline<\/a>.<\/p>\n
Multimodal Late-Interaction Encoding<\/h2>\n
"Traditional" semantic search relies on a simple, single-vector approach: an input is provided to a model, which then outputs a single vector representing the entire document.\nSingle-vector search has many advantages. It is fast and exceedingly simple to implement. However, it is also extremely lossy: while a single vector can capture paraphrases and general meaning well, it dilutes details and precise intent, and it struggles with information-dense documents, such as technical, legal documents and academic papers, especially in multimodal settings.<\/p>\n
ColBERT<\/a> pioneers a multi-vector approach, which replaced traditional single-vector representations with individual, lower-dimension token-level representations. These token-level representations preserve fine-grained information in both the query and the documents. In practice, it yields more accurate search results in out-of-distribution settings, where real-world papers, slideshows or long-context texts may look fundamentally different from the training data. For example, our 17 million parameter open-source ColBERT outperforms 8 billion parameter embedding models on the LongEmbed<\/a> benchmark, which measures the ability of embedding models to perform long-context retrieval tasks.<\/p>\n
In our experiments, we found that not only can multi-vector representations be compressed while continuing to capture fine-grained meaning, they also significantly outperform single-vector alternatives on image, video and audio search.<\/p>\n\n\n\n\n\n\n
Benchmark (NDCG@10)<\/th>\nBest Single Vector Model<\/th>\nmxbai-wholembed-v3<\/th>\n<\/tr>\n<\/thead>\n
OHR-V2<\/a><\/td>\n86.47 (Qwen 3 VL 8B)<\/td>\n91.26<\/strong><\/td>\n<\/tr>\n
Miracl-Vision<\/a><\/td>\n59.79 (Qwen 3 VL 8B)<\/td>\n66.02<\/strong><\/td>\n<\/tr>\n
Vidore 3<\/a><\/td>\n64.81 (Qwen 3 VL 8B)<\/td>\n67.9<\/strong><\/td>\n<\/tr>\n<\/tbody><\/table>\n
\nmxbai-wholembed-v3 is currently in internal evaluation and will be available soon. We are publishing an in-depth release about the model benchmark soon. The benchmarks shown are challenging multimodal evaluations. As of Jan. 2026, Qwen 3 VL 8B is the latest release and achieves SOTA performance across multimodal and text only benchmarks.\n<\/div>\n\n
mxbai-wholembed: A Unified Multimodal Encoder<\/h3>\n
Leveraging our team's previous frontier work on both embedding models<\/a> and ModernBERT<\/a>, among other things, we trained mxbai-wholembed<\/strong>, our unified multimodal late-interaction encoder.<\/p>\n
This unified model produces semantic unit-level vector representations, across text, image, audio, and video inputs, in a shared latent space, accurately capturing semantic relationships across modalities. Many previous state-of-the-art systems opted for modality-specific approaches, which introduced complexity by requiring modality-specific handling and yielded performance tradeoffs orthogonal to scaling laws. By sharing one latent space, we unlock "any-to-any" search and streamline architectural constraints by removing the need for any such modality-specific routing and storing. Throughout our experiments, we found that the bitter lesson of the effectiveness of gradient descent continues to hold true. Rather than competing, we observed that properly orchestrated multimodal training in a single, shared embedding space resulted in a lifting effect: improving the quality of image retrieval, for example, also improved performance for both audio and text retrieval.<\/p>\n
Scaling Laws<\/h4>\n
Our experiments showed that, despite the huge attention given to the BERT-scale model, scaling works for retrieval. This has also been independently demonstrated in the best paper award winner<\/a> at SIGIR, the premier Information Retrieval conference.<\/p>\n
Unlike the situation around LLMs, retrieval scaling laws are largely underexplored, poorly understood, and rarely used for downstream production settings. In spite of this, we found that larger scale models consistently allow for better cross-modality alignment and handling of complex information, such as composed queries, at the expense of significant efficiency constraints.<\/p>\n
After careful experimentation, we have settled on an infrastructure that, combined with a custom-designed inference engine, strikes the right balance at the Pareto frontier of latency and retrieval performance.<\/p>\n
This optimised stack has led to considerable scale increases: mxbai-wholembed has over 20 times the parameter count of our original multimodal model, and three times that of our previous one. With every generation, we saw considerable improvements in our models, especially in real-world edge cases poorly captured by standardized benchmarking.<\/p>\n
\"mxbai-wholembed<\/p>\n
\nmxbai-wholembed-v3 has 20 times the number of parameters of v1. This scale increase leads to considerable performance improvements.\n<\/div>\n\n
Dynamic vector allocation<\/h3>\n
mxbai-wholembed estimates the information density of a given input and decides the necessary number of vectors to represent it accordingly.  For example, a simple cat image may output a few vectors, whereas a complex slide deck may generate thousands of vectors. This dynamic allocation of representation capacity prevents semantic collapse on dense content often experienced by single vector representations. The optimal size of allocation is determined based on large-scale internal experiments to allow mxbai-wholembed to capture fine-grained information while keeping storage requirements low.<\/p>\n
Billion-scale Late-Interaction Retrieval<\/h2>\n
While late-interaction makes retrieval more accurate for real-world uses, it also creates a scale challenge. A single document can produce hundreds or thousands of vectors; a single collection of documents can then contain millions of vectors to search across.<\/p>\n
The Scale Challenge<\/h3>\n
Single-vector search efficiency is greatly helped by relying on very simple operations at the hardware-level, making storage and search straightforward. Additionally, it benefits from decades of approximate nearest neighbor (ANN) index research: algorithms like HNSW, SPFresh, and DiskANN bring search to near-constant time.<\/p>\n
This simplicity is why many semantic search platforms heavily favor single vector offerings: they remove considerable complexity and are extremely cheap to scale. However, the tradeoffs to retrieval quality are significant and theoretically demonstrated to be impossible to overcome with our current understanding (Weller<\/a>).<\/p>\n
Conversely, late interaction approaches have been demonstrated to greatly alleviate these tradeoffs, but break many of the simplicity assumptions that power large-scale single vector search. Specifically, they introduce three compounding challenges:<\/p>\n
    \n
  1. Limited indexing support for multi-vectors<\/strong>: The vast majority of current ANN indexing methods are not built for multi-vectors. With hundreds of vectors per document, scanning becomes prohibitively expensive and searches can take seconds instead of milliseconds.<\/li>\n
  2. Scoring is expensive.<\/strong>  A single-vector document requires a single inner product calculation: scoring across a million documents is ~ 1 million operations. With multi-vector methods, a document with 256 vectors scored against a 16-vector query requires 256 x 16 = 4,096 (inner product followed by score aggregation): scoring a million documents now requires 4 billion operations.<\/li>\n
  3. Storage scales with vectors.<\/strong> More vectors per document means more storage, more memory and more I\/O pressure.<\/li>\n<\/ol>\n
    silo: An S3-Native Retrieval Engine<\/h3>\n
    silo<\/strong> is a custom multi-vector retrieval engine designed to support late-interaction at billions-of-documents and trillions-of-vectors scale and maintain the flexibility of common ANN indexes. Three crucial optimizations bring the search latency under 50ms.<\/p>\n
    Two-Stage Retrieval<\/h4>\n
    \"Filtering<\/p>\n
    silo performs two-stage retrieval to make the search space tractable. Stage 1 pre-filters the search space using approximate searches and metadata filtering, and reduces it from billions of documents to a few thousand candidates. Stage 2 scores the candidate documents using MaxSim.<\/p>\n
    silo maintains and updates the document indexes at write time when documents are ingested, with no performance cost during retrieval. Consequently, there is no index building in preprocessing or latency hits on CRUD operations common in ColBERT-like methods.<\/p>\n
    Accelerated Scoring<\/h4>\n
    A few months ago, we wrote and released maxsim-cpu<\/a>, a high-performance Rust implementation of MaxSim on CPU. You can read more about it in our blog post<\/a>.<\/p>\n
    In production today, we went a step further and run a version of maxsim-cpu further optimized with custom kernels targeting the specific hardware of our inference pipeline, along with aggressive vector quantization to speed up scoring.<\/p>\n
    As a result, scoring a thousand candidate documents using MaxSim takes under 3 milliseconds and a single CPU processor is sufficient to handle thousands of QPS.<\/p>\n
    Object Storage with NVMe Caching<\/h4>\n
    \"silo<\/p>\n
    All vectors are stored in low-cost, replicated S3 object storage for scalability, durability and as the source-of-truth. Actively accessed vectors are loaded at query time into NVMe SSDs and cached in memory for computation.<\/p>\n
    On the write path, both vectors and metadata are persisted to a write-ahead log (WAL)<\/a> in S3, and their index and cache are populated synchronously.<\/p>\n
    Guarantees:<\/strong><\/p>\n