Skip to content

About

Rust library for generating vector embeddings and reranking locally!

Topics

Resources

Stars

1.1k stars

Watchers

5 watching

Forks

Latest commit

 

History

354 Commits

Folders and files

Repository files navigation

Rust library for generating vector embeddings, reranking locally!

Greptile for Open Source

Features

Not looking for Rust?

Supported Models

Text Embedding

Click to list models

Quantized versions are also available for several models above (append Q to the model enum variant, e.g., EmbeddingModel::AllMiniLML6V2Q). The Q variants of the English BGE v1.5 models and of paraphrase-multilingual-MiniLM-L12-v2 are half-precision (fp16) models. EmbeddingGemma additionally ships a 4-bit build as EmbeddingModel::EmbeddingGemma300MQ4.

Sparse Text Embedding

Click to list models

Image Embedding

Click to list models

Reranking

Click to list models

✊ Support

To support the library, please donate to our primary upstream dependency, ort - The Rust wrapper for the ONNX runtime.

Installation

Run the following in your project directory:

cargo add fastembed

Or add the following line to your Cargo.toml:

[dependencies]
fastembed = "7"

Text Embeddings

use fastembed::{TextEmbedding, TextInitOptions, EmbeddingModel};

// With default options
let mut model = TextEmbedding::try_new(Default::default())?;

// With custom options
let mut model = TextEmbedding::try_new(
    TextInitOptions::new(EmbeddingModel::AllMiniLML6V2).with_show_download_progress(true).with_intra_threads(4),
)?;

let documents = vec![
    "Hello, World!",
    "This is an example passage.",
    "fastembed-rs is licensed under Apache 2.0"
];

 // Generate embeddings with the default batch size, 256
 let embeddings = model.embed(documents, None)?;

 println!("Embeddings length: {}", embeddings.len()); // -> Embeddings length: 3
 println!("Embedding dimension: {}", embeddings[0].len()); // -> Embedding dimension: 384

Sparse Text Embeddings

use fastembed::{SparseEmbedding, SparseInitOptions, SparseModel, SparseTextEmbedding};

// With default options
let mut model = SparseTextEmbedding::try_new(Default::default())?;

// With custom options
let mut model = SparseTextEmbedding::try_new(
    SparseInitOptions::new(SparseModel::SPLADEPPV1).with_show_download_progress(true),
)?;

let documents = vec![
    "Hello, World!",
    "This is an example passage.",
    "fastembed-rs is licensed under Apache 2.0"
];

// Generate embeddings with the default batch size, 256.
// This model emits one score per vocabulary entry per token.
// So keep the batch size small for long documents.
let embeddings: Vec<SparseEmbedding> = model.embed(documents, None)?;

Image Embeddings

use fastembed::{ImageEmbedding, ImageInitOptions, ImageEmbeddingModel};

// With default options
let mut model = ImageEmbedding::try_new(Default::default())?;

// With custom options
let mut model = ImageEmbedding::try_new(
    ImageInitOptions::new(ImageEmbeddingModel::ClipVitB32).with_show_download_progress(true),
)?;

let images = vec!["tests/assets/image_0.png", "tests/assets/image_1.png"];

// Generate embeddings with the default batch size, 256
let embeddings = model.embed(images, None)?;

println!("Embeddings length: {}", embeddings.len()); // -> Embeddings length: 2
println!("Embedding dimension: {}", embeddings[0].len()); // -> Embedding dimension: 512

Candidates Reranking

use fastembed::{TextRerank, RerankInitOptions, RerankerModel};

// With default options
let mut model = TextRerank::try_new(Default::default())?;

// With custom options
let mut model = TextRerank::try_new(
    RerankInitOptions::new(RerankerModel::BGERerankerBase).with_show_download_progress(true),
)?;

let documents = vec![
    "hi",
    "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear, is a bear species endemic to China.",
    "panda is animal",
    "i dont know",
    "kind of mammal",
];

// Rerank with the default batch size, 256 and return document contents
let results = model.rerank("what is panda?", documents, true, None)?;
println!("Rerank result: {:?}", results);

Locally Available Models

Alternatively, local model files can be used for inference via the try_new_from_user_defined(...) methods of the respective structs.

Similarity Search

Helpers in the similarity module score and rank the vectors embed returns, so a quick in-memory search needs no extra crate:

use fastembed::similarity::{cosine_similarity, top_k};

// `embeddings` is the Vec<Embedding> from model.embed(...)
let query = &embeddings[0];

// Score two vectors directly ([-1.0, 1.0], higher = closer)
let score = cosine_similarity(query, &embeddings[1]);

// Or rank the corpus: (index, score) pairs, best first
let hits = top_k(query, &embeddings, 5);
println!("Closest: {:?}", hits);

For larger corpora or persistence, push the vectors to a vector search engine (e.g. Qdrant) and query there.

Inference-free Sparse Embeddings

use fastembed::{SparseInitOptions, SparseModel, SparseTextEmbedding};

let mut model = SparseTextEmbedding::try_new(
    SparseInitOptions::new(SparseModel::OpenSearchNeuralSparseDocV3Gte).with_max_length(8192),
)?;

//  This model emits one score per vocabulary entry per token.
// So keep the batch size small.
let documents = model.embed(vec!["Hello World"], Some(4))?;

// Queries are from the tokenizer and a precomputed IDF table alone, without any inference.
let queries = model.query_embed(vec!["Hello World"])?;

Qwen3 Embeddings

Qwen3 embedding models are available behind the qwen3 feature flag (candle backend).

[dependencies]
fastembed = { version = "7", features = ["qwen3"] }
# The candle version that fastembed uses, for `Device` and `DType`.
candle-core = "0.11"
use candle_core::{DType, Device};
use fastembed::Qwen3TextEmbedding;

let device = Device::Cpu;
let model = Qwen3TextEmbedding::from_hf(
    "Qwen/Qwen3-Embedding-0.6B",
    &device,
    DType::F32,
    512,
)?;

// Text-only usage with the Qwen3-VL embedding checkpoint is also supported. `embed` runs each
// text as given, and `Qwen3VLEmbedding::embed_texts` adds the prompt of the model card.
// let model = Qwen3TextEmbedding::from_hf("Qwen/Qwen3-VL-Embedding-2B", &device, DType::F32, 512)?;

// A query carries its task instruction, and a document goes in as it is.
let embeddings = model.embed(&[
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:What is the capital of China?",
    "The capital of China is Beijing.",
])?;
println!("Embeddings length: {}", embeddings.len());

For multimodal text/image usage with Qwen/Qwen3-VL-Embedding-2B:

use candle_core::{DType, Device};
use fastembed::Qwen3VLEmbedding;

let device = Device::Cpu;
let model = Qwen3VLEmbedding::from_hf(
    "Qwen/Qwen3-VL-Embedding-2B",
    &device,
    DType::F32,
    2048,
)?;

let image_embeddings = model.embed_images(&["tests/assets/image_0.png", "tests/assets/image_1.png"])?;
let text_embeddings = model.embed_texts(&["blue cat", "red cat"])?;

println!("Image embeddings: {}", image_embeddings.len());
println!("Text embeddings: {}", text_embeddings.len());

Quantized GGUF files (for example from Qwen/Qwen3-Embedding-4B-GGUF or Qwen/Qwen3-Embedding-8B-GGUF) load with the same qwen3 feature. The weights stay quantized in memory, and the tokenizer comes from the original model's repo:

use candle_core::Device;
use fastembed::Qwen3TextEmbedding;

let device = Device::Cpu;
let model = Qwen3TextEmbedding::from_hf_gguf(
    "Qwen/Qwen3-Embedding-4B-GGUF",
    "Qwen3-Embedding-4B-Q8_0.gguf",
    "Qwen/Qwen3-Embedding-4B",
    &device,
    512,
)?;

// A query carries its task instruction, and a document goes in as it is.
let embeddings = model.embed(&[
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:What is the capital of China?",
    "The capital of China is Beijing.",
])?;
println!("Embeddings length: {}", embeddings.len());

Nomic Embed Text v2 MoE

The nomic-embed-text-v2-moe model is available behind the nomic-v2-moe feature flag (candle backend). First general-purpose MoE embedding model with 100+ language support.

[dependencies]
fastembed = { version = "7", features = ["nomic-v2-moe"] }
candle-core = "0.11"
use candle_core::{DType, Device};
use fastembed::NomicV2MoeTextEmbedding;

let device = Device::Cpu;
let model = NomicV2MoeTextEmbedding::from_hf(
    "nomic-ai/nomic-embed-text-v2-moe",
    &device,
    DType::F32,
    512,
)?;

let embeddings = model.embed(&["search_query: ...", "search_document: ..."])?;
println!("Embeddings length: {}", embeddings.len());

BERT Embeddings

BERT-family encoders are available behind the bert-text feature flag (candle backend). Any BERT checkpoint with CLS pooling loads through BertTextEmbedding, which returns L2-normalized vectors; mixedbread-ai/mxbai-embed-large-v1 is the tested one.

[dependencies]
fastembed = { version = "7", features = ["bert-text"] }
candle-core = "0.11"
use candle_core::{DType, Device};
use fastembed::BertTextEmbedding;

let device = Device::Cpu;
let model = BertTextEmbedding::from_hf(
    "mixedbread-ai/mxbai-embed-large-v1",
    &device,
    DType::F32, // this checkpoint's LayerNorm is not f16-safe
    512,        // truncate inputs longer than this many tokens
)?;

let embeddings = model.embed(&[
    "What is the capital of France?",
    "Paris is the capital of France.",
])?;
println!("Embeddings length: {}", embeddings.len());
println!("Dimensions: {}", embeddings[0].len());

On Apple silicon, enable metal instead (it enables bert-text) and pass &Device::new_metal(0)?. On NVIDIA GPUs enable cuda and pass &Device::new_cuda(0)?. The Metal build runs attention through candle's fused SDPA kernel and fuses the projection bias, residual, LayerNorm and GELU epilogues. CUDA and CPU use plain tensor attention.

BGE-M3 Joint Embeddings

The BGE-M3 model produces dense, sparse, and ColBERT embeddings simultaneously in a single forward pass.

use fastembed::{Bgem3Embedding, Bgem3InitOptions, Bgem3Model};

// With default options
let mut model = Bgem3Embedding::try_new(Default::default())?;

// With custom options (supporting custom max length up to 8192 tokens)
let mut model = Bgem3Embedding::try_new(
    Bgem3InitOptions::new(Bgem3Model::BGEM3Q)
        .with_max_length(1024)
        .with_show_download_progress(true),
)?;

let documents = vec![
    "Hello, World!",
    "This is an example passage.",
    "fastembed-rs is licensed under Apache 2.0",
    "i dont know"
];

// Generate all three representations in a single forward pass
let output = model.embed(documents, None)?;

println!("Dense dimension: {}", output.dense[0].len()); // -> Dense dimension: 1024

let sparse_emb = &output.sparse[0];
println!("Sparse non-zero tokens: {}", sparse_emb.indices.len());

println!("ColBERT token count: {}", output.colbert[0].len());

Bgem3Model::BGEM3Q is dynamically quantized, so the vectors of a text vary slightly with the other texts in its batch.

Model cache

Models download on first use and load from cache afterwards (no network needed at runtime once cached).

  • FASTEMBED_CACHE_DIR — cache location (default: .fastembed_cache). Equivalent to TextInitOptions::with_cache_dir.
  • HF_HOME — if set, models are cached directly in this directory, and it takes precedence over the above.
  • HF_ENDPOINT — Hugging Face mirror base URL, for restricted networks.
  • HF_TOKEN — Hugging Face access token, for gated repositories and higher rate limits. Without it, the token saved by huggingface-cli login is used when present.

DirectML (Windows)

To run models on a GPU via DirectML on Windows, enable the directml feature:

[dependencies]
fastembed = { version = "7", features = ["directml"] }
# The ort version that fastembed uses, for the execution provider.
ort = "=2.0.0-rc.15"

Then pass a DirectML execution provider when initializing a model:

use fastembed::{TextEmbedding, TextInitOptions, EmbeddingModel};
use ort::ep::DirectML;

let model = TextEmbedding::try_new(
    TextInitOptions::new(EmbeddingModel::AllMiniLML6V2)
        .with_execution_providers(vec![DirectML::default().into()]),
)?;

When DirectML is detected, fastembed automatically disables memory pattern optimization and parallel execution on the ONNX Runtime session, as required by the DirectML execution provider.

Automatic device selection

When no execution provider is registered, ort enables ONNX Runtime's automatic device selection with AutoDevicePolicy::MaxEfficiency, which registers CoreML on macOS. Registering ort::ep::CPU does not prevent this, because the CPU provider is built in and registers nothing. Set a policy to control the selection:

use fastembed::{AutoDevicePolicy, EmbeddingModel, TextEmbedding, TextInitOptions};

let model = TextEmbedding::try_new(
    TextInitOptions::new(EmbeddingModel::AllMiniLML6V2)
        .with_auto_device(AutoDevicePolicy::PreferCPU),
)?;

AutoDevicePolicy::PreferCPU selects a CPU device, which in a standard ONNX Runtime build is the built-in CPU execution provider.

Error handling

The ONNX models return a typed fastembed::Error. The type is re-exported from the crate root. The enum is #[non_exhaustive]. New variants can be added in minor releases without breaking match arms. The candle models return candle_core::Error.

use fastembed::{Error, Result, TextEmbedding};

fn load() -> Result<TextEmbedding> {
    let model = TextEmbedding::try_new(Default::default())?;
    Ok(model)
    // ...
}

To handle an error, match on the variant that applies:

use fastembed::{Error, TextEmbedding, TextInitOptions, EmbeddingModel};

match TextEmbedding::try_new(TextInitOptions::new(EmbeddingModel::AllMiniLML6V2)) {
    Ok(model) => { /* ... */ }
    Err(Error::ModelRetrieval { file, source }) => {
        eprintln!("could not fetch {file}: {source}");
    }
    /*
    ...
    */
    Err(Error::Ort(err)) => {
        eprintln!("ONNX runtime error: {err}");
    }
    Err(e) => eprintln!("{e}"),
}

LICENSE

Apache 2.0

About

Rust library for generating vector embeddings and reranking locally!

Topics

Resources

Stars

1.1k stars

Watchers

5 watching

Forks

Releases

Used by

Contributors

Languages