Summary

NomicV2MoeTextEmbedding::from_hf(..., DType::BF16, ...) (or F16) loads successfully but every subsequent embed() call fails with:

embed failed: dtype mismatch in mul, lhs: BF16, rhs: F32

Same error pattern with DType::F16. Only DType::F32 works end to end.

Reproduction

use candle_core::{DType, Device};
use fastembed::NomicV2MoeTextEmbedding;

let device = Device::new_metal(0).or_else(|_| Ok::<_, anyhow::Error>(Device::Cpu)).unwrap();
let model = NomicV2MoeTextEmbedding::from_hf(
    "nomic-ai/nomic-embed-text-v2-moe",
    &device,
    DType::BF16,            // ← bug: model loads, embed() will fail
    512,
)?;
let _ = model.embed(&["search_query: hello world"])?;  // ← errors here

Tested against fastembed 5.13.4 with default-features = false, features = ["nomic-v2-moe", "ort-download-binaries-native-tls"], candle-core 0.10.2, on Apple Silicon (M-series, Metal device).

Root cause (likely)

Some tensor in src/models/nomic_v2_moe.rs (likely a hardcoded constant — RoPE table, attention bias, layer-norm gamma/beta, or the gating router weights) is built with DType::F32 regardless of the dtype the rest of the model was loaded into. When that constant gets multiplied against a BF16 activation later in the forward pass, candle's mul rejects the dtype mismatch.

The DType parameter in from_hf is correctly threaded into the safetensors weight loader (the model loads fine and accepts batches), but at least one call site in the inference path doesn't .to_dtype(self.dtype) an intermediate.

Impact

For users wanting to reduce memory (BF16 halves weight memory and roughly halves activation memory), there's no functional path. We had to revert to F32 and pay the full ~9.7 GB physical footprint per actively-loaded MCP on Apple Silicon (most of it Metal MTLBuffers — weight mirror + kernel cache + activation pool, all proportional to dtype size).

Suggested fix

Audit src/models/nomic_v2_moe.rs for Tensor::new, Tensor::ones, Tensor::zeros, Tensor::arange, etc. that don't take a dtype parameter — these default to F32. Or for explicit DType::F32 usages that should instead use the model's stored dtype. Then add a regression test that runs embed() with each of DType::F32 | F16 | BF16 and asserts shape/no-error.

Happy to send a PR if a maintainer can point me at the right file region.