Add topk-io/Iso-ModernColBERT model meta - #4812

Can you run tasks to reproduce results?

I checked this branch again and did a best-effort local run.

What passed locally:

.venv312/bin/python - <<'PY'
from mteb.models.model_implementations import MODEL_REGISTRY
meta = MODEL_REGISTRY['topk-io/Iso-ModernColBERT']
print(meta.name)
print(meta.revision)
print(meta.framework)
print(meta.modalities)
print(meta.extra_requirements_groups)
print(meta.loader)
PY

Output confirms the registry entry resolves to MultiVectorModel with the pinned revision a43b93e62b11ff205b7f935c1ce2207bfed2d283 and pylate as the extra requirement. I also confirmed pylate, torch, transformers, and datasets are installed in the local env, and the model weights load successfully through mteb.get_model('topk-io/Iso-ModernColBERT').

I then tried to run the relevant small retrieval task:

.venv312/bin/mteb run \
  -m topk-io/Iso-ModernColBERT \
  -t NanoMSMARCORetrieval \
  --eval-splits train \
  --batch-size 64 \
  --output-folder /tmp/mteb-iso-moderncolbert-results \
  --verbosity 2 \
  --no-co2-tracker \
  --overwrite-strategy always

That run got through model loading, task loading, corpus standardization, and entered the PyLate MultiVector retrieval path, but I stopped it after several minutes on my local CPU-only run before it produced final metrics. NanoMSMARCORetrieval only has a train split in this checkout; using --eval-splits test fails with KeyError: 'test'.

Also, the currently failing lowest-deps CI job looks unrelated to this PR: it fails while downloading sentence-transformers/all-MiniLM-L6-v2 from the HF CDN with SSLEOFError, not while loading or testing topk-io/Iso-ModernColBERT. I tried to rerun the failed workflow, but GitHub requires repository admin rights for that rerun.

NanoMSMARCORetrieval only has a train split in this checkout; using --eval-splits test fails with KeyError: 'test

Yes, you don't need to run it on train

Thanks, that makes sense. I reran this on a proper non-train eval split instead of using the Nano train split.

Completed local run:

import mteb

model = mteb.get_model("topk-io/Iso-ModernColBERT", device="mps")
evaluation = mteb.MTEB(tasks=[mteb.get_task("NFCorpus")])
results = evaluation.run(
    model,
    eval_splits=["test"],
    output_folder="/tmp/mteb-iso-moderncolbert-nfcorpus-mps",
    overwrite_results=True,
    co2_tracker=False,
    encode_kwargs={"batch_size": 16},
)

Run details:

• task: NFCorpus
• split: test
• dataset revision: ec0fa4fe99da2ff19ca1214b7966684033a58814
• model revision: a43b93e62b11ff205b7f935c1ce2207bfed2d283
• MTEB version: 2.15.4
• device: Apple MPS via Python API; the CLI in this checkout parses --device as an int, so --device mps is rejected
• evaluation time: 246.7s

Key metrics from NFCorpus.json:

• main_score / ndcg@10: 0.36991
• map@10: 0.14613
• recall@10: 0.17644
• precision@10: 0.26625
• mrr@10: 0.586329

So the registered MultiVectorModel path loads the pinned model and completes a retrieval task on a held-out test split.

They report 37.31 for nfcorpus. Do you have ideas why scores have this difference?

I checked a few likely causes. I do not see evidence that the registry entry is loading the wrong revision/model, but I also cannot exactly reproduce the model-card number from the MTEB path.

What I verified locally:

• The difference is small in absolute terms: my MTEB run is 36.991 NDCG@10 vs the model card's 37.31 for BEIR NFCorpus, so about 0.319 NDCG points.
• The model card says those numbers come from an internal evaluation harness on BEIR, with Iso evaluated in bf16 and exact MaxSim for the NDCG@10 table.
• The MTEB run uses mteb/nfcorpus at dataset revision ec0fa4fe99da2ff19ca1214b7966684033a58814 through the MultiVectorModel / PyLate wrapper.
• I reran with explicit bf16 loading and the card's example batch size:

model = mteb.get_model(
    "topk-io/Iso-ModernColBERT",
    device="mps",
    model_kwargs={"torch_dtype": torch.bfloat16},
)
# dtype torch.bfloat16, query_length=32, document_length=300
# NFCorpus/test main_score remains ~0.37

• I also tested whether the gap was caused by the 300-token PyLate document default vs the model's 8192 ModernBERT context. Forcing document_length=8192 did not explain it; it lowered this MTEB run to 36.115 NDCG@10.

So my best guess is that this is a small harness/data-path difference rather than a metadata issue: TopK's internal BEIR evaluation vs MTEB's mteb/nfcorpus wrapper and PyLate retrieval/scoring path. I can ask the model authors for the exact BEIR eval command/harness if you want the PR to match the card number more tightly before merge.