Skip to content

Add perplexity-ai/pplx-embed-v1-late-0.6b model meta - #4813

Merged
Samoed merged 3 commits into
embeddings-benchmark:mainfrom
wgu9:add-pplx-embed-late-model
Jul 31, 2026
Merged

Add perplexity-ai/pplx-embed-v1-late-0.6b model meta#4813
Samoed merged 3 commits into
embeddings-benchmark:mainfrom
wgu9:add-pplx-embed-late-model

Conversation

@wgu9

@wgu9 wgu9 commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #4691.

Adds the ModelMeta entry for perplexity-ai/pplx-embed-v1-late-0.6b to
mteb/models/model_implementations/pylate_models.py.

It is a PyLate / ColBERT late-interaction (MaxSim) retrieval model —
continued training of perplexity-ai/pplx-embed-v1-0.6b with a token-level
128-dim projection. It loads through the same MultiVectorModel loader as the
other ColBERT entries (with trust_remote_code=True, since the repo ships
custom modeling code).

Metadata sources (model card / HF API / config)

  • revision 35e14f7c0f85720323342965e72f119dc1a937c0 (current main)
  • license mit, n_parameters 595,778,560 (safetensors total)
  • embed_dim 128 (1_Dense/config.json out_features, confirmed in the card)
  • max_tokens 512 (document_length in config_sentence_transformers.json)
  • adapted_from perplexity-ai/pplx-embed-v1-0.6b (HF base_model)
  • languages left None: the base model card only states "multilingual"
    without enumerating language codes (consistent with other entries that omit
    a language list rather than guessing).

Verification

from mteb.models.model_implementations import MODEL_REGISTRY
MODEL_REGISTRY["perplexity-ai/pplx-embed-v1-late-0.6b"]  # registers cleanly
  • ruff check / ruff format --check: clean
  • pytest tests/test_models/test_model_meta.py -k "name_and_revision or hashable or without_prefix": 731 passed

@wgu9
Add perplexity-ai/pplx-embed-v1-late-0.6b model meta
b76cd22 
Closes embeddings-benchmark#4691. Adds the ModelMeta entry for perplexity-ai/pplx-embed-v1-late-0.6b,
a PyLate/ColBERT late-interaction (MaxSim) embedding model with 128-dim
token-level vectors, continued-trained from perplexity-ai/pplx-embed-v1-0.6b.
Metadata sourced from the HF model card/API and config files (revision, MIT
license, 128-dim projection, 512 document length, ~596M params).
LuuOW
LuuOW reviewed Jun 14, 2026
View reviewed changes

@LuuOW LuuOW left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical audit: Implementation verified for architectural consistency and engineering integrity.

@Samoed

Samoed commented Jun 14, 2026

Copy link
Copy Markdown
Member

Can you run BEIR to reproduce resultls? Would be greate to have review from @bwang-pplx

@Samoed Samoed changed the title Add perplexity-ai/pplx-embed-v1-late-0.6b model meta (#4691) Add perplexity-ai/pplx-embed-v1-late-0.6b model meta Jun 14, 2026
@Samoed Samoed added the new model Questions related to adding a new model to the benchmark label Jun 15, 2026
bwang-pplx
bwang-pplx approved these changes Jun 17, 2026
View reviewed changes

@bwang-pplx bwang-pplx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Samoed thanks for the ping! the PR looks correct

@wgu9
Merge remote-tracking branch 'origin/main' into add-pplx-embed-late-m…
b1e9ecf 
…odel

# Conflicts:
#	mteb/models/model_implementations/pylate_models.py
@wgu9

wgu9 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, I refreshed this branch on current main and resolved the registry conflict with the now-merged topk-io/Iso-ModernColBERT entry. The PR diff is back to a single pylate_models.py metadata addition.

Validation after the refresh:

  • .venv312/bin/python -m pytest tests/test_models/test_model_meta.py -k "name_and_revision or hashable or without_prefix" -q -> 745 passed
  • .venv312/bin/ruff check mteb/models/model_implementations/pylate_models.py tests/test_models/test_model_meta.py
  • .venv312/bin/ruff format --check mteb/models/model_implementations/pylate_models.py tests/test_models/test_model_meta.py
  • git diff --check origin/main...HEAD
  • registry smoke: MODEL_REGISTRY["perplexity-ai/pplx-embed-v1-late-0.6b"] resolves to MultiVectorModel at revision 35e14f7c0f85720323342965e72f119dc1a937c0, with extra_requirements_groups=["pylate"] and n_embedding_parameters=155_311_104.

For BEIR reproduction: I tried to run NFCorpus/test locally on Apple MPS through the MTEB Python API with batch_size=16 and again with batch_size=64. The model loaded successfully and entered the PyLate full-corpus retrieval path, but I did not get final metrics in a reasonable local window. I interrupted both runs; both stacks were in document encoding inside MultiVectorModel._pylate_full_corpus_search -> _encode -> pylate.models.colbert.ColBERT.encode -> skiplist_mask.

I also verified a small same-wrapper encode smoke on MPS using NFCorpus task metadata: model._encode([{"text": [..]}], batch_size=2, task_metadata=task.metadata, hf_split="test", hf_subset="default") completed and returned two multi-vector embeddings, first shape (6, 128).

So I can confirm the metadata/loader/wrapper path, but I do not want to claim a BEIR score I could not complete locally. The earlier review from @bwang-pplx says the metadata looks correct; if you need an exact BEIR score before merge, this likely needs a maintainer/GPU-backed run rather than my local MPS run.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the stale label Jul 10, 2026
@wgu9

wgu9 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Still interested in getting this merged. The model-author review from @bwang-pplx is approved, the branch was refreshed against main, and CI is green. @Samoed, is a full BEIR reproduction still required before this metadata-only entry can merge? If so, I can run it and post the exact command and results here.

@Samoed

Samoed commented Jul 15, 2026

Copy link
Copy Markdown
Member

I'll try to run it, but this would take a lot of time

@github-actions github-actions Bot removed the stale label Jul 16, 2026
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the stale label Jul 31, 2026
@Samoed
Samoed merged commit 7c2712d into embeddings-benchmark:main Jul 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Samoed Samoed Samoed approved these changes
+2 more reviewers
@LuuOW LuuOW LuuOW left review comments
@bwang-pplx bwang-pplx bwang-pplx approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

new model Questions related to adding a new model to the benchmark stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add model: perplexity-ai/pplx-embed-v1-late-0.6b

4 participants

@wgu9 @Samoed @LuuOW @bwang-pplx