MTEB

New on MTEB: shekar-ai/Noql, a 12M-parameter Persian embedding model. Of the models with a published size and results on all 52 tasks of MTEB(fas, v2), it is the smallest. It averages 58.20; the next-smallest, MCINext/Hakim-small (39M), averages 68.55. Results submitted by amirivojdan.

1/ When we built MAEB, we cut audio at 30 seconds per clip to keep evaluation affordable. Many models support much longer audio, and at least 16 of our audio tasks have longer clips (up to 21 minutes), so those models were only scored on the start of each recording.

Submitted my two accepted posters for EMNLP today, if you are there an want meet up feel free to reach out! I also have a not so @mteb.org themed poster for a workshop on socio-cultural alignment -- stay tuned for that one

New on MTEB: litillabs/octen-law-8b-v1 takes 1st of 164 on MTEB(Law, v1) at 76.77, 4.19 clear of the next model. It declares two of the eight tasks as training data. Recomputed without them, the lead widens: 81.77 to 75.00. Model and results by narcolepticchicken.

New benchmark: MTEB(slk, v1) is MTEB's first Slovak board — 31 tasks across 7 task types, 17 built for it: parliamentary sentiment, fact-checking NLI, pharmacy Q&A, news clustering. Of the 8 models scored on all 31, intfloat/multilingual-e5-large-instruct leads. Contributed by andrejridzik.

On LMEB (long-horizon memory retrieval), opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini (11M active) sits on the Pareto frontier — nothing smaller scores higher, at any size up to 7.4B active. Newly scored, not newly released: one of 12 models that got LMEB results this week.

New benchmark: BRIGHT-Pro, a new multi-aspect retrieval benchmark on real StackExchange questions across 7 domains, built for agentic search — scored on surfacing several reasoning aspects, not one passage. AQ-MedAI/Diver-Retriever-4B-1020 leads at 55.88. Contributed by yilunzhao.

New on MTEB: malteos/most-embed-de, a German retrieval model fine-tuned from Nemotron-3-Embed-1B, takes 1st of 106 models with full coverage on a custom 6-task German-retrieval subset at 58.07 — beating the runner-up at roughly 1/15th the active parameters.

1/ MTEB's audio and video coverage grew substantially this month — 25 new tasks across 16 pull requests, spanning music, speech, environmental sound, and video retrieval. 10 of the 16 cite a published paper.

lightonai/mLateOn and mDenseOn are multilingual extensions of LateOn and DenseOn — eight languages plus English, same backbone. On English retrieval, both beat the English-only models they extend: 23rd of 197 vs 25th, and 28th vs 31st. Added by NohTow, results by paulomouraj.

New benchmark: MTEB(kor, v2) is now the default Korean board, superseding the 6-task v1. 20 tasks across 6 task types, adding clustering and NLI and with an increased focus on retrieval. Added by daegonYu.

New on MTEB: webAI-Official/webAI-ColVec1.1-8b, a visual document retrieval model, takes 1st of 41 on ViDoRe(v3) at 64.95. Its 4B sibling places 3rd at 63.90 — and both beat the lab's previous generation, which sits 6th and 7th.

Most embedding models publish their weights. Far fewer publish what it took to build them. Of 171 models with complete MTEB(Multilingual, v2) results: 153 open-weight, 10 fully open (weights + training code + data), 8 proprietary.

New on MTEB: Hanno-Labs/dinghy-law-4b-v1, a legal-domain embedding model, takes 1st on MTEB(Law, v1) at 71.22. Its 0.6B sibling places 4th at 65.83 — the smallest model in the top eight by an order of magnitude.

The best embedding score available at the end of 2023 needed a 304M-parameter model. Today the same score comes from an 11M one — 28x fewer active parameters.