Jindřich Libovický

I’m heading to #EMNLP2026 in Budapest 🇭🇺! Here’s where you can find us 🧵👇

aclanthology.orgCUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering DatasetJindrich Libovický, Jindrich Helcl, Andrei-Alexandru Manea, Gianluca Vico. Transactions of the Association for Computational Linguistics, Volume 14. 2026.

🎤 Monday, 16:45, G13-14: I’ll present our TACL paper CUS-QA on regional question answering and its evaluation (w/ @jindrahelcl.bsky.social , @andrei-a-manea.bsky.social & @gianlucavico.bsky.social ) aclanthology.org/2026.tacl-1....

arxiv.orgPredicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the given language are more aligned to English within the mo...

🌍 Tuesday morning poster session: Adnan Al Ali presents their paper on predicting how well an LLM performs across languages by comparing its hidden states with those for English (w/ @kathaem.bsky.social & Alex Fraser) arxiv.org/abs/2608.03446

Jindřich Libovický reposted

Yuval Pinter

@uvp.bsky.social

if you're at #colm2026, why not come listen to some fun #tokenization talks? Continental 1, all day long tokenization-workshop.github.io/schedule/

tokenization-workshop.github.ioSecond Tokenization Workshop @ COLM 2026 | Schedule

tokenization-workshop.github.ioSecond Tokenization Workshop @ COLM 2026 | Speakers

TokShop is just one week away! 🎉 I'm proud of the lineup of invited speakers we've put together. Meet them here: tokenization-workshop.github.io/speakers

Tokenization Workshop (TokShop) @COLM2026@tokshop.bsky.social

🎉 Our last invited speaker announcement — TokShop is next week! Welcome Artidoro Pagnoni @artidoro.bsky.social (Meta Superintelligence, FAIR), lead author of the Byte Latent Transformer (BLT) and co-creator of QLoRA. Best paper award & orals at ACL/NeurIPS.

alphaxiv.orgTokenization: A Survey for Modern NLPTokenization is presented as a core language-model design choice that shapes sequence length, computational cost, multilingual equity, evaluation, and security—not merely as preprocessing. The...

Everything you always wanted to know about tokenization but were afraid to ask! 🧩 @mcognetta.bsky.social l and @uvp.bsky.social nagged 30+ researchers (myself included) until we ended up with the most comprehensive survey on tokenization in NLP: www.alphaxiv.org/abs/2609.tok...

Marco@mcognetta.bsky.social

🚨 [Token][ization] Paper Alert 🚨 Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!

2027.aclweb.orgMain Conference PapersACL 2027 Call for Papers.

ACL announcing the deadline for ACL 2027...

ACL 2027@aclmeeting.bsky.social

🔔CFP for the #ACL2027NLP main conference is up! 2027.aclweb.org/calls/main/ 📝Submission deadline to ARR is Jan 4, 2027.

21 Sept

I really enjoyed giving a talk at the Machine Translation Marathon 2026 about our research on tokenization evaluation. Thank you for having me! For those interested, the slides from my presentation are available here: docs.google.com/presentation...

2 Sept

Very happy to share that Adnan Al Ali's first big PhD project has been accepted to #EMNLP2026! 🎉 Joint work with @kathaem.bsky.social, me & Alex Fraser. Paper: arxiv.org/abs/2608.03446 What predicts multilingual LLM performance in a language? 🧵

2 Sept

Answer: similarity of that language's representation to English. Unless two languages are very close (e.g., Czech/Slovak), alignment with English matters much more than similarity to the source language. Want to know more? Catch Adnan at #EMNLP2026 in Budapest!

aclanthology.orgLLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric ValidationLukáš Eigler, Jindřich Libovický, David Hurych. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). 2026.

7 July

Very proud of my student Lukáš Eigler presenting this at the ACL Student Research Workshop. TL;DR: you can validate NLP evaluation metrics with synthetic LLM judgments — the rankings track human ones almost perfectly. 📄 aclanthology.org/2026.acl-srw.125

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation aclanthology.org/2026.acl-srw... by Lukáš Eigler, @jlibovicky.bsky.social & David Hurych Rankings from synthetic LLM-generated data almost perfectly match real human judgments. 🤖⚖️ at Student Research Workshop

6 July

Dušan made something like TensorBoard for tokenization 📊 Token length, entropy, vocab overlap, JS divergence, alignment scores: sliced by language family, script, region, speakers, data availability. 🔗 Code: github.com/ufal/TokCollate 🔗 Demo: quest.ms.mff.cuni.cz/tokcollate

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

Dušan Variš presents joint work w/ @abyste.bsky.social & @jlibovicky.bsky.social: TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages aclanthology.org/2026.acl-dem... A dashboard for your tokenizer 👀 See which languages you left behind. Open-source, MIT.

aclanthology.orgTokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across LanguagesDušan Variš, Abishek Stephen, Jindřich Libovický. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 2026.

6 July

Congrats Dušan and @abyste.bsky.social ! 🎉 Btw. the demo ships pre-loaded with FLORES-200, so you can compare 21 tokenizers across 222 languages right out of the box. 📄 aclanthology.org/2026.acl-demo.41

Tokenization Workshop (TokShop) @COLM2026

2 July

AC observation this ARR cycle: I'm getting way more reviews and way earlier, than usual. I suspect it's the draconian desk-reject threats working, and not a sign of better reviewing culture. I worry that this gets read as 'threats work, do more of them, discipline the lazy researchers'.

Jindřich Libovický reposted

@tokshop.bsky.social

23 June

📣 TokShop 2026 deadline extended! 🗓️ New submission deadline: Friday, June 26, 2026 (AoE) Research papers (up to 9 pages) and extended abstracts (up to 2 pages) are welcome. Submit: openreview.net/group?id=col... More info: tokenization-workshop.github.io

arxiv.orgCHALIS: A Challenge Dataset for Language Identification in Difficult ScenariosWe present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic no...

15 June

Is "Je to obrovský problém, tvrdí české ministerstvo obrany." Czech or Slovak? Both. Language ID systems must pick one: and that's just one way they break. Our new benchmark CHALIS exposes how fragile they really are. 📄 arxiv.org/abs/2606.06088 🤗 huggingface.co/datasets/michal-tichy/CHALIS

15 June

This dataset is a part of master thesis of my student Michal Tichý, who just defended his thesis! 🎉 Congrats!

arxiv.orgLLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric ValidationValidating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalabl...

9 June

Lukáš Eigler defended his thesis (co-supervised with David Hurych, @valeoai.bsky.social) 🎉 Congrats! #NLP metric validation needs 🐌💰 human judgment data. Our fix: generate synthetic data for metric validation. ✅ Tested on MT, QA, summarization. To appear #ACL2026 SRW: arxiv.org/abs/2603.09403

jlibovicky.github.ioHow I grade 200 exams every yearThree years ago, I took the introductory machine learning course over from Milan Straka, and one of the problems I had to deal with was: how do I grade 250 written exams without it consuming my entire...

5 June

I have to grade 200+ ML exams/year. To survive, I coded a bunch of tools: unique exam generation, handwriting recognition, and per-question stats. Check out the details on my blog. 📊 jlibovicky.github.io/2026/06/04/G...

22 May

Send your work to TokShop! tokenization-workshop.github.io

Tokenization Workshop (TokShop) @COLM2026@tokshop.bsky.social

Announcing First Call for Papers: Second Tokenization Workshop 🔡 📣 ▶️ Non-archival submissions of two types: Research papers (up to 9 pages) ▶️ Extended abstracts (up to 2 pages) Submission deadline June 23, 2026 (AoE) Acceptance notification on July 24, 2026 (AoE) tokenization-workshop.github.io

14 May

TokShop is happening again!

Tokenization Workshop (TokShop) @COLM2026@tokshop.bsky.social

TokShop will be at #COLM2026! 🗓️ October 9th, 2026 📍 San Francisco, USA More details and a call for papers coming soon.

Institute of Formal and Applied Linguistics

30 Mar

Nice work by my student @gianlucavico.bsky.social on a topic close to home: crowdsourcing Piedmontese to test LLMs on non-standard orthography. New dataset covering tokenization, classification & translation.

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography by @gianlucavico.bsky.social and @jlibovicky.bsky.social aclanthology.org/2026.vardial... New Piedmontese dataset tests tokenization, classification & translation! 🗣️

Jindřich Libovický reposted

@ufal.mff.cuni.cz

28 Mar

Replying to @ufal.mff.cuni.cz

On the Credibility of Evaluating LLMs using Survey Questions by @jlibovicky.bsky.social aclanthology.org/2026.mme-mai... Survey-based LLM value evals are unreliable — prompting & decoding choices skew results drastically.

26 Mar

So proud of my new PhD student Adnan Al Ali for presenting their master's thesis work at EACL! 🎓 A great contribution to understanding bias in AI text detectors across languages.

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors by Adnan Al Ali, @jindrahelcl.bsky.social and @jlibovicky.bsky.social aclanthology.org/2026.eacl-sr... AI text detectors are suspected of biasing against non-native speakers, but not in Czech 🇨🇿

26 Mar

I'm at #EACL2026 in Rabat 🇲🇦. Find me and talk to me about tokenization or multilingual model eval. Also, check out our work on how to eval morphological plausibility of your tokenizer if you don't have gold segmentation data, but you happen to have morphosyntactic features 👇

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features by @abyste.bsky.social & @jlibovicky.bsky.social aclanthology.org/2026.finding... TL;DR: Morpho-syntax features replace gold segmentation data for tokenization eval.

20 Mar

Thanks for organizing this! I am definitely coming, and everyone interested in tokenization at #EACL2025 should too. 🫵

Ilker Kesen@ilkerkesen.bsky.social

📢I'm organizing a BoF session at #EACL2026 called Tokenization & Beyond, aiming to gather researchers exploring tokenization and alternatives such as byte-level and pixel-based approaches. Sign up using the form if you're interested! #NLProc @eaclmeeting.bsky.social

ufal.mff.cuni.czIntroduction to Machine Learning with Python | ÚFAL

9 Mar

3rd run teaching ML to 250+ bachelor students (with great materials originaly by @straka-milan.bsky.social). Core philosophy: explain the math, implement algorithms from scratch, Kaggle-style competitions, all auto-graded. ufal.mff.cuni.cz/courses/npfl... But look what LLMs did to the course 👇

9 Mar

Some students find the assignments too time-consuming. Fair. But here's what the data shows over 3 years: 📉 Forum questions dropped ~4× 📈 Full bonus points: 20% → 27% → 52% 📉 Avg. test attempts: 2.7 → 2.4 → 1.9 Asking less, achieving more, iterating less. 🤔

recodex.mff.cuni.czrecodex.mff.cuni.cz

9 Mar

None of this would work without my TAs: Dušan Variš, Tomáš Musil, Jan Bronec, @gianlucavico.bsky.social , Adnan Al Ali, Kristýna Onderková, and @straka-milan.bsky.social taking care of ReCodEx: recodex.mff.cuni.cz. Thank you 🙏

4 Mar

Spent time making AI-generated images of Bayes' Rule, Laplace Smoothing, Markov Chains & Shannon Entropy for class today 🎨🤖 Even though the images are objectively hilarious, none of the 50 students in the room laughed. Or even smiled. 💀

16 Feb

I reviewed papers evaluating LLM values using sociology questionnaires. Different methods, different results. Didn't trust them, so I tested it myself. Methodology matters. Short answers vs CoT, squared err vs KL div.: each changes which populations an LLM "aligns" with. www.arxiv.org/pdf/2602.04033

arxiv.orgarxiv.org

16 Feb

More interesting: humans are predictably inconsistent in their values. LLMs capture this but overgeneralize: they become more stereotypically consistent than actual humans. After several rejections, finally publishable. To appear at the Multilingual Multicultural Evaluation workshop at EACL 2026.

arxiv.orgCUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering DatasetWe introduce CUS-QA, a benchmark for evaluation of open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art l...

3 Feb

We have updated the pre-print on CUS-QA, benchmark for regional knowledge about Czechia, Slovakia and Ukraine arxiv.org/abs/2507.22752 Now, there are results of retrieval-augmented generation and more detailed analysis of model performance depending on the topic of the question or visual context.

Jindřich Libovický@jlibovicky.bsky.social

🧵 We're releasing CUS-QA - a new benchmark for testing LLMs on regional knowledge! Find out what your model knows about Czechia 🇨🇿, Slovakia 🇸🇰, and Ukraine 🇺🇦! 👉 Textual and visual questions, answers, and human judgment on model outputs! huggingface.co/datasets/ufa... www.arxiv.org/abs/2507.22752

huggingface.coufal/cus-qa · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

arxiv.orgEvaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic FeaturesWe present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentati...

2 Feb

We (= mostly @abyste.bsky.social) developed a way to evaluate how morphological a #tokenization is w/o gold segmentation labels. arxiv.org/abs/2601.18536 The key: align subword tokens with morphological features from UniMorph using IBM Model 1. To appear in EACL 2026 Findings.

2 Feb

👉 What do we do? We use the good old IBM1 model to align subwords with morphological features from Unimorph and we show it captures the same thing as morpheme boundary recall. 👉 Why it matters? For many languages good segmentation data is missing. Morphological features are more widely available.

23 Dec 2025

Happy holidays! 🎄🎅🤩🎁

quest.ms.mff.cuni.czWelcome to CrowdTranslation

10 Nov 2025

Attenzione! 🇮🇹 Know Piedmontese or Neapolitan speakers? @gianlucavico.bsky.social is collecting crowd-sourced translations to evaluate LLM performance on these regional languages. Partecipate!

Gianluca Vico@gianlucavico.bsky.social

We’re collecting crowd-sourced translations in Piedmontese and Neapolitan. 🎯 Goal: see how well LLMs understand these languages. 👉 Participate here (in IT🇮🇹): - Piedmontese: quest.ms.mff.cuni.cz/crowd-transl... - Neapolitan: quest.ms.mff.cuni.cz/crowd-transl... Anyone can join, no need to be fluent!

21 Oct 2025

With @andrei-a-manea.bsky.social, we posted a survey on multilingual vision-language models 👉 arxiv.org/pdf/2509.22123 We reviewed 31 models+21 benchmarks. There's a tension between language neutrality (same results across languages) & cultural awareness (context matters differently across cultures)

arxiv.orgarxiv.org

21 Oct 2025

☝️🧵 Most current approaches emphasize langauge neutrality: about two-thirds of VL benchmarks use translation-based evaluation. This makes sense because we can explicitly train for language neutrality when we have parallel data. But... 🧵👇

21 Oct 2025

Cultural awareness is trickier. Different data for different cultures means we can't really compare performance across cultures in a straightforward way. And there's no clear optimization target for cultural awareness beyond curating diverse training data.

1 Sept 2025

So proud of my PhD student @andrei-a-manea.bsky.social for his first first-author publication! 🎉 He presented this work last week at TSD. Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders arxiv.org/pdf/2504.21681

arxiv.orgarxiv.org

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz

Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders by @andrei-a-manea.bsky.social & @jlibovicky.bsky.social TL;DR: Explore how parallel datasets improve cross-lingual transfer in vision-language models. arxiv.org/abs/2504.21681

1 Sept 2025

Most vision-language models only work in English. We explore how different parallel data types (machine-translated vs authentic captions) affect cross-lingual transfer. Key finding: authentic data can outperform machine translation, and multilingual training beats bilingual approaches. #NLP

huggingface.coufal/cus-qa · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

View full thread

The results are... humbling 😅 Even the best models: >40% accuracy on textual questions <30% on visual questions Often perform better in English than the local language (!!) Visual QA with regional images is especially challenging.

For evaluation researchers: Simple string-overlap metrics (BLEU, chrF) work surprisingly well for factual QA. 🤔 When answers are mostly named entities, exact matches matter more than we thought. LLM-as-judge 🦙🧑‍⚖️ correlates best with human judgment, though.

huggingface.coufal/cus-qa · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

The problem: Most QA benchmarks focus on globally known facts. But real users ask about local geography, culture, and history. We collected questions from native speakers in Czechia 🇨🇿, Slovakia 🇸🇰, and Ukraine 🇺🇦 about facts locals know but outsiders don't.