Post

🤔Consider the scenario LLaMA 65B scored 0.637 on HELM's MMLU LLaMA 65B scored 0.488 on lm-eval-harness's MMLU Same model. Same benchmark name. Different prompts, settings, extraction methods. 💡Which score is right? Both? Neither? We can't compare. 🤷

This has real costs! 🔬 Signal buried in noise, can't tell if differences reflect model capability or just setup 📦 Evaluation debt piles up silently across the ecosystem 🔎Redundant re-runs of expensive evaluations 🌟That's where Every Eval Ever comes

What we built: 📋 Metadata schema for cross-framework comparison 🔧 Validation via Hugging Face Jobs 🔌 Converters (Inspect AI, HELM, lm-eval-harness) 📊 Community repo organized by benchmark/model/run ✨ Captures scores AND context: settings, prompts, example-level data