Raphaël Merx on Mythryl: “Incredible paper, finding that large companies can game the LMArena…”

Incredible paper, finding that large companies can game the LMArena through statistical noise (via many model submissions), over-sampling of their models, and overfitting to Arena-style prompts (without real gains on model reasoning) The experiments they run to show this are pretty cool too!

Sara Hooker@sarahooker.bsky.social

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

2/5/25, 1:28 pm0 reposts4 likes

Post