Benjamin Muskalla on Mythryl: “As agents get more clever, so do their attempts at benchmark…”

As agents get more clever, so do their attempts at benchmark hacking. Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard. These were our findings: poolside.ai/blog/through...

poolside.aiThrough the looking glass of benchmark hackingOutlining some of the reward hacks we’ve encountered and what strategies we are exploring to resolve them.

11/5/26, 5:38 pm0 reposts1 like

Post