Post

Fable 5 is the new highest scorer on bluffbench! The eval measures models' ability to accurately describe plots that show counterintuitive patterns. Six months ago, the strongest models were still in the single digits. simonpcouch.github.io/bluffbench/

Viktor Rognås

@vrognas.com

11 June

Are the thinking modes similar enough to compare; would it make sense to also compare the highest available thinking (eg xhigh or max for fable 5), since that is what users probably will use for tricky data assessments? Also, it would be interesting to see how a human would score.

Simon P. Couch

@simonpcouch.com

Yeah, the thinking levels aren't perfectly apples-to-apples. Our default thinking level in Posit Assistant is medium and, among folks who have opted-in to data collection, many leave it as-is. :) At least on this subset of the eval, I'd suspect humans would do very very well, but we haven't tested.

11/6/26, 3:57 pm0 reposts1 like

Viktor Rognås

@vrognas.com

12 June

Thanks for the reply, makes sense that you compare the Posit Assistant defaults.