Post

Are the thinking modes similar enough to compare; would it make sense to also compare the highest available thinking (eg xhigh or max for fable 5), since that is what users probably will use for tricky data assessments? Also, it would be interesting to see how a human would score.

Yeah, the thinking levels aren't perfectly apples-to-apples. Our default thinking level in Posit Assistant is medium and, among folks who have opted-in to data collection, many leave it as-is. :) At least on this subset of the eval, I'd suspect humans would do very very well, but we haven't tested.