Post

The "Labs are nerfing their models post-release" conspiracy is being joined by its antagonist: "Labs are secretly routing to completely different, stronger models".

The thing to keep in mind is that routing to a newer model will cause all sorts of severe regressions in production systems. Coding harnesses meant for interactive use are *slightly* less sensitive to this, but I still don't buy it. I'm going to say 80% that the silent re-routing one is false too.

I think Google is doing A/B testing on its AI Studio Playground.

I'd consider this part of the *barely plausible* category. Though I'll up my P(routing) if you say that you've personally observed this. :) Everything I see on X about these claims are vibes and one-off results (typically some 3D modeling prompt that is gonna be high variance to begin with).

What sort of regressions are you thinking of? I could see it for small/old models where prompt engineering is a thing, or for changing between model families... but for modern models, as long as context is >= expected and structured responses are equivalent, I don't see what would cause regression.

Lots of model releases (even within family) changing the style and length of their responses. Different tool-call behavior, different calibration points in decision prompts.