Post

We upgraded AI SRE to use Opus 4.7 yesterday after running a bunch of benchmarks against various incidents to check how it performs. For anyone looking at a similar upgrade, some takeaways…

Lawrence Jones

@lawrencejones.dev

19 Apr

1. Token usage was marginally increased: 4.7 uses a different tokeniser that will produce more tokens for the same content, which impacts costs. In practice we only saw 5-10% more usage, so pretty minor.

Lawrence Jones

@lawrencejones.dev

2. Effort levels have 'inflated': replacing 4.6 for 4.7 lead to a decrease in performance for us when using the same effort levels. We had a collection of medium effort 4.6 which only started performing better when we moved to xhigh on 4.7.

19/4/26, 4:36 pm0 reposts0 likes

Lawrence Jones

@lawrencejones.dev

19 Apr

3. Models are already smart enough: this model is obviously better and does improve our performance, but we only saw an uplift of 75% -> 81% accuracy on a dataset of 'hard' incidents.

Lawrence Jones

@lawrencejones.dev

19 Apr

This is what we'd expect given previous upgrades, and is candidly quite comforting: No matter how good the model gets, it's the harness that's going to get us into the high 90%+ accuracy, raw intelligence isn't going to solve this problem for us!