Kato Coaching

The engineer who reads every line of every AI-written PR is now the slowest part of your release. Review metrics don't always show it. I counted:

Tests green, a bot has reviewed it, three more waiting, so you approve it. Nothing records whether a person read it. I counted seven repos:

Asked AI for 30 rows of signup test data. Got 30 tidy, valid, useless rows. Treating it as a pair instead of an oracle, constrain, generate, verify, surfaces the bug the happy-path data missed. Two-minute demo on a real form.  #softwaretesting 

The argument for not testing a prototype is that you don't know if it's worth keepin. That's no longer true.  I'm partnering with SmartBear on a sponsored series of content about tools for the AI era. Opinions are entirely my own.

Buying a testing agent: easy. Knowing if it works, recalibrating it, watching its reasoning, drawing its authority line: not in the box. The five questions I'd start with: https://kato-coaching.com/fighting-ai-with-ai-easy-part/ Paid partnership w/ SmartBear.

Holy guacamole. Long-ish read, pretty jaw dropping. bobdahacker.com/blog/fifa-hack

You can't manually test AI-generated code at AI-generation speed. That math isn't mathing. 40% of teams generate 41%+ of their code with AI. 92% still test by hand. Source:

Everyone in QA can describe their AI's bad habits without thinking. Nobody finishes "what I really want from this tool is…" If you've been generating tests with AI and correcting the same handful of things every week, what would actually help?

The research on code review has been consistent for over a decade: it's strong at knowledge sharing, weak at catching defects. When teams insist on "humans review every line" of AI output, they're doubling down on the thing we know doesn't work. kato-coaching.com/human-in-the-loop-code-review

The AI keeps making the same test mistake because the decision to stop is buried in a Slack thread it's never read. Write the rule. Put it in the file your AI reads at generation time. Walkthrough: https://youtu.be/ccFaZdGUSkw #AIinTesting #QA

> 65% of QA teams describe AI output as "noisy, incorrect, or requiring rework." > > Mapping where the verification time actually goes. 5-min survey, no opt-in: >

Last night Alan and I talked about the new cool kid on the block, and how it's changing testing. Or maybe not?  I'll link Apple podcasts, but it's available everywhere you get your podcasts. 

Our free web development bootcamp launch party is today! Join: @vivaldi.com's @brucel.bsky.social will be showing us the lovely side of HTML Me for how good notes drive learning @freecodecamp.bsky.social's @quincy.bsky.social for a fireside chat about learning We would love to see you there!

It’s no secret that I love working with claude code. If you’ve not tried it yet but would like to: turns out I’ve got three passes for a free week of it. Grab it here claude.ai/referral/WY0...

I have a Claude Code skill that creates google slides for me, and then adds speaker notes. Since the update to Sonnet 4.7 it constantly "forgets" how to do the latter, and tells me it's incapable of it. Anybody else have this problem? Suggestions?

The "testing mindset" isn't a personality trait. It's trained. Developers often don't think about quality the way QA engineers do because they haven't had the same deliberate practice. That's fixable, but only if the culture reinforces it too.

Before you conclude AI isn't worth it, measure how long correction actually takes. One experiment, a real task, less than a day.  Free guide: resources.kato-coaching.com/5-ai-experiments #SoftwareTesting #QA #AIinTesting

The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. His team suggests using AI to plan the hiding spots. Before handing a task to AI, run it through a simple framework. Here is how the Easter Bunny uses this framework to avoid a disaster. 🐇🥚

Getting the task type wrong makes the model choice irrelevant. The scorecard tells you what kind of solution actually fits before you commit to one.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

TIL: it's much easier to get Claude Code to play nicely with the Google workspace MCP than it is to get Gemini to use it. What the hell? Yes, I am experimenting with a multiple agent setup, but man, it's painful.

AI gave you 12 scenarios. Now what? The list is raw material, not a test plan, and your judgment is what turns it into one. New video on how to triage what AI gives you: https://youtu.be/0awmamL1l2c #softwaretesting #AIinTesting

Agile gave us JIRA. DevOps gave us pipelines. Neither answered the harder question. AI evals is doing the same thing: the tooling is excellent. The question underneath it remains.  https://kato-coaching.com/the-hard-part-of-ai-evals-isnt-the-tooling/ #AITesting #QualityEngineering #SoftwareTesting

Most AI pilots produce impressions, not evidence. An acceptable error rate isn't a feeling. It's a number you write down before the experiment starts.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

"The profession keeps arriving at “quality is systemic” as though it’s a fresh revelation, every time a new technology makes the verification frame obviously inadequate. It happened with continuous delivery. [...] It is happening now with AI agents."

Every AI demo works because the vendor picked the use case. When you're back at your own codebase, the conditions are different. Define your problem before you evaluate the tool.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

The vague prompt gets you a generic list. Two minutes of context - scope, users, risk angle - gets you scenarios worth thinking about. Short video on how to close that gap.  https://youtu.be/55SLTvKq7rE #softwaretesting #AIinTesting

The AI evals field calls it a rubric. If you've done serious testing work, you probably know it by a different name.  https://kato-coaching.com/the-anatomy-of-a-metric/ #SoftwareTesting #AITesting

When AI adoption is handed to IT, you get a deployment. Deciding what good output looks like and how to verify it belongs to the people doing the work. https://resources.kato-coaching.com/scorecard?utm_source=Bluesky&utm_medium=post&utm_campaign=scorecard  #QualityEngineering #TechLeadership

When teams say the AI tool isn't working, I ask about the process before the AI arrived. Usually the requirements were always vague. The tool is just making that visible. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AIinTesting #QA

Six months into an AI pilot with no success criteria is just six months of accumulated impressions.  You can't pass or fail a hope. 

I am working on a food tracker for my very specific needs, and Claude just turned into every developer I ever worked with.

"The hard part is scalability, not automation." That line from session one of "AI evals and analytics" confused me. Session two explained it.  Full write-up in my blog: https://kato-coaching.com/the-ai-evals-field-chose-a-flawed-tool-and-stuck-with-it/ #AIEvals #SoftwareTesting #QA

Updated my website this week — it should finally be clear what I do and how to work with me. Courses, workshops, and 1:1 coaching for QA professionals.  kato-coaching.com

If the output is slop regardless of how you phrase it, the problem isn't the prompt. It's the use case.  Free scorecard:

Before I wrote today's post, I defined what good looked like: gives value, doesn't rely on outrage, sounds like me. Writing to a clear brief changes the experience. So does diagnosing a draft. Testers do this before they run anything.

The AI correction loop usually starts before the tool is opened, at the moment someone chose the wrong use case for it. Testers already know how to ask whether a tool suits a problem. That skill just hasn't been applied here yet. More soon.

The skills QA professionals already have (defining success criteria, testing behaviour, not trusting metrics at face value) are exactly what's missing from most AI integrations. I'm learning AI evals to understand why.  Session one: kato-coaching.com/what-i-dont-understand-about-ai-evals-yet/

Ran my workshop "Deciding Fast" on Tuesday with a software team in Sweden. Everyone in one room sharing computers, no breakouts. It's built for remote. I adapted. 6/8 Good or Excellent. Best response to "what will you do differently?": "Set clearer success condition." #SoftwareTesting #AITesting

"Tell me what you're uncertain about." "Push back if my assumptions are wrong." Only 30% of people give instructions like these. The model defaults to confident and agreeable.  https://www.anthropic.com/research/AI-fluency-index #SoftwareTesting #AITesting #AILiteracy

The strongest predictor of AI fluency, per Anthropic's research: iteration. Treating the first response as a draft, not an answer. 5.6x more likely to question reasoning. Familiar territory if you work in testing.  https://www.anthropic.com/research/AI-fluency-index #SoftwareTesting #AITesting

Six months in and nobody can say whether the AI is actually working. Not anecdotally, but with evidence. That gap is the most common thing I see in QA teams right now.  How do you measure if the new licence is worth the money? #softwaretesting #QA #AItools

18% of testers I surveyed said their top AI frustration is not bad output. It is that the tools have no sense of test strategy. The AI is not wrong. It is indiscriminate. #AITesting #SoftwareTesting #TestStrategy

I timed myself running tests manually, then gave the same task to an AI tool and timed that too. The result was not what I expected. Full breakdown with video later this week.