Post

A friend sent me an app that lets you run "user research" with AI participants and asked what I thought about it. I obviously have my (very strong) opinions about it but I thought it could be a fun discussion. So what do you think about using LLMs for user research? #designsky #uxsky

So my gut reaction could be described as “violently against”. But I try to see both sides and I can see a sliver of merit. LLMs are statistics applied to language. Theoretically, you could feed it the right data and ask for general consensus for some high level topics.

But if high level market info is news to you, then I think you have much bigger problems because you clearly don’t know the market you’re designing for. Founders should at least have a gut feeling about what problems exist and how they want to solve them. Follow your gut over fake research.

If you’ve done research interviews, you know gold is found in the highly specific follow up questions. My best insights have consistently been when someone makes an offhand comment and I say “tell me more about that”. There’s a joke that interviewees say all the good stuff once you stop recording.

Data leads to confidence. Misplaced confidence if you can’t verify the data or check if it’s lying. Great article about that from @asmartbear.com: blog.asmartbear.com/whos-lying/

So in summary, I wouldn't touch it. If I was going to use an LLM, it would be in surfacing insights from interviews I conducted myself. Which I've done on a few projects and it's massively helpful.

I’ve been testing a GPT for this!