Bad AI doesn’t smell

In her excellent book, Just Enough Research, Erika Hall wrote about the fact that bad surveys don’t smell. I’ve written about that myself over here. Here’s Erika explaining what she means:

A bad survey won’t tell you it’s bad. It’s actually really hard to find out that a bad survey is bad — or to tell whether you have written a good or bad set of questions. Bad code will have bugs. A bad interface design will fail a usability test. It’s possible to tell whether you are having a bad interview right away. Feedback from a bad survey can only come in the form of a second source of information contradicting your analysis of the survey results.

If you replace ‘survey’ with ‘AI tool’ then the point still stands. In fact, I’d argue that it’s even more true for AI tools. With a survey, you have control over the questions (assuming you wrote them), so you can at least put safeguards in place to increase the confidence you’ll have in your results. With AI tools though, you don’t have that level of control. 

Ingrained bias

On the whole, we humans tend to think of ourselves as being quite sceptical. That scepticism is always dependent on context though. We suffer from authority bias. We give information extra credibility if we trust the person or organisation providing it, even when we lack the expertise to assess its accuracy. We also have an innate truth bias, a tendency to assume that information communicated by others is true, unless there's a reason to doubt it. We can't independently verify everything we hear or read, so we often accept information by default.

We’re constantly told that “AI” is incredibly intelligent (‘super’ intelligent in fact!) so authority bias is likely to come into play without us realising. Add this to us accepting responses as true by default, and we can see why we don’t immediately smell the bad AI.

While we’re talking about biases, algorithm appreciation is another consideration. This was a new one for me, that I only found out about while writing this post. In essence, algorithm appreciation is a psychological bias where people trust or follow advice from an algorithm more readily than from a human; even when the AI's imperfections or mistakes are explicitly shown. 

All of this helps us to understand why people are more likely to trust AI than humans, even when it’s wrong. 

What’s the impact on user research?

If people tend to accept everything that AI tells them, the implications for user research findings are concerning. 

Let’s take a scenario where research has been undertaken. Proper research with humans, not some sort of random made-up research with so-called synthetic users. The recordings of this research have been pulled together and transcripts generated. There is a temptation to ask AI to synthesise and analyse the findings. This is likely to be even more tempting for someone who wasn’t involved in running the research and who doesn’t want to watch back hours and hours of recordings in full. 

The challenge here is that while AI might do a good job pulling out the important points from the research, it’s likely to make mistakes based on:

• A mistranscription of a crucial word
• A lack of visual cues (like body language)
• A misunderstanding of sarcasm, or generally misinterpreting tone
• A lack of context for the overall project
• A hallucination
• A general lack of understanding of real humans

All of this means that you end up with confident, wrong insights from your research.

How to avoid AI atrocities

To avoid AI ruining your research it’s vital that someone who was involved in running the research is there to check the work of your chosen AI tool. This one’s non-negotiable as far as I’m concerned. Without that safeguard you might as well have not bothered running the research in the first place. If a human wasn’t actually involved in running the research then it’s not research and any ‘results’ shouldn’t be taken seriously

If you do decide to use AI to help with your analysis, prompts need to be well thought out in order to minimise any issues. While AI can be a time-saver for researchers, it can also provide a very quick way to get misleading, incorrect answers. It’s best to ask specific questions, and also ask for participant quotes to back up any insights. Don’t just ask the robots to analyse the data and give you their findings. That said, it can be beneficial to ask for this type of analysis after you’ve completed your own analysis, as a way of sense-checking and to pull out any further areas of investigation.

The best way to avoid AI mistakes is to run research the old-fashioned way; get out there and run research in the field and complete your own analysis of the findings. It may take a little longer, but truly valuable research is worth the effort. 

Learn more at Clear Insight

If you want to learn about running better research then get yourself a ticket to next year’s Clear Insight conference. Talks that cover running human-led research include: 

Bringing the human back into research, Veronica Nagiub - Everyone is talking about AI moderation, synthetic users and automated insight generation. But Veronica argues that the biggest challenge facing researchers hasn't changed; understanding people.

Beyond the script: Decoding guarded behaviour, Sayani Mitra - In sensitive or high-pressure situations, people may offer polite, rehearsed or coded responses shaped by fear, professional expectations, social pressure or self-protection. Sayani argues that knowing how to interpret these moments is what separates us from AI. 

Stop researching behind a screen, Maria Zielinska - In a research world increasingly dominated by remote calls, AI tools and synthetic users, in-person fieldwork has become a rare but highly useful UX capability. In her interactive session, Maria will help us all develop this increasingly valuable skill.

If you’d like to know how to bring the human back into your research and how to stop AI from misinterpreting your studies then get yourself a ticket to Clear Insight today.