Fraud exists in Customer Research, and if you let it get into your own study, you’ll end up with worse insights. Here’s how we spot and handle it.
- First, acknowledge when you have made a mistake
- In this case, I should NOT have approved this candidate for the study without a direct message asking for more details. My fault, my cost.
- Next, learn how to use the Timeline of Progress to listen and map out the story – gaps and logical inconsistencies imply someone that is making things up
- Last, if you suspect you’ve been duped, go back and compare their story to the answers they gave in the screener – if it fails to match up, it’s probably BS
Watch the interview below and see how he struggles to provide details, leaves long pauses for simple questions, and evades the questions by giving wishy-washy answers.
Watch the full interview below:
Telltale BS Signals
Here’s my review and thoughts for Janis (and the UserInterviews team) about why I think this participant was making this up:
Here are the UserInterviews responses, and why I now doubt these are valid:
The first mistake I made was not requiring a LinkedIn profile. I relaxed this rule for Janis’ study – where we were talking to people like Khalid who are students, I thought it would be ok.
- If a Social Profile is present, click it, review it, and make sure the job title matches the profile provided
The skillshare course WAS interesting to me – sure, we had seen a few sketchnote interviews already, but the responses below indicated that this was not just about taking better notes.
These answers are the reason I wanted to capture this story:
- Someone looking to communicate more clearly at work with visuals sounds interesting
- There are alternative choices mentioned in the screener – indicating there was some shopping and possibly a struggle to decide
- We have a clear picture of the end outcome
Unfortunately:
I should have passed these through GPTZero before admitting him to the interview.
- Normally, UserInterviews flags copy that has been pasted as “possible AI” – it didn’t here, as you can see in the image
- I suspect they typed this in to get past the problem, or started typing and then pasted the rest
- As I point out in this article – wordy answers that are too well written are often AI
The rest of the screener is relatively OK. Until you paste it into GPTZero:
Look at the percentages:
- 88% chance of being AI
- 0% change of being Human
In this case – I needed to get the Men-in-Black red memory eraser out and forget this ever happened.
Can we trust Jobs-to-Be-Done research?
Episodes like this DO cause me to stop and pause, as it makes you wonder whether all the research is valid or not. This is good, because it means we still have our critical faculties and we’re thinking clearly.
Issues with qualitative research:
- We rely on the memory of others, and we know memory is fallible
- Fraudsters exist out there who want to get a quick buck from sharing a made up story
- We need to interpret what interviewees say and make business decisions based on these interpretations
This is obviously non-ideal, but we don’t live in a perfect world.
To overcome this we rely heavily on techniques taught in Law Enforcement Interrogations and Hostage Negotiations (Chris Voss’ excellent “Never Split the Difference“. We dig deep into any answer and probe the person to make sure they can give us a clear, coherent, consistent story.
For those of you trying to analyse these interviews as we run them – please forget this one. We’ll publish more genuine customer stories soon.