Evaluating the honesty of a user interview
Hey IH!
I've been building a side project for a little while and maybe I'm being a defeatist, but I'm having trouble deciphering if the users I'm interviewing are being honest with me. My project is for ride sharing drivers and taxis and whenever I've gotten into an uber or taxi I always chat with the driver and do a small interview with them to assess the problem I'm solving. At the end of the chat I give them the actual pitch for my app and I get responses from lukewarm to semi-excited. It's great to hear user feedback, but I'm a bit concerned that they are just being polite and telling me what I want to hear.
I'm wondering: when you talk with potential users about your product- how able are you to trust what they say? Can you tell if they are just trying to just be polite/inoffensive?
Out of curiosity have you read / applied The Mom Test?
This!
Well, I know what I'll be doing tonight haha
Just from glancing at the first chapter It seems like most of my questions aren't terrible at least, but pitching the actual product isn't useful... I need to continue to focus on the problem.
Thanks for the recommendation!
Yeah I actually thought those were some reasonable questions. Super helpful resource though because the default is basically to get rubbish feedback by asking rubbish questions but there is an art/science to it!
Lean B2B is another awesome book that dives deep on extracting high quality information out of people.
This book is less than 3 hour read: The Mom Test. It's all about asking right questions, right way.
It will take some time to make it a habit though, but totally worth it.
I'd encourage this video that GV did on the subject:
https://www.youtube.com/watch?v=U9ZG19XTbd4
IMO, if you do nothing else, do the following:
This comment was deleted 8 years ago
Hey Matthew - good for you for going outside of your comfort zone to get feedback!
On my previous project, I was a total noob at getting feedback. And basically, I set up a situation where people I interviewed felt compelled to say "yes" to me even if they meant no.
Here are a few things I learned from those mistakes:
In person, interviewees will tell you "yes" just to be polite. It's much harder for them to tell you "your idea sucks" even if that is what you think on the inside because of the close physical proximity. You have to really probe to get past that. I ask "why" multiple times and / or "how much would you pay for this?" as two ways to get a better quality answer if I feel someone is just saying yes because they think that's what I want to hear.
You have ask open ended questions. This gets the other person talking and gives you a chance to listen and probe where you want to go next.
You have to mask your level of excitement along the way. If you get too giddy because of something they say (or don't say), they'll read your body language / voice / tone and it'll impact their answers. it kind of gets back to point #1 above that people want to be polite.
Hope this is helpful. If you have more questions, feel free to send me an email. Happy to chat more!
Cheers,
Jonathan
Hey Jonathan-
Thanks for passing on the knowledge- I'll put it to good use!
Glad to hear this was helpful and actionable.
If you are interested, I'm launching a "30 day feedback challenge" soon on Tribe of Five (www.tribefive.me). Let me know if you are interested to join?
Details:
The goal is push you slightly outside your comfort zone so it's top of mind to go out and get feedback each day!
I believe that if you consistently and deliberately seek feedback, you'll get way better at it in just a month. And along the way, you'll level up on your ability to collect real feedback and then get actionable takeaways that'll help your project.
Let me know if you are interested. I can save you a spot!
Cheers!
Jonathan
Some organizations will hire people from outside the team to do the interviews. I've heard the following reasons;
(1) outsider interviewers won't get giddy or defensive during the interview
(2) interviewees feel more comfortable being honest when they know the person who they're talking didn't make it.
So, if you have a friend who can conduct it (and record it), that could work.
You will never get honest advice if you are doing this interview in the middle of a paid for uber/lyft/taxi ride.
People here have given good advice on actual interview skills, but picking your environment is just as important.
In a ride sharing situation the driver's livelihood is dependent on you leaving them a good review post ride. There is no chance in hell that you as a driver are going to go out on a limb and poo poo on an idea a customer is telling you about.
You need to get these ride sharing people into a coffee shop or other neutral setting.
When you’re right you’re right, and of course, you’re right.
I realized this was a problem very recently after a very awkward Interaction that and it’s another piece that fueled me posting this question. I think I can have small innocuous questions I can ask during drives but I need to perfect a better user interview and do neutral space interviews.
It still depends on what question you ask - if you explicitly ask them for "one thing you don't like about the idea, even if it is really small" and that's ALL you ask, then they aren't put in an awkward situation compared to "what do you think?"
If you need to give them more context, talk about how sometimes just one tiny annoying thing can bug you - anybody using devices can relate to that.
I explicitly tell people: "I have trouble getting negative feedback, and that's really what I'm looking for. Positive feedback is great, too, but any negative feedback you can give me is the most valuable." And folks typically oblige!
Yeah, I've got a variation of this question for my new user interview questions. I'm really interested to see what people tell me.
This comment was deleted 8 years ago
Next to the mom test there is the money test; Ask them to (pre) sign up on the spot, when they are hesitant or don't want to, the problem is probably not that big of an issue for them.
My fav answer so far, would have written the same. Your end goal is (probably) to be profitable, and if you can be at least somewhat profitable on just a pitch/idea/prototype you're on a good path.
This is something I regret not doing more and will try do it more in the future.
Agree- I think at the very least I should be asking every person for at the very least to commit to giving me their email address. It's a stronger indicator than I'm giving it credit for I think.
my suggestion: don't ask about the idea or your app, ask about "what problem would you think I will face with this idea?" and ask if they want to be an early adopter. Extra hustle point if you ask to pay before release.
I really like the idea of probing the user for more practical info. Maybe it's time to revamp my questions and focus less on the solution and more on the problem.
I really appreciate this answer- thanks for the idea!
There are companies that specialize in feedback of your product that way you don't have to guess if they are being honest. But you have to pay them for this feedback so I'm not sure if you are willing to do that.
There are a few things you can do to elicit honest feedback:
Say that you are part of a research group or survey team, doing research on behalf of the (your) company. This little white lie means that people will be more honest as they don't think that negative comments are directly tied to you!
Another thing we say, is we've received a lot of feedback but we're trying to understand why this wont work / is a bad idea. Or you can ask, what don't you like about this / what wont work?
Careful how you phrase this though, you don't want them to think that everyone else thinks it's great and make them feel stupid for "not getting it".
Hope that helps!
Check out this book - Talking to Humans. It's short and you can get it free. It deals with exactly this topic. All the best.
This is EASY!
I learned it at the Founder institute - ask them to tell you one thing they don't like about your idea or product name.
You can take the conversation then more to what else they have heard about that is similar. If they can relate to the problem, how do they or friends solve it at present?
I built https://roastmy.site to get feedback for your website, and you can also get honest and sometimes very frank feedback about your idea
Give it a go if you want!
Always an issue. Maybe a little out of the box thinking is needed. If you ran a small contest with a prize of gift card for finding the biggest fault or issue people will now look for problems and not just stroke your ego. I do think in a survey or poll asking if and what they would pay for it provides the data you are looking for. All ideas are great if they don't cost me anything. Most people wont give negative feedback. They really want you to succeed but you are asking for honesty. You really have to make it a point that you want to know what is wrong and missing. They have to have some stake or gain to give you an honest review. Otherwise it does not matter to them.
"It's great to hear user feedback, but I'm a bit concerned that they are just being polite and telling me what I want to hear."
That's a legitimate concern. It's hard to get raw, unadulterated reactions to ideas because even the question affects the answer, or the fact that the feedback was solicited vs. volunteered.
I commend you for just chatting people up like you described. There's no silver bullet but I imagine your doing that over and over again with different people it will give you a good sense of what's a real knee-jerk reaction vs. espoused/just being nice.
I feel like partly what inspired me to post this was that in my last few user interviews. I was hearing the same things over and over...I was questioning my approach because the conversations weren't really varying and evolving so I wasn't learning anything new which was frustrating/seemed like a bad sign.
I think a fresh take on my interview questions and style is needed at this point.
Why are you not sure they are honest? And how will you know they are?
The more I think about this question and the tips/other questions asked in this thread, the only answer I have is I don't think you can ever know if your user is being honest- you can only question your approach, craft your interviews carefully, do more interviews and repeat.
I don't know that I'll truly know for sure until people start paying for the service, but if I can de-risk my project with well thought out questions then the best possible consumer research will keep me from flying blind.
You might want to check clean language. I like their premise - which is to remove the asking person as much as possible from the question itself - more than the actual questions they use.
https://en.wikipedia.org/wiki/Clean_Language
That depends on what you want to learn from them. If you are just pitching, they likely saying something nice about it.
What questions are you asking now? Want to learn if your ‘solution/idea’ is worth pursuing?
Better to ask how they currently solve the problem. What do they currently do to get the job done. Don’t pitch your product just yet.
Sure, so to give context, I'm building an app that does fare wait-time prediction for drivers at the airport to help them know if they should wait for passengers to arrive to take them back to the city or if they should go back to the city on their own.
My conversation usually goes:
-do you do many rides to the airport?
-when you're there, do you wait for a fare to take you back to the city?
-how long do you wait normally?
-if you knew you'd be waiting longer than X minutes do you think you'd just go back to the city and not bother waiting?
-I'm building an app that will tell you that- does that sound like something you'd use?
I've spoken to ~50 drivers and I feel like I'm starting to understand their options and how they make decisions. I guess with my last question I'm trying to assess if they want what I'm building- that typically is how I've gotten contact info from the drivers that seem enthusiastic.
I always think of the marketing side of things. like how much it will cost to get a user versus how much extra can they make. Also, does the data change that much? would they download it once then delete it once the learn the pattern.
There are patterns in the data for sure- there are definitely time of day and day of week effects, but there are lots of smaller things that you can't know up front- for example plane size or the demand for rides based on that plane.
The cost of acquisition is a puzzle piece I have yet to figure out. I have some very manual approaches to recruiting users but I'm still figuring out how much to charge given I expect the user base to be price sensitive but I don't see this being insurmountable.
First, you're already ahead of the game when it comes to many side project builders—you're talking to your potential customers! Keep it up.
The way you describe it, you're more or less a physical manifestation of a marketing web site. On a marketing web site, you'd describe the problem you're solving, pitch the solution, and use a call to action to move them into your sales funnel. You're missing the last step: the ask!
Get their email address, phone number, or some other form of contact information to follow up with them. That will give you a sense of whether they're truly interested or just blowing smoke. Keep those people updated on your progress, engage them for product feedback as you iterate, and then prompt them with another ask: sign up for the actual product.
If you can keep these potential users engaged to end up as customers, it signals you're onto something. If you're having a hard time gathering contact info (think ~25% conversion from conversation to getting their email as a ballpark), then you may not be solving a strong enough problem.
Each stage of the funnel helps you understand where the problem might lie. If you can't get contact info, or responses, it may not be compelling enough (either the messaging or the problem). If it's a failure to convert to customers, it may be execution of the solution.
I've actually been taking down the phone numbers of the people that seemed the most enthusiastic- so as soon as I have my beta ready I've got a list of people to call.
I actually feel a little silly I didn't think of it like this! It never occurred to me to draw the parallel between gathering their contact info in person to a "drop in your email" landing page.
Thanks a lot for this- it gives me renewed faith that people weren't just being polite!
Even better! Phone number is great for mass texts as well, you just want any contact info to follow up.
I'm making an assumption here, but only tweak I'd make is to ask everyone you talk to about it for their contact info; let that be the black and white answer to whether they're interested. Some people aren't as overly enthusiastic in demeanor as others.
Fundamentally, try to break the process down into small steps to measure. It will give you more insight into where your current bottleneck might be, as it will certainly move around. Small, measurable steps will give you a greater sense of objective success, along with control, rather than feeling like you have this big black box that isn't working.
My man!
I'm so glad I posted this question. I'm so inside my own head with this problem I feel like I'm starting to stumble around in the dark. You're 100% right about breaking things down into smaller pieces.
I'll be asking every user I speak with for their email/phone number from here on out. Thanks for your input!!
This comment was deleted 8 years ago