I'm interviewing my users for feedback at the end of every beta sprint, here's what I learned so far

Right now I'm building NotePlan 3 and it's in a closed beta. Means only a handful of chosen users will get to see it and give me feedback. And with every user I'm sharing it with I have an interview = usability test.

First of all, how do I get those beta users? That's easy. NotePlan has already a core user base who are following my progress building the app since years ago. So I sprinkled a few hints on twitter, in the change log on the AppStore and on Reddit to get a few of them reach out to me. When someone reaches out, I check if they are open for a video call (or screen sharing call) and they come on a list.

Next step is coding. I build in 2 week sprints. So I have something ready to share at the end of those two weeks. Then with this shiny new version I schedule calls with a handful of users over zoom. 4-5 zoom meetings, also called Usability Test, are enough to get a first impression of what to focus on next.

Here is how the interview goes:

  1. I share the new version in this call with the interviewee.
  2. I explain that I'm not allowed to reply to questions or explain anything substantial, except there is something not finished and I need to point out a few bugs.
  3. I ask the user to speak out loud his or her thoughts.
  4. I let them share their screen, get my notebook out and start writing notes as fast as possible.
  5. The user explores the app, tries to figure out stuff, tells me how he expects things to work.
  6. I watch the user interacting with the app while shutting my mouth.

What kind of feedback do you get? A ton of things which people won't email you, because they are rather small, but irritating. Especially if it's something in the user interface and you are not sure yourself how to build it. So instead of guessing around, I just have interviews to see what people expect to happen. And then I build that.

So after a round of interviews I figure out the most important points and get to work fixing them. Then in the next round of interviews I check if I was successful in improving the app. Till date I have changed the sidebar and naming of things in NotePlan at least 5 times. And each time the interviews go more smoothly than before.

This is how you can iterate your beta as fast as possible. In between the interviews I also get a lot of emails by the existing group of closed beta users. This closes the feedback loop and that very tightly. On top of that you get feedback which you would normally not get. You see the actual reactions of your users before they figure out something new. And without burning through too many users you can make rapid changes and test them back to back.

I'll keep documenting my progress. Let me know if you got questions!

  1. 1

    this is great, would love to hear more about the process such as how you decide who are the customers you actually interview!

    1. 1

      At first, I asked users who were very actively emailing me with feedback. Asking them right then will yield the best results.

      Then I built a beta version of the next big upgrade and tweeted, emailed people that I'm building it, and if they want they can participate in a closed beta. Whoever replied, I asked if a screen sharing call is ok where I share the beta. This obviously only works, if you have already some user base.

      An alternative I used recently is using a paid service like rapidusertests.com. The quality is usually better and you have more takeaways in less time. But it costs money.

Trending on Indie Hackers
Customer acquisition when broke... 17 comments Facebook is (becoming) the new Yellow Pages 10 comments How do you read this logo? 6 comments What's the biggest challenge you face? 5 comments Our landing page only attracts users outside of the US during pre-registration. Any idea why? 2 comments How do I Build a Beverage Company in Public? 2 comments