I stopped explaining my product to testers. Here’s what happened.
I’m testing Zaivy with one rule: I don’t explain how to use it.
I’m trying to separate three problems:
“I don’t understand the product.” → UX problem
“I understand it, but don’t know the answer.” → real business uncertainty
“I understand it, but don’t care.” → value problem
The third one might be the most important.
For anyone who has tested an early product this way:
What signals did you use to tell these apart?
Your third case (understands it, doesn't care, ghosts) is the one that stays invisible in the session — nothing tells you it happened until the return visit that never comes. Cancel flows are one of the few places that same case becomes explicit instead of silent: someone has to click cancel and an exit survey puts the question directly in front of them, so "I understood it, I just don't need it" finally gets labeled instead of just showing up as absence later. Built CancelKit around exactly that gap. Once you have enough testers to segment by which feature they opened first before disappearing, do you think that tells you more than the 7-day-return signal alone?
Yeah, I think it would. The 7-day return tells me whether there was enough value to come back, but what they opened first and where they stopped could help explain why they did or didn’t return. I’m still at such a small tester count that I don’t think segmentation would tell me much yet, but once there’s enough usage I could see those signals being much more useful together.
The three leave different traces if you stop asking and watch. Not understanding shows up as backtracking: they open something, close it, open it again. Understanding without an answer shows up as a pause on a field with the cursor in it, and they usually say so out loud. Understanding without caring is the quiet one: they complete the task cleanly, say it was nice, and never come back.
So the third is only visible after the session, with nobody watching. The signal I would trust is a return visit inside seven days with no prompt from you. Everything said in the room is polite; the return is not. And if they do come back, what they open first tells you what they actually valued, which is often not the thing you built the demo around.
The return visit point is really useful. I’ve been thinking mostly about whether someone can get through the flow without me explaining it, but an unprompted return seems like a much stronger signal of actual value. I’m keeping the beta small, so I’m going to watch for that with the first few testers. Appreciate this.
One trap now that you are going to watch for it: the measurement dies the moment you prompt. If you email a tester to ask how they got on, their next visit is not a return, it is a favour, and with a small beta you only get one shot per person. So decide the window first, seven days or fourteen, and do not contact anyone inside it. Keep a note of who you did contact so you can exclude them later.
The other thing at small numbers: a first return from someone you personally invited still carries some social obligation. The second one does not. If you only get to trust one number, make it that, and write down what they open first when they come back.
That’s a really good distinction. I hadn’t considered that contacting them during the window would contaminate the return signal, or that the first return could still be partly social obligation. I’m going to set a 7-day no-contact window for each tester and treat a second unprompted return as the stronger signal. Appreciate you pointing that out.