I stopped tracking daily signups for 3 weeks. Here's the metric that actually told me what was broken.
For months I checked one number every morning: daily signups.
Felt productive. Kept me busy. Didn't move the needle.
Three weeks ago I stopped looking at it.
Not because signups don't matter. Because I realised I was using it as a comfort metric. If it went up, good day. If it stayed flat, anxiety. If it dipped, spiral. None of which was helping me build anything.
The metric I switched to: session depth. How far into Genie 007 are people getting on day one? Are they actually using the voice command feature or just doing basic dictation and stopping? Where exactly does the session die?
Week one of tracking this properly: I found 68% of new users never get past the second screen.
Not because they didn't like the product. Because there was a friction point I'd built in intentionally. A setup step I genuinely thought was "onboarding." It was actually a wall.
Removed it. Session depth jumped. More people reached the moment where Genie actually does something impressive: you say a rough thought out loud and it delivers the finished action. The email you meant to write. The post. The form filled. That moment is where people go from "hm, interesting" to "I need this every day."
Signups hadn't changed. But the people who signed up were getting further, and staying.
The week changed because my thinking changed. Not my marketing.
I've been building in public for a while now and the honest truth is: the metrics you check most often are rarely the ones that tell you what's actually broken. They're the ones that feel safe to check.
What's your comfort metric? The one that feels productive but isn't really doing the diagnostic work?
Interesting take. Would you still recommend this approach to someone starting today?
I'd be curious whether that 68% drop was similar across acquisition sources. If one channel brought in people who were just curious, it might look like a product problem when part of it is who arrived in the first place. I'd compare the share who reach their first useful voice action, then come back a week later, by source.
I'm with SL Marketing, so my bias is to check that before deciding which campaign is actually bringing in good users. Signups are easy to count, but they can make the wrong channel look like a winner.
You're right and that was exactly what we found when we did segment it. The 68% drop wasn't uniform at all — paid traffic was close to 80% drop-off at session depth, organic search held much better, around 45%. The organic users arrived with a specific intent, they'd read something and came looking for a solution to a named problem. The paid users were vaguer. They clicked an ad, got excited about the headline, but hadn't yet decided they had the problem we were solving. So the onboarding wall hit them differently — one group was trying to get past it, the other group was still figuring out whether they wanted to. It completely changed how we thought about the problem. The fix for organic wasn't the same as the fix for paid. We ended up creating a softer intro flow for paid traffic that explained the "why" before asking them to set anything up. That moved the paid drop-off from 80% to 58%, without touching the organic flow at all.
What was that setup step on the second screen that turned out to be a wall? It is crazy how often things added to help onboarding actually just block users from reaching the core value. Did you end up moving that setup task to happen later in the user flow, or did you realize it wasn't actually necessary at all?
The wall was two things stacked on top of each other on the same screen. First, a microphone permission prompt — which makes sense to ask for, but feels alarming if you don't understand why a web app needs your mic before you've even seen what it does. Immediately after that, a language/accent selection screen where you had to pick your primary language and speaking style before you could proceed. Both of those are genuinely necessary for the product to work. But we were asking for them before the user had experienced any value at all. They hadn't seen a single useful thing yet. We moved the language selection to post-first-use — after someone had completed one task with the product in whatever default settings we applied. The mic permission we couldn't move entirely, but we added a one-line explainer above it that said what it was used for. Just that change dropped the exit rate on that screen from 62% to 38%. The lesson was that necessary friction is still friction. You don't always have to remove it, but you do have to earn the right to ask for it.
The framing I would add is that signups were not only a comfort metric, they were the only one that moved daily, which is why they were so hard to put down. Session depth needed three weeks before it said anything at all. Most people choose metrics by importance and then check them at a frequency the metric cannot support, and daily noise fills the gap. The other thing worth watching: that setup step presumably existed for a reason. Now people skip it, it is worth checking whether the ones who get further actually stick, or whether the wall just moved to day three where it is harder to see.
The frequency mismatch point is sharp. Signups feel actionable every morning because they move every day, even when the signal is noise. Session depth doesn't give you anything meaningful for two or three weeks, which is exactly why most people never commit to it as a primary metric — the silence is too uncomfortable. You end up switching back before the data has time to tell you anything. On your second point: yes, and it's something we're still watching. The drop-off at day three did go up slightly after we moved the setup step. Not dramatically, but it's real. Our read is that the people who hit day three and leave are doing so for different reasons than the people who were leaving on screen two — they've had more time with the product, so the exit is more considered. But you're right that we can't call the change a win until we have 60-day retention numbers by cohort. The wall may have moved rather than gone away. That's one of the things that makes session depth useful — it would show us a different pattern by week than by day one.