We 5x'ed free trial signups but conversion dropped to almost 0%

Danny from XO here. We ran an experiment on our newest acquisition, GrowthBar. The product historically was gate-kept by a cc requirement and a 5 day free trial. It was getting 8-10 new free trials per day and new customer subscriptions was roughly around 35-40 per month (~13% conversion).

We thought... why not open the flood gates and remove the credit card. At the very least, the # of new customers wouldn't get worse right? WRONG!

We went from ~70s signups a week to nearly 350. It was pretty great, but because we also extended the free trial to 14 days (from 5 days), we had to wait for our amazing experiment to bear fruit. Honestly, we thought in our heads, even if we get a 5% conversion per month from these sign-ups, we are nearly doubling the new customers because of how much we are growing top of the funnel.

It's a scary thing when after the 14 days you start hearing crickets. The panicky feeling where you feel like you screwed up a business 30 days after acquisition. Thats honestly what I felt because 14 days went by and we got nothing. On top of that, we were experiencing almost a 6-7% increase in churn from the previous months, so the compounding nature of this was freaking us out. Here are some data points...

• Activation was interesting. 50% of the free trial users never touched a single thing in our platform. 50% of the active users used up at least 1 feature to it's limit. This told me that people were using it and these weren't just bots trying to eat into our AI costs.... but still no conversion.
• Onboarding email sequence engagement was poor. To be expected I guess, but we realized that activation overall was an area we need to focus on. We have videos in our emails and we have in-app product tours, but engagement with those were really poor. Hotjar shows almost every user x'ing out of each video tutorial and proceeding to get lost in how to use one of our core features.

Anyways, after 30 days of extreme highs and extreme lows, we reverted everything back. It's been 7 days today since we changed everything back and included the credit card, so I'll report back on if things are a bit back to normal.

At the end of the day, this experiment was going to happen regardless. It was low hanging fruit that we needed to test, and we did. It very clearly failed, but that's okay (sort of). We have a lot more things we need to test, validate, and improve in our product and customer experience.

This is obvious... but 3 big things I was reminded of:

1. Growth isn't easy - What worked for one of our companies may NOT work... at all for another. Sometimes there is no "playbook", but listening to customers is the right place to start.
2. Experiments fail - It happens. It's not going to destroy the business per say, but just like a scientist, there needs to be planning, a set of controls, and a hypothesis to test.
3. Track everything - Like #2, if we let this thing play out for much longer, I think I would have had a heart attack. Tracking data everyday was helpful because we were very aware of what was going on, so we could control when and how it ended.

If you want some simple graphs of some of the data from this experiment, you can check out the daily recap from the morning I talked about this here: https://www.dailyrecaps.xyz/trial-and-error/