My cancellation survey was lying to me, so I stopped trusting it

I run a small SaaS called CurseCut. Every month I'd look at the cancellation survey and think okay, people are leaving because of long file issues, missed curse words, price. It sucks but I guess it makes sense, moving on.

Except none of it was actually true.

One user cancelled saying the product couldn't process their long files. They had 56 successful tasks in their history. Files up to 177 minutes, zero failures. They were still processing files three days after cancelling. The real reason? They didn't want the $39/month recurring charge but had credits left to burn.

Another user cancelled three separate times across nine months. Each time a different reason. First time: the product made mistakes. Second time: missed curse words. Third time: long files again. I dug into the actual task data. 55 tasks. 100% success rate. He had tried the premium AI model exactly once, decided he didn't need it, and went back to the default. The stated reason was whatever option got him out of the modal fastest.

A third user cancelled the same day they signed up. Said the product couldn't process their files. Their one task had completed successfully in 36 seconds.

I was making product decisions based on this polluted data. Prioritizing fixes for problems that didn't exist. Missing the actual pattern, which was that my credit model was letting people subscribe, build up a balance, cancel, and coast for months.

I couldn't see any of this from the survey. I only found it by cross referencing task history, Stripe events, login timestamps, and credit balances for every single cancellation. The first time I did it manually it took two days and I still got one case wrong.

So I built something that does it automatically. Every cancellation triggers a pipeline that pulls the behavioral data, treats the stated reason as a hypothesis, checks it against what the user actually did, writes up a draft analysis & creates a draft PR. It posts to Discord and takes about 60 seconds.

I've only ever run this on my own product. I want to know if the patterns are universal or if I just have weird users.

If you run a Stripe subscription product and you've ever looked at your cancellation data and thought something feels off about this, drop a comment. I'm not selling anything. I just want to see if what I built transfers outside of my own product.