The conversation data that changed our product

If you’re building with AI — especially agents — you need to expect misalignment between what you build and what users actually do.

Agents are flexible by nature. Unlike traditional software, which forces users into specific patterns, AI agents invite open-ended behavior.

People assume the agent can do more than it can. And they’ll happily “misuse” it if it helps them solve their problems.

That’s what happened to us at Jotform — and we used it to our advantage by making a strategic pivot. Here’s how you can do the same.

How we discovered the pivot signal (and what you can copy)

When we first built Jotform AI Agents, the idea was simple: Let users talk to a form instead of typing.

But within two weeks of going live, we noticed a major gap between what we built and how people were using it.

They weren’t using it to complete forms. They were using it to answer questions.

Questions like:

• “Where do I park?”
• “What’s the refund policy?”
• “Can I upgrade my ticket?”

90% of prompts were customer support queries. We’d built a form-filler. But what users needed was a support agent.

Here’s how we picked up on it — step-by-step — and what I’d recommend any indie founder do if you're building in this space.

1. Track the actual conversations — not just the activity.

We weren’t guessing. We had visibility into thousands of live interactions through our platform.

And what we found was this: less than 10% of the prompts that our users sent were about completing forms.

They were asking the agent support-style questions:

• “Where do I park?”
• “What’s the refund policy?”
• “Can I upgrade my ticket?”

What you can do:

If you’ve launched an AI agent, take a sample of conversation logs and run a basic LLM prompt over them to classify user intent.

You don’t need a full pipeline — just prompt an LLM with something like:

What's the user trying to do? Choose one:
- Ask for information
- Complete a transaction
- Navigate something
- Fill in data
- Other

Even tagging 100–200 rows manually or in a notebook will show you where the real usage is coming from.

Don’t assume your usage matches what you intended. Log the actual conversation intents. It’s where the real product insights are.

2. Watch for multi-account patterns.

One person using your AI agent in a new way? Interesting. Lots of people doing the same thing? That’s a signal.

We saw it right away:

• A festival using it for event questions
• A Shopify store handling “where’s my order?”
• A university answering student FAQs

Different worlds. Same use: support.

What you can do: Check for this every week. You don’t have to do it by hand.

Here’s a simple setup:

• Pull the past week of conversation logs for each account (use a cron job, Zapier, whatever’s easiest)
• Run the text through an LLM to tag what the user was trying to do (ask a question, buy something, fill data, etc.)
• Save the results to a Google Sheet, Airtable, or Notion table
• Count which tags show up most often, and in how many accounts
• Write a tiny script that pings Slack or email if a new tag shows up in 3+ accounts

What to look for:

• The same questions coming up again and again
• Different industries using your agent in the same way
• A lot of chats that still end up with a human (a sign the agent isn’t ready for that job yet)

You don’t need dashboards. You just need a simple way to spot when a pattern is forming.

3. When the data is clear, pivot quickly — and quietly.

We didn’t make a big announcement.

We didn’t shut down the form-completion features.

We just started building out the real use case.

That meant:

• Adding WhatsApp, Messenger, phone, and voice channels
• Creating a native Shopify integration
• Reworking onboarding flows to ask: “What do you want your AI agent to do?” — and tailoring setup accordingly

What you can do:

If you spot an emergent use case with real volume and user retention:

• Add small improvements that smooth out the experience — things like better onboarding copy, a tweak to how queries are routed, or even a lightweight feature patch.
• Create a landing page for that use case, and send traffic to it.
• Make it easier for future users to start with that use case. Tailor your onboarding, prompts, or templates accordingly.
• Continue supporting the original vision. You don’t have to burn it down — just widen the lane.

4. Spin out simple use cases

Once we accepted that AI agents were being used for customer service, we realized: Some users don’t need full customization. They just need a chatbot on their site that answers FAQs.

That’s why we quietly launched Noupe, a stripped-down version of Jotform AI Agents focused entirely on one thing: Paste your website URL → Get an AI chatbot trained on your content → Add it to your site with one line of code.

We built it because the pivot signal told us there’s a segment of users who don’t want to build an agent.

They just want an agent.

What you can do: Not all users want full power. If you’re seeing consistent demand for a simple version of your AI product:

• Build a separate, fast-start version — something users can set up in seconds, without needing to customize or configure anything.
• Use it as a lead-in to your broader platform
• Pay attention to adoption — if lots of people stick with the simple version, you can either keep supporting it as-is or break it out into its own product.

If you’re building in this space, don’t wait for churn to tell you something is off. The signal is in the conversations.