The most useful UX test you’re not doing

As founders, we can be bad at testing our own products. We already know the flows. We know where to click. We know what each screen means.

New users don’t.

And that’s where most products break. Not bugs — UX.

AI can help you see your product like a stranger again.

Here’s how.

Level 1: You click, AI reviews

Let’s start with the simplest setup. Then, we’ll look at more advanced ones later.

Open your app in a private window. Pretend you're a brand-new user. Do one thing a first-time user should do, like sign up and send an invoice.

Record your screen while you do it (Mac/Windows has this built-in).

Then, drop that video into any AI chat tool you use. Ask it to act like a confused new user and tell you:

What feels unclear?
What would make you bounce?
What’s missing or awkward?

That’s it.

You’ll spot small UX issues that cost you users, and you’ll know exactly what to fix.

Level 2: Your script clicks, AI reviews

Now let’s make it faster and more repeatable.

This version utilizes test scripts to automate clicks and allows AI to perform the review.

You’ll need a test tool like:

Playwright
Cypress
Selenium

(If you already have automated tests, you’re halfway there.)

The idea:

Your test script walks through the flow
It saves the text from each screen
AI reviews the text like a new user

What to change in your test

After each step, save the text from the screen.

In Playwright, that might look like this:

const body = await page.textContent('body');
fs.appendFileSync('flow-log.txt', \`\\nStep 3:\\n${body}\`);

Do that for every screen.

You’ll end up with a plain text log like:

Step 1 - Homepage  
Step 2 - Signup form  
Step 3 - Dashboard  
Step 4 - Invoice form  
Step 5 - Confirmation

Ask AI for feedback

Paste that log into ChatGPT, Claude — or any other tool of your choice — with this prompt:

Act like a new user. 
Here's what the app shows, step by step. 
Tell me where the flow is unclear or confusing. 
Say what you would do at each step.

Now the AI reviews the flow — just like Level 1 — but with no manual clicking.

You can do this:

After a big change
Before shipping
During onboarding reviews

You’ll get fast, repeatable feedback — every time the test runs.

Level 3: Automate it all

This is where the AI does everything:

Opens your app
Clicks around
Fills in forms
Checks what happens
Tells you what’s broken or confusing

It acts like a QA engineer — but faster and tireless.

You’ll find these in:

QA platforms with AI built-in
Your CI/CD pipeline (if you integrate tools)

This needs:

Engineering time
Setup and test design
Good test coverage

But once it’s live, it’s powerful:

Catches regressions
Flags confusing UX
Saves human QA time

This is worth doing when:

You have multiple flows to test
You ship often
You want full regression coverage

How to set it up

You have 2 options:

Option 1: Use a QA platform with agents

These tools give you built-in agents:

QA Wolf
Reflect.run
Testim
Walnut
Others

What to do:

Sign up
Record or define your test flow
Let the agent run it
Get alerts when it breaks

No code needed for most of these.

Option 2: Do it with code

If you have dev resources:

Use Playwright or Selenium to write the test flow
Plug in an AI model to read results and flag issues
Add it to your CI/CD pipeline (so it runs on every deploy)

It takes time to set up, but you get full control.

That's it.

Quick note: Don’t jump straight to agents.Only invest in them once:

You’ve fixed the basics
You know what good UX looks like for your first-time flow
You’ve already used Level 2 a few times

If not, you may end up automating chaos.