We tested AI builders and the first screen wasn’t the interesting part
One thing I’m becoming less interested in with AI app builders is the first screenshot.
A lot of tools can produce something that looks convincing very quickly now.
What I care about more is what happens when you actually use the thing:
Does the workflow make sense?
Does the logic hold up?
Can you change it without starting over?
What happens once the app has real rules instead of just nice screens?
We’ve been documenting some of these experiments while working on built.new:
I’m obviously involved with built.new, but I’d be interested in hearing from people using other builders too:
What’s the first thing you test after the “wow, it generated an app” moment wears off?
First thing to test: change one business rule and see whether the builder edits it or regenerates the screen. Second: make the tool open its own output in a real browser and fix what it finds. The benchmarks & research section of shipwithmuse.live has a head-to-head where four coding agents (Muse Spark 1.2, GPT 5.6 Sol, Kimi K3, Grok 4.5) each built two landing pages, then had to find and fix their own bugs in a browser before handing over. That self-debug step says more than the first screenshot. (I help curate it)
Appreciate the honesty here, most people only share the wins.
Appreciate that. I think the failures are usually more useful anyway. A polished first screen tells you almost nothing about what happens when the logic gets stressed or the user does something unexpected.
Nice progress. What is the next thing you are focusing on?
The next thing I want to look at is iteration rather than generation.
Not “can it build the first version?” but “what happens after 10–20 meaningful changes?”
That feels like a much better test of whether a builder is actually useful long term.
Good write-up. What would you do differently if you started again?
I’d define the test cases before looking at the output.
It’s too easy to get impressed by a polished UI and then unconsciously judge everything else more generously. Next time I’d decide the failure cases first, then run every builder through the same checks.
Nice progress. What is the next thing you are focusing on?
I’m also interested in testing what happens when you change one core assumption after the app already works. That’s usually where “generated app” starts turning into “maintainable product.”
Interesting approach. What was the hardest part to get right?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
What made you pick this stack over the alternatives?
Makes sense. Are you planning to charge for it, or keep it free for now?
Nice progress. What is the next thing you are focusing on?
Makes sense. Are you planning to charge for it, or keep it free for now?
Great breakdown. What feedback have you had from early users?
Good point. Did you test that with users before committing to it?
Really relatable. How much time do you put into this each week?
How did you decide this was worth building in the first place?
Nice work shipping it. What has been the biggest challenge since launch?
Nice work shipping it. What has been the biggest challenge since launch?
Nice work shipping it. What has been the biggest challenge since launch?
That distinction matches what I’ve seen: the first useful test is to take the generated happy path and then change one assumption—an input shape, validation rule, or failure case. If the builder can’t explain or preserve the change, the screenshot was mostly theatre. I’d also test whether a new user can recover after the first error without knowing the implementation. Do you have a repeatable “break it” checklist yet, or are you still collecting cases?
That’s exactly the direction I’m heading in.
I don’t have a finished checklist yet, but the repeatable tests I care about are starting to look something like:
change one core assumption,
submit invalid input,
trigger the same action twice,
test a permission boundary,
break the happy path,
then ask the builder to change the rule without breaking the rest.
The recovery point you mentioned is especially interesting. A product can technically handle an error and still be unusable if the user has no idea what to do next.
I’m still collecting cases, but I’d much rather end up with a small repeatable “break it” suite than another screenshot comparison.
Really solid approach — curious how you're thinking about this, what's been the hardest part to figure out so far?