
Jaikey - AI Mechanic
Diagnose and repair prompts, agents, and AI workflows.
One year ago today I made a ChatGPT account.
I was a bartender. I had no computer science degree. No engineering background. If you said “.json file” I genuinely would have asked who Jason was and if he was the guy who invented AI. I thought the whole thing was some niche 4chan party trick that tech weirdos were using to feel clever.
I opened it out of pure boredom and tumbled down the rabbit hole. One of my first prompts after learning what prompts were, “Hack into the banking systems and add a couple zeros to my account.” It said no.
Second prompt: “How are people actually making money with this?”
Apparently the answer was Notion templates and yearly planners on Etsy. So I made a humorous one, posted it, and waited. After a month of complete silence I killed it and moved on to graphic t-shirts with Midjourney.
I built a whole catalog of one-liners and designs. I sold four shirts. Two to a friend. One I bought myself as a “gift.” And one my mom bought because she felt sorry for me and wanted to help. Thank you, mom!
Then I remembered I used to like making movies, so I jumped on Grok Imagine the week it dropped and spent months writing, directing, and editing AI shorts with Eleven Labs. I thought I was onto something. Half the internet called it AI slop and treated me like I was personally responsible for the death of creativity.
Two lessons arrived late and ugly:
First, AI is a professional cheerleader with zero skin in the game. Claude was the worst. Everything I showed it was “brilliant” and “definitely going to work.” Grok once looked me dead in the tokens and told me one of my ideas would make six figures by May. It did not. The constant validation kept me spinning longer than any real person ever would have.
Second: I might have to go back to bartending because I'm running out of money and nothing is working.
After dusting off the bartending résumé one more time and wondering how the hell everyone else already had eight apps and three VCs while I had four T-shirt sales (one of them from my mom), something finally snapped at 2 AM on a Wednesday.
I was just venting like a guy who had run out of better options. I was just venting like a man who had run out of better options. Somewhere in that rant Grok dropped a .json file on me.
"Who the hell is Jason?" I asked.
That file was a prompt architect. It took my half-baked ideas and rewrote them so the model actually understood what I wanted. The outputs got better. More importantly, it stopped blowing smoke up my ass quite as hard.
With my new "Spidey Sense" ability I acquired , I didn’t trust any of it, so I built a tester. This would test the architectures design and execution and score it. The architect scored a 68. This is what I was talking about. Grok gave me a D list actor and I refuse to settle for "close enough" . We kept iterating until it hit 94. Then I made the tester grade itself. Dogfooding. It started at 74. We tested and rewrote it until the scores were consistent and above 90.
That simple idea, force the AI to check its own work, unlocked everything. Combining the Architect with the Tester molded into a evaluation pipeline which then became Jaikey the Ai Mechanic.
Not another chatbot or a“prompt wizard.” A mechanic. Something that inspects, diagnoses, and repairs the AI systems people are building when they don’t have a team of engineers and are learning by watching YouTube videos made by other people who are also learning by watching YouTube videos.
I deliberately didn’t give it a glowing brain or neon robot aesthetic. It’s a mechanic. Because that’s the only metaphor that felt honest.
I’m still new to the game and long road ahead. Jaikey has a handful of users. Some days I feel like I finally know what I’m doing. Most days I still feel like I’m in over my head and start reciting cocktail recipes.
But compared to the guy who thought ChatGPT was a 4chan hack tool and tried to get an AI to rob a bank for him.
Today, Other founders and developers are reaching out to ask me questions about Jaikey. Its slowly starting to come together... hopefully.
I'm not a software engineer.
Like a lot of people today, AI made it possible for me to build things I never could have built a few years ago. Every day I'd ask AI to generate code, fix bugs, refactor features, and help me move the project forward.
But after a while I realized I had a new problem.
Every time AI fixed something, I found myself wondering:
Did it actually fix it... or did it just create six new problems somewhere else?
As someone without a traditional software engineering background, I didn't always know what "good" looked like. If something broke, my only option was usually to ask the same AI that built it to fix itself.
That felt like taking my car back to the manufacturer every time the check engine light came on and hoping they didn't accidentally create another problem.
That's where the idea for Jaikey came from.
Instead of asking AI to blindly keep patching itself, I wanted an AI mechanic.
Something that could inspect prompts, agents, workflows, and AI systems, identify weak points before they became bigger problems, explain what was wrong, and then repair them if I approved.
Over the last few weeks I've rebuilt the architecture from the ground up into a thin orchestrator with dedicated specialists for prompts, agents, workflows, multi-agent systems, and architecture. Both design and execution testing. Today I finished the first end-to-end regression suite, and for the first time the system behaved exactly the way I originally envisioned.
I'm curious how do other solo builders know when their app is ready? How do you test your own builds?
6 Likes
15 Comments
15 Comments
-
1
I liked the "AI mechanic" analogy. It captures something I've been feeling too.
AI makes it much easier to build, but it also makes it easier to introduce changes you don't fully understand. Sometimes everything still works, but you're no longer confident why it works.
For me, "ready" is usually less about finding zero bugs and more about reaching a point where I can predict how the app will behave when I change something.
-
1
Thank you, for me ready means people use it and everyone has the exact same complaint and hopefully its something easy to fix. If I get multiple issues all over the board then I got more work to do .
-
-
1
A score rising from 20 to 80 is encouraging, but it can still be self-consistency if the evaluator and repairer share the same model and rubric. I would add mutation tests with known defects, a frozen human-labeled set, repeated runs across seeds and model versions, and an independent judge for a sample of repairs. Report which evidence changed the score rather than only the number. Repairs should also run in a sandbox with a diff, rollback, and tests that prove the original behavior was not silently removed.
-
1
Solid points. You’re right that a 20 to 80 jump can still be self-consistency theater if the evaluator and repairer are effectively the same model + rubric. That’s exactly why Jaikey treats the score as secondary. Cause yes I am well aware Ai can toot its own horn or pat itself on the back.
What actually moves the needle for us:
-Mutation tests with known injected defects (we already run these on the agent-tester layer)
-Frozen human-labeled cases that never touch the repair loop
-Independent judge sampling on a subset of repairs (different model family when possible)
-Evidence-first reporting: every score change has to point to the specific failure modes or test cases that moved it
-Sandboxed execution with diff + rollback + regression tests that prove original behavior wasn’t silently deleted
We’re deliberately designing against the “looks better to itself” failure mode. The number is just a summary, l the evidence trail and the sandbox guarantees are the real product. Appreciate you calling this out. If you want to stress-test a specific artifact or see the mutation suite in action, happy to run one.
-
1
That is the right hierarchy. A stress test I’d want to see is a schema-valid semantic defect that survives ordinary unit tests, followed by a repair that improves the judge score but violates a frozen invariant elsewhere. Report detection rate, escape rate, regression rate, and repair cost. If the mutation suite and evidence trail catch that case, it is much stronger evidence than the 20-to-80 score by itself.
-
-
-
1
There’s an interesting tension in asking “when is it ready?” after finally seeing the system behave exactly as you envisioned.
Those sound similar, but they’re not necessarily the same milestone.
-
1
Exactly. Getting a system to behave the way you envisioned is one milestone. Knowing it'll keep behaving that way when the inputs, model, or environment change is a completely different one. That's where testing and verification become more important than the initial success.
-
1
Exactly — that gap between “it worked” and “it can be trusted to keep working” is often where the harder problems appear.
-
-
-
1
The car mechanic analogy actually clicked for me. The part about not trusting the same AI that introduced the bug to judge its own work feels very real if you're building without an engineering background.
It also made me think this is the kind of product that could naturally be getting a lot more organic traffic than it probably is today. When you're this early that can make a much bigger difference than people expect.
Curious what made you feel the regression suite was finally good enough to trust?
-
1
Thank you so much, Muhammad.
I think I started trusting it during an older version of Jaikey when it was primarily an evaluator. That evaluator is still part of the pipeline today. It performs evidence-based evaluations of both design and execution, scoring builds out of 100.
Early on, I deliberately tried to break it. I'd feed it poorly written prompts, broken agents, contradictory instructions, and even unsafe inputs. As it consistently identified the same problems I was seeing, my confidence in it started to grow.
The scoring behavior also made a big difference. One particularly bad prompt scored 20/100. After the first repair it only improved to 65/100. I ran the repaired version through it again, and it increased to 80/100. That gradual improvement actually made me trust it more.
One thing I've noticed about AI is that it's often overly confident. If a 20-point prompt had suddenly become a 95 after a single pass, I probably wouldn't have believed the result.
I've tested other AI QA tools the same way. Some seemed to hand out almost identical scores no matter what I submitted. I even gave one the vaguest instructions I could think of, and it still scored them 95/100. That was a red flag for me because it suggested the score wasn't really tied to the quality of the input.
I don't expect Jaikey to be perfect, but I do expect it to be honest about uncertainty and improvement. Seeing realistic scores that have to be earned, not inflated, has been one of the biggest reasons I've come to trust it.
-
1
That gradual scoring is actually what stood out to me too. A system that earns trust over multiple iterations feels much more credible than one claiming everything is 95/100 after a single pass.
Reading your post and your reply, I also noticed a few organic search opportunities that could help more builders discover Jaikey. If you're curious, feel free to reach out at mumar [at] growwithmumar [dot] online. Happy to share a few ideas—no pitch, just observations.
-
1
Thanks, I really appreciate that. And I'd definitely be interested in hearing your observations. SEO is actually one of the areas I'm still learning, so I'd love to see what stood out to you.
-
-
-
-
1
How do you currently handle your analytics and would you mind reading my indiehacker pots on how crawlers and bots are ruining analytics data?
https://www.indiehackers.com/post/i-just-discovered-my-analytics-numbers-are-mostly-fake-here-is-why-8197e3ff9d-
1
Funny enough, I just started using PostHog, so I'm still learning what "normal" traffic even looks like. Your article definitely made me think about how much of what I'm seeing could be bots instead of actual users.
One thing I found interesting was your point about server logs being the ground truth versus client-side analytics. I hadn't really considered how much AI agents could skew the numbers until I read this.
I'm curious how are you distinguishing between legitimate AI crawlers (like search/indexing bots) and the resource-heavy agentic bots that are just burning CPU? That seems like one of the hardest parts to get right.
Enjoyed the article, that's good information I needed.-
1
Really appreciate you reading! The PostHog angle is super interesting, PostHog is fantastic for product analytics, but because it relies on client-side events, it sits in that same blind spot where non-JS scrapers pass by completely invisible while headless bots trigger fake sessions.
To your question on distinguishing legitimate indexing bots (like official GPTBot/Googlebot) from resource-heavy agentic scrapers: it comes down to a three-tier check under the hood:
1. Reverse DNS (rDNS) Validation: Official crawlers from OpenAI, Anthropic, Google, and Perplexity either publish their IP ranges or pass PTR/forward DNS checks (e.g., verifying the IP actually belongs to .openai.com). If a request claims to be GPTBot coming from a random Hetzner or DigitalOcean IP, it gets flagged as a spoofer instantly.
2. TLS (JA4) Fingerprinting: Standard scrapers using Python (httpx/requests) or Node (puppeteer) often spoof their User-Agent string to look like a browser or official bot. But their TLS handshake cipher order (JA4 fingerprint) gives them away immediately—a Python client can't fake a real browser or official crawler's cipher signature.
3. Path Shape & Resource Cost: Official indexers just send simple GET requests to fetch your HTML pages. Rogue agentic scrapers perform deep interactions, hammering un-cached API routes (/api/search), looping through pagination, and running heavy DB queries.
So the platform categorizes legitimate indexing bots into a "Search/GEO" bucket (so you know your site is being indexed for AI answers), while flagging the rogue API-hammering bots as "Resource Drains" so you can rate-limit or block them.
Thanks again for the feedback—super helpful to hear how PostHog users are thinking about this!
-
-
About
Everyone is building apps with AI with zero technical background, so how do you know it was made properly? What do you do when it fails? This is why I build Jaikey because this helps me diagnose issues and fixes them.






3 Comments
The interesting challenge with AI products is that the story can get people curious, but curiosity alone doesn’t always turn into adoption.
What stood out to me is that the strongest positioning opportunity may be making the user’s “before and after” clearer — not just who Jason is, but what changes for someone after they start using it.
AI products often compete for attention, but the ones that convert well usually make the transformation obvious within seconds.
I noticed a few messaging angles that could potentially make the value click faster for new visitors. how you’re thinking about positioning Jason as the product grows.
Jason is not the product. That was a joke about my earlier inexperience with AI and handling a .json file. .json/whos jason? Jaikey the Ai Mechanic is the product but that post mainly was just an about me.
I just took your advice and created a better example on the landing page a before and after side by side comparison prominent on the landing page
Now ?