I dogfooded my code review tool on 67 of my own files. Now I need harsher feedback.
Hey IH,
I’m building a small code review tool for the “before I ask a human to review this” moment.
The honest story: I tried recruiting a few friendly testers first and got basically no signal, so I turned the tool on my own codebase instead. That was useful in a different way. It scanned 67 real files from the product and caught several false-positive patterns I had to fix:
- env-backed secrets being mistaken for hardcoded credentials
- normal Python
.join(...)calls being treated like JS/Python confusion - safe
with open(...)usage being flagged as unclosed files JSON.parseinside try/catch needing better context awareness
That dogfood pass got the benchmark to 64/64 and made the product quieter, which matters because noisy review tools lose trust fast.
What I built:
Check AI Code is a quick first-pass scanner for Python, JavaScript, TypeScript, Java, and Go. It combines static rules with optional AI-enhanced explanations. It is not a replacement for CodeQL/Sonar/Snyk; it is meant as a practical sanity check before human review.
What I need now:
If you write or review AI-generated code, I’d love one real-file test and blunt feedback:
- Did it catch anything you would actually fix?
- Did it produce any false positives?
- Was the explanation useful or too generic?
- Where did the flow feel confusing?
Link: https://checkaicode.com
There’s a free tier and a short Pro trial, no card required. If you test it, the most useful feedback is not “nice tool” but a concrete false positive, missed issue, or confusing step.
Happy to share more about the stack too: Next.js, Prisma, static rules, Semgrep-style checks, and DeepSeek for Pro explanations.
the friendly-testers-give-no-signal thing is so real, people who like you just say 'looks good' and you walk away learning nothing. respect that you pointed it at your own code instead. before i run it on a python repo, does it scan the whole thing at once or is it still file by file? kinda matters for bigger repos.
the with open() being flagged as unclosed is exactly the kind of false positive that makes me drop a review tool fast, so good catch fixing those. gonna try it on a small TS project this weekend. does it have a git diff mode so it only checks what changed, or full files every run?
Yeah, this is a very fair question. Right now the product is still mostly file-first/full-file: paste or upload the file you want checked. It does not yet have a proper git diff mode.
Your point is exactly right though: for real PR review, diff-first is the more natural workflow. I am going to treat that as a priority direction instead of making people re-scan whole files every time.
You asked for blunt, so here it is. The real headline is buried: you got basically no signal from testers. That is the problem to solve, not the benchmark. Asking a stranger to run their own file through a brand-new tool and then write you feedback is a high-effort favor with zero payoff for them. You will get the same silence here that you got from your friends.
Flip the ask. The best content is already sitting in this post: the four false positives your own tool threw and how you fixed them. Show that, do not ask people to go find their own. "Here is what my code reviewer got wrong and how I made it quieter" proves the judgment without making anyone work.
For real signal, go to the moment of pain. Find people dumping AI-generated code into PRs, offer to run their actual PR and hand back the result with one question. Do the work for them. And make the low false-positive rate your headline claim, because in a category full of noisy tools, quiet enough to trust is the only thing that sells. The dogfooding is your proof for exactly that.
You are right. I over-weighted the benchmark because it was measurable, but the real problem was that my ask was too expensive. Asking strangers to bring their own code, run a new tool, and then write feedback is a lot.
The next pass should probably be: show the false-positive fixes as concrete proof, then do the work on a real PR/public example and ask one focused question. Also agree on "quiet enough to trust". That is a better headline than another score.
the false positives you killed are all syntax-shaped. secret vs env, file handles, parse context, that flavor. the class i'd actually stress-test is the quiet one: does it stay silent on an authz hole? endpoint parses fine but never checks "is this row mine". or a fix that loosens a guard and still looks clean. that's the stuff that's bitten me in my own AI-written code way harder than a hardcoded key ever did, and it's the worst place for a scanner to go quiet because you read silence as safe. so i'd want to see the miss rate on the permission and logic stuff, not the false-positive count. human review is slowest exactly there.
This is the part I was under-testing. Most of my dogfood was about getting the scanner less noisy on normal code, but authz / ownership / business-logic miss rate is the dangerous layer. A clean result should not imply "safe" there.
I am going to build a small negative set around row ownership, relaxed guards, tenant checks, and endpoint auth before treating the scanner as useful for reviewer trust.
The friend-testers gave no signal, dogfooding caught 4 false-positive classes story is honestly more useful than the final 64/64 benchmark. testers being polite isn't the same as testers being engaged, and most of us only learn that after spending weeks waiting for feedback that wasn't going to come. is there a specific false positive type you're most worried about missing now? if so i can pull a python file from my own work and run it tomorrow.
That is generous, thank you. The thing I worry about most now is less one specific false positive and more the opposite: clean-looking Python/JS where the scanner stays quiet but misses an ownership/auth/business-logic issue.
If you do try it, please only use a non-sensitive file or a public snippet. The most useful note would be: did the result make you trust it more, or did it miss the thing you were actually worried about?
I checked one of my JS files for the Figma icon plugin and it came out 100/100 - Clean bill of health. No issues detected. The code was completely written using vibe coding 👍
Thanks for trying it on a real JS file. A 100/100 clean result is useful, but I am trying not to treat clean as automatically safe.
If you remember the experience: did the result make you more comfortable shipping that vibe-coded file, or did it feel like the tool might still be missing the kind of issue you would actually worry about?
Dogfooding is underrated. Using your own product daily surfaces things no beta user will ever tell you. What's the harshest thing you discovered about your own tool?
Dogfooding your own product first — respect.
I'm Minchul, 57, former construction manager from Korea.
Not a developer, but I tested my own tool
hundreds of times before showing anyone.
"Noisy tools lose trust fast" —
that applies to every product, not just code review.
Good luck finding real testers!
One thing I'd pressure-test before spending too much time on benchmark scores.
The product description makes it sound like the buyer is the developer uploading code.
The pain in the post sounds closer to the reviewer who keeps getting pulled into low-quality AI-generated code that should have been cleaned up earlier.
Those are very different buyers.
If that positioning line is blurry, you can improve detection quality and still make the product harder to sell than it needs to be.
I'd be careful not to let the conversation stay entirely around false positives and model quality. The buyer decision may matter more than the next few points of accuracy.
Happy to put the tighter positioning angle in writing if useful.
This is a very useful framing. I think you are right: the sharper buyer is probably not just "the developer uploading code", but the person who has to review AI-generated code and decide whether it is worth trusting.
I am tightening the positioning around that: a quiet pre-PR sanity check before human review, not a broad "scan my code" tool. The buyer decision point matters more than another few benchmark points.
Appreciate the context.
The buyer decision point is the interesting part here — especially the shift from code scanning to helping someone decide what deserves human review.
Would be good to understand how you're thinking about that positioning change and what you're learning from it.
What's the best email to reach you on?
Thanks, happy to continue the conversation. You can reach me at support@checkaicode.com.
The positioning shift I am testing is: not broad code scanning, but a quiet pre-PR sanity check that helps reviewers decide what deserves human attention before a PR gets noisy.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.