Built a security tool with real proof it works — can't tell if the niche is too small to matter
Spent the last few days building Verdict, a GitHub Action that runs security-critical PRs through two independent AI models (different companies, no shared context) plus a deterministic scanner, and blocks merge if they disagree instead of averaging it away.
Didn't want to just claim it works, so I ran it on a real PR with an actual RCE and a prompt-injection attempt baked into a code comment. Both models caught the vulnerability and both called out the injection by name instead of falling for it. Merge got blocked for real, no override: https://github.com/xKazeex/verdict-action/pull/2
Here's my actual problem. There's real competition in this space now -- a few other multi-model review tools, some more polished than mine -- and the market I originally built this for (x402 crypto payment devs) is small. I've had a genuinely good conversation going on IH about the underlying design question (when should an AI product show disagreement instead of picking an answer), which is more engagement than I expected honestly. But that's about the idea, not the tool -- zero stars, zero issues, nobody's actually poked at the repo itself yet.
So genuinely asking: is "cross-provider AI review with a hard merge gate" a real wedge, or does this get absorbed into GitHub's native tooling or an existing player within a year? I've got real proof it works. I don't have proof anyone wants it yet. If you build dev tools, curious how you'd read this.
Honestly, you're right that I don't have an answer to my own question yet. Actually putting it on the line: if you've got a payment/auth path in something you're building, want to try running this against a real PR of yours? No pressure either way -- but "would a stranger actually let this block their merge" is a much better test than anything I can do to my own repo.
The RCE test proves the mechanism works. What would convince you the wedge is real—teams actually enabling the action on production repos, or willingness to block merges without human review?
The strongest signal here is the blocked merge on a real RCE—not the model count. I’d test the wedge with a narrow design-partner cohort: ask teams with compliance or review pain to install it for two weeks, then measure confirmed issues caught, review-time change, and override rate. If those numbers hold, configurable repo policies and a gradual rollout could make the gate feel safer than an all-or-nothing switch.
That test you described is basically exactly what I should be doing instead of just posting and waiting. If you (or anyone on your team) touches payment/auth/agent code, want to be that design partner? Genuinely happy to help wire it in and just watch what happens -- real data beats me guessing at metrics.
The wedge isn't the multi-model part — it's the hard merge gate. Most tools surface disagreements; yours prevents bad merges. That's the behavior buyers pay for. If GitHub Copilot adds this, they'll build it into their workflow, not a separate action. Distribution advantage goes to incumbents. Your moat: make the gate configurable per repo policy so teams adopt incrementally. Also: x402 is too narrow. The real wedge is "CI gate that actually blocks" — any team with compliance requirements needs this.
Been sitting with your point since you posted it -- you're right that I led with the wrong thing. Reworking the framing around the gate, not the model count. If you're up for it, I'd genuinely value your read on whether it holds up against a real repo, not just the pitch -- happy to set that up if you've got something with actual merge-worthy code.