I built Code Factory because I kept getting fooled by passing tests
I did not build Code Factory because I thought the world needed one more AI
coding tool.
I built it because I kept falling into the same trap. A coding assistant would
write the code, write the tests, and leave me with a nice clean wall of green
checks. I would think I had made real progress. Then I would actually use the
feature, or let somebody else try it, and the thing I cared about would fail.
The tests had passed. The thing had still failed.
That got old fast.
The annoying part was not just fixing the bug. I had already mentally moved on.
I had trusted the green check, started the next thing, and then had to backtrack
through the code and tests to figure out what they had really checked. It wasted
time, killed momentum, and made every “looks good” result feel less useful.
Eventually I stopped asking whether the test suite was green. I started asking:
could this test have caught the actual thing that broke?
What was going wrong
Usually, nothing was dramatically broken in the test framework. The test just
was not proving what I thought it was proving.
Sometimes a test checked that a function returned something, not that it
returned the right thing. Sometimes a mock made an integration look healthy
while the real path was never exercised. Sometimes the asserted condition was
so broad that an obviously broken implementation could still satisfy it.
AI makes this easier to miss because it can produce the code, the tests, and a
very confident explanation all at once. It looks finished. The output is neat.
The test run is green. That is exactly when it is easy to stop looking.
I was not trying to make an assistant sound smarter. I wanted a way to keep
myself from confusing a lot of activity with actual proof.
So I started building Code Factory for myself
The first part is intentionally simple: take a plain-English outcome and turn
it into a local, app-shaped starting point that I can inspect and build on.
The first command is intentionally simple:
factory mvp "Build an approval tracker" --root .
It is useful, but it is not automatically “done” just because a scaffold
exists.
The other part is the bit I wanted most. Code Factory keeps asking:
- What requirement is this change trying to satisfy?
- Which declared checks support that claim?
- What evidence is missing or stale?
- What should be verified next?
Graph Ops makes that evidence path visible. Proof Review gives me a concrete
handoff for a diff. The Verifier Plane keeps a worker's claim separate from the
evidence used to check it.
And there is one idea I care about a lot: if a control is supposed to protect a
change, try breaking the control. Delete it. Invert it. Corrupt it. If the
evaluator still passes, the control was not doing enough work to deserve my
trust.
If it does not, I would rather know before I rely on the result.
What it is not
Code Factory does not run around on its own, decide it is finished, and publish
things. It does not look for credentials, upload source, deploy a service, or
approve its own work. It does not tell you that a generated MVP is ready for
production just because it has files and a passing test run.
That is deliberate.
I still want the human decision in the loop. I just want that decision to have a
better paper trail: what changed, what was checked, what was challenged, and
what is still an assumption.
It is local-first because I wanted to use it on my own work without handing over
a working tree. If it is useful, the same receipt and proof model can grow with
a team. If it is not useful, it should be easy to ignore.
Why I am sharing it for free
Honestly, I built this to save myself time and aggravation. I was tired of
getting a reassuring test result and then discovering later that it had not
covered the part that mattered.
I do not think I am the only person dealing with that now that assistants can
produce so much code so quickly. So Code Factory is free and open source under
MIT or Apache-2.0. Use all of it, use one piece, read the receipts, or tell me I
am adding too much ceremony.
I would genuinely like blunt feedback from people who maintain real codebases or
review AI-assisted changes:
- Where would this save you time?
- Where would it be overkill?
- What would you need a test to prove before you trusted it?
If you have been burned by a passing test that did not mean what it seemed to
mean, I made this for that feeling.
Start with the source and a local command:
pip install factoryline-code-factory==0.28.2
factory mvp "Build an approval tracker" --root .
factory studio --root .\my-mvp
This is a real problem nobody talks about. AI writes tests that pass AI-written code — it's a closed loop that looks green but proves nothing. I've seen the same thing building API integrations for 7 different LLM providers. The tests would pass locally but fail in production because the mock data didn't capture edge cases in real API responses. The fix that worked for me: integration tests against real endpoints with small credit budgets. Expensive but catches what unit tests miss.
The quickest proof is not another internal evaluator. It is a corpus of real failures where tests passed and users still hit the defect. I would publish 20 anonymized cases and show whether Code Factory identified the missing evidence before revealing the known regression, including false positives and review time added. Then narrow the first buyer and workflow to one wedge, such as AI-assisted pull-request review for teams with five to twenty engineers. Without that benchmark and narrow entry point, the breadth may look like ceremony; with them, the evidence model becomes measurable.
The trap one level up from fooled-by-tests bit us this year: fooled by the checker. We shipped a guard that validated coverage by parsing declarations out of the same file it was checking, so "parsed equals declared" passed by construction and physically could not fail. Breaking the control only catches this when the evaluator's expectation comes from an independent source; when the control and the evaluator share ground truth, they break together and stay green.
So one more question for your list: where does the checker get its expectation, and can that source drift with the code it checks? And break-the-control deserves to be a habit, not an idea. We now mutate every new guard once before trusting it, the way you'd test a smoke detector with actual smoke.
This is a really important distinction: passing tests prove that the checks passed, not necessarily that the intended outcome is correct.
I especially like the idea of challenging the control itself. If deleting or corrupting a test still lets the evaluator pass, that tells you something much more valuable than another green check.
The visual makes the core idea much clearer. The distinction between proving a requirement and simply getting a green test result is especially interesting.
That seems like a useful way to make “done” a much harder claim to make casually.
For anyone who prefers the short visual version, here is the current Code Factory proof loop:
Start with an outcome, fuzzy PRD, or risky AI-assisted diff. Scope the proof, find the gap, bind the evidence, then use Graph Ops for one evidence-backed next action.
The important boundary: it stays local-first. Code Factory does not silently publish, deploy, sign, or approve work—the reviewer still owns that decision.
"Try breaking the control" is the strongest idea here — that's the line between a test and a guardrail. Generated suites mostly prove the happy path stays happy. Add one question to the list: "what did this change break on purpose?" If the diff can't answer that, the tests aren't testing intent yet.