AI Needs Knowledge Quality Gates
AI Can Generate Code Faster Than We Can Verify Its Knowledge.
One thing has become increasingly clear to me while building NAEOS:
The problem isn't only giving AI more context.
We also need to know whether that context is trustworthy.
Imagine an AI coding agent retrieves five pieces of engineering knowledge.
One is outdated.
One contradicts another.
One references an API that no longer exists.
One was never approved.
And one is correct.
The AI doesn't necessarily know which one is authoritative.
More context doesn't automatically mean better decisions.
Sometimes it just means more opportunities for confusion.
That's why I've started thinking about knowledge validation as a first-class part of AI engineering.
Before knowledge becomes available to an AI agent, we should be able to validate things like:
Is the schema valid?
Is the owner defined?
Is the reference still valid?
Does it contradict another approved rule?
Are its dependencies available?
Is the version current?
Has it passed the required review?
Is it allowed by organizational policy?
Is it duplicated somewhere else?
And validation shouldn't necessarily happen only once.
Knowledge changes.
Systems change.
Dependencies change.
Policies change.
So knowledge may need continuous validation throughout its lifecycle.
I'm beginning to think about this similarly to how we treat source code.
We don't blindly deploy every line of code.
We run tests.
We perform static analysis.
We enforce policies.
We review changes.
We validate dependencies.
Why should the knowledge that guides AI engineering be treated differently?
If an AI agent is going to use engineering knowledge to make decisions, that knowledge itself needs quality gates.
This is becoming one of the core ideas behind NAEOS:
Don't just retrieve knowledge. Verify it.
Because an AI with less knowledge but high-quality context may be more useful than an AI with unlimited access to unreliable information.
I'm curious:
What should happen when two pieces of engineering knowledge contradict each other?
Should the AI choose the newest one?
The most recently approved one?
The one with the highest authority?
Or should it stop and ask a human?
