Is anyone building a true governance layer between apps and LLMs?

Hey builders,

I’ve been working on something that started as a small architectural experiment but is slowly turning into a bigger idea.

While building AI applications, I kept running into the same problem.

Most apps today connect directly to an LLM like:

App → LLM API → Response

But once you scale beyond simple chatbots, this approach becomes messy very quickly.

You start needing things like:

• user memory
• policy enforcement
• tool access control
• identity / session state
• response constraints
• logging and auditability

And suddenly the application logic becomes tightly coupled with the model.

So I started experimenting with a separate governance layer between the app and the model.

The architecture looks more like this:

App
↓
Governance Proxy
↓
LLM

I’m calling this NEES GP (Governance Proxy).

The idea is that this layer becomes the system of control for AI behavior.

Instead of the model deciding everything through prompts, the proxy manages:

• user identity and session state
• emotional / intent interpretation
• policy and guardrails
• tool permissions
• response filtering
• audit logging

So the model becomes more like a reasoning engine, while governance stays outside.

One important distinction:

This is not RAG.

RAG mostly solves context retrieval.

What I'm exploring is behavior governance.

The proxy decides:

• what tools the model can access
• what policies apply
• how responses should be constrained
• how memory and identity evolve over time

In other words, the proxy becomes the AI operating layer for the application.

A few questions I’m exploring:

Does AI infrastructure need a dedicated governance layer separate from orchestration frameworks?

Could this architecture make AI systems easier to audit, debug, and scale?

Could governance proxies eventually become a standard component of AI stacks?

Right now I'm building a working prototype to test this idea in a real product environment.

Curious if anyone else here is exploring something similar.

Would love to hear how other builders are thinking about:

• policy enforcement
• agent behavior control
• AI observability
• state management outside prompts

Thanks