Where should the multi-model compatibility layer live?

I'm curious how other people are handling a problem that only shows up once an app talks to more than one model provider.

We run coding agents against Claude, OpenAI-compatible models, and a few others. At first, we put provider-specific logic inside each app. That worked until streaming and tool calls started diverging:

• SSE event boundaries are not consistent.
• OpenAI tool_calls and Claude tool_use have different shapes.
• Similar failures come back with different status codes and error bodies.
• Usage and finish reasons are not normalized either.

We recently moved that logic behind a small gateway. It gives us one internal event model and keeps the applications talking to one contract.

The tradeoff is obvious: the gateway becomes part of the critical path, and debugging needs to preserve the raw provider events. But it has been much easier to add or replace a model without touching every agent.

I'm not trying to pitch a finished product here. I'm more interested in the architecture choices people have made.

If you run agents across multiple providers, where does this compatibility layer live for you: inside each client, in a shared SDK, or behind a gateway? And what would make you avoid a gateway?

I wrote down a few of the edge cases we ran into here: https://api.luoluocoder.com