Thinking about an API-based approach for AI evaluation

I'm exploring an idea where AI teams could connect an API to their existing AI system and continuously evaluate its responses.

The goal would be to make evaluation fit into the workflow they already use, rather than asking teams to completely change their setup.

I'm still exploring the best approach.

For founders building AI products: would you prefer an API-based evaluation layer, or a separate dashboard/tool?

I'd really value your thoughts.