Launched SemanticTest - Testing framework for AI agents (open source)

Hey everyone!

Just launched SemanticTest after scratching my own itch building calendar0.app

The Problem:

Testing AI agents is a pain. Traditional unit tests fail because AI outputs aren't deterministic. Manually testing each release doesn't scale.

Existing eval frameworks are complex and hard to integrate.

The Solution:

Use AI to validate AI. An LLM judges whether outputs are semantically correct instead of exact string matching.

What it does:

✅ Validates semantic correctness (not just exact matches)

✅ Tests tool calls and multi-turn conversations

✅ Works with any LLM (OpenAI, Anthropic, local models)

✅ Simple JSON config - no complex setup

Try it live (no signup): https://semantictest.dev/playground

For developers:

📚 Docs: https://docs.semantictest.dev

📦 NPM: npm install @blade47/semantic-test

Current status:

- 100% open source

- Using it in production for calendar0.app

- Just hit v1.0.5

What I learned building this:

- Sometimes simple tools are better than more comprehensive but complex ones

- AI agents testing is actually more important than I expected

Would love your feedback!

What's your experience testing AI systems?