I built a small LLM app, then added evals and guardrails

Cold Lead Decoder takes a company domain and returns a structured lead card:

• company summary
• positioning signals
• likely pain points
• a grounded cold-email opener
• follow-up angles

The first version was easy to demo.

The harder part was making it behave when inputs are messy:

• thin websites
• invalid model output
• prompt-injection attempts
• generic openers
• unreachable domains
• schema failures

So I added:

• Zod as the output contract
• repair retries when schema validation fails
• degraded states instead of raw 500s
• SSRF protection on fetched URLs
• banned-phrase checks for generic openers
• a small eval harness with fixed fixtures
• a live /eval page for operational metrics

It is not a scaled product. No users/revenue story here.

The point was to practice turning one LLM feature from “cool demo” into something more testable and inspectable.

Live: https://coldl.vercel.app
Eval page: https://coldl.vercel.app/eval
Code: https://github.com/nikabokuchava/cold-lead-decoder

Curious how other people are testing LLM features before they trust them in real workflows.