The hidden risks of LLMs: why we're building ModelRed

We’re building ModelRed, a security platform for organizations using large language models.

The idea came from watching how quickly companies are plugging LLMs into production — customer support, finance, healthcare, internal tools — without much thought about security. Attacks like prompt injections, jailbreaks, and data leaks aren’t rare anymore. They’re happening, and most teams don’t have a way to even know when it happens, let alone prevent it.

With ModelRed, we want to make that easier.

• We’ve built 100+ probes that automatically test models for vulnerabilities.

• It works with providers like OpenAI, Anthropic, Hugging Face, AWS, Azure, and GCP.

• You don’t need to rebuild your stack — our SDKs (Python, JS), APIs, and CI/CD hooks plug right into existing workflows.

• Beyond testing, we also provide real-time monitoring and alerts so you know the moment something goes wrong.

One thing we’ve noticed is that the risks go beyond just using pre-trained models. As organizations start doing fine-tuning and retrieval-augmented generation (RAG), they’re introducing new attack surfaces. A poisoned dataset, a bad chunk of retrieved text, or even a subtle manipulation during fine-tuning can sneak in vulnerabilities that aren’t obvious until it’s too late. Without continuous checks, you can end up deploying a model that looks fine on the surface but is wide open to exploitation.

Right now we’re in private beta and running on a waitlist. A few early partners are already testing it, and we’re learning a lot about how teams want to integrate security into their AI workflows.

What’s surprised us most is how manual AI security still is. Many companies either rely on human red-teamers (expensive and slow) or… nothing at all. We’re trying to turn that into something continuous, automated, and part of daily operations.

If you’re building with LLMs and want to kick the tires, you can join the waitlist on modelred.ai. We’d also love to hear from this community:

• How are you thinking about security for the AI tools you’re building?

• Do you run any kind of testing before deploying models?

• As fine-tuning and RAG become more common, how are you making sure your data pipelines are safe?

Excited to share more as we keep building.

— The ModelRed team