Making the Most of your LLM: The tech of Validate My SaaS

Making the Most of your LLM: The tech of Validate My SaaS

I was on a month-long work-ation in Brazil when I had the idea for Validate My SaaS. About 6 months of working on it nights and weekends, then leaving FAANG to plunge into indiehacking full time, I happily launched the most comprehensive competitor analysis tool for new SaaS startup ideas.

Honestly, I probably went overkill with the tech. I may have found more success being quicker to market with a product that just wrapped asking ChatGPT “so what do you think about my startup idea”, but those tools already exist and aren’t data you can rely on.

As an engineer, I had a lot of fun building Validate My SaaS, and also learned some great lessons. So here they are!

Context

First a little context: Validate My SaaS offers a Competitor Analysis report for a user based on their startup idea. It looks something like this:

The simplified high level flow is this:

snapshot of a Validate My SaaS Report

The user fills out a form describing their startup idea. The report generation takes about an hour and we send them an email once it’s ready.

On the backend, we

1. Do live web scraping to find the most current and relevant competitors based on the startup idea

2. For the relevant competitors, we extract and format a bunch of data like features, pricing, popularity metrics, trust pilot reviews, etc.

For each report we scrape about 500 web pages and process about 1 Million tokens with an LLM, to cut through the noise and present our customers with a thorough, data-rich report about their relevant competitors.

So with that overview of the process, let’s dive in.

Minimize the Context

Even if an LLM has a large context window, too much context can distract and decrease the quality of the response. Not to mention increasing both the cost and time to process.

I’ve found it very helpful to use a simple olds-school tool called “code” to narrow down what content we feed into the LLM.

Look at the bottom of Asana’s home page for example:

None of that is really helpful for our use case. So we cut it all out before sending to the LLM.

A more targeted example: if a chunk of text does not contain any numbers or currency symbols, it’s unlikely it contains information about a product’s price.

There’s a lot you can do here with simple regex. We’re able to cut 50% of tokens out of a web page before sending it to an LLM.

Best of Both Worlds / Multi-pass Filtering

Some models are fast, cheap, and dumb. Others are more accurate, but you pay for that accuracy (with both speed and money). What happens when you need both?

One of the most critical parts of our pipeline is accurately gauging the relevance of a competitor. If it’s not accurate, it drowns our customer in irrelevant data.

So we developed a multi-pass filtering process. GPT 3 does NOT provide the accuracy we need. But, it IS accurate within some order of magnitude. So we first send a product through GPT 3 to get a sense for its relevance. If it’s not even close, we can already filter those out. We then process again with GPT_4 for a more accurate relevance. This does mean we process some products 2x, but most products don’t survive the GPT_3 stage so their cost is cut 10x.

Scores > Decisions

Concisely, I’ve found it much more helpful to have the LLM give something a score (ex from 0-100) rather than a binary decision (is this a relevant competitor or not). This gives you much more control over the cut-off threshold and opens up a lot of other possibilities such as taking the top-most scoring items, or combining scores across different axes.

GAR > RAG

Shortly after starting work on VMS, RAG (Retrieval Augmented Generation) became very popular. Especially thanks to Perplexity.

Validate My SaaS does not use RAG, but something even better for our use case: Generation Augmented Retrieval. This allows us to find new or updated products that LLMs have NOT been trained on. To take it to 10x, we also use adaptive-GAR, where our retrieval engine adapts its search based on the success of its results.

Shipping Fast

There was a lot of work developing and refining Validate My Saas. We have about 20 different prompts in the system. How to ship something like this fast? Do things that don’t scale.

Each major step of our pipeline saves its state as a JSON blob, which the next step usually uses as its input. One of the first things we built was an admin dashboard that shows each step, and allows a human (that’s me) to change the results and run the pipeline again from that point.

For the first few months, our landing page noted that a report may take up to 24 hours to deliver. Because I manually reviewed (and usually improved) every report as we refined the process until it consistently delivered results I was happy with.

Closing

Hopefully some of these techniques we developed are helpful with your own product. If you are still in the early stages of your product and want to understand the competitive landscape BEFORE you build, of course check out Validate My Saas. Or if you just want to discuss what you’re working on and get some pointers for working with LLMs, just reach out.