Why I Built an AI Scouting Tool for a Market That Hates Black Boxes

I built an AI scouting engine for a niche I actually play in... here's what the architecture looks like 2.5 months in

Dynasty fantasy football is a strange market. The players are obsessive, analytical, and deeply skeptical of anything that feels like a black box. They've been burned by generic rankings and ADP-driven consensus. So, when I started building FPIS (Five Pillars Prospect Intelligence System, fivepillars.dev), I made one design decision that shaped everything else: the scoring had to be explainable at every layer.

The product:

FPIS is an AI-powered scouting tool for dynasty fantasy football. You input a prospect's name and position. The system evaluates them across five weighted pillars — Production (30%), Athletic Profile (20%), Film/Tape (25%), Contextual Factors (10%), and Projection (15%) — and returns a structured scouting report with a composite score, pillar breakdown, dynasty verdict, and player comps. (Each Pillar can be adjusted to the weight that suits a given user)

The "Five Pillars" framework isn't just a UI pattern. It's the actual scoring contract enforced through the Claude/Gemini prompt. Every output is validated against a Zod schema before it touches the frontend. If the model hallucinates a score outside 1–10, the response is rejected.

The Stack (and why):

Frontend: React SPA on Vercel. Vite build, React Router, Clerk for auth.

Backend: Express on Railway. The analysis engine lives here — it calls either Claude Sonnet 4.6 (Pro tier) or Gemini 2.5 Flash (Free tier) depending on the user's subscription.

Billing: Stripe + Clerk metadata. Webhooks write tier and billing_interval to Clerk publicMetadata, which the frontend reads without an extra API call.

Data: 175+ prospects seeded from nflverse, enriched with CFBD athletic data and Sleeper ADP. Persisted to a Railway volume. A merge strategy on every deploy preserves accumulated scores while refreshing ECR rankings.

The biggest architectural lesson: keeping the scoring logic in the prompt contract, not in the code. The Zod schema is the source of truth. The frontend never transforms scores, it just renders what the schema validated. This made iterating on the prompt fast without touching the UI.

The cost optimization that made Free tier viable:

Running Claude Sonnet for every free analysis was burning through our budget. We switched to Gemini 2.5 Flash for free users and dropped the per-analysis cost from about 8 cents to less than half a cent. The tradeoff? Not much thus far but looking forward to your thoughts on this part. Lesson learned here... I’m continually fine tuning not only my approach but what I am researching as well. Delighted to find comparable (if not better) resources at the fraction of the price.

Now Free Tier runs on Gemini at ~$0.003/analysis. Pro tier runs on Claude Sonnet for the quality ceiling. The model split isn't just a cost decision; it's a quality delta that justifies the upgrade.

The social sharing mechanic I built this week:

Dynasty players share player takes on X and Discord constantly. The problem: sharing a prospect link returned to a generic static OG image — no name, no score, nothing that communicates value in a feed.

I set up a server-side route to generate OG images using Satori and @resvg/resvg-js. Each prospect gets their own 1200×630 PNG with their name, a color-coded position badge, and a gold composite score — the whole thing renders in about 80ms. I load the fonts (Inter) once the module starts, so there's no file I/O on every request. It's cached for 24 hours.

We use edge middleware to detect bots trying to access prospect pages. When a bot shows up, we send back a preview image instead of the full page. Real users just see the app normally.

Here's the funnel: someone spots a prospect card on X, clicks through to see the public profile. They can see the overall score and ranking, but the detailed breakdown is hidden. That gets them interested, so they click 'Unlock Pro Analysis' and land on our pricing page.

What I'd do differently:

The leaderboard schema has gone through three versions. I should have locked it earlier. Every schema migration on a JSON file that Railway persists to a volume is a manual migration — there's no rollback. I now treat the schema version field as a hard contract.

I also underestimated the public/free experience early. The app was built for Pro users first. The sharing mechanic and public prospect pages only got built when a user asked "why does sharing only work for Pro?" The answer was obvious in hindsight — the conversion hook must exist before the gate, not behind it.

Where it is now:

The pipeline has 175+ prospects with ECR rankings, nflverse athletic enrichment, and accumulated Five Pillars scores for every analyzed player. The Big Board, rankings pages, and per-prospect pages are all live at fivepillars.dev.

The target is a clean exit to a larger fantasy data company. The code is written to that standard: environment variables for everything, modular scoring logic, a schema-versioned data store, and documentation that a buyer could deploy in under an hour.

Future Ideas/Add-ons:

I'm thinking about some bigger features down the road. There's potential to refactor things and bring in NFL prospects for the current redraft fantasy football world — I'll probably tease that as 'coming soon' to build anticipation.

Dynasty fantasy football is still pretty niche, but it's growing. I wanted to start here and expand from there rather than bite off too much. I've been toying with the idea of running a local LLM to power that work while keeping the site live and active in the meantime.

The real dream? Building a fantasy hockey app that goes toe-to-toe with Fantrax, Sleeper, and Yahoo. But that's years away — total pipedream at this point, honestly.

STATS: MRR $0 (well $9.99 of my own money), and 2 email addresses captured at the email gate :)