# I Built an AI That Reads Your Portfolio Screenshot and Roasts Your Investment Decisions

Let me tell you about the dumbest thing I ever built that turned out to be the smartest.

It started when a friend texted me a screenshot of his Robinhood portfolio and asked "how am I doing?" He had 23 positions. Some tech, some random penny stocks, something called a "leveraged inverse ETF" that he clearly didn't understand.

I spent 40 minutes going through each position, looking up fundamentals, checking correlations, assessing concentration risk. By the end, I had a rough picture: he was overexposed to tech, had three positions that were essentially the same bet, and his "diversification" was an illusion.

Then I thought: why did that take me 40 minutes? An AI could do this in seconds.

The Screenshot Problem

The first version of the Portfolio Scanner was a simple form. Enter your tickers and quantities. Done. It worked fine technically, but nobody used it.

Here's what I didn't account for: most retail investors don't know their exact positions off the top of their head. They'd have to open their brokerage app, write everything down, switch back, type it all in. Too much friction.

So I added screenshot upload. Take a photo of your portfolio screen, drop it in, and the AI extracts everything automatically — tickers, quantities, cost basis, current value. It uses vision AI to parse brokerage screenshots from Robinhood, Schwab, Fidelity, E*Trade, and most other platforms.

Usage jumped significantly almost immediately.

What the Scanner Actually Does

Once your portfolio is loaded, the scanner runs three layers of analysis:

The Health Check — This is the quick diagnostic. Overall risk score, total return, annualized performance. Think of it as your portfolio's vital signs. Green, yellow, or red.

Diversification Analysis — This is where it gets interesting. The scanner maps every position to its sector, industry, market cap tier, and geographic exposure. Then it identifies hidden correlations.

Most people think owning Apple, Microsoft, and Google means they're diversified. They're not — they're making three versions of the same bet on large-cap tech. The scanner catches this and shows you exactly where your concentration risk lives.

The AI Diagnosis — This is the part people either love or hate. The AI writes a candid assessment of your portfolio as if a hedge fund analyst was reviewing it. It doesn't sugarcoat. If your portfolio is a mess, it'll tell you. If you're overweight in one sector, it'll explain why that's risky. If you have a position that makes no sense given your other holdings, it'll flag it.

I've had users tell me the diagnosis felt "personally attacking" — which, honestly, means it's working.

The Technical Challenge

Parsing brokerage screenshots sounds straightforward until you actually try it. Every platform has a different layout, different font sizes, different ways of displaying gains and losses. Some show percentages, some show dollar amounts. Some use green and red, others use arrows.

The vision AI model had to be robust enough to handle all of these variations. I ended up building a preprocessing pipeline that normalizes the screenshot before extraction — adjusting contrast, identifying table structures, isolating the relevant data regions.

It's not perfect. Some screenshots need manual correction, especially from less common brokerages. But the accuracy rate on first try is high enough that most users don't need to touch anything — and across dozens of different brokerage interfaces, that's something I'm genuinely proud of.

The system also handles the edge cases that would trip up a simpler approach: fractional shares, options positions displayed inline with stocks, crypto holdings mixed in with equities. Each of these required specific handling logic.

The Feedback That Shaped It

The earliest feedback was fascinating. People didn't just want to know if their portfolio was "good" or "bad" — they wanted to know what to do about it.

So I added actionable suggestions. If you're overexposed to tech, the scanner suggests sectors with low correlation to rebalance into. If you have too many small positions that aren't meaningfully contributing to returns, it suggests consolidating.

One user uploaded his portfolio and discovered he was unknowingly holding three different ETFs that all tracked essentially the same index. He was paying three sets of expense ratios for the same exposure. The scanner caught it. He consolidated and saved about $400 per year in fees.

That's the kind of thing that makes this worth building. Not the fancy AI — the practical, "I just saved real money" moments.

Why Screenshots Matter

There's a deeper product insight here that I think applies to any tool builder: meet your users where they are, not where you want them to be.

I wanted users to carefully input their portfolios with exact share counts and cost bases. Users wanted to take a screenshot and move on. The screenshot approach is messier from an engineering standpoint. But it removed the single biggest barrier to usage.

Sometimes the best product decision is accepting imperfect input and making it work anyway.

If you want to try it, go to stockexpertai.com and look for the Portfolio Scanner. Take a screenshot of whatever brokerage you use, upload it, and see what it says. Fair warning — it doesn't pull punches.

Day 2 of my series on building Stock Expert AI. Tomorrow: the AI assistant that argues with you about your stock picks — and sometimes wins. www.stockexpertai.com