Never average the ratios: designing a composite score users can actually audit

I build Intrinsiqq, a stock analysis tool. One of the harder things I have had to design is a single picture that answers "what kind of companies do I actually own?" for a portfolio of arbitrary holdings.

The answer I landed on is a radar chart with five axes: Profitability, Growth, Balance Sheet, Cash Flow and Valuation. Each axis is 0 to 100.

Getting from a pile of holdings to those five numbers turned out to be much less obvious than it looks, and almost every interesting decision was a scoring architecture decision rather than a finance one. If you are rolling up per-item metrics into a group-level number anywhere in your product (per-repo health into org health, per-endpoint latency into service health, per-customer signals into an account score) most of this transfers directly.

Here is what I got wrong first, and what the current design does instead.

Rule 1: score first, then average. Never average the raw ratios.
This is the one that matters most, and it is the mistake I actually shipped in the first version.

The obvious way to get a portfolio-level price to earnings ratio is to take the weighted average of the ratio for each holding. It reads like the right thing. It is not.

Ratios do not survive averaging. A price to earnings ratio has earnings in the denominator, and earnings can be small, zero or negative. One holding with barely positive earnings produces a ratio in the hundreds and drags the portfolio average somewhere meaningless. One holding with negative earnings produces a negative ratio, which is not a cheap valuation, it is a company losing money, and averaging it in makes the whole portfolio look cheaper the more money that one company loses.

The same trap exists well outside finance. Average the response times of your endpoints weighted by traffic and one dead endpoint failing instantly makes your service look fast.

The fix is to invert the order of operations. Instead of aggregating the raw values and then interpreting the result, you interpret each holding first and aggregate the interpretations:

For each holding, map each raw metric onto a bounded 0 to 100 sub-score.
Take the position-weighted average of those sub-scores.
Once every input is bounded before it is aggregated, no single holding can dominate an axis no matter how extreme or broken its underlying number is. The worst any holding can do to an axis is contribute a zero at its own weight.

That sounds like a small reordering. It is the difference between a chart that is occasionally nonsense and a chart that is always in range.

Rule 2: use explicit thresholds, not percentile curves
Each raw metric passes through a table of bands. Above a certain level of profitability you earn full marks, below it you step down through a small number of tiers.

The tempting alternative is to score on percentile against the rest of the database. It normalizes everything, has no hand-picked constants, and calibrates itself as the data grows.

I decided against it, for three reasons.

Your score becomes a function of your database. If a holding scores 72 today and 68 next month because I ingested a few thousand more companies, that is not a signal about the company, it is my data pipeline leaking into the user's number. Percentile scoring means a result changes when nothing about the thing being scored changed.

You cannot explain a percentile. Telling someone their Profitability is 61 is unanswerable if 61 means "better than 61% of a set you cannot see". A threshold is answerable: this company's return on invested capital cleared the second band, so that component scored where it did. The user can disagree with where I drew the band, which is a far healthier relationship than not being able to interrogate the number at all.

Percentiles hide absolute badness. If everything in the database is unprofitable, the best of a bad set still scores near the top. Thresholds say what they mean.

The honest cost of bands is cliff effects. Two companies sitting either side of a boundary get visibly different scores for a nearly identical business. That is real and I have not solved it. I accepted it because a visible, explainable cliff is better than an invisible, unexplainable curve, and because these axes are meant to be read as coarse shape rather than precision measurement.

Rule 3: missing data is excluded, never zero-filled
Some holdings will not have every input. Data gaps happen: a company genuinely has no meaningful figure for something, or my pipeline has not extracted it yet.

Zero-filling is the default everywhere because it is one line and it never crashes. It is also a lie. Scoring a holding zero on Cash Flow because I could not compute its cash flow tells the user their company is bad at something when the truth is that I do not know. My data gap becomes their bad news, and they have no way to tell the two apart.

So a missing input stays missing all the way through, and the aggregation skips it rather than substituting anything for it.

Two consequences follow, and both are easy to forget.

Renormalize the weights. If you drop holdings from an axis, the remaining weights no longer sum to one. Each axis therefore has to build its own denominator from only the holdings that made it into that axis, rather than dividing by the portfolio total. Otherwise every axis with a gap is quietly scaled down, and a partial answer renders as a bad score. This is the subtle bug that comes free with doing the right thing on nulls, and it is worth writing a test for specifically.

Show coverage. Excluding data silently is just a more sophisticated lie than zero-filling it. Every radar carries a coverage line: how many holdings actually contributed, and what share of portfolio value they represent. Both numbers matter independently, because eight of eleven holdings sounds perfectly fine right up until you learn the three missing ones are most of the money.

If you exclude data from a score, you owe the user a coverage number. I would make that a hard rule for any composite score in any product.

Rule 4: one scoring rule for heterogeneous things is quietly wrong
The Valuation axis blends several different multiples together. My first version applied the same blend to every holding, which is clean, uniform, and wrong for a meaningful minority of them.

Banks and insurers do not have meaningful free cash flow or EBITDA in the way an operating company does. They are spread and leverage businesses, and the convention every bank analyst uses is to anchor on book value instead. Scoring a bank on a cash flow multiple is not a slightly worse estimate. It is a category error that produces a confident number out of an input that does not apply to that kind of business at all.

So the axis branches by sector. Financial-sector holdings are valued on the measures that mean something for a balance-sheet business, and everything else is valued on the operating-business measures. Same axis, same 0 to 100 output, two different models underneath.

Two things in that decision are worth naming separately.

The branch is a class-specific model, not a special case. If your scoring system covers entity types that genuinely behave differently, uniformity is not fairness. It is applying the majority's model to the minority and calling it objective. The place to look is the classes where an input is undefined rather than merely unusual, and that is where you branch.

The blend within each branch is weighted, and the weighting is an opinion I hold deliberately. Not every multiple is equally informative, so the one I consider most informative for an operating business counts for more than the others. There is no way to build a composite score without embedding opinions like that. The only real choice is whether you state them or bury them. I state them, in the tooltip, on the axis, next to the number they shaped.

Rule 5: if you cannot show the arithmetic, do not show the number
A composite score is an authority claim. You are asking someone to trust a number whose derivation they cannot see, about their own money.

So the rule I hold myself to: any composite score must be fully decomposable down to its raw inputs within two clicks.

Under the chart there is a "How is this calculated?" link. It opens a table with every holding as a row: its position weight, every raw input that fed the model, and its five sub-scores. Hovering any sub-score shows the components that produced it, each with its raw value, the band it landed in, and the points it earned, followed by how they combined. The full band tables sit above the table. Nothing is hidden behind an explanation of the explanation.

The useful side effect is that building this caught my own bugs faster than any test did. When you are forced to render the arithmetic for a real portfolio, a mis-set band or a double-counted metric is obvious on sight. Cash flow growth originally sat on the Growth axis, and seeing it laid out beside revenue growth and earnings growth made it immediately obvious that it belonged on Cash Flow with the other two cash metrics, which is where it lives now.

The wart I have not fixed
Honesty compels this bit.

The band tables currently exist in two places. Once in the server module that computes the scores, and once in the client component, because the breakdown view needs to render the exact math and cannot reach into a server-only module. There is a comment in the client file that says, roughly, if the server thresholds change, update these too.

That is a comment doing a type system's job, and one day it will be wrong.

The right fix is a shared, dependency-free module holding nothing but the bands and the pure scoring functions, imported by both sides, with a test asserting the two agree at every boundary value. I know exactly what to do. I have not done it, because the tables have been stable and the duplication has not bitten me yet. Writing this paragraph has moved it up my list, which is probably the real argument for writing about your own architecture in public.