Ostryno
AI powered Creator Vetting Tool
Been building a creator-vetting tool for the last few weeks and made a call I keep second-guessing.
Every tool in this space — HypeAuditor, Modash, the free checkers — hands you a number. Audience Quality Score: 74. Credibility: 82%. It's what buyers expect and it's what the category trained them to want.
I'm not doing it. Two reasons. A single number invites a decision it can't support — nobody can tell you what separates a 74 from a 79, including the tool. And it puts me in the position of publishing a judgment about a named person's audience, which I don't think I can stand behind commercially.
So instead the report leads with audience composition — what share looks real, what looks suspicious — then the individual signals with the actual thresholds published, then a plain-English read of what the pattern suggests. You can disagree with my interpretation because you can see the working.
The bet is that people doing the vetting would rather have the evidence than the verdict. The risk is that a score is what makes a tool feel decisive, and without one this feels like homework.
Pricing will be $19/$49/$149. It's not live yet — I'm collecting a waitlist while I finish the report page. For context HypeAuditor and Modash both start at $299 and HypeAuditor gates its tiers behind a demo, which is the gap I'm going after.
Genuine question for anyone who's vetted creators for money: did you ever act on one of those scores, or did you always go look at the comments yourself anyway?
8 Comments
Say a creator reads 18% suspicious in October and 26% in December. The brand's first question is whether the audience changed or your thresholds did, and you'll be tuning those thresholds for months, so each report needs to store the threshold version and the follower snapshot it ran on. Without both, nobody can check October's report against December's, and that comparison is exactly what a $149 buyer will make.
You're right, and it already bit me. I changed the YouTube signal weights on Saturday and left formulaVersion at "v2.0", so two reports claiming the same version were computed with different weights. The label was worse than useless because it looked like a guarantee.
Fixed it by deriving the version instead of typing it. Every report now stores a hash of the constants that decide a score, so two reports are comparable if and only if that hash matches. Change a threshold and it moves on its own. The resolved weights and benchmark are stored in readable form next to it, along with the follower count at run time.
The follower snapshot was already there, so your second half was covered. The first half wasn't, which is the half I'd have sworn was fine.
The score-versus-evidence tradeoff is the real test. Have people on the waitlist said they'd actually make a creator-buying decision from the evidence alone, or do they still want a single number to act on quickly?
Honestly no, the waitlist is tiny and I'd be making it up. It's a conviction from watching brands on my marketplace act on scores they couldn't explain.
Would a real/suspicious percentage be enough for you, or does it still need to collapse to one number?
That marketplace behavior is an interesting signal. If you’re open to it, what’s the best email to reach you on?
hello @ ostryno . om
can't post links but its dot c o m
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
Pricing will be $19/$49/$149. It's not live yet — I'm collecting a waitlist while I finish the report page. For context HypeAuditor and Modash both start at $299 and HypeAuditor gates its tiers behind a demo, which is the gap I'm going after.