We scored 237 apps in one Shopify category because the star ratings were useless
I spend a lot of time crawling the Shopify App Store, and this is the clearest case I've run into of ratings just not doing their job.
Picked one category — returns and exchanges apps, 237 live listings — and looked at what the storefront actually shows shoppers. Of the 111 apps that have any reviews at all, 101 sit at 4.5 stars or higher. When almost the whole category clusters at the top, the rating stops telling you anything about which one to pick.
The other 126 of the 237 have never been reviewed at all — not low-rated, just never reviewed. And five apps hold 54% of the 10,345 reviews the whole category has earned, so the review count is as lopsided as the star rating is compressed.
I re-read every shortlisted listing live on 21 September 2026 rather than trust a stale crawl, so the pricing below is current: entry prices in the category run from $4.99 to $179. And of the negative reviews that do exist, 57% are about support, not the product itself — worth knowing if you're trying to judge a tool by its complaints.
The caveat: this is one category, one snapshot, one day. I don't know yet whether returns apps are unusually skewed or whether any mature Shopify category looks like this once a handful of players have spent years compounding reviews.
If you're shipping into a crowded category with zero reviews yet, genuine question — does that actually cost you conversions, or only once a buyer has something to compare you against?
(I work on BestAppify, that's where the numbers come from)
https://merchants.bestappify.com/blog/best-shopify-apps-to-reduce-returns-52
Have you seen any evidence that merchants actually choose differently when given deeper comparison data, or is the rating problem mainly obvious analytically but weak as a buying trigger?
Good write-up. What would you do differently if you started again?
Honestly, I'd lock the live re-check in earlier. We scored the 237 apps from our own crawl, then went back and manually re-read every shortlisted listing on the live App Store on 21 September to get today's prices and ratings — that second pass is what caught the stuff our crawl was already a few days stale on. Next time I'd build that live re-check into the pipeline from day one instead of bolting it on at the end. (I work on BestAppify, that's where the numbers are from.)
Makes sense. Are you planning to charge for it, or keep it free for now?
The writeups like this one stay free — that's how people find us. The paid side is the tracking underneath: we crawl 27,115 live Shopify apps and 2,495 keywords, and that ongoing monitoring is the actual product. A one-off category score like this is basically a demo of what the tool sees day to day. (I work on BestAppify.)
Curious how long it took before you saw the first real results?
We don't actually log a "time to first result" number for something like this — it's not a metric we track. What I can tell you is the crawl date: the scoring ran off our data and every shortlisted listing was re-read live on 21 September 2026, so what's published is only a few days old. Beyond that I'd just be guessing, so I won't. (I work on BestAppify.)
How did you decide this was worth building in the first place?
The trigger was looking at the raw listings and realizing the stars told us almost nothing: 101 of the 111 returns apps with any reviews at all sit at 4.5 stars or better, and the median rated app has just seven reviews. When nearly everyone's a 5-star app with single-digit review counts, star rating stops being a real signal — that gap is what the scoring was built to fill. (I work on BestAppify.)
Interesting take. Would you still recommend this approach to someone starting today?
Mostly yes — the alternative is trusting star ratings that don't discriminate between apps, which is worse. The honest caveat: 126 of the 237 apps in this category have never been reviewed at all, so for over half the list you're scoring on things other than review signal, and that's softer than I'd like. If you're doing this yourself, budget real time for the manual re-read step — that took longer than the crawl did. (I work on BestAppify.)
How did you decide this was worth building in the first place?
Rating compression is the same problem in SEO tools. Everyone's site health score clusters around 85-95 because the tools grade generously to avoid churn. When the scale only uses the top 10% of its range, the number stops being information and starts being decoration.
The 57% negative reviews about support rather than product is the more useful finding. It means the actual differentiator in a mature category is not feature parity — it is response time after something breaks. That shifts the competitive advantage from engineering to operations, which most solo founders underestimate.
On the zero-reviews question: we sidestep it at UtilitySEO with a free scan that requires no signup. If someone can try the tool in 30 seconds, the review count matters less because they form their own opinion before ever looking at ratings. Harder to pull off when the product requires a Shopify integration to evaluate, but worth asking whether a preview or sandbox mode could serve the same function.