We tried to write one honest sentence about Shopify's AI endpoint. It took a 190-store experiment.
What we built. Shelfglance scans a Shopify store for what AI shopping
assistants can actually read from it - product structured data, crawler
access, server-rendered pages, llms.txt - grades it, and keeps a public
directory. 10,099 stores scanned so far, every report page public. It sells a
$29/month watch that re-scans a store daily and emails you only when something
changes, and a $149/month board for agencies with up to 25 client stores.
The sentence. Since 2026 every Shopify store answers an agent-commerce
endpoint (/api/ucp/mcp, Universal Commerce Protocol) that merchants never
switched on. Its schema declares a category filter. We wanted to tell
merchants what leaving the product category blank costs them - "agents filter
by category, blank means invisible". Obvious, plausible, and one link away
from being in ten thousand report pages.
So we tested it first.
The experiment. 200 stores, picked deterministically from the corpus (sort
hostnames, take every Nth, so anyone can redo it). Five calls per store: a
control with no filter, an impossible category, the store's own category, the
same id unwrapped, and a price filter of one cent as a second control.
Queries were three words from each store's own product titles - generic
words like "gift" gave numbers that were measured honestly and were still
wrong.
What it found. 190 stores answered. 186 ignored the category filter
completely: the impossible category returned the full unfiltered set. 4
rejected every value, including the category their own products carry.
0 filtered. Meanwhile the one-cent price filter worked on 150 of them - same
envelope, same store, one field changed. So the request was fine; the filter
just does nothing today.
The sentence we wanted to write was false. A blank category excludes a
merchant from nothing, because nothing is being filtered out of anything.
What it cost. In money, nothing: 1,000 HTTP calls from the same Cloudflare
Worker the scanner already runs on. In time, most of a day - and half of that
was being wrong. The first full run reported 177 stores refusing the request,
which looked like an even better headline. It was our bug: we were passing the
category back as the object the API returns instead of the string its schema
asks for, and calling a correct refusal a discovery. Fixing one .value
turned the finding from "Shopify breaks" into "Shopify ignores", which is a
much smaller and much truer thing.
What we did with it. Published the method, the mistake, and all 190 rows
as a CSV, so every row names a domain you can hit yourself with one curl. And
did not publish the sentence. The report pages now say what the scan
recorded, at the severity it recorded, and nothing a store can disprove by
viewing source.
The takeaway for anyone building on a new API. Test what an endpoint does,
not what its schema says. Schemas ship before behaviour on young protocols,
and the gap is exactly where a confident marketing claim goes to die.
Study and data: https://shelfglance.com/research/ucp-category-filter
Happy to answer questions about the method or the corpus.
Running a 190-store experiment just to craft an accurate summary of an API endpoint demonstrates how marketing claims often obscure the messy operational reality of platform integrations. By replacing marketing hype with hard empirical testing, this data-backed approach not only builds immense technical credibility with developers, but also exposes the hidden quirks and rate limits that businesses face when deploying AI at scale.
Killing the sentence after all that work is the part I would have fought myself on.
The thing I keep wondering about is the grade. If the category filter is inert today, does a blank category still cost a store points on its public report page? A merchant reads "defect" very differently from "risk once this actually ships", and right now you are one of the few people who knows which one it is.
Publishing the wrong run is the part I respect most — most teams would have shipped the 177-store "Shopify breaks" headline. The schema-vs-behavior gap shows up in analytics too: GA4 documents events that silently stop firing after a consent change, and dashboards keep reporting like nothing happened. Behavior tests are the only honest check.
The 190-store test is the strongest signal here.
Are merchants paying more attention to actual behavior changes, or mainly to the static readiness score?
The discipline here is the same one that decides cold-email copy, from the seller side: a claim survives only if the reader can disprove it with one click. We sell a $19 outreach kit and the single most common edit we make to any template is replacing a superlative with a checkable number — "tested your endpoint before writing" beats "cutting-edge solution" every time, because it survives forwarding.
Question from the other end of your corpus: we ship an llms.txt and keep our pages server-rendered for the same reason you scan for it. Across 10,099 scans, do you see any signal yet that agent-readable stores actually receive agent traffic — or is agent commerce still a write-ahead log, protocols deployed before the readers exist? I'm deciding how much effort to keep spending on agent-readable surfaces at near-zero traffic scale, and your 190-row sample is the best data I know of.
The part worth stealing here isn't the finding, it's the second control. The one-cent price filter is what turns "the category filter doesn't work" from a guess into a fact — same envelope, same store, one field changed, so you've ruled out your own request being malformed. Most people testing a young API skip that and end up publishing their own bug, which is exactly what your first run was.
I do adjacent work (a bot that answers from a site's own crawled pages, so what's actually readable on a store matters to me), and the same rule keeps saving me: the schema tells you what the vendor intends, the response tells you what ships. Structured data is the worst offender — pages declare fields that no consumer reads, and if you write copy off the declaration you're selling a benefit nobody receives yet.
Two questions on the method: (1) do you re-run the 200-store panel on a schedule? The interesting artifact is the date the filter starts working, and you're the only one positioned to catch it — that's a better recurring reason to pay $29 than "we watch your store." (2) Any pattern in the 4 that rejected every value, including their own category — theme, app, plan tier? Four out of 190 is small, but if they share something it's a hint about which path actually implements the filter.
Publishing the mistake alongside the CSV is why I'd trust the next number you post. Rare.