Is there demand for an LLM change-history API?
I crawl the docs of 15 LLM vendors and record every change: deprecations,
shutdown dates, price moves. Vendors overwrite these pages, so nobody keeps the
history ? the page just says something different than it did last month.
The open question is whether that history is worth selling as an API:
before-and-after diffs, when a date moved, what a price used to be.
It is also the kind of dataset that only gets more useful with age. Right now
it is months of history, not years, which is either the reason to wait or the
reason to start.
Before I build the billing side, two things I would rather hear from people who
have paid for a data API than guess at:
- Has a deprecation or a silent price change ever actually cost you something?
Or did you find out late and shrug? - When a data API charges per row returned instead of per request, does that
read as fair, or as a meter you cannot predict?
The strongest signal seems less whether people find the history useful and more whether a past change has already caused enough damage to justify paying for prevention. Have you found cases where a deprecation or pricing change created a concrete cost or incident?
Thanks for pushing on this — it's the right question and the one I was least ready for.
Honest answer: I don't have customer incident data, so I can't tell you someone lost money. What I do have is a case where the deadline itself moved.
On July 28 Google's deprecation page listed October 16, 2026 as the shutdown date for gemini-2.5-pro, -flash and -flash-lite. On August 3 the date was gone. As of today the same page reads "No shutdown date announced" for all three — 33 days later, with no replacement date.
So a team that read that page in late July scheduled a migration against an October deadline. A team that read it a week later saw no deadline at all. Same first-party page, no notice in either direction.
That part I can prove. Whether it has cost anyone real money I genuinely don't know, which is why I asked rather than assumed. I'm going to go looking for that number before I build anything else — thanks for making me say it out loud.
That deadline change is a useful concrete signal. The next thing I’d be curious about is whether any teams actually changed a migration plan because of the conflicting dates — that would get you closer to consequential demand.
Thanks for staying on this instead of letting me settle for the interesting version.
Straight answer: not that I can see, and I don't have a way to see it — I'd be inferring from traffic, which isn't evidence anyone rescheduled anything.
So you've turned it from a data question into an interview question. If you know anyone who had Gemini 2.5 in a migration plan this summer, that's the conversation I need. Otherwise I'll go find them, and that's my next step. Thanks again.
That’s the right conversation to have next. If you’re open to it, what’s the best email to reach you on?
Happy to. The contact form at aichangewatch.com/contact comes straight to me — send anything through there and I'll reply from my own address, rather than posting one in a public thread.
If it's easier the other way round, say the word and I'll reach out to you first.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
For the API, I'd want separate fields for when your crawler first observed a change and when the vendor says it takes effect, plus the captured source. Otherwise a polling delay can look like a vendor announcement date. That distinction would matter if I were reconstructing why a migration was scheduled.
Thank you — this is the most practically useful note I've had on the shape of the data, and the one I'd most want to get right.
You're right that collapsing those into one field is the trap. I ended up with four separate dates per record for that reason: when my crawler first observed the change, the date the vendor's own record gives for it, when the vendor announced it, and the shutdown date it points to. Before and after source snapshots plus a diff sit alongside each one, which I think covers the captured-source part of what you're describing.
Where your point still lands is that announcement date. It's populated for 22 of the 267 records that carry a shutdown date, because Anthropic and AWS are the only vendors I've found who publish one at all. For everyone else it stays null — and your reason is exactly why I leave it null rather than filling it in from when my crawler noticed.
So nothing to change on my side here beyond keeping that discipline — you've confirmed the part I was least sure about. Appreciated.
Everyone in this thread is debating whether "history" is valuable in general, but I think the sellable wedge is narrower and doesn't need that question answered first: a deprecation-date-change alert, not a full history API. The Gemini example you gave is the product, basically — you're not selling "here's the diff," you're selling "we'll tell you the moment a vendor quietly moves a date you're already planning around." That's a much smaller, cheaper thing to build and sell than a general-purpose API, it's usable immediately by anyone doing migration planning regardless of whether they'd ever query historical data, and it sidesteps the whole "is history worth paying for" debate by monetizing the one moment everyone in this thread agrees is a real pain: finding out late.
The most concrete reframing anyone's given me, so thank you. You're right that it's a sharper wedge than the one I asked about, and it's closer to built — the Gemini one surfaced as an automatic event, which is how I found it.
The homework you've left me is frequency: that withdrawal is the only date movement I hold, 3 records from one incident. But 28 deprecations were announced with no date at all, so "a shutdown date just appeared for something you're running" is the same machinery firing more often.
I'm going to build that version. Thanks for saving me from the broader one.
Glad it landed. Frequency over rarity is the right instinct — that's the kind of thing that only shows up once you've got the taxonomy split out like you did. Good luck building it, would love to see where it lands.
Your real measurement problem isn't "do people care about history" - it's "how many customers actually measure the cost of silent breaking changes?" Most teams don't measure what they don't see. If a vendor changes a price or deprecates something quietly and your team never measures the impact, you don't have a cost to justify a prevention tool. You probably have lost time or overspending without even knowing it, but if you can't measure it, it doesn't count in a purchasing decision. The per-row pricing question is measurement granularity: "I know what I'm paying for" feels fair, but "I can't predict my bill" feels like you're hiding costs. The sale swings on whether your measurement system lets them feel in control of the number.
This is the most useful reply I've had on the idea, so thank you.
You named the thing I couldn't. "How many teams measure the cost of a change they didn't see" is the real question, and I suspect the answer is near zero — which makes this a vitamin until someone has a number. I'll be carrying that framing into every conversation from here.
On per-row: that's the sharpest version of that objection I've heard, and it lands. "I can't predict my bill" is exactly what I'd be answering on every call, and it sounds like predictability is worth more here than precision. I'll be weighing that seriously before I settle on a meter.