List Price Is Not Your Cost: What AI Features Actually Spend in Production
Gartner predicts that by 2030, running inference on a one-trillion-parameter model will cost providers over 90% less than it did in 2025. The same press release predicts that total inference spending will go up. The two statements are not contradictory. They are the whole point.
AI pricing is collapsing at the token layer and exploding at the task layer. If your finance plan assumes the former means your bill will shrink, you are optimising the wrong number.
The most obvious reason is that reasoning models bill for thoughts you cannot see. OpenAI's documentation says reasoning tokens "are not visible through the API" but still count against the context window and are billed as output tokens. Anthropic's extended thinking mode does the same thing: when summarised thinking is enabled, you pay for the full chain of thought the model generated, not the short summary shown to you. The provider is selling you a black box and charging by the weight of the box, not what comes out of it.
Recent work by researchers at Stanford, UC Berkeley, CMU and Microsoft Research puts a number on the mismatch. In their sample, the cheapest-looking AI model was actually more expensive than a pricier rival in 32% of pairwise comparisons once reasoning tokens were counted. In extreme cases, the real cost was 28x higher than the list price suggested. A separate study from Michigan, Stanford, MIT and Google DeepMind found that an agentic coding workflow on SWE-bench Verified burned 4.17 million tokens per task and cost $1.857. A single-turn code-reasoning pass on the same benchmark used 1,190 tokens and cost $0.016.
That is not a rounding error. It is a different product architecture. And the input-to-output ratio in those agentic runs was 153.85, which means most of the spend was not the model's answer. It was the context repeatedly shovelled back into the prompt. The lever that controls your bill is not output length. It is context management.
The same paper reports another uncomfortable fact: run-to-run cost variance on identical tasks reached 30x, and the models' own cost estimates correlated with actual cost at no better than 0.39. Your budget is being decided by a process that cannot predict itself.
This is why the published discounts are both generous and misleading. Anthropic's cache read is 90% off. OpenAI's gpt-5-nano prompt caching is 90% off. Gemini's implicit caching on 2.5+ models is 90% off. OpenAI's Batch API cuts everything 50%. Those numbers are real, but they apply only to the tokens that qualify. A cache miss, a reasoning model that was not cached last turn, a user who leaves a chat and returns three hours later — none of those qualify.
GitHub's engineering team wrote about how they hit a 94% cache hit rate on Anthropic models in VS Code agentic workloads. The method was not a checkbox. They anchored cache_control at four stable boundaries: the end of tool definitions, the end of the system prompt, and two rolling anchors in the conversation. For OpenAI models, they set 24-hour prompt cache retention and saw relative hit rates on GPT-5.4 improve by 679% at 40 to 60 minute intervals.
Ninety per cent off is an engineering target, not a default. Cache misses are expensive too. Anthropic charges 1.25x the uncached input rate for a five-minute cache write and 2x for a one-hour entry. From GPT-5.6 onward, OpenAI bills cache writes separately at 1.25x. If your hit rate is 5%, you are paying a premium to maintain a cache you barely use.
The other hidden multiplier is retries. McKinsey's Enterprise AI FinOps survey, published in July 2026, found that about 60% of the cost of an agentic task goes to refining, checking and re-verifying the answer rather than the first generation. That is a different way of saying the same thing: the cheapest token is the one you did not have to spend twice. But you will not know you spent it twice unless you are measuring per-task cost, not just monthly bill.
That is where most teams stop. Menlo Ventures' enterprise generative AI survey from late 2025 found enterprise spend hit $37 billion in 2025, up from $11.5 billion in 2024. ETR's Summer 2026 macro survey found that 47% of organisations were moderately or significantly over their AI budget plan, only 6% were under, and 10% had no formal AI budget at all. When overspending happened, just 17% said they would pause or reduce. Overspending is the baseline.
There is a sensible response, but it is not glamorous. Stop obsessing over list price per million tokens. Start measuring cost per completed task. That means instrumenting your pipeline so you can see where retries happen, where context is being wasted, where a cheaper model would have been good enough, and where the expensive one actually matters.
The companies that get this right route tasks by cost and quality, not by brand. They use a reasoning model only on the 5% of inputs that need it. They run batch inference where latency does not matter. They pin snapshots, so a silent update does not suddenly double their spend. And they keep a live comparison against alternatives, because the best model for the price changes every quarter.
That routing layer is exactly what MuseSpark AI is built for: low-cost model access across providers, with stable endpoints so you can move traffic as prices and capabilities shift without rewriting your application.
The list price is a starting gun. Your real cost is what you pay after retries, cache misses, hidden reasoning tokens and a workflow that turned a five-cent prompt into a two-dollar task. Gartner is right that inference will get cheaper. They are also right that we will spend more. The difference between those two truths is your architecture.
