Why AI inference costs keep falling faster than anyone predicted

For most of the past three years the headline number in artificial intelligence was the cost of training. It made for clean coverage: a single figure, a single moment, a single decision by a single company. But the number that decides which products survive is the cost of inference — what it costs to answer one request, one time, for one user.

That number has been falling steadily, and faster than most planning assumptions written even eighteen months ago. The decline is not the result of one breakthrough. It is the compound effect of four separate improvements landing at once.

Four forces compounding

• Hardware: newer accelerators deliver more usable memory bandwidth per dollar, and memory bandwidth — not raw floating point throughput — is the binding constraint for most serving workloads.
• Quantisation: running weights at lower precision has moved from a research trick to a default deployment step, cutting memory footprint with measurable but generally acceptable quality loss.
• Serving software: batching, paged attention and prefix caching turn idle accelerator time into completed requests. The same hardware now serves several times the traffic it did two years ago.
• Model design: mixture-of-experts routing and distillation mean fewer parameters are activated per token, which lowers cost without a proportional drop in capability.

Why the curve matters more than the benchmark

A capability that is technically possible but economically absurd is not a product. When a feature costs a dollar per user interaction, it ships behind an enterprise contract. When it costs a fraction of a cent, it ships as a default in a free tier. The falling curve is therefore a schedule for which product categories become viable and when.

The interesting question is no longer whether a model can do the task. It is whether the task can be done at a price the product can absorb.

The margin problem moves downstream

Cheaper inference does not automatically mean healthier businesses. As unit costs fall, usage expands to fill the available budget: longer contexts, multi-step agent loops, background jobs that run whether or not a user is watching. Several application companies report that their cost per request dropped while their cost per customer rose.

This is a familiar pattern in computing. Cheaper storage did not reduce storage spending; it produced more data. Cheaper bandwidth produced video. Cheaper inference is producing agents.

What to watch next

Three signals are worth tracking through the rest of the year. First, whether context windows keep growing faster than the caching techniques that make them affordable. Second, whether smaller specialised models displace general-purpose ones for high-volume tasks. Third, whether power availability, not silicon supply, becomes the real ceiling on serving capacity.

The last of those is increasingly the constraint operators talk about privately. Chips can be ordered. Substations cannot.