Small models are quietly eating the easy work
Classification, extraction and routing are moving off frontier models and onto systems small enough to run on a laptop.
Ask a team which model they use and you will usually hear the name of a frontier system. Ask which model handles most of their requests and the answer is increasingly something far smaller.
The workload was never uniform
Production traffic is dominated by tasks that are structurally simple: deciding whether a message is a refund request, pulling a date out of a document, choosing which tool to call next. These tasks reward consistency and low latency far more than reasoning depth.
Once teams instrument their traffic, the split becomes obvious — a small fraction of requests need the strongest available model, and the rest do not.
Three reasons the shift is accelerating
- Latency: a locally served small model answers in tens of milliseconds, which changes what can sit in an interactive loop.
- Cost predictability: fixed infrastructure is easier to forecast than per-token billing that scales with user enthusiasm.
- Data control: regulated teams prefer not to send document contents to a third party when a smaller model on their own hardware performs adequately.
The evaluation burden
The catch is measurement. Routing work to a cheaper model is only safe if you know what quality you are giving up. Teams that make the transition successfully almost always build a task-specific evaluation set first and treat model choice as a tunable parameter rather than a company-wide decision.
That discipline is the real prerequisite. The models are ready; most measurement practice is not.
Sources
About the author
Editor, AI & Infrastructure
Elena Marsh writes about machine learning systems, inference economics and the data centre build-out. She previously worked as a platform engineer before moving to full-time technology reporting.
