How to evaluate a model provider without guessing

Public benchmarks tell you how a model performs on a public benchmark. They tell you very little about how it will handle your documents, your customers, your latency budget or your compliance obligations. A structured evaluation takes about a week and prevents most of the expensive surprises.

1. Build a task set before you look at vendors

Collect fifty to two hundred real examples of the task, with the outputs you would consider correct. Write the grading criteria down before running anything. This artefact outlives any individual vendor decision and is the single highest-value item in the process.

2. Measure latency at the percentile that matters

Average latency hides the experience. Measure time to first token and total completion time at the ninety-fifth percentile, under load, from the region your users are in.

3. Model cost against realistic usage

• Count input tokens honestly, including system prompts and retrieved context, which usually dominate.
• Model the retry rate, because failed and repeated calls are billed calls.
• Check whether caching is available and what it actually discounts.

4. Read the data terms

Establish retention periods, whether inputs may be used for training, where processing happens geographically, and what the subprocessor list looks like. If a regulated customer will ask, get the answer in writing before you integrate.

5. Price the exit

Keep provider-specific logic behind a thin internal interface and keep your evaluation set portable. The goal is not vendor neutrality for its own sake; it is being able to switch in a week when pricing or quality changes.

A reasonable decision rule

Choose the cheapest provider that clears your quality bar on your own evaluation set, meets your latency target at the ninety-fifth percentile, and gives you acceptable data terms. Revisit quarterly, because all three variables move.