How to choose an LLM for your product
A practical selection framework — capability, latency, cost, data terms and switching risk — instead of chasing benchmark leaderboards.
Benchmark tables are the worst way to pick a model for a product. They measure general capability on public tests; you need specific capability on your data, at your latency budget, at a price that survives contact with your usage.
How to actually select a model
Decides the choice
- Twenty of your hardest real cases, run against every candidate
- First-token latency against your interaction budget
- Cost modelled on your real prompt shape, retries included
- Data residency, retention and training terms
Doesn't decide it
- Public benchmark leaderboards
- Headline price per million tokens in isolation
- Context-window size as a specification
- Which model a competitor announced
Start from your hardest real example
Take twenty of the genuinely difficult cases from your domain — the ambiguous ones, the ones with missing context, the ones your best human gets wrong sometimes. Run every candidate model against them. This tells you more in an afternoon than a month of reading evaluations run by someone else on someone else's data.
Latency is a product constraint, not a detail
A model that is two seconds slower changes what you can build. Interactive assistants need first-token latency under a second to feel alive; batch document processing can happily take a minute. Decide which you are building before you fall in love with a model.
Price the actual workload
Cost per million tokens is meaningless until you multiply by your real prompt shape. Long system prompts and large retrieved contexts dominate spend — a frugal-looking model with a fat context can cost more than a premium model used carefully. Model the whole request, including retries.
Read the data terms before the benchmarks
Where does your data go, is it retained, is it used for training, which region does it sit in, and what happens under your customers' contracts? For regulated buyers this is the first question, not the last, and it eliminates candidates faster than any capability test.
Design so you can switch
Assume the best model for your job changes within a year — because it will.
Cost and latency per feature
So a swap can be judged, not guessed
An eval suite behind the adapter
Runnable against any candidate
Prompts in version control
Versioned artefacts, not inline strings
One adapter layer
No provider SDK calls scattered through the codebase
Design so you can switch
Assume the best model for your job will change within a year, because it will. Keep provider specifics behind one interface, keep prompts in version control, and keep an eval suite you can run against any candidate. Switching should be a day of work and a decision, not a rewrite.
- One adapter layer, no provider SDK calls scattered through the codebase
- Prompts as versioned artefacts, not inline strings
- An eval suite that runs against any model behind the adapter
- A cost and latency dashboard broken down per feature