API design for AI products
Non-deterministic, slow and occasionally wrong — AI endpoints break the assumptions REST conventions were built on.
A conventional API returns the same answer for the same input, quickly, or an error. AI endpoints violate all three assumptions, and pretending otherwise pushes the problem onto every consumer.
Long operations should be jobs
Anything that might take more than a couple of seconds wants a job model: submit, receive an id, poll or subscribe. Holding an HTTP connection open for ninety seconds fails at every proxy between you and the client.
What an AI response should carry
Consumers need enough to decide whether to trust the result.
Request id
Traceable to your logs when support asks
Model and prompt version
So behaviour changes are attributable
Provenance
Citations or source ids for grounded answers
Confidence
Or an explicit uncertainty flag
The answer
Structured, validated
Return confidence and provenance, not just an answer
Consumers need to decide whether to trust the result. Give them what they need to do that: a confidence signal, the sources used, the model version and whether any fallback was triggered.
- Model and prompt version, so behaviour changes are attributable
- Confidence or an explicit uncertainty flag
- Citations or source ids for grounded answers
- A request id that traces to your logs for support
Make partial results first-class
Extraction that resolves eight of ten fields should return eight fields and two nulls with reasons, not a 500. Design the response so partial success is expressible rather than an error case.
Version behaviour, not just shape
In an AI API, the contract includes how it behaves. Swapping the underlying model can change output quality dramatically without changing a single field. Communicate model changes like breaking changes, because for your consumers they often are.
Behaviour is part of the contract
- Swapping the underlying modelTreat as a breaking change
- Operation may exceed a few secondsReturn a job id, not a held connection
- Extraction resolves 8 of 10 fieldsReturn 8 and 2 nulls with reasons
- Client may retryIdempotency key, or pay twice
Idempotency is not optional
Clients retry slow requests. Without idempotency keys, a retry on a generation endpoint means paying twice and, worse, duplicating any side effect.