RAG in production: beyond the demo
Retrieval-augmented generation is easy to prototype and hard to productionize. Here's what actually matters when you take RAG live.
A RAG demo takes an afternoon. A RAG system your customers trust takes real engineering. The gap between the two is where most AI projects quietly stall.
At DSME we've shipped retrieval systems into support, underwriting and clinical workflows. The patterns that separate a convincing demo from a dependable product are surprisingly consistent.
Chunking is a product decision, not a default
The naive approach — split documents every 500 tokens — throws away the structure that makes retrieval accurate. Headings, tables and semantic boundaries carry meaning. We chunk with the document's own structure and attach metadata so answers can be cited and filtered.
What to score in a RAG eval harness
Run on every deploy. Retrieval and generation fail differently — score them separately.
- Retrieval precision / recall5
Against a golden set — is the right chunk even returned?
- Answer faithfulness5
Is it grounded in the sources provided?
- Citation accuracy4
Do the links actually support the claim?
- Latency and cost per query3
Budgets, enforced in CI
Evaluate before you trust
You cannot improve what you cannot measure. Every RAG system we ship comes with an automated eval harness scoring retrieval relevance, answer accuracy and hallucination rate on a curated set — run on every deploy.
- Retrieval precision/recall against a golden set
- Answer faithfulness (is it grounded in the sources?)
- Citation accuracy — do the links actually support the claim?
- Latency and cost budgets per query
Guardrails and graceful failure
The best RAG systems know when they don't know. We design for confident abstention — routing low-confidence queries to a human with a drafted answer rather than inventing one. Users forgive 'let me connect you'; they don't forgive a confident wrong answer.
Design for confident abstention
- Retrieval returns nothing relevantSay so — do not improvise
- Sources contradict each otherSurface the disagreement
- Confidence below thresholdRoute to a human with a drafted answer
- Answer is well groundedAnswer, with citations