RAG in production: beyond the demo
Retrieval-augmented generation is easy to prototype and hard to productionize. Here's what actually matters when you take RAG live.
A RAG demo takes an afternoon. A RAG system your customers trust takes real engineering. The gap between the two is where most AI projects quietly stall.
At DSME we've shipped retrieval systems into support, underwriting and clinical workflows. The patterns that separate a convincing demo from a dependable product are surprisingly consistent.
Chunking is a product decision, not a default
The naive approach — split documents every 500 tokens — throws away the structure that makes retrieval accurate. Headings, tables and semantic boundaries carry meaning. We chunk with the document's own structure and attach metadata so answers can be cited and filtered.
Evaluate before you trust
You cannot improve what you cannot measure. Every RAG system we ship comes with an automated eval harness scoring retrieval relevance, answer accuracy and hallucination rate on a curated set — run on every deploy.
- Retrieval precision/recall against a golden set
- Answer faithfulness (is it grounded in the sources?)
- Citation accuracy — do the links actually support the claim?
- Latency and cost budgets per query
Guardrails and graceful failure
The best RAG systems know when they don't know. We design for confident abstention — routing low-confidence queries to a human with a drafted answer rather than inventing one. Users forgive 'let me connect you'; they don't forgive a confident wrong answer.