LLM observability: what to log and what to watch
You cannot debug what you did not record. The traces, metrics and samples worth capturing before your AI feature meets real users.
Conventional monitoring tells you the request returned 200 in 800ms. For an AI feature that is close to useless — the interesting failures all return 200 with a confident, wrong answer.
Log the whole chain, not the call
One user question can involve a rewrite, a retrieval, a rerank, two model calls and a tool invocation. Logging only the final call leaves you unable to answer the one question that matters: which step went wrong? Trace the chain with a shared id.
- The original input and the rewritten query
- Retrieved chunks with scores, and which ones the answer actually used
- Full prompt as sent, including the resolved system prompt and prompt version
- Raw model output before parsing
- Tokens, latency and cost per step
One question, many steps — trace all of them
Logging only the final model call leaves you unable to say which step failed.
Post-processing
Validation, repair, fallback taken
Model call
Raw output before parsing, tokens, latency, cost
Prompt assembly
Full prompt as sent, plus prompt version
Retrieval
Chunks returned, with scores
Query rewrite
What you actually searched for
User input
Original text, before any rewriting
Watch the metrics that predict complaints
Refusal rate, retrieval-miss rate, fallback rate and the share of low-confidence answers all move before your satisfaction scores do. Alert on those and you find problems while they are still small.
Metrics that move before complaints do
- Refusal rate climbingRetrieval is missing
- Fallback rate climbingPrimary path is failing quietly
- Low-confidence share climbingContent or questions have shifted
- Satisfaction droppingYou found out last
Sample and read the transcripts
Automated evals catch what you thought to test. Reading fifty real conversations a week catches everything else — and it is the single habit that most reliably improves an AI product. Rotate who does it.
Mind what you are storing
These logs contain user input, which frequently contains personal data. Decide retention deliberately, redact at capture rather than at read time, and make sure your logging story is one you would be comfortable explaining to the customer whose data is in it.