Getting reliable structured output from an LLM
Parsing JSON out of prose is a losing game. Schema enforcement, validation and repair patterns that hold up under real traffic.
The moment an LLM feeds another system rather than a human, free text becomes a liability. You need a contract, and you need it enforced at the boundary rather than hoped for.
Use the provider's schema mode
Constrained decoding — where the model is forced to produce output matching a schema — removes an entire category of bug. If your provider supports it, use it. Prompt-only instructions to 'respond in JSON' will hold most of the time, and most of the time is not a specification.
Shape is not sense
Schema mode gives you
- Valid JSON every time
- Correct field types
- No missing required keys
- No parsing of prose
You still have to check
- A date in the future for a past event
- A total that does not match the line items
- A valid enum that is implausible given the input
- Confident values extracted from a blank field
Validate anyway
Schema mode guarantees shape, not sense. A field can be correctly typed and completely wrong: a date in the future for a past event, a total that does not match the line items, an enum value that is valid but implausible given the input. Validate semantics after you validate structure.
Design for partial success
Extracting fifteen fields from a document, you will often get twelve right and three uncertain. All-or-nothing handling throws away good work. Return per-field confidence, accept what is solid, and route only the uncertain fields to a human.
Recovery ladder for malformed output
Escalate in order — each rung costs more than the last.
- 1
Repair
Fix trivially malformed output programmatically before rejecting it.
- 2
Retry with the error
Feed the validation failure back as context and ask again. Once.
- 3
Escalate the model
Fall back to a stronger model for cases that still fail.
- 4
Escalate to a human
Show the model's attempt as a starting point rather than a blank form.
Repair, then retry, then escalate
A tiered recovery path keeps quality high without burning cost on hopeless cases:
- Repair — fix trivially malformed output programmatically before rejecting it
- Retry once with the validation error fed back as context
- Fall back to a stronger model for the cases that still fail
- Escalate to a human, with the model's attempt shown as a starting point
Keep the raw response
Store what the model actually returned alongside the parsed result. When something looks wrong three weeks later, the raw output is the difference between a five-minute diagnosis and an afternoon of speculation.