Human-in-the-loop patterns that actually scale
Review queues that become bottlenecks, and the designs that don't. How to keep people in the loop without capping your throughput.
'We'll have a human check it' is the most common answer to AI risk and the least designed. Done casually, it caps your capacity at the reviewer's attention and creates a queue nobody wants to own.
Review by exception, not by default
Routing everything to a human means you have automated nothing. Route by confidence and consequence: high-confidence low-stakes items go straight through, everything else surfaces. The design work is in calibrating that threshold honestly and revisiting it with evidence.
Make approving cheaper than doing
If reviewing a suggestion takes as long as producing it, reviewers will stop reading and start rubber-stamping — which is worse than no review, because it manufactures false assurance. Show the evidence, highlight what is uncertain, and make accept a single keystroke.
Capture the correction
Every human override is a labelled example you paid for. If you are not storing what was changed and why, you are throwing away the most valuable dataset the system produces.
Review that scales vs review that stalls
Scales
- Route by confidence and consequence, not everything
- Evidence shown before the recommendation
- Accept in one keystroke; corrections captured as training data
- Approval rate tracked per reviewer
Stalls
- Every item queued for a human
- Reviewing takes as long as doing
- Overrides discarded rather than stored
- Nobody notices a reviewer approving 100% of items
Watch for automation bias
People agree with confident machines, especially under time pressure. Counter it deliberately:
- Show the evidence before the recommendation, not after
- Surface disagreement between sources rather than resolving it silently
- Track approval rate per reviewer — a rate near 100% means nobody is reviewing
- Periodically seed known-bad cases and measure whether they get caught
Watching for automation bias
- Approval rate near 100%Nobody is reviewing
- Recommendation shown before evidenceYou are anchoring the reviewer
- Sources disagree, resolved silentlyThe reviewer cannot see the doubt
- Seeded bad cases get caughtReview is genuinely working