Computer vision in the wild: what breaks between the lab and the loading dock
A vision model that scores beautifully on a curated test set can fall apart on its first night shift. Here's what actually changes when CV leaves the lab — and how to build for it.
Computer vision has a peculiar failure mode: the demo is genuinely impressive, the test-set accuracy is genuinely high, and the system still falls over in its first week on site. The model didn't get worse. The world it was trained on just wasn't the world it was deployed into.
We've put vision systems into logistics yards, workshops and inspection lanes, and the lesson repeats: the model is maybe a third of the work. The rest is optics, physics, operations and the slow drift of reality away from your training data. This is what breaks, roughly in the order it breaks.
Lighting is the first thing production takes away
Training data gets collected on good days — daylight, clean lenses, someone standing at the right angle. Production is a loading dock at 5am: sodium lamps, headlight glare, hard shadows that move with the sun, rain on the housing and a lens fogging as the temperature swings.
The fix is only partly in the model. Augmentation for exposure, blur and glare helps, but the higher-leverage move is treating illumination as infrastructure: controlled lighting where you can add it, camera exposure settings you actually manage, and a collection plan that deliberately includes night shifts and bad weather from day one. If your training set has no 3am footage, your accuracy number does not apply at 3am.
Camera placement is an engineering decision, not an install detail
The single cheapest accuracy improvement in most deployments is moving the camera. A model asked to read a container code from thirty degrees off-axis, half-occluded by a forklift, is being asked to compensate for a mounting decision made by whoever had the ladder that day.
Do a site survey before you train anything. Decide the field of view, the working distance, the mounting height and what passes through the frame — then collect training data from that exact geometry. And plan for the mount to be disturbed: cameras get bumped, remounted and cleaned, and a system with no way to detect that its viewpoint has shifted will degrade silently for weeks.
The edge is a different runtime, not a smaller cloud
Most industrial CV can't ship every frame to a datacenter — bandwidth, latency and cost all say no. That means inference on a box at the site, and the box changes the engineering. Quantized models behave slightly differently than the checkpoint you validated; thermal throttling shows up in summer; power cuts and dropped connectivity are routine, not exceptional.
Design for the box from the start: validate the deployed artifact, not the lab checkpoint; buffer and reconcile when the link drops; and build remote update and rollback before the first unit ships, because driving to a site to reflash a device does not scale past three sites.
- Validate the quantized, exported model — the thing that actually runs — against your eval set
- Assume disconnection: queue events locally and reconcile when the link returns
- Ship remote model updates with staged rollout and one-command rollback
- Monitor device health (temperature, frame rate, disk) alongside model metrics
Drift arrives on a truck
Vision drift is rarely subtle statistical creep. It arrives as a physical event: a new packaging supplier changes the label layout, a customer repaints their fleet, the yard adds a lamp post in front of the camera, winter puts high-vis jackets on everyone. Each one moves the input distribution overnight, and none of them will be announced to your team.
You catch this with monitoring that looks at inputs, not just outputs: confidence distributions per camera, detection counts per hour against historical norms, and a sampled stream of low-confidence frames going to human review. When a camera's numbers step-change, you want an alert that day — not a complaint from operations a month later.
Labeling is an operation, not a phase
Teams budget labeling as a one-time cost before training. In production it is a standing pipeline: hard cases flow from the field, get labeled against a written guideline, and feed the next training round. The systems that keep improving are the ones where that loop is cheap and routine.
Two things make it work. First, an unambiguous labeling guide with the edge cases decided in writing — is a pallet half out of frame 'present'? — because inconsistent labels put a hard ceiling on model quality. Second, smart sampling: label the frames the model found difficult, not ten thousand more of the ones it already gets right.
Ship the feedback loop, then the accuracy
The instinct is to hold deployment until the model is excellent. The better move is to deploy earlier with a human checking the output, because the site itself is the only data source that matters and every reviewed frame is training data you couldn't have collected any other way.
A vision system with 90% accuracy and a working correction loop will beat a 95% system frozen at launch within a quarter. In the wild, the durable asset isn't the model — it's the pipeline that keeps remaking the model as the loading dock keeps changing.