Computer vision papers report benchmark deltas; production systems report incidents. Between the two sits a gap that most tutorials never mention. These notes collect what changed in my practice after shipping vision models into environments with dust, glare, motion blur, and users who point cameras at things no dataset anticipated.
Datasets are the model
Across every project, the ranking of interventions by impact was stable: fixing label noise beat augmentation, augmentation beat architecture changes, and architecture changes beat hyperparameter tuning — usually by an order of magnitude each. A ResNet trained on clean labels reliably outperformed a state-of-the-art transformer trained on the raw crowd-labeled set.
The practical consequence is unglamorous: budget more time for dataset audits than for model experiments. We institutionalized a weekly 'worst 50' review — the fifty lowest-confidence predictions from production, relabeled by hand. Half of our accuracy gains over six months came from that ritual alone.
Relative accuracy impact by intervention
Indicative — each step was roughly an order of magnitude over the next
Evaluation is a product decision
A single accuracy number hides the question that matters: which errors are cheap and which are catastrophic? A false negative in a safety inspection app is not the same currency as a false positive in a photo organizer. We stopped reporting accuracy and started reporting cost-weighted confusion matrices, with the weights set by product, not engineering.
This reframing changed model selection more than any benchmark. The 'best' model on F1 lost to a slightly worse one whose errors were concentrated in categories users could easily correct.
Closing note
None of this diminishes the research frontier — it locates it. The hard problems in applied vision are distributional: the world your model meets is not the world it trained on, and the gap is where engineering lives. Ship the boring pipeline, audit the data, price the errors. The demo was never the hard part.