The Defect Detector That Was Too Good on Camera Day

A manufacturing partner came to us with a defect detection model performing at 99.2% validation accuracy but an unexplained rise in field-reported defects reaching customers.

The cause wasn't the model architecture. It was that the validation dataset had been collected under a calibrated lighting rig, while the production line's lighting varied by up to 30% in intensity across a shift due to a skylight the original data collection hadn't accounted for.

The model had, in effect, learned to detect defects under one lighting regime and silently generalized worse under the rest. Standard validation metrics gave no signal — accuracy on the held-out set stayed flat because the held-out set shared the same lighting bias.

This is the kind of failure that never shows up until you specifically test for environmental variance, which is why it's now a required check in every deployment we advise on.