“One overturned false call, one lesson” is a fair description, but it is easily read as an automatic process: report it and things improve. What actually happens has three stages — attribution, admission, verification — and if any is missing, feedback is only bookkeeping. More importantly the path runs both ways: the same mechanism that moves a boundary towards accuracy can move it towards passing real defects, and the second direction makes the false-call numbers look better every time.
ACI in this article stands for Automated Cognitive Inspection — the next-generation term for AI AOI: keep the A and the I, and replace the O (optical) with C (cognitive), because the decision basis is cognition rather than the imaging method. The full argument is set out in From AI AOI to ACI.
1. What one overturned call actually gives the model
Start with the information content. An overturned review says that on one image, one decision was judged wrong by a person. That is one bit on one sample, and on its own it cannot move any model.
What makes it useful is attribution. “This image at this designator is fine” is worth almost nothing, because that designator will not be there next time. “Components of this class in this surface condition are fine” is worth a great deal, because it holds for every component of that class from then on.
So the first step of feedback is not training but classification. An overturn that cannot be attributed to a class is only a log entry, however completely it was recorded.
2. The unit of learning is the component class, not the image
Once the criterion hangs on the component, the description of good is organised by component class. What an overturn does is return that sample to the good set of its class, pushing that class's boundary outward a little.
How far depends on the denominator. Where a class already holds several hundred samples, one overturn barely moves it; where it holds a dozen, one overturn matters a great deal.
Feedback works visibly on new part numbers and seems inert on mature ones — not a malfunction, a different denominator.
That explains a common puzzle on the line, and it carries a practical implication: for mature classes, no single piece of feedback should be expected to produce a visible effect. What to watch is where overturns of the same class cluster over a period.
3. “Sharper with use” has three preconditions
The claim holds under conditions. Miss any one and feedback learning degenerates into a table of recorded false calls:
- Attributable: an overturn must land on a component class and a variation type. It cannot land anywhere if review only records “OK”.
- Retained: the sample itself must be kept, not only the conclusion. Keeping the conclusion alone is knowing the answer without the question, and nothing can be retrained from it.
- Verifiable: after an update, what changed must be measurable. Otherwise there is no telling accuracy from looseness — and in the short term the two look identical.
The one most often missing is the first, and it is the only one that has to be solved at the review station: whether the interface lets a person pick a reason in one gesture decides whether the other two have any raw material.
4. The reverse risk: feedback can also teach a model to pass real defects
The definition of a false call hands the verdict to review, and review makes mistakes. Parts waved through during the shifts when output is behind will, if admitted to the good set as usual, push the boundary some distance towards real defects.
This drift has three properties that together make it hard to notice. It is one-way, because correcting it requires someone to actively pull a part back, and nobody pulls back a part that has already been judged. It is slow, moving a little at a time. And it makes the false-call figures look better on every step.
Put differently, the process of a model going wrong presents, on every metric anyone looks at daily, as improvement. It does not surface by itself; it is usually found in hindsight after a batch escapes.
5. Hence a frozen holdout set
There is only one direction of remedy: keep a set of samples that takes no part in any feedback update, including confirmed real defects. Run it after every update and ask one question — is what used to be caught still caught?
The whole value of that set is in the word frozen. Once it starts being updated along with everything else, it shares an origin with what it is meant to judge and its discriminating power falls to zero: it will report that all is well after any change, and that report is word for word identical to a genuine all-is-well.
This is the rule that correction and verification must not share one basis, in its feedback-learning form. Split the holdout from training data by time or by line, and change it only by an explicit decision — never by quietly adding a few more.
6. Two kinds of problem feedback cannot fix
Telling these apart saves a great deal of fruitless investigation into why something is never learned:
- Not visible in the image: the defect leaves no signal at the current illumination, viewing angle or resolution. No amount of feedback learns something that is not there; what needs changing is the imaging, not the model.
- The criterion is itself a geometric specification: coplanarity and height difference have explicit tolerances, and a rule layer answers them more reliably than a model approximating them.
A simple way to tell: pull up the missed parts and have a person look at the same images. If a person cannot see it either, it is the first kind. Only when a person sees it at a glance and the model did not report it does feedback learning apply.
7. Who is allowed to teach it
Feedback is a write permission — it changes not this part's verdict but the verdict on every similar part afterwards. Treating it as one means arranging three things:
- Separate the roles: the person reporting a false call and the person approving a sample into the set need not be the same, least of all when output is behind.
- Apply in batches rather than instantly: instant effect sounds fast, but it leaves no buffer between one wrong admission and a change in what the line accepts.
- Version it and make it reversible: a system that cannot say what a given update changed can only be rolled back wholesale when something goes wrong.
8. How to verify that a piece of feedback took effect
“It has not been reported since” is not enough. The thing may simply not have appeared again, and “did not appear” and “was learned” are the same number on a report.
What works is two checks together. Pull historical samples of the same class and re-run them to see whether the verdicts changed as expected — that shows it learned. Then run the frozen holdout and confirm that what used to be caught still is — that shows it did not learn the wrong thing.
The first check alone tests whether it learned; the second alone tests whether it went wrong. Only both together answer whether the update is worth shipping.
9. When feedback learning is not needed
The cases where this does not apply deserve the same clarity:
- False-call volume is already low: when review is not the bottleneck, the upkeep of this machinery does not return its cost.
- Product life is very short: the model never accumulates enough for clusters to show before the model changes, and the denominator never builds.
- There is no stable review process on site: the quality of feedback learning is capped by the quality of review, and where review is casual it only goes wrong faster — in that case not deploying it is the safer choice.
The third deserves emphasis. Feedback learning is not a module that runs on its own; it is a way of scaling human judgement. Where that judgement is unsteady, what gets scaled is the unsteadiness.
On why the good set is a closed one and which way its boundary moves, see Defects are an open set, good parts are a closed one.
The full argument for not tuning and verifying on the same data is in A threshold is a curve, not a number.
What this looks like in the product: Feedback learning.
FAQ
Does the model change immediately when a false call is reported?
Usually not, and it should not. One overturn is one bit on one sample, and its effect on the boundary depends on how many samples that component class already holds — barely anything in a mature class, a great deal on a new part number. Instant effect also removes the buffer between one wrong admission and a change in what the line accepts; batched updates with an approval step are safer.
Can feedback learning teach the model the wrong thing?
Yes. Review holds the verdict on what counts as a false call, and parts waved through when output is behind will push the boundary towards real defects if admitted to the good set as usual. The drift is one-way, slow, and makes the false-call figures look better each time, so it never surfaces by itself. The only remedy is a frozen holdout that takes no part in updates, run after each update to confirm that what used to be caught still is.
How do you prove a piece of feedback took effect?
Not by “it has not been reported since” — the thing may simply not have appeared. Do two checks together: re-run historical samples of the same class to see whether verdicts changed as expected (it learned), and run the frozen holdout to confirm nothing regressed (it did not learn the wrong thing). Either alone leaves the shipping decision unanswered.
Full solution for this scenario: the full inspection solution for AI AOI Software
This article was generated by AI. Customer cases are simulated scenarios based on real product capabilities and figures are illustrative; see product pages for official benchmarks.