Catching missing items at an appliance packing station

The station sits at the end of the packing line. Two foam inserts are placed on the belt — one with the kettle cavity, one with cavities for the manual and accessories. The operator puts the manual into the right-hand cavity, slips a plastic bag over the kettle, sets the bagged kettle into the left cavity, adds the base, then lays the film over the top and the unit moves on to the case sealer.

The packing itself is not hard. What is hard is that a missing item leaves no trace on the floor.

A missing manual becomes a complaint and a re-ship once the customer opens the box. A missing base means the unit cannot be assembled at all — a full carton comes back, plus freight and re-packing labour. All of that cost lands outside the factory, while the two seconds that produced it looked exactly like a good unit.

The usual fallback is weighing the carton. The magnitudes do not work: the carton weighs kilograms and a manual weighs tens of grams, well inside the natural variation of the foam and cardboard themselves. Even when a difference does show up, all it says is "lighter than expected" — not which item is missing, so the carton still has to be opened.

That hands the problem to vision — and packing materials are exactly what traditional AOI handles worst. White foam sits on white foam, the bag is transparent, and most of the manual is tucked into a cavity with only a corner showing. Irregular shapes, edges defined by reflection, foreground the same colour as background: template matching cannot frame them and fixed thresholds cannot separate them. On top of that, appliance models change often — each model has its own foam and its own checklist, so a template built today is obsolete in a few weeks.

We solve this layer with few-shot semantic segmentation: instead of drawing a box around the object, trace its outline. Each class starts from 1–10 sample images and runs within minutes of labelling. A new model does not need a retrained network — it needs a few more images.

The logic is not written, it is wired. The operator library is on the left; drag onto the canvas and connect, and that is the pipeline.

The split is deliberate: recognising is the model's job, deciding whether it counts is the event layer's job, and calling OK or NG is the state machine's job. Keeping them apart is what makes a model changeover or a rule change a local edit instead of a rebuild.

This layer answers one question only: in this frame, which pixels belong to which class.

We use 6 classes: foam insert, manual, bagging, kettle placement, base placement, closing. Why not simply "item present / absent"? Because "incomplete" on a report gets ignored, while "missing manual" gets acted on within a minute. The granularity of the class list is decided by the sentence the NG record has to produce, not by what the model can do.

Why segmentation instead of detection boxes: neither the bag nor the foam cavity is rectangular. A box over the kettle only says "the kettle is somewhere around here"; an outline that follows the cavity edge says "the kettle actually dropped into the cavity". For soft packing materials, that is the whole difference.

Each class starts with 5–10 sample images. Run once, pull the misclassified frames back into the training set, run again — all inside this one interface, with no offline training environment. Moving to another appliance model, where the foam shape and the checklist both change, works the same way: add images, do not rebuild.

Segmentation emits a mask on every frame. But "the manual is visible in this frame" and "the manual went into the box" are two different statements.

The operator pulls a manual from the bin, carries it across the station, and sets it down — the manual is in frame the whole time. If visibility alone counted, then carrying material past, picking it up and putting it back, even material from the neighbouring station drifting into view, would all register as a placement. This kind of false trigger is more dangerous than a miss: a miss lets one unit through, while a false trigger makes the system report OK on a unit that really is incomplete.

Placement events (E1–E4) all AND two conditions: the target appears inside its designated cavity and stays still for a number of consecutive frames. Material that has landed in a cavity does not move; material in a hand never stops moving. That single condition removes pass-throughs and test fits. The cavity region is drawn on screen and the frame count is typed in — changing either touches no model.

E5 and E6 use a different combination: Object present (the closing action) AND Count check (items placed). An explicit closing action is needed as the settlement signal because a packed unit, unlike a machined part, does not leave on its own. Laying the film over the top is the one moment on this station that reliably marks "I am done with this unit". Anchoring to it gives a definite settlement point instead of guessing when a unit ends.

Events are scattered points; the state machine accumulates them into a verdict per unit. This diagram is the entire decision logic — there is no code.

Manual and base: a set, not a sequence

Either can go in at any point and the unit is still good; operators each have their own comfortable route. So they hang on the self-loop at S1: each trigger increments the checklist count and the state does not change. What the system cares about at settlement is whether the set is complete, not the order in which it filled up.

Bagging and kettle placement: this pair does have an order

The kettle must be bagged before it goes into the cavity — an unbagged kettle rubs against the foam in transit and the finish gets scratched. That is a process constraint, not a habit, so the diagram has to enforce it.

The mechanism is an intermediate state, S2 bagged: bagging (E2) moves the state to S2, kettle placement (E3) is only accepted from S2, and once accepted the state returns to S1 to keep filling. That forms a gate — if the kettle goes in unbagged, E3 lands on S1 and is not accepted, the count does not increment, and by the closing action the unit is necessarily short one item and falls through to NG. No separate "wrong order" rule is needed; the gate structure already catches it.

Settlement: the verdict lands with the closing action

On an appliance line, models change more often than the line does. Keeping the decision logic on a state diagram is what keeps those changes inside the configuration layer.

1. Use segmentation, not detection boxes, for soft packing items. Foam cavities and transparent bags are not rectangles; only an outline can show that something landed inside.
2. Let the NG report define the class list. Decide what action the record should trigger on the floor, then work backwards to the granularity.
3. False triggers are more dangerous than misses. Suppress them in the event layer with regions and stillness conditions, rather than hoping the model figures it out.
4. Encode physical constraints in the diagram instead of writing rules. One intermediate state enforces bag-before-place; no separate order check is needed.
5. Keep the decision logic on the state diagram. A model changeover is a diagram edit, not a code change — which is what makes this sustainable on a line like this.