Large-Model Vision · Process Checks ／ In Practice

Home Appliances: Packing Process SOP Monitoring — Field Deployment

A station camera recognises parts and motion nodes in real time, and an SOP state machine confirms every step — missing parts, skipped steps and wrong order are visible on the spot, and every record is traceable.

by Leo 中文版

Key Points

The problem. A missing part does not surface at the factory — it surfaces when the customer opens the box; the cycle is fast, the parts are soft, and models change often.
Five operators, one tensor flow. Video-stream camera, preprocessing, semantic segmentation, event detection and a state machine, connected in order from frame capture to process judgement.
Few-shot start. Five target classes — base board, foam paper, carton, back panel, manual — each settled with 1–10 on-site images in the labelling tool.
Order enforced by a state machine. The S0–S6 graph manages the seven-step sequence; unexpected events simply self-loop — a skipped step never advances and never reaches the end.
One engineering translation. "Pack everything in the right order" is broken down into targets, sequence, states, exceptions and outputs — and lands in software configuration.

CASE 01Live packing station: the overhead view tracks parts and motion nodes while the workflow panel ticks each SOP step off.

Packing is the last manual step before a product leaves the factory, and the one most prone to "looking fine": once the box is sealed, a missing manual, a missing back panel or unfilled cushioning is invisible from the outside. This station ships one piece every 30 seconds across a seven-step sequence; the difficulty lies in white, thin, soft parts, a fast cycle and frequent model and carton changes.

01/6The Problem

Why missing parts only surface when the customer opens the box

Break it down and this step is hard for three reasons: once the box is sealed, its contents are invisible, and weighing or sampling at final inspection catches only part of the problem; asking a person to watch a 30-second cycle endlessly does not scale, and reviewing recordings after the fact is barely feasible; and whenever the carton spec or the sequence changes, the old approach of training a dedicated appearance model needs new samples and retraining, which cannot keep pace with changeovers.

So the design choice is to define the problem as "did the packing steps happen in the right order" rather than "train yet another dedicated appearance model" — a vision model sees parts and motions, a state machine judges the sequence, and results stream out at the edge. Changing the carton or the sequence is mostly reconfiguration, not a resident training effort.

The core conflict is not "can we see clearly" but "we can see, yet we keep no evidence": every step before sealing needs a judgement that can be reviewed.

02/6Tensor Flow

Five operators wired into one flow on the canvas

The vision-agent software runs at the edge, and configuring, executing, displaying and exporting a check all happen in one application. This case is built entirely with zero-code drag-and-drop: operators are pulled out of the library on the left and connected along the data flow — no algorithm code required.

Packing process check tensor flow canvas: video-stream camera, preprocessing, semantic segmentation, event detection and state machine connected in order — UI 01The packing-check tensor flow canvas. Video-stream camera, preprocessing, semantic segmentation, event detection and state machine are connected in order, forming the flow from frame capture to process judgement.

Video-stream camera: feeds the overhead view of the packing station to every downstream operator.
Preprocessing: denoise, enhance and normalise so the input meets the model's requirements.
Semantic segmentation: finds the pixels of each part and draws contours and class labels that hug the shape.
Event detection: combines recognition results through primitives into event signals such as "part in place".
State machine: advances the SOP steps on events, maintains the current process state and gives the basis for every judgement.

All three sub-editors open by double-clicking their node: semantic segmentation → the labelling tool (UI 02), event detection → the GPU event orchestrator (UI 03), state machine → the state graph editor (UI 04). How this configuration ticks steps off live is shown in the header video.

03/6Few-Shot Segmentation

See the part pixels first, then talk about ticking steps

Packing parts are mostly white, thin and soft, often behind a clear bag: foam paper creases, the manual sits askew, the back panel hugs the carton wall. Bounding boxes fit such targets poorly, so this project uses pixel-level semantic segmentation — 1–10 on-site samples per class are enough to settle it in the labelling tool, and the output masks drive presence and area checks.

Semantic segmentation labelling tool: defect type set to carton, a brush/polygon tool column on the left, two canvases in the middle, image status and image list on the right, and Save Labels / Transfer / Detect / Training Curve along the bottom — UI 02Double-clicking the Semantic Segmentation node opens the labelling tool. Left is the labelling canvas, right the detection result — the green contour follows the carton and bag edges, not a rectangle. The bottom row is one training loop: Save Labels → Transfer → Detect → Training Curve.

Class	Samples (typical)	Purpose	Judgement
Base board	4 imgs	Start / end in place	mask appears → E0 / E6
Foam paper	5 imgs	Cushion laid	mask area ≥ threshold → E1
Carton	5 imgs	Carton in place	mask area ≥ threshold → E2
Carton (flipped)	same class	Flip confirmation	segmentation + angle check → E3
Back panel	5 imgs	Back panel loaded	mask area ≥ threshold → E4
Manual	4 imgs	Documents placed	segmentation + count check → E5

"Carton flipped" is not a separate class: the same carton mask changes its axis ratio and orientation clearly before and after the flip, and an angle-check primitive tells them apart — one class less means one labelling round less. Training and inference both run locally on the edge GPU, so station footage never leaves the factory; on a model change, add samples and run one more transfer round.

04/6Event Judgement

Turning pixels into "what happened" in the primitive orchestrator

Segmentation only sees pixels; the business needs judgeable events like "foam paper in place" or "manual placed". The event list is defined in the state machine, and the judgement logic is assembled in the GPU event orchestrator: drag judgement primitives from the library, set their parameters, and compose them with AND / OR / NOT into the firing condition — changing a judgement means touching parameters, not retraining a model.

GPU event orchestrator: primitives library on the left, a canvas in the middle holding an Object-present primitive with target classes, detection region and tile window parameters, and the event list plus node properties on the right — UI 03The GPU event orchestrator. Pick an event → drag judgement primitives → set target classes and detection region. E1 "foam paper in place" uses a single Object-present primitive; E3 "carton flipped" and E5 "manual placed" each combine two primitives with AND.

05/6State Machine

Order enforced by the state graph; a missing part is visible on the spot

The workflow panel on the right is not a display list but an SOP state machine: S0–S6, seven states waiting for seven steps, advanced by seven events E0–E6. Sequence enforcement needs no extra NG rules — it comes from "every unexpected event self-loops": a skipped step does not advance, and a missed step never reaches the end.

State graph editor: circular state nodes chained by edges carrying event labels, with the event tensor and state-action tensor listed on the right — UI 04The state graph editor. `S0` waits for the start base board; `S1–S6` wait in turn for foam paper, carton, flip, back panel, manual and the end base board; `E0–E6` connect neighbouring states, and E6 returns to S0 for the next piece.

Rule table editor: a matrix of current state times event, with target states on the diagonal and self-loops everywhere else — UI 05The rule-table view of the same configuration: the diagonal holds target states, everything else is "↻ self". A skipped step does not advance and a missing step never reaches the end — no extra NG rules required.

A piece counts as OK only when all seven events advance in full; if the manual never goes in, S5 waits forever, and even if the end event fires it does not advance — that piece is never recorded as OK.

06Reflections

Translating shop-floor requirements into rules the system can execute

The deepest lesson of this project is that a shop-floor requirement needs one complete "engineering translation" before the system can act on it. "Pack everything according to the SOP" sounds like a single sentence; in practice, which parts to recognise, how each step is judged, what counts as a miss and how results are kept — each item needs an explicit definition.

The requirement went through three layers of decomposition:

Business requirement: pack per SOP before sealing; missing parts and skipped steps must surface on the spot.
Judgement rules: recognise five part classes, verify the seven-step order, keep evidence per step and emit OK / NG.
Software configuration: five operators wired into a tensor flow, with the S0–S6 state graph defining the advancement.

Only after these three layers aligned did the model's output enter the real process. Measured on this station, a piece takes 32–39 seconds; anomalous pieces deviate clearly — cycle time itself is a usable inspection signal. The boundary of this approach is equally clear: it fits manual packing stations with many parts, many steps and frequent changeovers; small parts hidden in deep cavities inside the box need process cooperation or a close-up camera.

Looking back, four questions ran through the whole implementation: what does the system watch, when is a step done, what counts as a miss, and how are results kept.

LEO

Leo

Vision-intelligence architect · lead for industrial process checks

PhD in computer vision, researching few-shot segmentation and video action understanding; 12 years on industrial-vision front lines, with 30+ lines and 100+ stations delivered.

Advocates moving process checks from "one dedicated model per station" to "few-shot segmentation + event orchestration + an SOP state machine", decoupling judgement logic from the perception model so a sequence change never means retraining.

Currently focused on edge-GPU inference optimisation, cross-domain generalisation and on-site data loops — making the same configuration method repeatable across factories and processes. Articles record real configurations, judgement logic and field results from live projects.