When a sensor fails, it is obvious. When an annotator misreads a task boundary, the file still looks fine. Quality control for human data is the work of making those quiet errors visible before they reach a training run.
The review ladder
Every batch moves through four stages. Automated checks catch the mechanical failures: broken sync, dropped frames, sensor gaps, schema violations, sessions that are shorter than their manifest. A first reviewer pass checks task boundaries, labels, and free-text notes.
A second reviewer independently checks a stratified sample of the batch. Disagreements go to adjudication, where a lead decides the correct label and records why. The reason matters as much as the decision: it becomes training material and, eventually, a new automated check.
Sampling, not hoping
We do not review a fixed percentage and call it coverage. Samples are stratified by site, task type, annotator, and label class, so a rare category cannot hide behind a strong average. Control items with known answers are mixed into the queue to keep reviewers calibrated.
Agreement between annotators is tracked, but it is not treated as proof of quality. Two people can agree on the wrong boundary. Where possible, we compare against the recorded task and the field lead's notes.
An error budget
Each label class carries an acceptable error rate set by the partner's use case. A benchmark of gross activity can tolerate more ambiguity than a manipulation policy that depends on exact contact points. When a batch exceeds its budget, the line stops, the cause is found, and affected files are re-reviewed.
What we measure
- Disagreement rate, split by label class and site.
- Ambiguity rate: items reviewers flag as underspecified.
- Time to correction for every batch that fails a check.
- Reviewer drift against control items over time.
The pipeline is built so that volume does not dilute the standard. More data means more sampling, not less attention per batch.