Data labelling without a labelling army
Weak supervision, model-assisted labelling and active learning — how small teams build training and eval sets that hold up.
Labelling is the tax on supervised learning, and the instinct is to outsource it in bulk. For most business problems that is both expensive and worse than the alternatives.
Cheaper ways to get labels
- It's the evaluation setLabel by hand, with a domain expert
- Large training set neededModel proposes, human corrects
- Past decisions already recordedMine them as weak supervision
- Budget is limitedLabel what the model finds hard, not a random sample
Label the evaluation set yourself
If you label nothing else by hand, label the eval set, and have someone who genuinely understands the domain do it. A few hundred carefully judged examples are worth more than tens of thousands of cheap ones, because everything you decide afterwards is measured against them.
Let a model do the first pass
Model-assisted labelling — the model proposes, a human corrects — routinely cuts effort by most of the way. The trap is anchoring: reviewers accept plausible-looking suggestions. Sample a portion blind and compare, so you can measure how much correction is really happening.
Mine the labels you already have
Businesses are full of implicit labels nobody thinks of as training data: resolved ticket categories, approved and rejected applications, corrections editors made, refunds issued. Weak supervision from these is noisy and abundant, which is often the better trade.
Label what the model finds hard
Random sampling spends most of your budget on cases the model already handles. Prioritising uncertain and disagreeing cases concentrates effort where it changes the outcome.
When labellers disagree
- High agreement between annotatorsThe task is well defined
- Low agreement on a subsetThe definition is ambiguous there
- Low agreement overallFix the task, not the model
- Nobody measured agreementYou do not know which of these you have
Measure your labellers
Have several people label the same subset and compute agreement. Low agreement between humans is not a labelling problem — it means the task definition is ambiguous, and no amount of modelling will fix that.