Data Labelling and Annotation
Labelled datasets that a model can actually learn from, built on a written guideline, measured agreement between annotators and audit sampling at every batch rather than on trust.
Labelling is treated as the cheap part of a machine learning project and it is where most of them quietly go wrong. Not because annotators are careless, but because the instructions were ambiguous and nobody measured whether two people applying them produced the same answer.
Data annotation is the process of adding structured labels to raw data — categories on text, bounding boxes or masks on images, spans on documents, timestamps on audio — so that supervised models have examples to learn from. Its quality is bounded by the clarity of the guideline and measurable through the agreement between independent annotators.
Agreement first, volume second
Before any large batch is labelled, several annotators independently label the same sample and we measure how often they agree. Low agreement is diagnostic, and it is nearly always the guideline rather than the people:
- Ambiguous category boundaries. Two labels that could both apply, with no stated precedence.
- Unstated edge-case rules. What to do with a partially visible object, an ambiguous clause, a sarcastic sentence, a document in the wrong language.
- Granularity confusion. Whether to label the whole invoice or each line item, the sentence or the phrase.
- Missing 'none of these' option. Which forces annotators to guess, and turns the catch-all category into noise.
The guideline is revised until agreement is acceptable, and only then does volume labelling begin. Skipping this step does not save time; it moves the cost to the point where a model has been trained and nobody can explain why it is inconsistent.
How we run an annotation programme
Write the guideline with your experts
Definitions, precedence rules, worked examples and a growing edge-case register. It is a living document, and it is frequently the most reusable artefact the whole project produces, because it makes every future dataset faster to build.
Sample deliberately, not conveniently
What gets labelled determines what the model learns. We sample for coverage of rare classes and hard cases rather than taking the most recent ten thousand records, because a dataset drawn from convenience teaches a model that the easy case is the only case.
Audit every batch
A percentage of each batch is independently re-labelled and compared. Quality is reported per annotator and per class, so a drift in one person's interpretation or one difficult category is caught within days rather than at model evaluation.
Adjudicate disagreements with a domain expert
Cases where annotators disagree go to your expert, and the resolution feeds back into the guideline. These cases are the most informative part of the dataset and they are exactly what a majority vote would have thrown away.
Use the model to accelerate the humans
Once an initial model exists, it pre-labels and humans correct, which is substantially faster than labelling from scratch. We measure for automation bias — the tendency to accept a confident wrong suggestion — and periodically hold out blind batches to check that correction rates have not quietly collapsed.
Handle sensitive data properly
Where data contains personal or confidential information, we agree the handling model before anything is shared: redaction, pseudonymisation, restricted annotator pools, or working entirely within your environment. This is settled in week one, not negotiated once a batch is already in flight.
Active learning pays for itself on large datasets
Rather than labelling randomly, the model selects the examples it is least certain about, so annotation effort concentrates where it changes the model. On large corpora this routinely reaches target accuracy with a fraction of the labels, and it is the main reason to instrument the loop early.
What we annotate
| Data type | Typical annotation | Feeds |
|---|---|---|
| Text and documents | Categories, entity spans, relations, clause types | Classification and extraction |
| Customer feedback | Aspects, sentiment, intent | Sentiment and intent analysis |
| Images | Bounding boxes, segmentation masks, classes, attributes | Object detection |
| Scanned documents | Field locations, table structure, document type | Document processing |
| Video | Tracked objects, event boundaries, activity labels | Video analytics |
| Model outputs | Preference rankings, quality ratings, failure categories | LLM evaluation |
The last row is increasingly the largest. Evaluating generative systems needs judged examples exactly as supervised learning does, and the same discipline about guidelines and agreement applies.
How the engagement runs
The guideline is proven on a pilot sample before volume labelling is authorised.
Guideline and sampling
Definitions and precedence rules written with your experts; sampling strategy agreed for coverage of rare cases.
Pilot and agreement
Independent labelling of a shared sample, agreement measured, guideline revised until it holds.
Volume annotation
Batched labelling with audit sampling, per-annotator quality reporting and expert adjudication.
Model-assisted acceleration
Pre-labelling with correction where a usable model exists; blind batches to verify quality holds.
Delivery
Dataset, guideline, agreement measurements and edge-case register handed over.
What you receive
A dataset with a known quality figure, and the guideline that lets you extend it without us.
Labelled dataset
In your chosen format, with train, validation and test splits defined deliberately.
Annotation guideline
Definitions, precedence rules, worked examples and the edge-case register.
Agreement report
Inter-annotator agreement by class, with the confusions named and explained.
Quality audit trail
Per-batch audit results, per-annotator quality and adjudication decisions.
Sampling record
What was sampled and why, including deliberate over-sampling of rare classes.
Handling record
How sensitive data was treated, by whom and under what controls.
Is this the right engagement?
Worth being direct. Data Labelling and Annotation is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A supervised model is planned and no labelled data exists.
- An existing dataset produces inconsistent models and the labels are suspect.
- Rare but important classes are under-represented in what you have.
- Generative outputs need judged examples for evaluation.
- Your experts can commit time to adjudicate hard cases.
Choose something else if
- Nobody can define the categories, even after a workshop.
- The data cannot leave your environment and no in-place arrangement is possible.
- A pretrained model already performs acceptably; measure before labelling.
- The volume is small enough that your own team can label it in a fortnight.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
How much labelled data do we actually need?
For fine-tuning a modern pretrained model, typically several hundred to a few thousand examples per class, with more for rare classes and subtle distinctions. Consistency beats volume: a thousand examples labelled to an agreed guideline will train a better model than ten thousand labelled to three different interpretations of it.
Can we use synthetic or model-generated labels instead?
Partly, and it is often worth doing. Model pre-labelling with human correction is much faster than labelling from scratch, and synthetic data can fill gaps for rare classes. What neither replaces is a human-labelled test set, because a model evaluated on labels generated by a model measures agreement rather than correctness.
How do you measure annotation quality?
Inter-annotator agreement on shared samples, reported per class rather than as an overall figure, plus independent re-labelling of a percentage of every batch. Both are reported to you throughout, so quality is a number you can see rather than an assurance you have to accept.
What happens with confidential or personal data?
The handling model is agreed before any data moves: redaction or pseudonymisation where it does not destroy the signal, restricted annotator pools with signed agreements, or annotation entirely within your own environment. Where the data genuinely cannot leave, we set up and train your team to run the programme instead.
Who owns the dataset afterwards?
You do, in full, on final payment, along with the guideline, the agreement measurements and the edge-case register. The dataset usually outlives several generations of model, which is why we treat the guideline as a deliverable in its own right rather than as internal working material.
Often paired with this
Most clients combine two or three engagements from the Data Engineering and AI Readiness pillar. These are the ones that most often run immediately before or after.
Data Cleaning, Quality and Enrichment
Profiling, deduplication, standardisation and enrichment, with automated checks that stop quality decaying again.
Read more →Vector Database Design and Implementation
Vector stores designed around chunking, filtering and reindexing, tuned against measured retrieval quality.
Read more →Data Pipeline Development
ETL and ELT pipelines that survive schema drift, late data and reruns, with tests and monitoring from day one.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.