Data Cleaning, Quality and Enrichment
Fixing the data you have and, more importantly, stopping it from degrading again — because a one-off cleanup restores quality for about as long as it takes the next load to run.
Every organisation has been through a data cleanup. Most have been through several, because the cleanup fixed the records and left untouched the form that allows a free-text country field, the integration that creates a duplicate customer on every failed retry, and the absence of anyone accountable for the result.
Data quality work covers profiling to find what is actually wrong, cleaning and standardisation to fix it, deduplication and entity resolution to merge records that refer to the same real-world thing, enrichment to add missing attributes, and automated checks to detect decay before it reaches a report or a model.
Profiling first, because assumptions are usually wrong
We measure the actual state of the data before proposing to fix anything, and the findings are reliably surprising to the people who own the system:
| Dimension | What we measure | Typical finding |
|---|---|---|
| Completeness | Nulls and blanks by field and by segment | A required field empty for one channel since a form change |
| Validity | Values against expected format and range | Free-text entry where a picklist was assumed |
| Uniqueness | Duplicate and near-duplicate records | Materially more customers in the system than exist |
| Consistency | Same entity across systems | Three spellings of the same company name |
| Timeliness | Age of records against the decision that uses them | Data fresh enough for reporting, too stale for pricing |
| Accuracy | Sample verified against reality | The only dimension that cannot be measured from the data alone |
The last row matters. Accuracy requires an external check — a phone call, a physical count, a source document — and where accuracy is critical we build that verification step in rather than inferring it from internal consistency.
The work itself
Standardise with rules that are written down
Addresses, names, phone numbers, country codes, product identifiers and dates normalised to an agreed canonical form. The rules become part of the pipeline rather than living in a script on somebody's laptop, which is where the last cleanup's rules currently live.
Deduplicate with human-reviewed thresholds
Matching combines deterministic keys with fuzzy comparison, and the threshold is set with your domain experts against real examples. Two thresholds are set, not one: automatic merge above the high bar, human review between the bars, no action below. Aggressive automatic merging destroys data in ways that are effectively unrecoverable.
Resolve entities across systems
The same customer, supplier or product represented differently in CRM, billing and support is resolved to a single identity with a stable key and a crosswalk, so that downstream systems can join reliably instead of each inventing their own matching logic.
Enrich only where it changes a decision
Missing attributes filled from internal sources, third-party data or derivation. Every enrichment is justified by the decision it improves, and third-party sources are checked for licence terms and for how their coverage varies by region, because coverage gaps produce systematically biased results.
Treat missing data as information
A null often means something — a step was skipped, a channel does not collect it, a customer declined. Blanket imputation destroys that signal and injects confident fabrication into models. We distinguish types of missingness and handle each deliberately, recording the choice.
Instrument checks so decay is visible
Automated tests on completeness, validity, uniqueness and distribution run with every load and alert when they fail. This is what makes the improvement durable, and it is the part most often cut from scope by people who will be running another cleanup in eighteen months.
The upstream fix beats the downstream fix
Where a quality problem originates in a form, an integration or a process, we say so and quantify it. Cleaning the same defect every night forever is a permanent tax; fixing the form is usually a fortnight of somebody else's work. Both are in the report, with the cost of each.
Why this determines whether AI works at all
Model performance is bounded by data quality in ways that are hard to see and expensive to discover late:
- Duplicates split history. One customer as three records means three thin behavioural histories rather than one useful one, and every propensity model built on it underperforms.
- Inconsistent categories fragment signal. The same product under four names cannot be learned from as one product.
- Silent nulls become confident zeros. Which models learn as a real value and act on.
- Leakage from careless joins. A field populated after the outcome it is meant to predict produces excellent evaluation scores and useless production behaviour.
Where a model build is planned, this work is the cheapest available way to improve its eventual accuracy, and it happens first for good reason.
How the engagement runs
Profiling comes before proposals, because the real defects are rarely the assumed ones.
Profiling
Completeness, validity, uniqueness, consistency and timeliness measured across the relevant datasets.
Prioritisation
Defects ranked by the decisions they affect; upstream causes identified and costed separately.
Cleaning and resolution
Standardisation rules, deduplication with reviewed thresholds, entity resolution, justified enrichment.
Automated checks
Quality tests instrumented into pipelines with alerting and an ownership route for failures.
Handover
Rules, thresholds, crosswalk and monitoring handed over with the reasoning documented.
What you receive
Cleaner data, and the instrumentation that keeps it that way after we leave.
Profiling report
Measured quality by dimension and dataset, with the defects that matter ranked.
Cleaned datasets
Standardised, deduplicated and resolved, with every change auditable back to the original.
Matching rules and thresholds
Deterministic and fuzzy logic with the review band your experts agreed.
Entity crosswalk
Stable identities mapped across systems so downstream joins are reliable.
Automated quality checks
Tests running with every load, with alerting and a named owner for failures.
Upstream findings
Root causes in forms, integrations and processes, each with the cost of fixing versus cleaning.
Is this the right engagement?
Worth being direct. Data Cleaning, Quality and Enrichment is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Reports from different systems disagree and nobody can settle which is right.
- Duplicate customers, suppliers or products are known to exist at scale.
- A model or migration is planned and the data has not been assessed.
- A previous cleanup has already decayed.
- Quality problems are discovered by users rather than by monitoring.
Choose something else if
- The real requirement is labelled examples for training. That is annotation.
- The data is fine and the problem is that nobody can access it. See data strategy.
- You want a one-off cleanup with no monitoring, which we will do but do not recommend.
- No domain expert can review matching decisions, which makes deduplication a gamble.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
How do you decide what to fix first?
By the decision each defect affects. A field that is thirty per cent null but feeds nothing important ranks below a field that is two per cent wrong and drives pricing. The profiling report ranks defects by consequence rather than by severity, because fixing everything is how these programmes run out of budget before they run out of scope.
Will deduplication merge records it should not?
Not automatically. Two thresholds are set with your experts: above the high bar records merge, between the bars they go to human review, below it nothing happens. Merges are recorded and reversible. Aggressive automatic merging is fast and produces damage that is very difficult to undo later.
Should we fix the data or fix the source system?
Both, in that order, and we quantify each. Cleaning restores usable data now; fixing the form, the integration or the process is what stops the defect recurring. The upstream findings are delivered with cost estimates so somebody can make that call with numbers rather than with opinion.
How do you handle missing values?
By first establishing what the absence means, because a null can be a skipped step, an unavailable channel or a genuine refusal, and each demands different treatment. Blanket imputation is avoided; where values are estimated, the estimate is flagged so that downstream models and reports can distinguish it from observed data.
How do we stop quality decaying again?
Automated checks running with every load — completeness, validity, uniqueness and distribution against expectation — with alerts routed to a named owner. Without that, the next cleanup is already scheduled; it just has not been budgeted yet.
Often paired with this
Most clients combine two or three engagements from the Data Engineering and AI Readiness pillar. These are the ones that most often run immediately before or after.
Data Pipeline Development
ETL and ELT pipelines that survive schema drift, late data and reruns, with tests and monitoring from day one.
Read more →Data Labelling and Annotation
Labelled datasets with a written guideline, measured annotator agreement and audited quality at every batch.
Read more →Legacy Data Migration and Modernisation
Migration with profiled sources, signed-off mapping, rehearsed cutover, parallel running and a tested rollback.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.