Data Pipeline Development
Pipelines that keep working when the source schema changes at midnight, the file arrives four hours late, or last month has to be reprocessed after a correction upstream.
Any competent engineer can move data from A to B once. The engineering is in what happens the two hundredth time, when the source added a column, the job failed halfway, the same file arrived twice and someone needs March recalculated by Thursday.
A data pipeline is an automated, repeatable process that extracts data from source systems, applies transformation, and loads it into a target store. ETL transforms before loading; ELT loads raw data and transforms inside the target platform, which is the more common pattern on modern warehouses because it preserves the original and makes reprocessing cheap.
The failure modes we build against
| What happens | Naive pipeline | How we handle it |
|---|---|---|
| Source adds or renames a column | Silent breakage or silent data loss | Schema contract, explicit evolution rules, alert on drift |
| Job fails at 60 percent | Partial data in the target | Atomic writes, transactional load, resumable stages |
| Same file delivered twice | Duplicated rows, wrong totals | Idempotent loads keyed on natural or hash keys |
| Records arrive days late | Historic aggregates silently wrong | Event-time windows and defined late-arrival policy |
| Upstream correction to old data | Manual, risky backfill | Reprocessing designed in from the start |
| Volume grows tenfold | Runtime and cost grow past the window | Incremental processing and partitioning from day one |
None of these are exotic. All of them arrive eventually, and the difference between a pipeline that lasts three years and one that is rewritten in eight months is whether they were designed for before or discovered after.
How we build pipelines
Land raw, transform after
Raw source data is landed unchanged and retained. Every transformation is derived from it and can be rebuilt. This costs a little storage and repays it the first time a business rule turns out to have been wrong for six months, because the fix is a rerun rather than a recovery project.
Make every stage idempotent
Running the same stage twice produces the same result, which turns failure recovery from a judgement call into a retry. It is the single design decision that most reduces the operational burden of a data platform, and it has to be there from the beginning.
Enforce contracts at the boundary
Expected schema, types and nullability are declared and checked on arrival. When a source changes, the pipeline says so loudly and immediately rather than quietly writing nulls into a table that six dashboards depend on.
Test the transformations
Business logic is covered by tests with known inputs and expected outputs, run in CI before deployment. Data pipelines are software and go wrong the way software goes wrong; the absence of tests is why so many of them are frightening to change.
Monitor freshness, volume and distribution
A pipeline that runs successfully but delivers a tenth of the usual rows has failed. We monitor row counts, freshness and value distributions against expectation, because the dangerous failures are the ones that do not raise an error.
Instrument cost as a first-class metric
Compute cost per run is tracked from the start. Warehouse bills grow quietly through inefficient joins and full-table rebuilds that nobody attributes to a specific job, and the fix is much cheaper before the pattern is established across forty pipelines.
Batch, incremental or streaming
The three patterns have very different cost and operational profiles, and the right answer is usually the least exciting one that meets the requirement:
- Full batch. Simple, robust, easy to reason about. Correct for small volumes and slowly changing reference data.
- Incremental batch. Only new or changed records are processed. The default for most production pipelines, and where most of the engineering care goes.
- Streaming. Continuous processing with sub-minute latency. Genuinely necessary for fraud, monitoring and live personalisation; expensive and operationally heavier than most requests for it assume. See streaming data engineering where it is justified.
Real-time is a requirement, not a preference
We ask what decision changes if data is one hour old instead of one minute old. Where the answer is 'nothing measurable', incremental batch delivers the same business outcome at a fraction of the cost and with far fewer people on call.
How the engagement runs
Sources are profiled before pipelines are designed, because the schema documentation is rarely accurate.
Source profiling
Actual schemas, volumes, update patterns and quality measured directly rather than taken from documentation.
Design and contracts
Pattern chosen per source, schema contracts written, target model agreed.
Build
Extraction, transformation and loading built incrementally, with tests written alongside.
Hardening
Failure injection, reprocessing rehearsed, monitoring and alerting configured.
Handover
Runbook, on-call guidance and a working session with the team who will own it.
What you receive
Pipelines your team can change without fear, with the tests and runbook that make that true.
Production pipelines
Deployed, scheduled, idempotent and incremental where volume warrants it.
Source profile report
Real schemas, volumes, update patterns and quality findings per source.
Schema contracts
Declared expectations per source with drift alerting.
Test suite
Transformation logic covered and running in CI.
Monitoring and alerts
Freshness, volume and distribution checks with routing to your on-call.
Runbook
Failure modes, recovery steps, reprocessing procedure and cost levers.
Is this the right engagement?
Worth being direct. Data Pipeline Development is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Data is moved by scripts, schedules or people that nobody wants to touch.
- Pipeline failures are found by users noticing a stale dashboard.
- Reprocessing history is a manual project rather than a command.
- Volumes are growing and runtimes are creeping toward the window.
- A new platform or warehouse needs populating reliably.
Choose something else if
- The architecture and target are not agreed yet. Start with data strategy and architecture.
- A single scheduled export meets the need and always will.
- The requirement is genuinely sub-second. See streaming data engineering.
- The source data is so poor that no pipeline can rescue it. Begin with data quality.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What is the difference between ETL and ELT?
ETL transforms data before loading it into the target; ELT loads raw data first and transforms it inside the target platform. ELT is the more common modern pattern because warehouse compute is cheap and elastic, and keeping the raw data means a business-logic error can be fixed with a rerun rather than a re-extraction from a source that may no longer hold the history.
Which orchestration and transformation tools do you use?
We work with the established orchestrators and transformation frameworks and choose based on what your team can operate after we leave, not on what is fashionable. A pipeline stack nobody in your organisation can debug at two in the morning is a liability regardless of its elegance.
How do you handle schema changes from source systems?
By declaring the expected schema as a contract and checking it on arrival. Additive changes can be configured to pass through; breaking changes stop the pipeline and alert, rather than being absorbed silently. Silent absorption is how nulls end up in a revenue table for a quarter before anyone notices.
Can you work with our existing pipelines rather than replacing them?
Usually yes, and often that is the better value. We assess what is worth keeping, add contracts, tests and monitoring to what stays, and rebuild only what genuinely cannot be made reliable. Wholesale replacement is a decision that should be argued for on evidence.
Who runs these after handover?
Your team, which is why the runbook and the working handover session matter as much as the code. Where you would rather not, ongoing operation can be arranged separately, but the build is deliberately structured so that it is a choice rather than a dependency.
Often paired with this
Most clients combine two or three engagements from the Data Engineering and AI Readiness pillar. These are the ones that most often run immediately before or after.
Data Cleaning, Quality and Enrichment
Profiling, deduplication, standardisation and enrichment, with automated checks that stop quality decaying again.
Read more →Real-Time Streaming Data Engineering
Event-time streaming with exactly-once semantics and late-arrival handling, built only where latency changes a decision.
Read more →Data Lakehouse and Warehouse Build
A governed lakehouse or warehouse with a modelled core layer, access control and predictable cost.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.