Real-Time Streaming Data Engineering
Continuous processing for the decisions that genuinely cannot wait, built with event-time correctness and honest costing — and a recommendation against it where hourly batch would have done the same job.
Streaming is asked for far more often than it is needed. The question that settles it is simple and rarely asked: what decision changes if this data is an hour old rather than a minute old? Where the honest answer is nothing measurable, incremental batch delivers the same outcome for a fraction of the cost and without a second on-call rota.
Streaming data engineering processes events continuously as they occur rather than in scheduled batches, typically over a distributed log, with processing semantics defined in terms of when events happened rather than when they arrived. It provides sub-minute latency at the cost of significantly higher operational complexity than batch processing.
When streaming genuinely earns its cost
| Use case | Why latency matters | Related work |
|---|---|---|
| Fraud and payment risk | The decision must precede the transaction completing | Fraud detection AI |
| Equipment and process monitoring | Intervention window is measured in seconds | Predictive analytics |
| Live personalisation | The session ends before a batch job would run | Recommendation engines |
| Dynamic pricing and inventory | Stale prices are sold against | Dynamic pricing |
| Safety and security alerting | Response time is the whole value | Video analytics |
| Operational dashboards | Usually does not | Incremental batch is nearly always enough |
The last row is included deliberately. A dashboard refreshed every fifteen minutes satisfies almost every operational team that asked for real-time, and it costs a small fraction of what streaming does to build and to run.
What makes streaming hard
Event time versus arrival time
Events arrive out of order and late, sometimes by hours when a mobile device has been offline. Processing by arrival time produces windows that are quietly wrong; processing by event time with watermarks and a defined late-arrival policy produces windows that are right. The choice has to be made explicitly, and it is the most common source of silently incorrect streaming results.
Delivery semantics
At-least-once delivery means duplicates; exactly-once requires either idempotent writes or transactional coordination, and costs throughput. We decide per stream against the consequence: a duplicated fraud alert is an annoyance, a duplicated payment is not.
State, and what happens when it grows
Aggregations, joins and sessionisation hold state that grows with cardinality and window length. State size drives cost and recovery time, and a stream that works in testing and falls over in month four is nearly always a state growth problem that was never bounded.
Backpressure and replay
Sources spike, consumers slow down. The system needs defined behaviour under load and the ability to replay from a retained log after a fix, which is what turns a bad deployment into a reprocess rather than a permanent hole in the data.
Operational load
Streaming systems fail at three in the morning and cannot wait until Monday. Runbooks, alerting and an on-call model are part of the design rather than an afterthought, and if your organisation cannot staff that, the honest recommendation is batch.
We will argue you out of it where the case is weak
A streaming build that nobody can operate is worse than the batch pipeline it replaced. Where the latency requirement does not survive the question 'what decision changes?', we say so in week one and propose incremental batch instead, with the cost difference written down.
How we build it
- Establish the real latency requirement per consumer, in terms of the decision each one drives rather than as a general preference for freshness.
- Design the event model. Schemas, keys, ordering guarantees and a registry, with compatibility rules so producers can evolve without breaking consumers.
- Choose semantics per stream. Delivery guarantee, windowing, watermarks and late-arrival policy decided against consequence and cost.
- Build with replay in mind. Retention sized for realistic recovery, reprocessing rehearsed before go-live rather than discovered during an incident.
- Serve both paths. Streaming results into the online store for immediate use and into the lakehouse for history, so analytics and models see the same events.
- Load-test to the spike, not the average. Peak concurrency and burst behaviour measured before launch, with autoscaling and backpressure verified under real load.
How the engagement runs
The latency requirement is challenged before anything is built, because half of the scope usually does not survive it.
Requirement and honest assessment
Latency need established per consumer; batch alternative costed and compared.
Event model and semantics
Schemas, registry, keys, windowing, watermarks and delivery guarantees decided per stream.
Build
Producers, processing topology and sinks built, with state bounded and replay implemented.
Load and failure testing
Peak load, spike behaviour, failure injection and replay rehearsed end to end.
Handover
Runbook, alerting, on-call guidance and working sessions with the team who will carry the pager.
What you receive
A streaming platform your team can operate, with replay rehearsed before you need it.
Deployed streaming pipeline
Producers, processing and sinks with the chosen semantics implemented and tested.
Event schemas and registry
Versioned schemas with compatibility rules for independent producer evolution.
Replay capability
Retention sized and reprocessing rehearsed, with the procedure documented.
Load test results
Behaviour at peak and under spike, with autoscaling and backpressure verified.
Monitoring and alerting
Lag, throughput, error rate and state size, routed to your on-call.
Runbook and on-call guide
Failure modes, recovery steps, escalation and the cost levers.
Is this the right engagement?
Worth being direct. Real-Time Streaming Data Engineering is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A decision genuinely degrades in value within minutes.
- Fraud, safety, live personalisation or pricing depend on current state.
- Event volumes are high enough that batch windows no longer fit.
- Your organisation can staff on-call for a continuously running system.
- Both immediate use and historical analysis need the same event stream.
Choose something else if
- Nobody can name a decision that changes with sub-hourly data.
- Incremental batch meets the requirement. See pipeline development.
- There is no capacity to operate a continuously running system.
- The underlying data quality would make fast wrong answers rather than slow right ones.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
Do we actually need streaming?
Frequently not, and we test it in week one by asking which decision changes if the data is an hour old. Fraud, safety, live personalisation and dynamic pricing usually survive that question. Operational dashboards and most reporting do not, and incremental batch serves them at a small fraction of the build and running cost.
What is the difference between event time and processing time?
Event time is when something actually happened; processing time is when your system received it. Events arrive late and out of order, so windows computed on processing time are quietly wrong. Event-time processing with watermarks and an explicit late-arrival policy is what makes the results correct, and it is the distinction most streaming projects get wrong first.
Can you guarantee exactly-once processing?
Yes, where it is warranted, through idempotent writes or transactional sinks — at some cost in throughput and complexity. We choose per stream by consequence: a duplicated alert is tolerable, a duplicated financial transaction is not, and paying exactly-once costs everywhere is a common and expensive over-correction.
How much does streaming cost to run?
Materially more than the equivalent batch pipeline, in infrastructure and in people. Compute runs continuously rather than for minutes a day, and someone has to be on call. We produce the comparison before you commit, because the total cost of ownership is what should decide this rather than the architecture diagram.
Can streaming and batch share the same data?
Yes, and they should. Streaming results are written to the online store for immediate use and to the lakehouse for history, so real-time consumers and analytical or model workloads read the same events rather than two definitions of the same thing.
Often paired with this
Most clients combine two or three engagements from the Data Engineering and AI Readiness pillar. These are the ones that most often run immediately before or after.
Data Pipeline Development
ETL and ELT pipelines that survive schema drift, late data and reruns, with tests and monitoring from day one.
Read more →Feature Store Implementation
One definition per feature, point-in-time correct training data and low-latency serving from the same logic.
Read more →Data Lakehouse and Warehouse Build
A governed lakehouse or warehouse with a modelled core layer, access control and predictable cost.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.