Streaming gets chosen for the wrong reason more often than almost any other data architecture decision.
The reason is rarely a documented latency requirement.
It is that streaming sounds modern, while proposing a batch pipeline in a design review can sound like settling.
So the team builds streaming:
Kinesis
A consumer application
Checkpointing
A dead-letter path
Monitoring and alerting
An on-call rotation
The pipeline delivers data four seconds after the event.
The dashboard is reviewed once each morning.
The cost isn't just the infrastructure bill. It is the operational surface: an always-running system with failure modes that can page a human, built to satisfy a freshness requirement nobody properly defined.
Real-time is a requirement, not a default.
Why Streaming Costs More Than Its Infrastructure Bill
The infrastructure cost is only the visible part.
The larger cost is operational.
A streaming pipeline is always running. That means it can always fail. It requires monitoring, alerting, replay procedures, capacity planning, and engineers who understand the system well enough to respond under pressure.
A micro-batch pipeline that runs every fifteen minutes has a different failure profile. It processes data in bounded windows. It can retry within the next window. A failure may delay freshness, but it does not necessarily require immediate human intervention.
Streaming also introduces more concepts for the team to own:
Backpressure
Out-of-order events
Checkpoint management
Replay behavior
Watermarks
Duplicate delivery
Consumer lag
Partition or shard capacity
These are not theoretical concerns. They become operational responsibilities.
There is also a concentration risk. Many organizations have only one or two engineers who deeply understand the streaming stack. Every change then depends on the same people.
That is not just a staffing inconvenience.
It is a reliability constraint.
Why Freshness Requirements Become Inflated
Most freshness requirements are inherited rather than derived.
Someone asks a stakeholder how fresh the data should be. With no cost or operational context, the answer is often “real time.”
That answer becomes a requirement.
Eighteen months later, the team is maintaining a streaming pipeline because of one unpriced conversation.
Ask a better question:
What decision does this data support, and how quickly after the event must that decision happen?
Consider the difference:
A fraud system may need to react within seconds.
An operational dashboard may need updates every fifteen minutes.
A daily revenue report may only need correct data by 8 AM.
A weekly cohort analysis may only need data by Monday.
These are different business requirements.
They should produce different architectures.
The other source of inflation is anticipated future need.
Teams build streaming because they might need real-time later. Then they pay the operational cost for years while the requirement never arrives.
When the requirement finally does arrive, it usually applies to one pipeline, not the entire platform.
The Five Questions to Ask Per Pipeline
1. What decision does this data drive?
Do not ask how fresh the data should be.
Ask what someone or something does with it.
If a human reviews the data during business hours, the actual requirement may be measured in minutes or hours.
If an automated system must react immediately, streaming becomes more defensible.
Architecture should follow the decision, not the preference of the person writing the requirements document.
2. Is there an automated action triggered by the data?
Automated response is the strongest reason to choose streaming.
Examples include:
Blocking a suspicious transaction
Triggering a safety response
Updating a user-facing experience immediately
Routing an operational event
Detecting a system failure within seconds
If nothing happens automatically and the data is only reviewed later, micro-batch may provide the required outcome with less operational burden.
3. What is the concrete cost of fifteen minutes of staleness?
State the consequence.
Does fifteen-minute-old data:
Lose revenue?
Increase fraud exposure?
Delay an operational response?
Violate a contractual requirement?
Create a safety issue?
Prevent a customer from completing an action?
If nobody can name the consequence, there may not be a genuine latency requirement.
Do not buy a permanently running system to avoid an inconvenience that has never been priced.
4. Does the volume make batch impractical?
Latency is not the only reason streaming may be necessary.
Sustained event volume can make batch processing impractical if each batch must process more data than arrived during the previous window.
You may be approaching a simple constraint:
Processing time is greater than the batch interval.
If a fifteen-minute batch takes twenty minutes to process, the pipeline cannot keep up. The backlog compounds.
That is a legitimate reason to consider streaming, independent of whether anyone needs sub-minute freshness.
Measure:
Events per second
Average and peak volume
Batch processing duration
Batch interval
Backlog growth
Recovery time after failure
Do not confuse a temporary implementation problem with a permanent streaming requirement. A faster batch job, better partitioning, or a larger processing window may solve the issue.
5. Does the team have on-call capacity?
Streaming belongs in the operating model, not only in the architecture diagram.
Ask:
Who owns the pipeline at 2 AM?
Who responds to consumer lag?
Who handles duplicate or out-of-order events?
Who replays a failed time range?
Who responds when capacity is exhausted?
Who maintains the dead-letter path?
If the team is already stretched, adding another always-on system creates reliability debt.
A streaming design that nobody can operate is not production-ready, regardless of how elegant the topology looks.
The Break-Even Test
The basic latency test is simple:
If the required decision latency is comfortably longer than the batch interval, micro-batch is usually sufficient.
For example:
Required decision latency: 45 minutes
Proposed micro-batch interval: 15 minutes
Processing time: 4 minutes
That pipeline has margin. Streaming is difficult to justify unless another factor, such as sustained volume or automated action, changes the economics.
Now consider:
Required decision latency: 30 seconds
Proposed micro-batch interval: 15 minutes
Micro-batch cannot satisfy the requirement. Streaming earns its additional complexity.
Also calculate whether batch can keep up:
Batch processing time < Batch interval
If processing time consistently approaches or exceeds the interval, the batch design is under pressure.
A pipeline may need streaming when:
The required latency is below the practical batch interval
Automated action must happen quickly
Sustained volume causes batch backlog
The business cost of delay exceeds the operational cost of streaming
Otherwise, micro-batch should be the default you test first.
What Choosing Honestly Returns
Teams that evaluate pipelines individually usually end up with a small number of streaming systems where the requirement is real and micro-batch everywhere else.
The first return is lower on-call load.
Fewer always-on systems mean fewer pages and fewer specialized systems competing for the same engineers.
The second return is faster delivery. Micro-batch pipelines are usually quicker to launch because the operational hardening burden is smaller.
The third return is better attention allocation. The streaming pipelines that remain receive the expertise they need instead of competing with six unnecessary streaming systems for the same two engineers.
There is a legitimate case for streaming.
When automated action, genuine sub-minute decisions, or sustained volume make batch the constraint, build streaming properly.
Do not stretch batch beyond what it can handle.
But do not build streaming to avoid looking old-fashioned in a design review.
The Decision Tree
Use this sequence for every pipeline:
What decision does the data support?
How quickly must that decision happen?
Is there an automated action?
Can batch process the volume within the required window?
Can the team operate another always-on system?
Then classify the result:
Micro-batch: Human-reviewed data, flexible decision timing, manageable volume, limited on-call capacity.
Streaming: Automated response, genuine sub-minute requirement, or batch throughput constraint.
Re-evaluate: Conflicting answers, undocumented requirements, or an architecture based primarily on anticipated future demand.
Write down the decision and the reason.
Revisit it when volume, consumers, latency requirements, or operating capacity changes.
The architecture should evolve when constraints change.
The Payoff Is Fewer Systems That Matter
Streaming is not bad.
Unnecessary streaming is expensive.
A micro-batch pipeline that meets the business requirement is often the better system because it is easier to build, operate, explain, and recover.
A streaming pipeline that satisfies a real automated or latency-sensitive requirement is worth the complexity.
The mistake is treating one pattern as the default for every workload.
Use the smallest architecture that satisfies the actual decision latency and volume constraint.
That is not conservatism.
That is engineering.
Before approving the next streaming pipeline, make the requestor answer five questions:
What decision does this data drive?
How quickly must the decision happen?
What is the cost of fifteen minutes of staleness?
Can batch keep up with the volume?
Who owns the system when it fails?
If the answers are vague, build the simpler pipeline first.
The goal is not to make data arrive as quickly as possible.
The goal is to make the right decision at the right cost without creating an on-call rotation nobody needs.
That’s it for today!
Did you enjoy this newsletter issue?
Share with your friends, colleagues, and your favorite social media platform.
Until next week — Amrut
Get in touch
You can find me on LinkedIn or X.
If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.