Here is how to build a Kinesis-to-S3-to-Athena pipeline so owning it remains sustainable.

The goal is not maximum throughput or minimum theoretical latency.

AWS services can handle those requirements.

The goal is fewer unnecessary pages, safe recovery when records fail, and landed data that Athena can query efficiently.

A streaming pipeline is successful when it delivers fresh data without turning two engineers into permanent babysitters.

Why Operational Load Decides Whether Streaming Was Worth It

The business case for streaming is made on latency.

The real cost is paid in on-call hours.

A pipeline that pages twice per month consumes more than its infrastructure budget. It interrupts engineers, reduces roadmap capacity, and creates a reliability tax that rarely appears in the original architecture decision.

Track these metrics:

• Pages per month

• Ratio of pages to real incidents

• Mean time to recovery

• Dead-letter record rate

• Average S3 object size

• Producer throttling events

The goal is not zero alerts.

The goal is for every page to represent a condition requiring immediate human action.

How Streaming Pipelines Create Unnecessary Pages

Self-resolving consumer lag

A traffic burst increases Kinesis consumer lag. IteratorAgeMilliseconds rises, then falls once the consumer catches up.

If every temporary spike pages someone, the alert is measuring movement rather than danger.

Use warning and page thresholds. Notify the team when lag increases but the consumer is recovering. Page when lag is sustained, recovery has stalled, or the stream is approaching a retention boundary.

Small files in Amazon S3

Firehose delivers objects when the configured size or time condition is met first.

With dynamic partitioning, buffering occurs separately for active partitions. Low-volume partitions may flush based on time before reaching the desired object size.

A stream can appear healthy while creating thousands of small S3 objects.

Nothing fails immediately.

Athena simply becomes slower and more expensive over time.

Untested dead-letter handling

A dead-letter prefix is not a recovery system.

It is only a location where failed records wait.

If replay has not been implemented and tested, the first malformed record becomes both a production incident and an engineering project.

Build replay before production.

The Three Decisions That Set Operational Load

1. Kinesis capacity: on-demand or provisioned?

On-demand reduces shard-management work and usually fits spiky traffic or small teams better.

Provisioned capacity can be more economical for high, predictable throughput, but the team owns capacity monitoring, scaling, and resharding decisions.

The question is not which mode is cheaper in isolation.

The question is whether someone owns capacity review and whether the traffic pattern justifies the operational work.

A provisioned stream nobody resizes is not optimized.

It is neglected.

Monitor producer and consumer throttling separately. Producer throttling is especially important because it can prevent records from entering the stream in the first place.

2. Firehose buffering: freshness or object size?

Firehose delivers when the size or time condition is met first. The configured values are buffering hints, not guarantees.

Set the largest buffer size and longest interval the actual freshness requirement allows.

Do not assume a 128 MB setting means every delivered object will be 128 MB. Dynamic partitioning creates separate buffers per active partition, and low-volume partitions may flush on time before reaching the size target.

High-cardinality partition keys can create too many active buffers and produce small objects.

Choose the partition key for both query behavior and delivery behavior.

3. Error handling: drop, block, or replay?

Dropping malformed records keeps the pipeline moving but silently loses data.

Blocking preserves data but allows one bad record to stop the stream.

A dead-letter path with tested replay is the correct default.

The replay process must define:

• Where failed records are stored

• How the original event ID is preserved

• How records are reprocessed

• How duplicates are prevented

• How successful replay is verified

• When a failed record is permanently quarantined

If replay cannot establish whether a record landed once, the pipeline is not recoverable.

Building the Pipeline

1. Size capacity from peak traffic

Estimate capacity from measured peak throughput, not average throughput.

Averages hide the burst that causes throttling.

Use historical peak traffic, add operating margin, and document the scaling trigger. For provisioned streams, define who reviews capacity and how often.

For on-demand streams, still monitor throughput, throttling, iterator age, and cost.

Removing shard management does not remove the need for capacity visibility.

2. Choose the partition key from query behavior

Use dynamic partitioning based on a field analysts actually filter on.

A useful key usually has:

• Predictable values

• Manageable cardinality

• Sufficient volume per partition

• A direct relationship to common Athena predicates

Do not partition by a field simply because it appears in every event.

A high-cardinality key can create too many active Firehose partitions and small S3 objects. A low-cardinality key may produce efficient objects but weak query pruning.

The correct key balances query behavior and delivery behavior.

3. Define a strict S3 layout

Use a predictable prefix such as:

dataset/event_date=YYYY-MM-DD/hour=HH/

Keep the layout stable so Athena partition projection can calculate locations without requiring a crawler for every delivery.

Decide the partition convention before the stream ships. Repartitioning later means migrating data, updating the table, and validating every consumer.

4. Configure buffering from the real latency requirement

Start with the longest interval the business can tolerate.

Then measure actual object sizes and delivery freshness.

Do not assume one global buffer setting produces one global object size. Dynamic partitioning creates separate buffers per active partition, and low-volume partitions may flush on time rather than size.

If files are consistently too small:

• Reduce partition-key cardinality

• Increase the interval if permitted

• Consolidate low-volume partitions

• Add a compaction process

• Reconsider whether the stream should partition at ingestion

5. Build replay before production

Send deliberately malformed records through the pipeline before launch.

Verify that they:

1. Reach the dead-letter location.

2. Retain enough metadata to identify the original event.

3. Can be replayed without manual reconstruction.

4. Land in the intended S3 prefix.

5. Do not create uncontrolled duplicates.

6. Remove or mark only after successful verification.

An untested replay path is not error handling.

It is storage with optimistic branding.

6. Use tiered iterator-age alarms

Use a warning notification for temporary lag and a page for sustained lag that threatens retention or indicates the consumer cannot recover.

The exact thresholds depend on stream retention, traffic, and recovery time.

Do not copy universal thresholds without measuring your pipeline.

The page threshold should answer:

If nobody acts now, is data loss or an unrecoverable freshness breach likely?

If the answer is no, notify.

Do not page.

7. Page on data-loss conditions

Producer throttling and failed writes deserve urgent treatment because they can prevent records from entering the stream.

Monitor producer success rates, write throttling, and failed producer operations.

Consumer lag is not automatically data loss.

Producer rejection can be.

8. Monitor object-size degradation

Track average and p95 object size in the S3 landing prefix.

Alert when object size falls below the threshold that makes Athena performance or request overhead unacceptable for your workload.

Do not treat 64 MB or 128 MB as universal laws. They are operating thresholds to test against query performance, partition count, and delivery latency.

9. Alert on dead-letter growth

One malformed record may be normal.

A rising dead-letter rate usually indicates an upstream schema change, transformation bug, or invalid payload pattern.

Monitor:

• Dead-letter records per minute

• Dead-letter rate as a percentage of total records

• Change from the trailing baseline

• Age of the oldest unreplayed record

The important signal is accumulation.

10. Put the action in the alarm

Every alarm should link to a runbook containing:

• What the alarm means

• The first diagnostic command

• Two likely causes

• Whether to page or notify

• Immediate remediation

• Escalation owner

• Recovery verification

An alarm without an action is a notification that forces a tired engineer to invent a procedure at 3 AM.

The Streaming Pipeline Alarm Matrix

Before production, review every alarm with the engineer who carries the pager.

Page only when immediate human action is required to prevent data loss, an SLA breach, or an unrecoverable failure.

Temporary iterator-age increase

Severity: Notify

Action: Monitor recovery.

If the consumer is catching up and the stream remains within its retention window, notify the team and observe recovery.

Sustained iterator age approaching retention risk

Severity: Page

Action: Investigate consumer capacity immediately.

Check consumer throughput, processing errors, throttling, and recent traffic changes.

Producer throttling or failed writes

Severity: Page

Action: Protect against data loss.

Check stream capacity, producer error rates, retry behavior, and recent traffic volume.

Dead-letter rate increase

Severity: Notify or page based on loss risk

Action: Investigate the schema or transformation failure.

Notify when records are safely retained, and replay remains within its recovery objective.

Page when the backlog is growing toward an SLA breach or records may be lost.

Object-size degradation

Severity: Notify

Action: Correct buffering or partition strategy.

Review Firehose buffering, dynamic partitioning, traffic distribution, and partition-key cardinality.

Aging replay backlog

Severity: Page if the recovery SLA is threatened

Action: Execute the replay procedure.

Page when unreplayed records approach the maximum recovery window, the backlog grows faster than it can be processed, or replay stops making progress.

If the person carrying the pager does not agree that an alarm deserves a page, the alarm design is unfinished.

A page is a request for immediate action.

Treat it like one.

What to Measure Monthly

Track:

• Pages per month

• Ratio of pages to real incidents

• Mean time from alert to confirmed data state

• Average and p95 landed S3 object size

• Dead-letter rate

• Oldest unreplayed record

• Producer throttle events

• Consumer iterator age

• Active dynamic partitions

• Athena bytes scanned for landed tables

Review alarm thresholds quarterly with on-call engineers.

Review provisioned capacity monthly.

Review object sizes and Athena scan behavior monthly because delivery configuration and query usage change independently.

The Measure of a Good Streaming Pipeline Is Silence

A well-built Kinesis-to-S3-to-Athena pipeline is one your team does not have to watch constantly.

It lands queryable data, keeps partition behavior predictable, sends malformed records to a tested replay path, and pages only when a human must act.

That is the evidence that the streaming decision was correct.

Not the architecture diagram.

Not the fact that data arrives in seconds.

The evidence is the number of unnecessary pages.

Take your most page-prone pipeline and complete the alarm matrix against its current configuration.

Where the matrix and the live system disagree is where your unnecessary pages come from.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

The reason is rarely a documented latency requirement.

It is that streaming sounds modern, while proposing a batch pipeline in a design review can sound like settling.

So the team builds streaming:

• Kinesis

• A consumer application

• Checkpointing

• A dead-letter path

• Monitoring and alerting

• An on-call rotation

The pipeline delivers data four seconds after the event.

The dashboard is reviewed once each morning.

The cost isn't just the infrastructure bill. It is the operational surface: an always-running system with failure modes that can page a human, built to satisfy a freshness requirement nobody properly defined.

Real-time is a requirement, not a default.

Why Streaming Costs More Than Its Infrastructure Bill

The infrastructure cost is only the visible part.

The larger cost is operational.

A streaming pipeline is always running. That means it can always fail. It requires monitoring, alerting, replay procedures, capacity planning, and engineers who understand the system well enough to respond under pressure.

A micro-batch pipeline that runs every fifteen minutes has a different failure profile. It processes data in bounded windows. It can retry within the next window. A failure may delay freshness, but it does not necessarily require immediate human intervention.

Streaming also introduces more concepts for the team to own:

• Backpressure

• Out-of-order events

• Checkpoint management

• Replay behavior

• Watermarks

• Duplicate delivery

• Consumer lag

• Partition or shard capacity

These are not theoretical concerns. They become operational responsibilities.

There is also a concentration risk. Many organizations have only one or two engineers who deeply understand the streaming stack. Every change then depends on the same people.

That is not just a staffing inconvenience.

It is a reliability constraint.

Why Freshness Requirements Become Inflated

Most freshness requirements are inherited rather than derived.

Someone asks a stakeholder how fresh the data should be. With no cost or operational context, the answer is often “real time.”

That answer becomes a requirement.

Eighteen months later, the team is maintaining a streaming pipeline because of one unpriced conversation.

Ask a better question:

What decision does this data support, and how quickly after the event must that decision happen?

Consider the difference:

• A fraud system may need to react within seconds.

• An operational dashboard may need updates every fifteen minutes.

• A daily revenue report may only need correct data by 8 AM.

• A weekly cohort analysis may only need data by Monday.

These are different business requirements.

They should produce different architectures.

The other source of inflation is anticipated future need.

Teams build streaming because they might need real-time later. Then they pay the operational cost for years while the requirement never arrives.

When the requirement finally does arrive, it usually applies to one pipeline, not the entire platform.

The Five Questions to Ask Per Pipeline

1. What decision does this data drive?

Do not ask how fresh the data should be.

Ask what someone or something does with it.

If a human reviews the data during business hours, the actual requirement may be measured in minutes or hours.

If an automated system must react immediately, streaming becomes more defensible.

Architecture should follow the decision, not the preference of the person writing the requirements document.

2. Is there an automated action triggered by the data?

Automated response is the strongest reason to choose streaming.

Examples include:

• Blocking a suspicious transaction

• Triggering a safety response

• Updating a user-facing experience immediately

• Routing an operational event

• Detecting a system failure within seconds

If nothing happens automatically and the data is only reviewed later, micro-batch may provide the required outcome with less operational burden.

3. What is the concrete cost of fifteen minutes of staleness?

State the consequence.

Does fifteen-minute-old data:

• Lose revenue?

• Increase fraud exposure?

• Delay an operational response?

• Violate a contractual requirement?

• Create a safety issue?

• Prevent a customer from completing an action?

If nobody can name the consequence, there may not be a genuine latency requirement.

Do not buy a permanently running system to avoid an inconvenience that has never been priced.

4. Does the volume make batch impractical?

Latency is not the only reason streaming may be necessary.

Sustained event volume can make batch processing impractical if each batch must process more data than arrived during the previous window.

You may be approaching a simple constraint:

Processing time is greater than the batch interval.

If a fifteen-minute batch takes twenty minutes to process, the pipeline cannot keep up. The backlog compounds.

That is a legitimate reason to consider streaming, independent of whether anyone needs sub-minute freshness.

Measure:

• Events per second

• Average and peak volume

• Batch processing duration

• Batch interval

• Backlog growth

• Recovery time after failure

Do not confuse a temporary implementation problem with a permanent streaming requirement. A faster batch job, better partitioning, or a larger processing window may solve the issue.

5. Does the team have on-call capacity?

Streaming belongs in the operating model, not only in the architecture diagram.

Ask:

• Who owns the pipeline at 2 AM?

• Who responds to consumer lag?

• Who handles duplicate or out-of-order events?

• Who replays a failed time range?

• Who responds when capacity is exhausted?

• Who maintains the dead-letter path?

If the team is already stretched, adding another always-on system creates reliability debt.

A streaming design that nobody can operate is not production-ready, regardless of how elegant the topology looks.

The Break-Even Test

The basic latency test is simple:

If the required decision latency is comfortably longer than the batch interval, micro-batch is usually sufficient.

For example:

• Required decision latency: 45 minutes

• Proposed micro-batch interval: 15 minutes

• Processing time: 4 minutes

That pipeline has margin. Streaming is difficult to justify unless another factor, such as sustained volume or automated action, changes the economics.

Now consider:

• Required decision latency: 30 seconds

• Proposed micro-batch interval: 15 minutes

Micro-batch cannot satisfy the requirement. Streaming earns its additional complexity.

Also calculate whether batch can keep up:

Batch processing time < Batch interval

If processing time consistently approaches or exceeds the interval, the batch design is under pressure.

A pipeline may need streaming when:

• The required latency is below the practical batch interval

• Automated action must happen quickly

• Sustained volume causes batch backlog

• The business cost of delay exceeds the operational cost of streaming

Otherwise, micro-batch should be the default you test first.

What Choosing Honestly Returns

Teams that evaluate pipelines individually usually end up with a small number of streaming systems where the requirement is real and micro-batch everywhere else.

The first return is lower on-call load.

Fewer always-on systems mean fewer pages and fewer specialized systems competing for the same engineers.

The second return is faster delivery. Micro-batch pipelines are usually quicker to launch because the operational hardening burden is smaller.

The third return is better attention allocation. The streaming pipelines that remain receive the expertise they need instead of competing with six unnecessary streaming systems for the same two engineers.

There is a legitimate case for streaming.

When automated action, genuine sub-minute decisions, or sustained volume make batch the constraint, build streaming properly.

Do not stretch batch beyond what it can handle.

But do not build streaming to avoid looking old-fashioned in a design review.

The Decision Tree

Use this sequence for every pipeline:

1. What decision does the data support?

2. How quickly must that decision happen?

3. Is there an automated action?

4. Can batch process the volume within the required window?

5. Can the team operate another always-on system?

Then classify the result:

• Micro-batch: Human-reviewed data, flexible decision timing, manageable volume, limited on-call capacity.

• Streaming: Automated response, genuine sub-minute requirement, or batch throughput constraint.

• Re-evaluate: Conflicting answers, undocumented requirements, or an architecture based primarily on anticipated future demand.

Write down the decision and the reason.

Revisit it when volume, consumers, latency requirements, or operating capacity changes.

The architecture should evolve when constraints change.

The Payoff Is Fewer Systems That Matter

Streaming is not bad.

Unnecessary streaming is expensive.

A micro-batch pipeline that meets the business requirement is often the better system because it is easier to build, operate, explain, and recover.

A streaming pipeline that satisfies a real automated or latency-sensitive requirement is worth the complexity.

The mistake is treating one pattern as the default for every workload.

Use the smallest architecture that satisfies the actual decision latency and volume constraint.

That is not conservatism.

That is engineering.

Before approving the next streaming pipeline, make the requestor answer five questions:

• What decision does this data drive?

• How quickly must the decision happen?

• What is the cost of fifteen minutes of staleness?

• Can batch keep up with the volume?

• Who owns the system when it fails?

If the answers are vague, build the simpler pipeline first.

The goal is not to make data arrive as quickly as possible.

The goal is to make the right decision at the right cost without creating an on-call rotation nobody needs.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

This is the system for fixing them in the order that creates the highest return per hour of engineering time.

The order matters.

Partitioning, partition projection, file format, and query predicates all interact. Make the wrong change first, and you may have to redo the work.

The optimization sequence is:

1. Find the dominant query predicates.

2. Verify the partition key matches them.

3. Confirm partition pruning is actually occurring.

4. Move to partition projection where the layout supports it.

5. Convert qualifying tables to Parquet.

6. Measure the result against the original baseline.

Do not start by converting every table to Parquet.

Start with the tables that cost you the most.

Why Scan Reduction Is the Primary Lever

Athena cost is closely tied to the amount of data scanned.

That makes scan reduction unusually measurable. If a table moves from full scans to scans that read one-twentieth of the data, the scan volume for those queries can fall dramatically.

The performance benefit arrives at the same time.

Analysts feel the difference between a query that takes ninety seconds and one that takes four. Faster queries also encourage more productive exploration, which makes efficient table design a business capability rather than merely a finance exercise.

The goal is not to make every query cheap.

The goal is to eliminate repeated, avoidable scans from the tables people use most.

Why Partitioned Tables Still Scan Too Much

A table can be partitioned and still receive almost no benefit from partitioning.

Three mistakes cause most of the damage.

The partition key does not match real query behavior

A table partitioned by ingest_date may receive queries filtered by event_date.

The table is technically partitioned. The common queries still scan across the relevant data because they do not constrain the partition key.

Partition based on how the data is actually queried, not only on how it arrives.

A function wraps the partition column

Queries that cast, transform, or apply date functions to the partition column may prevent effective pruning.

For example, this pattern can be problematic:

WHERE CAST(event_date AS DATE) = DATE '2026-09-01'

Prefer predicates that preserve the column’s native type and apply the comparison directly:

WHERE event_date >= TIMESTAMP '2026-09-01 00:00:00'
  AND event_date <  TIMESTAMP '2026-09-02 00:00:00'

The exact syntax depends on the column type. The principle is consistent: make the partition predicate explicit and comparable without wrapping the partition key in a function.

Partition metadata is incomplete or expensive

Data can exist in [internal] while Athena lacks the corresponding partition metadata.

The result is missing data, unexpected scans, or repeated crawler and repair work.

Tables with very large numbers of registered partitions can also develop meaningful metadata-management overhead. At that point, partition projection may be a better fit if the prefix layout is predictable.

Three Decisions Per Table

1. What should the partition key be?

You usually choose between an ingestion-time column and a business-time column.

Ingestion-time partitioning is simpler operationally. It matches how files arrive and how pipelines typically write data.

Business-time partitioning is often better for query pruning when analysts filter by the date an event actually occurred.

Look at query history before deciding.

The predicate that appears in most queries is stronger evidence than the partition convention your ingestion pipeline happens to prefer.

2. Should you use catalog partitions or partition projection?

Catalog partitions rely on registered metadata, usually maintained through crawlers, repair operations, or explicit partition updates.

Partition projection defines the partition scheme through table properties. Athena calculates partition locations from the defined pattern instead of requiring every partition to be registered individually.

Use partition projection when:

• The prefix layout is predictable

• Partition values follow a defined date or enum range

• The table has many partitions

• New partitions arrive regularly

• You want to reduce crawler or repair dependence

Keep catalog partitions when:

• The layout is irregular

• Partition values cannot be derived reliably

• The table has modest partition counts

• The data location changes unpredictably

Projection is not automatically better. It requires accurate ranges and predictable paths. A bad projection definition can cause Athena to consider locations that do not exist.

3. Should the table be converted to Parquet?

Parquet is most valuable for wide tables where queries select a small subset of columns.

Columnar storage lets Athena read only the required columns, while compression reduces the data transferred and processed.

Conversion is a stronger candidate when:

• The table is queried frequently

• The table contains many columns

• Queries select relatively few columns

• The current format is CSV, JSON, or another row-oriented format

• The table has meaningful scan volume

Conversion may return less when:

• The table is narrow

• Queries routinely use almost every column

• The table is rarely queried

• The conversion process costs more to operate than the scans justify

Do not assume Parquet is valuable because it is modern.

Measure how much of the table queries actually use.

The Optimization Sequence

1. Pull your top five tables

Start with the ranked table list from your scan analysis.

Work in order of aggregate scan volume. Do not spread engineering effort across twenty tables at once.

2. Extract the dominant predicates

Review the most common WHERE clauses for each table.

Record:

• The columns filtered

• The date ranges used

• Whether the partition key appears

• Whether functions wrap the partition column

• Whether scheduled queries use bounded predicates

This is the input to every later decision.

3. Compare the partition key with real query behavior

If the partition key does not match the dominant query predicate, address that before converting formats.

A perfectly compressed table with the wrong partition key can still scan far more data than necessary.

4. Check whether pruning is occurring

Compare bytes scanned with the total table size for representative queries.

A table where queries scan nearly the full table is a candidate for:

• A better partition key

• Corrected predicates

• Better partition metadata

• A different table layout

• A purpose-built aggregate

Do not assume pruning works because the table definition says it is partitioned.

Verify it in query execution data.

5. Move regular layouts to partition projection

For tables with predictable date or enum-based prefixes, define the projection properties and test a representative query.

Compare:

• Planning time

• Total runtime

• Bytes scanned

• Query success

• Results against the previous table

Do not remove the existing metadata process until you've validated the projected table.

6. Audit functions on partition columns

Search query history for casts and date functions applied directly to partition keys.

Fix the highest-volume queries first. A one-line rewrite in a dashboard definition can remove repeated full or broad scans.

7. Convert qualifying tables to Parquet

Write the converted data to a new prefix.

Use compression such as Snappy for a balanced combination of performance and compatibility.

Keep the original data until you validate:

• Row counts

• Representative query results

• Null behavior

• Timestamp behavior

• Partition coverage

• Cost and runtime

8. Avoid small files

Many small files create request and metadata overhead that can reduce Parquet's benefits.

Target a practical file size in the broad range of 128 MB to 512 MB, then validate against your workload. The right value depends on table size, query patterns, writer behavior, and partition cardinality.

Small-file problems tend to return unless the writer is configured to prevent them.

9. Re-run the measurements

After the table has been in use for at least a week, compare it against the baseline:

• Total bytes scanned

• Average bytes scanned per query

• p95 bytes scanned

• Query planning time

• Runtime

• Monthly scan volume

If scan volume did not fall, stop claiming success.

Find out whether the partition key, predicates, file layout, or workload was misdiagnosed.

The Athena Table Optimization Checklist

Use one worksheet per table.

Record:

• Table size

• Partition count

• Monthly query count

• Current average and p95 bytes scanned

• Current scanned-to-total ratio

• Top query predicates

• Current partition key

• Projection eligibility

• Average columns selected per query

• Current file format

• Average file size

• Owner

• Recommended change

• Before-and-after measurements

Complete the dominant-predicate section before changing the table.

Every other decision depends on it.

What to Measure

Track these per optimized table:

Scanned-to-total ratio

This is your primary pruning signal.

A low ratio generally indicates that queries are reading a limited portion of the table. A ratio near the full table size indicates that pruning may not be occurring.

Interpret the ratio alongside query intent. Some analytical queries legitimately require broad scans.

Query planning time

Rising planning time on a highly partitioned table may indicate that partition metadata is becoming expensive to manage.

Average file size

A declining average file size indicates that the writer is producing small files again.

Monthly bytes scanned

This is the number that ultimately matters to the bill.

Review the top-five ranking quarterly. Usage changes. A table that was insignificant last quarter can become the highest scanner after one new dashboard or reporting workflow goes live.

The Return Compounds

A correctly partitioned, efficiently formatted table reduces scan volume for future queries.

That is what makes this work different from negotiating a temporary discount or trimming an instance.

You are changing the unit economics of the table.

The return grows as usage grows.

Athena optimization is not a mystery, and it is not a platform-wide cleanup project.

It is a measurement problem followed by focused engineering.

Take your five highest-scan tables.

Bring the analysts who query them into the room.

Complete the predicate analysis before changing the table.

Then fix the highest-impact mismatch first.

The boring work is the profitable work.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

It is also one of the least diagnosed.

Athena charges based on data scanned. An analyst writes a query, gets the result, and moves on. The cost appears later as a line item with no obvious owner.

Then someone notices spend increased and asks the data team to investigate.

The data team looks at the total.

That is where the investigation usually stops.

The useful information is lower in the stack:

• Which queries scanned the most?

• Which tables were involved?

• Which workgroup or principal submitted them?

• Were they scheduled reports, dashboards, or exploratory queries?

• Did partition pruning actually happen?

Athena exposes execution metadata, including bytes scanned and execution time, through query execution history and APIs. Persist that metadata, then turn it into a cost-management table.

Why Expensive Scans Compound

An unpartitioned or poorly partitioned table doesn't cost you once.

It costs you every time someone queries it.

The worst table is often not the largest table. Large tables frequently receive attention early because their size makes optimization obvious.

The more dangerous table is often medium-sized, convenient, and queried constantly.

It started small. Nobody partitioned it. It became central to reporting. Now every dashboard refresh scans the table again.

The result is not just higher cost.

It is wasted engineering attention. Without attribution, the response becomes a broad request for “everyone to optimize their queries.”

Broad requests produce shallow effort.

The actual fix is usually concentrated in two or three tables.

First, Create a Query History Table

The queries below assume you have a normalized table called:

platform_analytics.athena_query_history

With columns such as:

• query_execution_id

• submitted_at

• query_text

• principal

• workgroup

• data_scanned_bytes

• execution_time_ms

• referenced_table

referenced_table should contain one row per table referenced by a query. A query that touches three tables should produce three history rows with the same query execution ID.

Do not pretend native query history is a permanent warehouse. Persist it to [internal] or another governed store if you need longer-term analysis.

Query 1: Which Tables Drive Total Scan Volume?

SELECT
    referenced_table,
    COUNT(DISTINCT query_execution_id) AS query_count,
    SUM(data_scanned_bytes) / POWER(1024, 4) AS tebibytes_scanned,
    AVG(data_scanned_bytes) / POWER(1024, 3) AS avg_gib_scanned
FROM platform_analytics.athena_query_history
WHERE submitted_at >= current_timestamp - INTERVAL '30' DAY
GROUP BY referenced_table
ORDER BY tebibytes_scanned DESC
LIMIT 25;

This ranks tables by aggregate scan volume over the last 30 days.

That is more useful than finding the single most expensive query. A table scanned moderately by hundreds of queries may cost more than one table scanned heavily once.

Start with the top three tables.

Do not optimize the entire lake.

Query 2: Who or What Is Driving the Spend?

SELECT
    principal,
    workgroup,
    COUNT(DISTINCT query_execution_id) AS query_count,
    SUM(data_scanned_bytes) / POWER(1024, 4) AS tebibytes_scanned,
    AVG(data_scanned_bytes) / POWER(1024, 3) AS avg_gib_scanned
FROM platform_analytics.athena_query_history
WHERE submitted_at >= current_timestamp - INTERVAL '30' DAY
GROUP BY principal, workgroup
ORDER BY tebibytes_scanned DESC
LIMIT 25;

This gives you attribution.

Not to assign blame.

To identify the type of fix required.

If scheduled dashboards dominate, optimize refresh queries and table design.

If one workgroup dominates, apply workgroup-level controls and review its consumers.

If exploratory users dominate, improve table documentation, examples, partition guidance, and query education.

The same cost requires different interventions depending on its source.

Query 3: Which Tables Are Being Scanned Almost in Full?

To run this query, maintain a second table with one row per table:

platform_analytics.athena_table_inventory

Required columns:

• referenced_table

• table_size_bytes

WITH table_scan_stats AS (
    SELECT
        referenced_table,
        COUNT(DISTINCT query_execution_id) AS query_count,
        SUM(data_scanned_bytes) AS total_scanned_bytes,
        AVG(data_scanned_bytes) AS avg_scanned_bytes
    FROM platform_analytics.athena_query_history
    WHERE submitted_at >= current_timestamp - INTERVAL '30' DAY
    GROUP BY referenced_table
)
SELECT
    s.referenced_table,
    s.query_count,
    s.avg_scanned_bytes / POWER(1024, 3) AS avg_gib_scanned,
    i.table_size_bytes / POWER(1024, 3) AS table_size_gib,
    s.avg_scanned_bytes / NULLIF(i.table_size_bytes, 0) AS avg_scan_ratio
FROM table_scan_stats s
JOIN platform_analytics.athena_table_inventory i
    ON s.referenced_table = i.referenced_table
WHERE s.avg_scanned_bytes / NULLIF(i.table_size_bytes, 0) >= 0.80
ORDER BY s.total_scanned_bytes DESC;

A high ratio means queries are scanning most of the table on average.

That makes partitioning, file format, predicate use, or table redesign a likely optimization target.

The ratio is a screening signal, not proof. A full scan may be intentional for some workloads. Confirm the query patterns before changing the table.

What to Do With the Results

For each of the top three tables, record:

• Owner

• Primary consumers

• Query count

• Total bytes scanned

• Average scan ratio

• Partition scheme

• File format

• Most common predicates

• Recommended fix

• Expected completion date

Then choose one optimization queue.

Do not send the results to everyone with a request to “use Athena more efficiently.”

Assign one owner to each table and fix the highest-impact problem first.

The usual candidates are:

• Add or repair partitioning

• Convert text formats to columnar formats

• Add date predicates to scheduled queries

• Reduce dashboard refresh frequency

• Separate raw and analytical tables

• Create an aggregate table for repeated reporting

• Move an unsuitable workload to another query engine

The first win should be measurable.

Compare bytes scanned before and after the change.

What to Measure Monthly

Track:

• Total bytes scanned

• p95 bytes scanned per query

• Top five tables by scan volume

• Top five principals or workgroups by scan volume

• Number of tables with average scan ratio above 0.80

• Athena cost per active analyst or consuming team

• Time required to identify the owner of an expensive table

The goal is not to make every query cheap.

The goal is to stop paying repeatedly for the same avoidable waste.

Athena costs become manageable when you have attribution, owners, and a recurring review cadence.

Every expensive Athena environment has the same failure mode:

The data needed to diagnose the problem exists.

Nobody turns it into a queue.

Run the three queries today. Find the top three tables. Assign owners. Fix one before optimizing anything else.

That is how a mysterious line item becomes an operating metric.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

The reasoning is understandable.

One pattern means fewer tools, fewer skills to maintain, fewer failure modes, and simpler runbooks. Standardization helps platform teams stay sane.

Then the portfolio grows.

A pipeline loads terabytes of raw data into the warehouse, performs an expensive transformation, and produces a small aggregate. The organization pays warehouse compute to process data that could have been reduced earlier.

Another pipeline transforms too aggressively before loading. Six months later, an analyst needs detail that was discarded during ingestion. Reconstructing it requires a source-system project nobody planned for.

The pattern choice is not a permanent platform standard.

It is a per-pipeline decision with cost, latency, flexibility, and compliance consequences.

What ETL and ELT Actually Trade

ETL means extract, transform, then load.

The data is transformed before it reaches the analytical destination.

ELT means extract, load, then transform.

The raw or lightly processed data lands first, and transformations happen inside the warehouse or lakehouse.

Neither pattern is universally better.

ETL is useful when you need to reduce data volume before loading, apply controls before broader access, or run stable and computationally heavy transformations outside the warehouse.

ELT is useful when you want to preserve raw data, iterate quickly on transformation logic, and let analysts modify SQL without requiring an engineering deployment.

The mistake is treating either pattern as an ideology.

Why the Wrong Choice Appears in the Warehouse Bill

ELT deliberately uses warehouse compute for transformation.

That is often a good trade. You preserve raw data, simplify ingestion, and keep business logic close to the analysts who change it.

The trade becomes less attractive when:

• The transformation runs on every load

• The input is much larger than the output

• The transformation is computationally heavy

• The output discards most of the source data

• The warehouse is already capacity-constrained

Loading two terabytes to produce a four-gigabyte aggregate may still be the right choice if the raw data has analytical value.

But if the raw data is never queried, must be reduced before access, and is expensive to transform repeatedly, ETL deserves serious consideration.

Do not optimize based on the size of the final table alone. Compare:

• Input volume

• Output volume

• Transformation frequency

• Compute cost

• Retention requirements

• Future analytical value of the raw data

Why Latency Changes the Answer

ETL adds a processing stage before the data reaches its destination.

That can increase freshness latency.

For a dashboard that genuinely requires data every five minutes, an additional transformation stage may be unacceptable. ELT, micro-batching, or streaming may be a better fit.

But many freshness requirements are inherited rather than measured.

A team says a dashboard needs five-minute freshness because that was written in an old requirements document. The dashboard is reviewed once each morning.

That is not a five-minute business requirement. It is an unchallenged assumption.

Before choosing a lower-latency pattern, ask how quickly someone must act on the data. Freshness is valuable when it changes a decision, not merely because the pipeline can be made faster.

Why Compliance Changes the Answer

Compliance does not automatically mean every sensitive pipeline must use ETL.

The real question is:

Can raw sensitive data land in the destination under an access model that prevents unauthorized users from seeing it?

If the answer is no, sensitive fields must be masked, tokenized, filtered, or otherwise controlled before they reach the broadly accessible analytical layer.

That may require ETL.

In other architectures, raw data can land in a tightly restricted zone while a governed transformation produces the analyst-facing tables.

The requirement is not “always use ETL.”

The requirement is that sensitive data does not become broadly accessible before the required controls are applied.

Document where the control happens, who can access the raw layer, and how the transformed layer is verified.

The Four Questions to Ask for Every Pipeline

1. What is the input-to-output ratio?

If a pipeline consistently reduces a large source into a much smaller analytical output, evaluate ETL.

A high reduction ratio does not automatically settle the decision. Raw data may still be valuable for future analysis, audits, or reprocessing.

But repeatedly paying warehouse compute to discard most of the input is a clear signal to investigate.

2. Does the pipeline contain sensitive data?

Identify whether the source includes:

• Personal information

• Financial information

• Health information

• Credentials or secrets

• Customer-level behavioral data

• Regulated operational data

Then define the first point at which unauthorized access is technically prevented.

If sensitive data must be controlled before it reaches the general warehouse, transform before that boundary.

3. How stable is the transformation logic?

Frequently changing business logic often belongs in ELT.

Analysts can iterate in SQL, compare versions, and update transformations without rebuilding the ingestion system.

Stable, computationally heavy transformations are stronger ETL candidates, especially when they reduce data substantially before loading.

Do not confuse “stable” with “important.” A metric can be business-critical and still change frequently.

4. What freshness requirement is real?

Classify the pipeline based on the action it supports:

• Daily reporting

• Hourly operational review

• Near-real-time monitoring

• Automated decisioning

• User-facing product behavior

Then measure the acceptable delay.

A pipeline that supports daily reporting does not need a five-minute architecture. A pipeline that triggers fraud controls may.

A Practical Decision Rule

Use ETL when most of the following are true:

• The transformation is computationally heavy

• The output is dramatically smaller than the input

• The logic is stable

• Sensitive fields must be controlled before broad access

• Extra processing latency is acceptable

Use ELT when most of the following are true:

• Raw data has ongoing analytical value

• Transformation logic changes frequently

• Analysts need direct SQL control

• The pipeline requires very low latency

• Warehouse capacity can support the transformation workload

When the signals conflict, resolve them in this order:

1. Security and compliance requirements

2. Real freshness requirement

3. Input-to-output economics

4. Transformation volatility

5. Platform simplicity

This keeps cost from overriding a control requirement and prevents inherited latency assumptions from dictating every design.

What Per-Pipeline Decisions Return

Teams that evaluate pipelines individually usually end up with a portfolio rather than a religion.

High-reduction and control-sensitive pipelines use ETL where it creates a clear economic or security advantage.

Stable-volume pipelines with frequently changing business logic use ELT where analyst flexibility is worth the warehouse cost.

The platform still has standards.

The standard is not “everything uses ETL” or “everything uses ELT.”

The standard is the decision process.

Track the choice for each pipeline:

• Selected pattern

• Reason for selection

• Expected cost profile

• Freshness target

• Data sensitivity

• Review trigger

Revisit the choice when the pipeline’s volume, consumers, sensitivity, or latency requirement changes.

Supporting two patterns creates real operational overhead.

It is still cheaper than forcing every pipeline into a pattern that fits half the portfolio.

A platform default is useful.

A default nobody is allowed to revisit becomes technical debt with a governance committee.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

The central data team is a bottleneck. Dashboard requests queue behind one another. Domain teams understand their own data best, but wait weeks for someone else to model it.

Decentralizing ownership sounds like the obvious solution.

So the organization announces a data mesh. Domains become responsible for their own data products. The central data team becomes a platform team.

Twelve months later, there are nine domain data teams, no shared discovery layer, four definitions of “active customer,” and an executive asking why two reports show different revenue numbers.

The bottleneck is gone.

So is the coherence.

Nobody necessarily made a bad decision. The organization decentralized ownership before building the capabilities that make decentralized ownership work.

Domain ownership without self-service infrastructure, data products, and federated governance is not a data mesh. It is decentralization with better branding.

Why a Premature Data Mesh Costs More Than the Bottleneck

The central data team may have been slow, but it was also quietly performing four jobs:

1. Enforcing consistent metric definitions

2. Helping people discover available datasets

3. Acting as a quality gate

4. Controlling access to sensitive data

When ownership is decentralized without replacing those functions, they do not automatically transfer to the domains.

They stop happening.

The cost appears as reconciliation work. Finance and product report different revenue numbers, and a senior analyst spends three days tracing the discrepancy.

That work rarely appears on a roadmap. It still consumes real capacity.

The second cost is duplicated infrastructure. Each domain builds its own pipelines, cataloging process, quality checks, and access workflow. Platform spending increases without a proportional increase in what the business can answer.

The third cost is loss of trust.

Once consumers cannot tell which dataset is authoritative, they create private copies. Those copies develop their own definitions, owners, and freshness characteristics.

You have not created a mesh.

You have created multiple competing versions of reality.

Why Capable Teams Decentralize Too Early

Data mesh is often adopted as an organizational fix for a technical constraint.

The pressure is immediate. The central team is overloaded, and business stakeholders want faster delivery.

Reorganizing is something leadership can do this quarter.

Building a self-service data platform is a multi-quarter engineering investment. For much of that period, the work produces infrastructure rather than visible business features.

When given a choice between movement now and results later, organizations usually choose movement.

That is why the sequencing fails.

Data mesh has four commonly cited principles:

• Domain ownership

• Data as a product

• Self-serve data infrastructure

• Federated computational governance

The first principle is easy to announce. The other three are harder to show in an org chart, so they become aspirations in a strategy document.

Domain ownership without the other three produces local autonomy without shared standards, discoverability, or reliable access.

That is not a mesh. It is a collection of silos.

The Seven-Signal Data Mesh Readiness Audit

Answer each question honestly.

Each “no” identifies a capability you need to build before expanding domain ownership.

1. Can a domain team publish a new data product without filing a ticket with the central team?

If every new dataset requires a platform engineer to create pipelines, configure storage, register metadata, or grant access, you have not removed the bottleneck.

You have renamed it.

A real mesh requires a paved path that lets domain teams publish within defined standards without needing custom platform intervention every time.

2. Is there one place where anyone can discover available data products?

A consumer should be able to find:

• What data products exist

• Who owns them

• What they contain

• How fresh they are

• What quality guarantees exist

• How to request access

Without a shared discovery layer, decentralized ownership becomes decentralized invisibility.

People cannot use data they cannot find.

3. Are your top ten business metrics defined and owned?

Do you have written definitions for metrics such as:

• Active customer

• Gross revenue

• Net revenue

• Churn

• Conversion rate

• Qualified lead

More importantly, does each definition have a named owner?

Federated governance does not mean every domain defines everything independently. It means accountable owners apply shared standards locally.

Without shared definitions, you do not have federation.

You have local interpretation.

4. Can one domain grant another domain access without involving a platform engineer?

Access friction creates shadow copies.

If a product team needs customer data and must wait two weeks for a platform engineer to process the request, someone will eventually copy the data into a place they control.

That may solve the immediate problem.

It also creates another dataset with another permission model, another freshness risk, and another definition of truth.

Self-service access is not just a convenience feature. It is a control against uncontrolled duplication.

5. Does every existing dataset have a named owner today?

Do not assume a reorganization will fix unclear ownership.

If ownership is ambiguous now, decentralization does not assign accountability. It distributes the ambiguity across more teams.

Before assigning domain responsibility for new data products, identify who owns the datasets already in production.

If nobody can answer, start there.

6. Do you have data quality contracts?

A data quality contract should make expectations explicit between producers and consumers.

It may cover:

• Schema

• Freshness

• Completeness

• Valid value ranges

• Availability

• Change notification

Without a contract mechanism, an upstream schema change becomes a downstream incident.

Domain autonomy is only safe when consumers know what producers have promised to maintain.

7. Does anyone treat data as a product today?

A data product has more than a table and a pipeline.

It has:

• A defined consumer

• A clear use case

• An accountable owner

• A quality expectation

• A freshness expectation

• A way for consumers to provide feedback

If nobody in the organization treats data this way today, a reorganization will not create product thinking by itself.

You will simply assign ownership of poorly defined datasets to new teams.

Score Your Organization

Use the following interpretation:

• 6 to 7 yes answers: You may be ready for a controlled data mesh rollout.

• 4 to 5 yes answers: Build the missing platform capabilities before expanding domain ownership.

• 0 to 3 yes answers: Do not begin with a data mesh reorganization. Your immediate priority is platform readiness.

The score is not a maturity badge.

It is a sequencing decision.

What Correct Sequencing Buys You

Organizations that build the platform layer first can decentralize once and keep the model stable.

Organizations that decentralize first often reverse the decision eighteen months later, after reconciliation costs, duplicated tooling, and loss of trust become impossible to ignore.

The platform investment pays back before the mesh arrives.

Self-service pipeline tooling reduces central team intake.

A discovery catalog reduces duplicate dataset creation.

Standardized access workflows reduce copying.

Data quality contracts reduce downstream surprises.

These capabilities are useful under a centralized model and necessary under a distributed one.

That is the key sequencing insight:

You do not need to wait for a data mesh to build a platform that supports one.

Fix the bottleneck with platform capability first. Move ownership when the capability exists to support it.

A reorganization is not an architecture.

If the platform underneath cannot support autonomy, the org chart change only relocates the problem to a place where it is harder to see.

Before you announce a data mesh, score your organization against these seven signals.

If you have fewer than five yes answers, do not start with decentralization.

Start by building the missing platform capabilities.

That is how you create a data mesh instead of a collection of silos with better branding.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

They fail because nobody encoded ownership, schema authority, partitioning, and access rules before the first dataset arrived.

The pattern is predictable:

A team creates an Amazon S3 bucket, points AWS Glue crawlers at it, and queries the data with Amazon Athena.

It works.

Six months later, the catalog contains hundreds of tables. Nobody knows which ones are authoritative. Analysts build private copies. Athena queries scan more data than expected. The platform team becomes the human directory for every dataset question.

The lake has become a data swamp.

The problem is not serverless architecture. The problem is that the structure was left in documentation, tribal knowledge, and manual cleanup instead of being encoded in infrastructure as code.

The cost of an unstructured AWS data lake

An unstructured data lake bills you twice.

The direct cost comes from inefficient queries. Without partitioning conventions enforced at ingestion, analysts can scan entire prefixes or large tables when they only need a small date range.

The second cost is trust.

When teams cannot answer which table is authoritative, they create their own copies. Eventually, the organization pays for duplicate storage and compute while executives receive reports that do not reconcile.

The platform team absorbs the organizational cost. They become the routing layer for questions that the data platform should answer automatically:

• Which dataset should I use?

• Who owns this table?

• Is this data raw, cleaned, or curated?

• Who can read customer PII?

• Is this table still supported?

• Why did this query scan so much data?

The fix is not another dashboard.

The fix is structure at creation.

Why data lakes accumulate hundreds of untrusted tables

The failure usually begins with a reasonable decision:

Let teams land data quickly and organize it later.

AWS Glue crawlers make this easy. Point a crawler at a prefix and it infers a schema, then registers a table in the Glue Data Catalog.

That reduces friction. It also removes accountability.

A crawler-created table may have a schema but no:

• Owner

• Data classification

• Quality contract

• Source system

• Lifecycle policy

• Statement of authority

By month eleven, the catalog may accurately describe what exists while providing almost no signal about what can be trusted.

The same pattern appears in permissions. Teams receive ad hoc S3 or IAM access during onboarding. Exceptions accumulate. A year later, answering “who can read this customer dataset?” requires reconstructing old tickets, policies, and team decisions.

That is not governance. That is archaeology.

The three design decisions that determine whether the lake holds

1. Separate zones with buckets or prefixes

A practical serverless data lake usually has three zones:

• Raw: Data as received from source systems

• Cleaned: Validated and standardized data

• Curated: Business-ready datasets for consumption

Separate S3 buckets provide cleaner IAM boundaries, independent lifecycle policies, and distinct encryption-key scopes.

Prefixes inside one bucket are simpler, but they create a weaker security boundary and make policy conditions easier to misconfigure.

Use separate buckets when you have regulated data, multiple teams writing data, or materially different retention requirements.

Use prefixes when one team owns the entire lake and the operating model is genuinely simple.

The important decision is not which structure sounds more elegant. It is whether the boundary will remain explainable as the organization grows.

2. Use crawlers selectively

Use Glue crawlers for the raw zone when the incoming schema genuinely changes or is outside your control.

For cleaned and curated data, declare table schemas in Terraform with aws_glue_catalog_table.

That gives you:

• Reviewed schema changes

• Version history through pull requests

• Explicit partition keys

• Consistent storage descriptors

• A clear owner for every table

A schema change should be visible before it reaches production, not discovered after an automated crawler silently changes the catalog.

3. Choose the permission model deliberately

IAM-only access is simpler and may be sufficient for a small analytics organization with limited sensitivity and few consuming teams.

Lake Formation becomes more valuable when you need permissions expressed in data terms:

• Database

• Table

• Column

• Data classification

• Consuming team

Do not adopt Lake Formation because it sounds more enterprise-ready. Adopt it when the organization needs governance that IAM paths cannot express cleanly.

The mistake is running both systems without deciding which one is authoritative. If you use Lake Formation, define the migration sequence, grant required access first, and remove legacy access only after you have verified the workload path.

The Terraform reference pattern

Build the lake from five modules:

1. Zone bucket module
   Inputs: zone name, retention period, KMS key ARN, versioning setting
   Outputs: bucket ARN, read policy, write policy

2. Catalog database module
   Defines the logical database for each data domain or zone.

3. Catalog table module
   Defines cleaned and curated schemas, partition keys, Parquet configuration, and table properties.

4. Lake Formation grant module
   Defines which teams can access which databases, tables, and columns.

5. Ingestion pipeline module
   Defines how data moves from raw to cleaned to curated.

Make the module interface fixed. Avoid one-off infrastructure for every new dataset.

Required metadata should be mandatory:

• Owner team

• Data classification

• Source system

• Retention category

• Zone

A table without an owner should fail the plan, not become a cleanup ticket.

Encode the partition convention in the ingestion and writer controls. For example:

dataset/ingest_date=YYYY-MM-DD/

Do not rely on a wiki to enforce a convention. Use controlled writer roles, validation, and policy checks so incorrectly shaped data cannot quietly enter the lake.

Add lifecycle rules when each zone is created. Retention and storage-class transitions are much cheaper to define at creation than retrofit across years of accumulated data.

The operating metrics

Review these four numbers monthly:

1. Percentage of catalog tables with an owner tag
   Target: 100%.

2. Athena bytes scanned at the 95th percentile
   Rising scan volume usually means partitioning or query discipline is degrading.

3. Lake Formation grants compared with consuming teams
   Grant growth that dramatically exceeds team growth signals exception accumulation.

4. Time required to answer who can read a table
   Target: less than ten minutes.

Review access grants quarterly. Review partitioning and scan behavior monthly with the analytics team.

Build a lake you can still explain in year three

A well-structured data lake may cost more effort in week one and less money, confusion, and risk every week after.

Structure encoded at creation is cheap.

Structure imposed on four hundred existing tables is a reorganization nobody has budgeted for.

If your lake is already in production, compare the current architecture against the five modules above. Identify which module has no clear equivalent. That gap list is your backlog.

The goal is not to build the most sophisticated AWS data platform.

The goal is to build one that remains trusted, governable, and explainable as the organization grows.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

1. Read new S3 files.

2. Transform the data.

3. Write to Amazon Redshift.

4. Finish successfully.

They run cleanly for months.

Then one fails halfway through the load, and the team discovers it cannot answer a basic question:

Which rows actually made it into Redshift?

That is when an ordinary pipeline failure becomes a data correctness incident.

The problem is not that the job failed.

The problem is that the pipeline has no reliable way to distinguish:

• Data that was read

• Data that was transformed

• Data that reached staging

• Data that reached the target table

• Data that was committed successfully

A production pipeline must make those states observable and recoverable.

Why Partial Loads Cost More Than Failed Loads

A clean failure is an alert and a rerun.

A partial load is different. Downstream dashboards may show incomplete numbers that still look plausible. Nobody notices until an analyst finds a total that does not reconcile, sometimes days later.

You cannot safely rerun the entire job if some rows already landed. The rerun may duplicate those rows.

The recovery process becomes manual:

• Reconstruct which files were processed

• Identify which rows reached Redshift

• Delete partial data

• Rerun a subset

• Validate the result under production pressure

The business cost is even higher. Decisions may have been made using incomplete data during the gap.

That cost rarely appears in the incident report.

Why Glue Job Bookmarks Are Not Idempotency

Glue job bookmarks are useful, but they solve a narrower problem than many teams assume.

A bookmark tracks source objects that a job has processed. It is not a record of what was successfully committed to the downstream target.

Imagine a job reads 100 files, writes data from 60 of them to Redshift, and then fails.

The bookmark may remain unadvanced. The rerun reads the 100 files again.

Without an idempotent write strategy, those 60 files can be inserted twice.

The bookmark did not fail.

It tracked source progress. It did not guarantee destination correctness.

Late-arriving data creates a second problem. A file containing yesterday’s records may arrive today. Depending on the source layout and bookmark behavior, the file may be missed, reprocessed, or require a broader partition scan.

You need both:

• A mechanism that finds the right source data

• A mechanism that makes the destination write safe to repeat

The Three Decisions That Determine Reliability

1. Staging merge or delete-and-insert?

Use a staging table plus MERGE when:

• You have a dependable business key

• Existing records can be updated

• You need exact upsert behavior

• Duplicate prevention matters across reruns

Use delete-and-insert by partition when:

• The data is append-only

• The partition is the natural unit of reprocessing

• The entire partition can be rebuilt safely

• You do not have a reliable primary key

Delete-and-insert is simpler, but only correct when the partition is fully reprocessable.

If new records arrive incrementally within the same partition, you may need to delete and rebuild the partition. Deleting only the rows seen in the latest run may leave stale or duplicate data behind.

2. Lookback window or event-driven reprocessing?

A lookback window reprocesses the last N days on every run.

This is simple and resilient. It catches late-arriving files without requiring a separate event workflow.

Start with a three-day window only as an operating default, not as a permanent truth. Measure actual arrival delays and resize the window based on evidence.

Use event-driven reprocessing when:

• Data volume makes repeated lookbacks expensive

• Late arrivals are rare but costly

• You can reliably detect affected partitions

• The team can operate the additional event workflow

The right window is the smallest one that captures your real arrival distribution with an acceptable miss rate.

3. Job-level or batch-level recovery?

Job-level retry reruns the entire job.

That is usually sufficient for shorter pipelines.

Batch-level checkpointing records progress as the job advances and allows recovery from a smaller unit. It adds complexity, but becomes valuable when a full run takes long enough that restarting from zero threatens the load window.

The decision is economic:

How expensive is a full restart when the job fails near completion?

If the answer is “it causes missed SLAs or creates a backlog,” finer-grained recovery may pay for itself.

The Production Pattern

1. Enable bookmarks and stabilize transformation contexts

Enable Glue job bookmarks, but treat them as source-discovery state, not destination idempotency.

Assign stable transformation context names. Refactoring an unnamed context can change how Glue identifies a transformation and may affect bookmark behavior.

Treat the context name like an interface. Change it deliberately, not incidentally.

2. Truncate staging at the beginning of every run

Write each run into a staging table with:

• The same business columns as the target

• A batch identifier

• Source file metadata

• Ingestion timestamp

Generate the batch identifier from the Glue run ID.

Truncate staging at job start, not job end.

If the previous job failed before cleanup, the next run starts from a known state. Cleanup at the end is not enough because failed jobs do not reliably reach the cleanup step.

3. Load staging before touching the target

Do not write directly into the production table.

First load the source data into staging. Then validate it.

Before the target changes, verify:

• Expected source objects were found

• Staged row count is within tolerance

• Required columns are present

• Business keys are not unexpectedly duplicated

• Transformation errors are below threshold

This separates ingestion failure from target corruption.

4. Merge into Redshift as one transaction

After validation, merge staging into the target within a single transaction.

The transaction should either apply the batch completely or apply none of it.

That converts a partial target write into a clean failure that can be retried.

Make sure the merge key is actually stable. A weak or non-unique key can create updates that look successful while corrupting the target’s logical grain.

5. Apply the lookback in the source query

Use the lookback window in the source predicate, typically against the partition column.

The bookmark and the lookback solve different problems:

• The bookmark tracks previously seen source objects.

• The lookback checks a bounded historical range for late data.

You need both when files can arrive after their logical event date.

6. Reconcile before committing

Compare source and staged counts before the merge.

At minimum, log and validate:

• Source object count

• Source row count, if available

• Staged row count

• Rejected row count

• Duplicate-key count

• Batch identifier

A nonzero variance should either fail the batch or trigger a defined tolerance rule.

Do not invent tolerance thresholds during an incident.

7. Prevent concurrent runs

Set maximum concurrency to one for pipelines sharing a staging table or target merge path.

Concurrent runs create race conditions around:

• Staging truncation

• Batch identifiers

• Merge order

• Late-arriving data

• Bookmark state

Retry storms are exactly when these bugs appear. Remove concurrency as a variable unless the pipeline was designed for it.

8. Log the state required for recovery

Emit structured logs containing:

• Batch identifier

• Glue run ID

• Source object count

• Staged row count

• Target row count

• Merge start and end time

• Final status

• Lookback window used

When someone asks which rows landed, the answer should be one query away.

9. Set the timeout below the load window

Do not rely on the default Glue timeout.

Set the timeout based on the operational window available for the pipeline. A hung job that consumes the entire window can delay every downstream run.

One late load can become a backlog if the pipeline has no recovery margin.

Recovery Runbook

Create a one-page runbook attached to the job alert.

It should classify at least four states:

1. Failed before staging write
   Confirm staging is empty for the batch. Rerun safely.

2. Failed during staging write
   Confirm the batch’s staged rows. Truncate staging, then rerun.

3. Failed before merge commit
   Confirm the target contains no committed rows for the batch. Rerun after validation.

4. Failed after merge commit
   Confirm target state using the batch identifier. Do not blindly rerun. Reconcile first.

The runbook should include queries for:

• Batch state

• Staged row count

• Target row count

• Duplicate business keys

• Source object list

• Late-arriving files

Fill in tolerance thresholds with the data owner before production. Deciding what variance is acceptable during an incident produces bad decisions.

What to Measure

Track these per pipeline:

• Merge duration as a percentage of total runtime

• Source-to-staging row-count variance

• Manual recoveries per month

• Late arrivals outside the lookback window

• Time from failure alert to confirmed data state

Targets:

• Row-count variance: zero unless explicitly explained

• Manual recovery: zero

• Confirmed data state: under fifteen minutes

• Late arrivals outside the window: rare enough to justify the operating cost

Review late-arrival behavior quarterly. Review failures and recovery time monthly.

Build a Load You Do Not Have to Watch

A production pipeline does not eliminate failure.

It makes failure safe, visible, and recoverable.

With bookmarks, staging, idempotent writes, bounded lookbacks, transactional merges, and structured recovery logs, a mid-load failure becomes an alert and a rerun instead of a data investigation.

A pipeline that only works when nothing goes wrong is not a production pipeline.

It is a demo that has not been tested yet.

Take your three highest-volume incremental loads and classify each against the four failure states above.

Any pipeline where you cannot answer which state a failure leaves you in is your first candidate to rebuild.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

Today they have exactly two services extracted. Both projects are frozen.

The pattern was fine: route traffic through a facade, peel off one capability at a time into its own service, shrink the monolith until there’s nothing left to strangle.

It’s the industry‑standard answer to “how do we decompose this without a rewrite” for a reason: it avoids the big‑bang rewrite that kills most modernization efforts.

But a pattern is not a plan.

• Service #1 was picked because an architect had a strong opinion

• Service #2 was picked because a team volunteered

• Service #3 never got picked because nobody has a reason for one seam over another

The strangler fig didn’t fail. Sequencing did.

Why a Stalled Decomposition Is Worse Than No Decomposition

A monolith that never started decomposing is a known, stable cost.

A monolith that stalled halfway is a compounding one.

• The facade/routing layer is now permanent infrastructure to maintain and debug

• Two teams now own two services plus the monolith = more surface area, not less

• “Independent deployability” exists for a tiny slice of the system, but the overhead is everywhere

The benefits were supposed to justify the cost across the whole estate, not two endpoints.

Worse, a stalled strangler erodes political capital:

• Every quarter without visible progress makes the next planning cycle ask, “Why are we still funding this?”

• A modernization program that can’t show acceleration starts to look like one that should be cut

The biggest risk with strangler fig was never “can we write the code.”
It was “can we keep organizational patience long enough to see compounding benefit.”

Why Sequencing Gets Skipped

Even teams that implement the pattern correctly stall because the literature explains how to cut a seam and says almost nothing about which seam you cut next.

So the default becomes:

• Whoever yells loudest claims the next capability

• Or the team that’s free “volunteers” for some random slice of the monolith

You get one extraction that proves the pattern is technically possible, but teaches you nothing about how to prioritize the other nine seams.

Momentum from the first service doesn’t transfer, because no one defined what made that seam a good choice in a way you can reuse.

Complicating it: most monoliths don’t have neat, obvious seams.

• Some capabilities share tables and code with three others

• Some are already almost independent, and you just haven’t admitted it

Without a scoring framework, teams default to:

• Extracting the easiest seam (often least valuable to isolate), or

• Extracting the most important seam (often the most coupled and riskiest)

Both patterns look busy, and both fail to converge.

The Sequencing Framework: Score Every Seam on 3 Axes

Before extracting anything, list candidate seams and score each one on three axes.

The scores decide the order. Not titles. Not volunteers.

1. Coupling: how entangled is this capability?

• Reads/writes mostly its own tables, with a narrow, clear interface → high score (good early candidate)

• Reaches into 5 shared tables that 3 other capabilities also mutate → low score (leave for later)

Early extractions should teach the team the pattern, not force data‑ownership wars.

2. Change frequency: how often does this code change vs the rest of the monolith?

• Changes weekly, competing for deploy slots and reviews → extracting it pays off immediately in deploy velocity

• Hasn’t changed in 2 years → clean seam or not, business impact of extraction is low

This is where the business case for extraction actually lives.

3. Blast radius: what else breaks if this service misbehaves after extraction?

• Narrow, well‑understood set of downstream dependents → safer to extract early

• Half the monolith calls into it transitively → extract later, when the team has more practice operating the new shape

First services should be low‑coupling, high‑change, low‑blast‑radius:

• Come out cleanly

• Prove value quickly

• Fail safely if something goes wrong

High‑coupling, high‑blast‑radius seams come later, when extraction and incident handling in the new world are muscle memory.

What Sequenced Decomposition Looks Like a Year In

A sequenced strangler doesn’t produce “two services and a stall.” It produces a visible curve.

After a year, you can point to:

• X capabilities now independently deployable

• Y% of changes flowing through services instead of the monolith

• Incidents that used to take down everything now contained to a single service

That curve is what protects the program when budget season comes.

A modernization effort that can show three good extractions in the first two quarters, each easier than the last because the team is getting better at the pattern, doesn’t have to beg to keep going.

It has evidence.

The payoff isn’t just an architecture that eventually decomposes.

It’s a program that never has to re‑justify itself every quarter, because the sequencing is the proof that it’s working.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

Move the workload as‑is, shut down the data center, save money.

The slide compared on‑prem cost to modeled cloud cost. The savings number looked solid. Finance signed.

Six months after cutover, the finance review shows the cloud bill higher than the data center line it replaced.

The migration went exactly as planned. The savings did not.

Nobody lied. The model was just incomplete.

A “lift‑and‑shift” is not a single decision. It is three cost decisions dressed up as one migration choice.

Two of those three almost never get modeled before cutover.

Why This Is a Board‑Level Problem, Not an Engineering Detail

A rehost that costs more than the data center it replaced is not a rounding error. It is a credibility hit.

The program’s ROI story was built on this math:

Per‑workload savings × number of workloads = portfolio business case.

When the first big wave lands 30–40% over estimate, that gap doesn’t stay local to one app. It becomes the number finance repeats the next time you ask for migration budget.

And the bad news shows up late:

• On‑prem: costs are sunk and mostly fixed. Overprovisioning hides in depreciation.

• Cloud: costs are variable and visible. Overprovisioning is a monthly bill someone actually reads.

The same workload that quietly overspent in the data center now triggers a finance review every month, and each review re‑litigates whether the whole migration was a mistake.

Your migration program’s reputation is set by the first few invoices, not the last one.

Why the Estimate and the Invoice Don’t Match

The gap between model and reality almost always traces back to the same three line items.

1. Instance right‑sizing

Most teams size target instances to match provisioned capacity, not observed utilization.

• On‑prem: server sized for a rare peak, used at 20–30% most of the year.

• In cloud: lifted 1:1 to an instance that now bills every hour for that peak capacity.

In the data center, that waste was a sunk CAPEX decision no one revisited. In the cloud, it is an OPEX leak on every invoice.

2. Storage class drift

Lift‑and‑shift tooling defaults to “safe” storage tiers: GP2/GP3 volumes, standard [internal], general‑purpose everything.

• Backups, logs, and archives land on the most expensive reasonable tier.

• Nobody circles back to reclassify them because “optimize storage” loses to “migrate the next wave.”

Cold data keeps paying hot‑data rates indefinitely.

3. Data transfer that never existed as a line item before

On‑prem, traffic between app, DB, and dependencies rides the LAN for free.

In cloud, the same paths can cross:

• AZ boundaries

• Region boundaries

• Out to the internet

Each with its own per‑GB charge that had no equivalent on the data center bill. It never showed up in the baseline, so it never made it into the model.

All three problems come from modeling what the workload is provisioned for instead of what it actually does.

The Three Questions That Catch This Before Cutover

Rehost is still the fastest, lowest‑risk way to move a big portfolio.

These three questions decide whether it’s also a cost win. They need answers before the business case is signed, not after the first invoice lands.

1. What does this workload actually use, not what is it provisioned for?

Pull real utilization:

• CPU, memory, disk, IOPS

• Over a window that includes representative peaks

Size instances to that pattern, with buffer, instead of mirroring the old server spec.

This single change closes most of the gap on most rehosts.

2. What is the access pattern for every dataset this workload touches?

Classify data by access:

• Hot: request path, low‑latency, high‑frequency

• Warm: regular but not critical

• Cold: backups, logs, archives, legal holds

Each class has a different cost‑optimal storage tier. Your model should have per‑dataset storage assumptions, not one blended number.

Defaulting everything to a single “safe” tier is how storage quietly eats the savings.

3. What does this workload talk to, and where do those calls cross billable boundaries?

Map dependencies:

• Databases, caches, downstream services, third‑party APIs

• Which connections stay inside an AZ vs. cross-AZ / region / internet?

This is the line item that has no on‑prem analog, which is exactly why it’s the one most often skipped.

Once you see the paths, you can:

• Co‑locate chatty services

• Avoid unnecessary cross‑region calls

• Decide consciously where to pay for egress

Answer these three and you’ve turned the “savings” slide from a wish into a forecast.

A forecast is the only thing finance should be signing off on.

What a Rehost Looks Like When the Math Holds

When you model these three line items up front, the rehost does what the slide deck promised.

• Right‑sized instances mean you pay for actual consumption, not inherited overprovisioning.

• Correct storage tiering means cold data costs cold‑data rates.

• Pre‑mapped data transfer means you can design away the worst egress paths before they start billing.

The first finance review after cutover becomes validation, not a surprise.

And that changes the trajectory of the entire program:

• Wave 1 shows invoices that match the model.

• Wave 2 budget gets approved off evidence, not optimism.

• The conversation shifts from “Can we trust these numbers?” to “How fast can we scale the factory?”

That is the real payoff: a migration program that can point at real bills and say, “The math holds.”

Free tool: Score your AWS platform’s predictability in 5 minutes
If this hit close to home, you probably have other places where ownership and accountability are fuzzy but invisible until something breaks.

To make this practical, I put together a free AWS Platform Predictability Starter Kit for readers:

• 5‑minute predictability checklist

• “Where are we bleeding?” team scorecard

• Platform risk radar with 12 early‑warning signals

• 10 executive questions with weak vs strong answers + debrief worksheet
  Most leaders run through these in a week of normal meetings and come away with a clear “top 3” to fix next.

👉 Grab the free PDF here: https://thecloudplaybook.gumroad.com/l/aws-platform-predictability-check

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

Someone opens the workload inventory, points at application one, and asks: rehost, replatform, or refactor?

The room debates for forty minutes. They land on an answer.

Then they point at application two, and the debate starts over, because nobody wrote down why they chose what they chose for application one.

Multiply that by 200 applications, and you get a migration program that spends its first quarter in conference rooms instead of moving workloads.

The strategy debate is not the hard part of migration.

It only feels that way because most teams run it as an open discussion instead of a decision tree.

The 6 R’s (rehost, replatform, refactor, repurchase, retain, retire) are not opinions to argue per workload.

They’re the output of four questions, asked in a fixed order.

Skip the order, and you get the meeting that repeats itself 200 times.

Why the Wrong R Costs More Than the Migration

Picking the wrong strategy rarely fails loudly. It fails as a slow leak you notice two quarters later.

Refactor a workload that should have been a simple rehost, and you’ve spent six months of senior engineering time re‑architecting an application nobody asked to modernize. Meanwhile, the business misses a data center exit you could have hit with a lift‑and‑shift.

Rehost a workload that should have been retired, and you’ve paid to move an app three people still use. Once it’s “already migrated,” nobody revisits it. You’ve made a zombie workload immortal.

The cost isn’t just engineering time. It’s factory throughput.

• A factory tuned for rehost velocity chokes on refactor‑sized workloads.

• A factory that treats everything as bespoke refactor never hits the wave‑based timelines the business signed up for.

Wrong R’s compound:

• Your best engineers get stuck on work that didn’t need them.

• The workloads that do need refactor never get the attention or budget.

Migration budgets are approved once, based on an assumed strategy mix. When the mix is wrong, the budget doesn’t stretch. It just runs out before the portfolio is done.

Why Smart Teams Default to Refactor or Rehost

Two patterns explain most bad decisions. Both come from forcing a whole portfolio through one mental model.

1. Modernization bias
   Architects who care about “doing it right” default to refactor, because that’s where the interesting engineering lives. A stable, low‑change, “good enough” app still gets nominated for re‑architecture, not because the business needs it, but because it’s more satisfying than rehosting.

2. Deadline bias
   Teams under a hard exit deadline default everything to rehost because it’s fast and predictable. Workloads that genuinely need refactoring get lifted and shifted in their brittle state, now with cloud spend on top of the technical debt.

Both biases share a root cause: strategy is being chosen from one vantage point (architect or program manager), not from the four questions that actually govern the right R.

The Four Questions That Sort Any Workload

Ask these in order. The order matters. Each question eliminates options before the next one.

1. Does the business still need this workload?
If usage is near zero, or it exists only because no one decommissioned it, retire beats every other option.

This comes first because retirement is the only R that shrinks your migration backlog. Every workload retired here is one the factory never touches.

2. Is there a commercial replacement that does this better?
If the workload is a homegrown system duplicating what a SaaS or managed product already does, repurchase often beats refactor or rehost.

This comes second because it can eliminate big, hairy workloads before you burn cycles estimating them.

3. Does it need to change, or just move?
If requirements, performance, and architecture are “good enough,” rehost or replatform gets you to cloud fastest.

• Rehost: move as‑is when change doesn’t pay for itself.

• Replatform: swap obvious managed services (e.g., self‑managed DB to RDS) when the operational savings justify it.

If there’s a real business reason to change (scalability limiting revenue, licensing that dies in cloud, compliance gaps), that’s the signal for refactor.

4. Does the deadline allow the strategy this workload actually needs?
This is a check, not a chooser.

If 1–3 say “refactor” and the data center exit is in 12 weeks, that conflict gets escalated. The business chooses: move the deadline or consciously downgrade the strategy.

Do not let the schedule silently pick a lesser strategy and pretend nothing changed.

Run every workload through these four and the R mostly picks itself. The 40‑minute debate collapses to four questions, many of which your CMDB already answers.

What a Portfolio Run Through This Tree Looks Like

Programs that apply these four questions consistently get a strategy mix, not a never‑ending strategy debate.

That mix is what the migration factory needs to:

• Plan capacity

• Sequence waves

• Forecast cost and timelines

What changes:

• Retire and repurchase decisions happen before wave planning, shrinking the portfolio.

• Rehost and replatform carry most of the volume on a predictable, assembly‑line cadence.

• Refactor is reserved for the few workloads where the business case genuinely justifies it, staffed with the senior engineers who should be there.

The visible outcome:

• Portfolio reviews that used to run for weeks compress into days.

• Strategy per workload becomes a lookup, not a debate.

• Engineering time that used to be spent re‑litigating strategy goes into moving workloads.

That’s the leadership payoff: not a cuter framework, but a portfolio that stops arguing with itself and starts moving.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

One team saw Secrets Manager’s per‑secret pricing, decided it was expensive, and standardized on Parameter Store for everything. Another standardized on Secrets Manager because it sounded “purpose‑built.” A third never decided at all:

• Database passwords in SSM parameters

• API keys in Secrets Manager

• A few credentials in KMS‑encrypted env vars because that was fastest for one service two years ago

Different routes, same destination: secrets live where habit or a cost comparison put them, not where the secret’s requirements put them.

And those requirements differ: a rotating database password is not the same as a static third‑party API key, which is not the same as a config value that’s merely sensitive.

Optimizing for the cheapest monthly line item while ignoring rotation, audit, and incident response is optimizing the wrong variable.

Why the Wrong Store Is a Real Cost

A secret in the wrong store is not a billing issue. It is:

• A rotation problem

• An audit problem

• An incident‑response problem waiting for its moment

Rotation cost:
A database credential in plain Parameter Store does not rotate itself. Someone has to remember. They don’t. A credential that should live 30 days lives 2 years. When it leaks, the blast radius is 2 years of access, not 30 days.

Audit cost:
Compliance frameworks want proof: scheduled rotation, scoped access, logged reads. A secret in the wrong store cannot show that history. The auditor asks for rotation records, and there are none, because that store doesn’t rotate.

Incident cost:
When a credential leaks, the first questions are:

• How old is it?

• What can it reach?

• How fast can we rotate or revoke it?

The store built for rotation answers quickly. The store chosen for price does not, and the incident stretches from hours into days.

The monthly price delta between stores is small.
The cost of a credential you cannot rotate, audit, or quickly revoke is very high, and shows up exactly when you can least afford it.

How Teams End Up With Secrets Scattered

Sprawl is not usually a decision. It’s the absence of one.

• The first secret goes “wherever” because one engineer had to ship one service.

• The second secret goes somewhere else because another engineer made a different local call.

• Neither choice is crazy in isolation. Together, they are the start of chaos.

Then cost anxiety accelerates it:

• Secrets Manager’s per‑secret line is visible on the bill

• Parameter Store’s usage is fuzzier

• A cost review flags Secrets Manager

• The team moves secrets to the “cheaper” store without checking whether it meets the secrets’ requirements

The bill goes down. Rotation and audit capability go with it.

This persists because nobody owns the decision. There’s a standard for compute, a standard for networking, and no standard for secrets. So the question “where does this go?” gets answered per engineer, per service, per deadline.

Sprawl is the predictable result of a decision nobody was assigned to make.

The Axes That Actually Decide

The “right” store is not about preference or price. It’s about four properties of the secret itself. Score the secret on these, and the store largely selects itself.

1. Does it need automatic rotation?
   This is the first cut.

   • Must‑rotate (e.g. most DB credentials, many service credentials) → Secrets Manager, with native rotation where possible

   • Does‑not‑rotate (static third‑party key, certain config values) → does not need to pay for rotation it won’t use

2. How sensitive is it and what’s the blast radius if it leaks?

   • High‑blast‑radius (prod DB with customer data) → strongest access controls, logging, and rotation

   • Lower sensitivity (non‑public but low‑impact config) → can live in a simpler store

3. How often is it read, and by what?
   Read pattern affects both architecture and cost:

   • Read on every request by a high‑traffic service → needs caching and cost awareness

   • Read once at startup → throughput less relevant

   Parameter Store and Secrets Manager behave differently under load; the wrong choice here is either a latency issue or a surprise bill.

4. What’s its compliance scope?
   In‑scope for FedRAMP / HIPAA / ISO? Then you may be forced into:

   • Demonstrable rotation

   • Scoped IAM

   • Auditable access history

   “We cannot demonstrate rotation” is a finding all by itself.

KMS‑encrypted environment variables sit at the bottom of this matrix on purpose. They are right for a narrow case:

• Value needed only at process start

• Not sensitive enough to justify a lookup per read

• Does not rotate

They are wrong for almost everything else. When secrets land here, it’s because no one applied the matrix at all.

What a Standard Changes

Teams that define where each class of secret belongs stop putting secrets in the wrong place.

• Rotating database credentials live where rotation is native, so they actually rotate, and a leak’s blast radius is 30 days, not 2 years.

• Static API keys live where they aren’t paying for rotation they don’t need.

• High‑sensitivity secrets live where access is tightly scoped and logged, so audits pull reports instead of doing archaeology.

• Low‑risk config values live in the simple store, so the expensive store is reserved for secrets that justify it.

When a credential leaks, the response is fast because the store was chosen for that risk class. The team knows its age, reach, and rotation path because the store was designed to answer those questions.

Secrets stop being scattered by accident and start being placed by design. The store follows the secret’s requirements, and the requirements were explicitly examined instead of assumed.

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

The console’s org‑wide enablement is a smooth process because AWS paper-overs the underlying complexity. Terraform does not. When you enable GuardDuty across an organization with Terraform, you run into reality:

• GuardDuty is regional

• Delegated administration has strict ordering

• Auto‑enable behaves differently for existing vs new accounts

• Member association is not the same as feature enablement

Teams that don’t know this up front get a rollout that half‑works:

• GuardDuty on in one region, missing in three

• New accounts get it, existing ones don’t (or the reverse)

• Delegated administrator set, member accounts not associated

Every one of these is a coverage gap that appears to be a success in the one dashboard you happen to be staring at.

This issue is the map of those gotchas and the code structure that avoids them.

Why the Gotchas Are Expensive

A half‑enabled detection system is more dangerous than an un‑enabled one, because it creates false confidence.

The team believes GuardDuty is running org‑wide. The audit evidence says it is enabled. But coverage has holes:

• A region never turned on

• Accounts that predate the auto‑enable setting

• A feature enabled in the delegated administrator but never propagated

The threat that lands in one of those holes is undetected, and nobody knows the hole exists until an incident review finds it. Discovering it after the fact means a multi‑week backfill project instead of a plan and an apply.

The gotchas are cheap to handle before rollout and expensive after. Doing this as code only pays off if the code encodes the handling so the gaps never open.

How the Gaps Open

These coverage gaps are not due to carelessness. They are the predictable result of GuardDuty’s architecture meeting an org‑wide rollout.

• Regional service. GuardDuty is regional. Enabling it is a per‑region action. Teams think in terms of “the organization,” focus on accounts, and then cover only their primary region while silently omitting the others.

• Auto‑enable subtlety. Auto‑enable controls whether new member accounts are enabled for GuardDuty. It does not retroactively turn it on for existing accounts. Teams that set auto‑enable and assume full coverage miss every pre‑existing account.

• Feature enablement vs association. Associating a member account with the delegated admin brings findings into view. It does not automatically enable additional protection features. Teams that associate all accounts and assume malware protection is on have coverage associated with accounts, not feature coverage.

The console flow hides most of this. Terraform surfaces it. That’s a feature, not a bug: the code forces explicit decisions, and explicit decisions do not turn into silent gaps.

The Decision Framework and the Gotchas

Structuring Terraform correctly requires resolving four things. Each has a gotcha.

Read more

Most teams do the same thing:

• Someone in the security account enables GuardDuty

• Delegated admin is set

• Auto‑enable is turned on for all member accounts

Now the estate has threat detection “everywhere.” The dashboard fills with findings. Everyone feels safer.

Six months later, those findings land in an inbox nobody reads. The volume is high, the severity is mixed, and nobody has decided who acts on what.

GuardDuty is generating thousands of dollars in value from detection and zero dollars in value from response, because the response side was never designed.

GuardDuty is not the failure. GuardDuty works.
The failure is enabling a detection system without first deciding what happens when it detects something.

Why the “Enable” Click Has a Long Tail

GuardDuty is easy to turn on and expensive to turn on incorrectly. The expense is delayed enough that the connection is easy to miss.

You pay in two currencies:

1. Dollars

GuardDuty pricing scales with the volume of events it analyzes. Volume‑heavy sources like:

• S3 data events

• EKS audit logs

• Malware protection

• RDS protection

Can multiply the bill in ways that surprise teams who enabled everything by default. An org‑wide enablement with every feature on, across a large estate, can produce a monthly bill that triggers a finance conversation nobody planned for.

2. Attention (the more damaging one)

A detection system that produces more findings than the team can triage trains the team to ignore them.

The high‑severity finding that matters arrives in the same flood as hundreds of low‑severity informational findings, and it gets missed.

An ignored detection system is worse than no detection system because it creates the appearance of coverage without the substance.

Both costs are set by decisions made before the enable, not after. That makes this a pre‑flight problem.

How Good Teams Still Enable It Wrong

The wrong‑way enable is not negligence. It’s a reasonable response to how the feature presents itself.

GuardDuty’s setup flow makes the easy path the default path:

• Enable

• Delegate

• Auto‑enable for all accounts

• Done

The flow does not ask:

• Who will read the findings?

• What will this cost at your current event volume?

• How does a critical finding reach a human with a pager?

It just turns detection on, and that feels like the responsible thing to do.

Security teams are under pressure to show coverage.

“Is GuardDuty enabled?” is a question for an auditor and a customer.
The fastest way to answer “yes” is the one‑click org‑wide enable.

The box measures whether detection is on, not whether the response works.

So detection gets enabled comprehensively, and response is never designed. Findings accumulate. Volume becomes noise. The team that enabled a security control to reduce risk has added an alerting channel that everyone learns to mute.

The gap between “enabled” and “operational” is invisible until an incident falls into it.

Four Decisions to Make Before You Enable Org‑Wide

GuardDuty org‑wide is worth doing. It’s worth doing after four decisions, not before.

Each one determines whether the system produces a signal or noise.

1. Who is the delegated administrator?

GuardDuty findings from every member account aggregate to a delegated administrator account.

That should be your security account, not the management account:

• Security tooling lives where the security team operates

• The management account stays minimal and boring

Choosing the wrong one is painful to reverse once findings are flowing.

2. What will each feature cost at your scale?

Foundational GuardDuty is one cost. The add‑ons are separate:

• S3 protection

• EKS audit log monitoring

• Malware protection

• RDS protection

Each scale with your actual usage patterns.

Before enabling, estimate:

• Monthly cost per feature

• Against your current log and data volumes

Then enable the features whose value justifies their cost. Not “all of them because they’re available.”

3. Where do findings go, and who owns each severity?

This is the decision that separates signal from noise.

Design this explicitly:

• High: pages a human (on‑call rotation/incident channel)

• Medium: goes to a queue where someone actually works (ticket system, triage channel)

• Low / Informational: logged for forensic use, not alerted

Without this routing, every finding goes to the same destination with the same urgency. The important ones drown.

4. What is your plan for the initial flood?

The moment you enable GuardDuty org‑wide, it surfaces findings for existing conditions. The first days produce a backlog, not a steady state.

Decide in advance:

• Who owns triage for the initial flood

• How you’ll distinguish pre‑existing conditions from new threats

• Which classes of “old but still present” findings will trigger immediate work vs backlog remediation

That keeps the launch from overwhelming the team on day one.

Enabling detection is a five‑minute decision. Designing a response is the actual work.
Do the actual work first.

A detection system is only as valuable as the response it triggers.

What Changes When Response Is Designed First

Teams that make these decisions before enabling get a security control that works, not a dashboard that decorates.

• Findings are routed by severity: critical ones go to a human, and informational ones go to a log.

• The team trusts alerts because alerts are calibrated, and trusted alerts are acted on.

• The monthly bill is a number someone chose, not a surprise that escalates.

When an auditor asks about threat detection, the answer isn’t just “it’s enabled.”

The answer is: “It’s enabled, findings route to these owners at these severities, and here is our mean time to acknowledge a high‑severity finding.”

That’s the difference between a checkbox and a capability.

GuardDuty stops being a system that generates findings nobody reads and becomes a system that generates responses that contain threats.

Detection was always the easy part.
Designing the response before you click enable is what makes the detection matter.

Coming Wednesday for paid subscribers

On Wednesday, paid subscribers get the full system for enabling GuardDuty org‑wide with Terraform, including the gotchas:

• Delegated administrator setup

• Per‑feature enablement decisions encoded as code

• Severity‑based routing and alerting design

• Initial‑flood triage plan

• The Terraform and org‑structure edge cases that break an org‑wide enable

If you want GuardDuty to be a control your team runs, not a dashboard you ignore, that’s the operating model.

Upgrade If You Need Implementation, Not Just Ideas

If you’re using these emails to guide real decisions on your platform, you’ll get more leverage from the paid version of The Cloud Playbook.

The free newsletter gives you patterns and language.

The paid newsletter turns those patterns into implementation kits you can ship inside a quarter:

• Concrete rollout plans (90‑day roadmaps for each pattern)

• Templates and checklists (policies, runbooks, tagging schemes, review checklists)

• Real examples from high‑stakes AWS environments (what we actually shipped and why)

If the paid side doesn’t save you more than the subscription in one incident, audit cycle, or bad migration you avoid, you should cancel and keep the playbooks.

Upgrade to the Paid Cloud Playbook

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

Then someone widened a permission set in the console during an incident, and nobody knows.

The permission set, which was scoped to read-only, now has write access because an engineer needed it at 2 a.m., and the console was right there. The change is live in every account to which the permission set is assigned. There is no pull request, no reviewer, no record of why. The access review three months from now will find it, or it will not.

Centralizing identity solved the problem of credential sprawl. It did not solve the sprawl of permissions inside the centralized system.

Why Console-Edited Permission Sets Undo the Work

The value of IAM Identity Center lies in the fact that access is defined in one place. That value evaporates the moment the one place is edited by hand.

A permission set is not a small object. It is a bundle of policies that applies to potentially dozens of accounts at once. A single console edit to a widely assigned permission set is one of the highest-blast-radius changes available in your entire AWS estate, and it is available to anyone with console access to the management account.

The cost is the same as that of any unversioned control. At audit time, you cannot show the history of who changed what and why. During an incident, you cannot tell whether a permission set is in its intended state or in the state someone left it months ago. And the privilege-escalation review that your compliance framework requires becomes impossible, because the thing you are reviewing changes underneath you without a trace.

You did the work to centralize. Managing permission sets by hand quietly gives it back.

How Teams Backslide After Centralizing

The backslide is not a decision. It is the path of least resistance reasserting itself.

Read more

Someone enables a guardrail during setup. Someone else enables three more during an audit scramble. A year later, the landing zone has 40 controls applied across five OUs, and no one can tell you why any single control is on, who turned it on, or what breaks if it comes off.

The controls work. They deny the actions they are supposed to deny. But the configuration lives in the console, not in a repository, and that means it cannot be reviewed, diffed, rolled back, or reasoned about as a system.

This is the gap between having guardrails and operating them.

Why Console-Managed Controls Become a Liability

Control Tower controls are the enforcement layer of your landing zone. They are also, for most teams, the least version-controlled part of the entire AWS estate.

The cost shows up at audit time. An auditor asks which controls enforce a given SOC 2 requirement, and the team produces a screenshot instead of a commit. The evidence is a point-in-time snapshot with no history, no author, and no justification.

The cost also shows up during incidents. A deploy fails because a proactive control blocks it. The engineer does not know the control exists, cannot find where it is configured, and cannot tell whether disabling it is safe. The control meant to prevent a mistake now causes an outage in developer velocity instead.

Controls managed by clicking are controls that drift. Someone disables one to unblock a launch and never re-enables it. The guardrail that your compliance posture depends on is now off, and nothing in your system knows.

How Smart Teams End Up Here

No one decides to manage controls by hand. The console path is simply the fastest way to unblock the immediate need.

Control Tower’s own interface encourages it. The landing zone is set up through the console, so the first controls are enabled there. The pattern is set before anyone asks whether it should be code.

Then the team scales. More OUs, more accounts, more frameworks in scope. Each new requirement adds controls, and each addition happens where the last one did: in the console, under deadline, by whoever is closest to the problem.

The team that would never deploy a Lambda function without a repository is managing the enforcement layer of its entire compliance posture with mouse clicks. The inconsistency is invisible because controls feel like configuration rather than infrastructure. They are infrastructure. They deny real actions in real accounts, and they belong under the same discipline as everything else you deploy.

The Three Control Types and How to Manage Each

Control Tower exposes three control types, and the controls-as-code decision varies by type.

1. Preventive controls are Service Control Policies. They stop an action before it happens: deny disabling CloudTrail, deny resource creation outside approved regions. These are the highest-value controls to manage as code because they are the ones an incident is most likely to touch. Manage them in Terraform against the OU, with the policy body in the repository so the exact denied actions are reviewable.

2. Detective controls are AWS Config rules. They evaluate resources after creation and flag non-compliance, such as an S3 bucket without encryption or a security group with open ingress. Manage these as code so the rule set is consistent across all accounts, and so that a new framework requirement becomes a pull request, not a console session.

3. Proactive controls are CloudFormation Guard hooks. They evaluate a resource before provisioning and block deployment if the evaluation fails. These are the controls most likely to surprise a developer, so managing them as code matters most for a different reason: the repository is where an engineer looks to understand why their deployment was blocked.

The axis to weigh is the blast radius against reversibility. A preventive control at the org root has the widest blast radius and the slowest reversal. It gets the most reviews. A detective control in a single sandbox OU has a narrow blast radius and instant reversal. It gets less. Match the review weight to the control’s reach, and do not apply the same ceremony to every control regardless of consequence.

The second axis is framework mapping. Every control should trace to a named requirement. A control that enforces nothing anyone can name is a control that will eventually break a deploy for a reason no one can defend.

Implementation Steps

Deploy controls as code in this order. The sequence matters because early mistakes are the expensive ones.

1. Inventory the current state first. Before writing any Terraform, export every control currently enabled, the OU it targets, and its type. This is your baseline. You cannot manage as code what you have not first written down.

2. Map each control to a requirement. For every control in the inventory, name the framework requirement or internal policy it enforces. Controls that map to nothing are candidates for removal, not import.

3. Import, do not recreate. Bring existing controls into Terraform state using import blocks rather than destroying and recreating them. Recreating a preventive control means there is a window when the guardrail is off. Import avoids the gap.

4. Structure the repository by OU and type. One module per OU, controls grouped by preventive, detective, and proactive. The structure should allow a reviewer to see all controls for a given OU in one place.

5. Enforce review by blast radius. Controls at the org root or a production OU require a two-person review. Controls on a sandbox OU can ship with one. Encode this in the pull request rules, not in a wiki no one reads.

6. Add drift detection. Run a scheduled plan against the control configuration. Any drift between the repository and the deployed state pages the owner. This is the check that catches the disabled-and-forgotten control before an auditor does.

7. Stage rollout through OU progression. A new control lands on the sandbox OU first, then a non-production OU, then production. Never apply a new preventive control to the org root as its first deployment.

What to Measure

Controls as code is a system, and the system should be measured.

1. Control coverage. The percentage of enabled controls that exist in the repository versus the console. Track this weekly during migration. The target is 100 percent, and anything managed in the console after migration is a regression.

2. Drift events per month. The count of times the deployed control state diverged from the repository. A healthy landing zone trends toward zero. Persistent drift signals that someone is still editing controls by hand.

3. Time-to-evidence for a control question. How long does it take to answer “which controls enforce this requirement” during an audit? With controls as code, this repository query takes minutes to run. An above-an-hour means the framework mapping is incomplete.

Review these in the platform's monthly operating review. Coverage and drift are the two metrics that indicate whether the enforcement layer is under control.

What Changes When Controls Are Code

The landing zone stops being a black box.

Every guardrail has an author, a justification, and a history. The audit question that used to take a day of screenshotting becomes a repository search. The control that blocks a deploy is documented where the engineer already looks. The disabled-and-forgotten guardrail is caught by drift detection within a day, rather than surfacing at the next audit.

The enforcement layer of your compliance posture becomes a system you operate, not a configuration you hope is still correct.

Upgrade If You Need Implementation, Not Just Ideas

If you’re using these emails to guide real decisions on your platform, you’ll get more leverage from the paid version of The Cloud Playbook.

The free newsletter gives you patterns and language.

The paid newsletter turns those patterns into implementation kits you can ship inside a quarter:

• Concrete rollout plans (90‑day roadmaps for each pattern)

• Templates and checklists (policies, runbooks, tagging schemes, review checklists)

• Real examples from high‑stakes AWS environments (what we actually shipped and why)

If the paid side doesn’t save you more than the subscription in one incident, audit cycle, or bad migration you avoid, you should cancel and keep the playbooks.

Upgrade to the Paid Cloud Playbook

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a reading topic, please feel free to contact me directly via LinkedIn or X.

The reference architectures are not wrong. They are incomplete in a specific way: they describe what the design looks like, not what the team commits to in order for the design to stay intact 18 months from now.

A checklist closes that gap. It explicitly names each design decision, the team that owns it, the verification cadence that prevents drift, and the artifact that proves the design is still in effect. The checklist does not produce a better design than the reference architecture. It produces a design that survives the conditions that erode reference architectures: org-chart changes, scope expansion, deadline pressure, and the silent drift that comes from boundaries without owners.

The checklist below is the one I would use with a regulated SaaS platform team.

It assumes SOC 2 Type II at a minimum, with a roadmap toward FedRAMP Moderate, HIPAA, or ISO 27001. The structure applies even if the team operates outside those frameworks; the cadences and evidence requirements are tighter for regulated workloads.

How to Use the Checklist

The checklist has seven sections. Each section contains design items, owners, verification cadences, and required artifacts.

For each item, the team commits to four things:

Read more

The single-account deployment was sprawling. The blast radius was unacceptable. Compliance was difficult to scope.

The team split workloads across accounts: production, staging, and dev. Then a separate account for the data team. Then accounts for individual customer tenants. Then a logging account. Then a security tooling account.

By year three, the organization has 40 accounts. AWS Organizations holds the top of the tree. Control Tower, if the team adopts it, holds the bottom. The middle is unclear.

The accounts work. Workloads run in them. Bills are paid. But the boundaries between accounts, the rules about what crosses what and who decides, are owned by nobody.

When something goes wrong, the cost is visible. The data engineer cannot get cross-account access to the bucket they need. The compliance team cannot find the audit log for an action in the production account. The platform team discovers that three accounts have public S3 buckets nobody owns. The security team flags an IAM role assumption pattern that has been running for 14 months without review.

The accounts are not the problem. The boundaries between them are. And boundaries without owners produce the same outcomes as no boundaries at all.

Why Account Sprawl Happens

Multi-account architectures rarely start sprawling. They become sprawling because account creation is rewarded and account boundaries are not.

The team that needs isolation creates an account. Provisioning is fast; the security team approves it, and the workload comes in. The team is unblocked. Nobody asks who owns the account in 18 months, when the workload changes shape, who decides which other accounts can call into it, who reviews the IAM policies written into it, or who confirms that logging is exported to the central account.

The account becomes orphaned at the moment it is provisioned. The team that owns the workload owns the workload. The platform team owns the foundation. Nobody owns the account.

Multiply this by 18 months of provisioning decisions. The org chart shifts twice. The platform team turns over a third of its engineers. The compliance scope expands to a new framework. The original team that requested the account no longer remembers why they needed it.

The boundary that was clear at creation time is now ambiguous. The account is still running. The workload inside it is operationally healthy. The boundary between this account and the others is undefined, and the security team has no record of who would answer questions about it.

What Boundary Ownership Actually Means

Boundary ownership is not the same as account ownership.

The team that operates a workload owns the workload. They are responsible for the application, deployment, on-call rotation, and cost. This is straightforward, and most organizations get it right.

Boundary ownership is the layer above. It governs the rules about what crosses the account, in either direction. Specifically:

• Identity boundary. Who can assume IAM roles in this account from outside, and what those roles can do? Who can assume IAM roles into other accounts from this one, and what those roles can do. Trust relationships, condition keys, session policies.

• Network boundary. What VPC peering, Transit Gateway attachments, or PrivateLink endpoints exist into and out of this account? What CIDR ranges are routable? What ports are open at the security group level for cross-account traffic?

• Data boundary. What S3 buckets, KMS keys, or other resources in this account can be accessed from other accounts? What buckets, keys, or resources in other accounts can this account access? Resource policies, bucket policies, KMS key policies.

• Audit boundary. What CloudTrail, Config, GuardDuty, and Security Hub data flows out of this account into the central audit account? Whether the flow is enforced, whether it is verified, and who reviews it.

• Cost boundary. Which cost center is the spend in this account allocated to? Whether the allocation is automatic via tagging policy or manual via accounting reconciliation. Who reviews variance?

Each of these boundaries has a different ownership profile. The team that runs the workload does not own the identity boundary. The platform team does. The compliance team does not own the network boundary. The networking team does. When ownership is explicitly named for each boundary, the team has a forwarding address for boundary-related questions. When ownership is implicit, the questions sit in nobody’s queue.

The Three Failure Patterns of Boundary Drift

Boundary ownership ambiguity produces three predictable failure patterns.

1. The shadow trust relationship. A workload in account A needs to read a bucket in account B. The team writes a bucket policy. The team writes a role with the right permissions. The role works. Nobody flags that the bucket policy now grants cross-account access. Six months later, a security review discovers the policy and cannot trace why it was written or whether it is still needed. Removing it might break the workload. Leaving it leaves an undocumented trust path. The team chooses to leave it. The trust path stays.

2. The audit gap. A new account is provisioned. The team configures the workload. The team forgets to enable CloudTrail forwarding to the central audit account. Eleven months later, the audit team is collecting evidence for SOC 2 Type II and discovers that the account's CloudTrail data has been missing for 11 months. The audit finding is significant. The remediation is a one-line configuration change that should have been made on day zero. Nobody owned the audit boundary, so nobody verified it.

3. The networking debt. Each new account is given a Transit Gateway attachment. The attachments accumulate. The CIDR ranges overlap or nearly overlap. The platform team does not realize this until a new account cannot peer because the CIDR space is exhausted. The fix is a re-IP project that takes a quarter. The cause was a boundary nobody was reviewing as accounts were created.

These patterns are not specific to one organization. They are predictable outcomes of the same root cause: boundaries without explicit owners drift, and drift compounds.

Four Questions That Surface Whether Your Boundaries Have Owners

Before the next account is provisioned, the platform team should be able to answer four questions in writing. If they cannot, boundary ownership is the work that needs to come before more accounts.

1. For each existing account, who owns the identity boundary? The named individual or team that approves cross-account role assumptions, reviews trust policies, and decides when a role can be assumed from a new principal. If the answer is “the platform team” without a specific person and a documented review cadence, the answer is no.

2. For each existing account, who owns the network boundary? The named individual or team that approves VPC peering, Transit Gateway attachments, and PrivateLink endpoints. The team that maintains the CIDR allocation registry. If the team cannot produce the registry, the boundary is unowned.

3. For each existing account, who verifies that audit data is flowing to the central account? Not configured. Verified. The difference is monthly: configuration drifts, forwarding breaks, and retention policies change. The team that owns audit boundary verification runs a monthly check. If no team runs the check, the audit boundary is unowned.

4. When a new account is provisioned, what is the first action that establishes ownership of each boundary? A defined process that names the identity boundary owner, the network boundary owner, the audit boundary verification owner, and the cost boundary owner before the account is handed to the workload team. If account provisioning does not include this step, every new account starts unowned.

These four questions are diagnostic. The honest answer reveals which boundaries are owned, which are partially owned, and which are not owned at all. Most organizations discover that one or two of the four are well known, while the others are not.

What Changes When Boundaries Have Owners

When boundary ownership is explicitly named, the failure patterns above stop occurring.

The shadow trust relationship is caught at creation. The identity boundary owner reviews the trust policy as part of the workload onboarding. Cross-account access has a documented justification, a documented review cadence, and a removal trigger if the workload changes.

The audit gap is caught within 30 days. The audit boundary owner runs a monthly verification. Configuration drift is detected before it becomes an audit finding. The CloudTrail forwarding question is answered before the auditor asks.

The networking debt is caught at the design phase. The network boundary owner maintains the CIDR registry and reviews each new attachment against the existing topology. Re-IP projects do not become quarter-long initiatives because the topology was managed instead of accumulated.

The platform team’s relationship to multi-account architecture changes. The team is no longer reactive to drift. The team is proactive about the boundary that prevents drift. The accounts continue to multiply as the workload portfolio grows. The boundaries stay coherent because they are owned.

Why This Is the Right Conversation Now

The teams most exposed to boundary drift are SaaS organizations expanding into regulated customer segments.

A SOC 2 Type II audit produces findings on boundary drift. A FedRAMP authorization fails on it. A HIPAA business associate agreement requires the organization to demonstrate control over the boundary, not just over the workload. A customer requesting tenant isolation in a separate account requires the organization to have a defensible model for what crosses the account boundary and what does not.

The cost of unowned boundaries is invisible until one of these events. By that point, the cost is high.

Teams that name boundary ownership before the next account is provisioned do not incur these costs in the same way. The boundary is owned. The boundary is reviewed. The boundary survives org-chart changes because it is documented. The audit, the customer, and the framework all encounter a defended boundary, not a boundary that has been drifting for two years.

On Wednesday, paid subscribers get the AWS Multi-Account Design Checklist for regulated SaaS platforms. The checklist covers account strategy, logging, identity, networking, tenant isolation, shared services, and audit evidence. Each section names the explicit owner, the verification cadence, and the artifact that confirms the boundary is intact. Use it to baseline your existing accounts before the next audit, or as the design document before the next account is provisioned.

Upgrade If You Need Implementation, Not Just Ideas

If you’re using these emails to guide real decisions on your platform, you’ll get more leverage from the paid version of The Cloud Playbook.

The free newsletter gives you patterns and language.

The paid newsletter turns those patterns into implementation kits you can ship inside a quarter:

• Concrete rollout plans (90‑day roadmaps for each pattern)

• Templates and checklists (policies, runbooks, tagging schemes, review checklists)

• Real examples from high‑stakes AWS environments (what we actually shipped and why)

If the paid side doesn’t save you more than the subscription cost in a single incident, audit cycle, or bad migration you avoid, you should cancel and keep the playbooks.

Upgrade to the Paid Cloud Playbook

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.

The conversation produces an outcome. It does not produce a defensible record. Six months later, when a similar workload arrives, the next team has the same conversation. There is no organizational memory. There is no consistency across teams. The compute estate fragments.

A scoring framework solves this. The framework does not eliminate judgment. It structures judgment so that the same workload produces the same recommendation regardless of which engineer scores it. It also produces a written record that future teams can reference and that leadership can audit when the cost trajectory deviates from the forecast.

The framework below is the one I would use with a real engineering team. It is calibrated for SaaS workloads, weighted for the dimensions that matter at 18-month horizons, and written at a level of detail that can be applied without further interpretation.

Why Most Frameworks Fail

Frameworks fail in three predictable ways.

1. Too many dimensions. The team scores 12 attributes. The scoring takes three hours. The team abandons the framework after the second use because it feels like overhead.

2. Wrong dimensions. The framework scores attributes that do not predict success. Language support, AWS region availability, and console UX. These attributes feel important, but rarely change the recommendation.

3. No threshold rules. The framework produces a score for each option, but no rule for what the score means. Three options score within 10 percent of each other. The team argues again. The framework produced no decision support.

The framework below is calibrated against these failure modes. Five dimensions, weighted by 18-month impact, with explicit threshold rules that produce a recommendation rather than a debate.

Read more

Lambda for the team that wants serverless.

ECS for the team that prefers containers.

EKS for the team that wants Kubernetes.

The conversation runs for an hour, the team picks the option that fits how it already works, and the service ships.

Eighteen months later, the choice is the most consequential architectural commitment the team has made. It governs cost, on-call load, hiring, audit posture, deployment cadence, and the team’s ability to take on the next workload type. It is harder to change than the database choice and almost as hard as changing the account topology.

The teams that get this right do not pick based on preference.

They pick based on five dimensions that determine whether the choice will compound value or compound debt: team maturity, operational burden, latency profile, tenancy model, and compliance exposure.

The teams that get this wrong pick based on what they already know.

Why Familiarity Is the Wrong Decision Criterion

Familiarity is the default tiebreaker, and it is the wrong one.

The argument for familiarity is operational. The team that picks the technology it knows ships faster, debugs faster, and produces fewer incidents in the first six months. This is correct. It is also short-sighted.

The argument against familiarity is structural. Most engineering teams will make compute choices for workloads that grow in volume, expand in tenancy, and acquire new compliance scope. Familiarity at year zero produces operational comfort. By year two, the workload no longer matches the compute model the team chose, and the cost of the mismatch is paid by every team that runs on the same platform.

The teams that pick on familiarity end up with three compute platforms five years later, each one chosen by a different team for a different workload, each one operationally expensive. The platform team inherits the burden of supporting all three.

The Five Dimensions That Should Drive the Choice

Compute selection is a multi-dimensional decision. The five dimensions below yield recommendations that differ from those of the familiarity heuristic.

1. Team maturity

   Lambda is the lowest operational burden of the three. ECS is moderate. EKS is high. The team’s operational maturity does not need to match the platform’s complexity at year zero, but it needs to match within 18 months. A team with no Kubernetes experience that picks EKS will be operating an architecture they cannot debug for two quarters. A team with deep container experience that picks Lambda will hit cold-start, package size, and orchestration limits within a year.

2. Operational burden

   Lambda offloads scaling, patching, and runtime management to AWS. ECS Fargate offloads node management, but the team still owns container lifecycle. EKS carries almost all of the operational burden with the team. The right question is not “which is easier.” The right question is “what fraction of the team’s capacity can be dedicated to running this platform six months from now?” If the answer is 0 percent, then the choice is between Lambda and Fargate. If the answer is 30 percent, EKS is on the table.

3. Latency profile

   Lambda has cold-start variance, which can be addressed but never eliminated. The variance is incompatible with sub-100ms P99 SLOs for low-volume services. ECS and EKS have warm capacity by default. For latency-critical workloads, the cold-start question is not theoretical. It is the deciding factor.

4. Tenancy model

   Lambda is shared-tenant by default, and tenant isolation is operationally awkward. ECS and EKS support tenant isolation through cluster topology, namespace boundaries, and IAM scope. For SaaS workloads that will need per-tenant isolation in the future, the tenancy commitment made today shapes whether that future is achievable.

5. Compliance exposure

   All three services support common compliance frameworks. The differences are in the operational effort required to produce evidence. Lambda’s logging and configuration boundary is the simplest. EKS produces the most evidence but also requires the most work to properly configure it. The compliance frameworks the team will face in the next 24 months should drive this dimension.

Why Lambda Is Misused

Lambda is the most misused compute service in AWS. The misuse is not technical. It is economic.

Lambda is priced per invocation and per millisecond of execution. At low volume, this is dramatically cheaper than running a container fleet. At high volume, it is dramatically more expensive. The cost crossover varies by workload, but for typical request-response services, it sits somewhere between 100 and 500 RPS sustained. Below the crossover, Lambda saves money. Above the crossover, Lambda costs 2 to 5 times as much as an equivalent ECS or EKS deployment.

Most teams choose Lambda based on their current volume. They forecast linearly. They do not model the workload at the volume it will reach in 18 months. By the time the cost trajectory becomes visible in the AWS bill, the architecture is load-bearing, and the migration cost is high.

The Lambda choice is not wrong. The Lambda choice without a written cost trajectory at 10x current volume is wrong.

Why EKS Is Often the Wrong Default

EKS has become the default container choice for engineering teams that want maximum flexibility. The argument is reasonable: Kubernetes is the broadest ecosystem, the most portable, the most powerful.

The argument also misses two structural realities.

First, the operational burden is not optional. EKS requires the team to operate cluster networking, ingress, pod scheduling, node lifecycle, secrets management, observability instrumentation, and security boundaries. AWS manages the control plane. The team manages everything else. The team that picks EKS is committing to roughly 0.5 to 1.0 FTE of platform work that does not directly produce product value.

Second, the flexibility is rarely used. Most workloads that pick EKS do not use the features that justify the operational burden. They run a deployment, a service, and an ingress. ECS Fargate would run the same workload with a fraction of the operational overhead and no portability cost; the team would actually exercise.

EKS is the right choice when the team has a portfolio of workloads that justify the platform investment, a strong commitment to Kubernetes patterns across services, or a specific feature, such as custom schedulers, that ECS does not support. It is the wrong choice for the team to pick it because the rest of the industry is using it.

Why ECS Is Underestimated

ECS does not have the brand prestige of Kubernetes. It lacks the elegance of Lambda. It has the property that matters more than either: it requires the least operational burden for the broadest range of workloads.

ECS Fargate runs containers without node management. It integrates with the rest of AWS through native primitives rather than abstractions. It produces logs, metrics, and traces through standard AWS instrumentation. It has been generally available longer than EKS and is operationally mature.

For most SaaS workloads, ECS Fargate is the correct default. It supports latency profiles, tenancy models, and compliance frameworks. It does not require the team to operate a Kubernetes cluster. It does not lock the team into Lambda’s pricing trajectory.

The teams that build on ECS as the default and reach for Lambda or EKS only when the workload demands it produce platforms with lower operational burden and better cost trajectories than the teams that pick on preference.

Four Questions That Drive the Right Choice

Before the team picks a compute service, four questions should be answered in writing. The questions take an hour. They prevent the wrong choice from compounding for years.

1. What is the team’s operational capacity for this workload at month 12

   Not month zero. The team has Slack at month zero because the workload is new. By month 12, the workload is one of several. If the team will not have 30 percent of an FTE available to run a Kubernetes cluster, EKS is not on the table.

2. At 10x current volume, what does this cost?

   Lambda’s cost trajectory hides at low volume and surfaces at high volume. Modeling 10x growth at decision time exposes the cost cliff. If the model shows Lambda becoming uneconomic at the forecast volume, the choice should be ECS or EKS.

3. What latency SLO does the workload need?

   Sub-100ms P99 with low volume is incompatible with Lambda cold-start variance unless provisioned concurrency is paid for. For workloads with strict latency requirements and low base load, ECS or EKS are the only viable choices.

4. What tenancy and compliance requirements will the workload need in 24 months?

   If the future includes per-tenant isolation for regulated customers, the compute choice today should not foreclose that future. Lambda’s tenancy model is the most awkward to evolve. ECS and EKS both support evolution but require different paths.

The team that answers these four questions makes a different choice than the team that picks based on preference. The choice is not always the same. It is always more defensible.

What Changes When Computing Is a Risk Decision

Engineering organizations that treat compute selection as a risk decision rather than a preference produce different outcomes.

The platform team consolidates on fewer compute models, reducing operational burden and improving the team’s ability to ship golden paths.

The cost forecast is accurate because each workload was modeled during growth, before the architecture shipped.

The compliance posture is consistent because the compute choice was filtered through the compliance scope at decision time.

The team’s hiring profile becomes coherent because computed commitments determine which kinds of engineers the team needs to recruit, develop, and retain.

The choice of a computer does not become invisible. It becomes the single most important architectural commitment the team makes. The teams that treat it that way ship platforms that compound. The teams that treat it as a preference ship platforms that fragment.

On Wednesday, paid subscribers get the complete ECS vs EKS vs Lambda decision framework: a scoring matrix that evaluates a candidate workload across team maturity, operational complexity, compliance impact, cost trajectory, and ownership.

The framework includes weighted scoring, the threshold rules that drive the recommendation, and three worked examples covering an internal service, a customer-facing API, and a regulated multi-tenant workload.

Upgrade If You Need Implementation, Not Just Ideas

If you’re using these emails to guide real decisions on your platform, you’ll get more leverage from the paid version of The Cloud Playbook.

The free newsletter gives you patterns and language.

The paid newsletter turns those patterns into implementation kits you can ship inside a quarter:

• Concrete rollout plans (90‑day roadmaps for each pattern)

• Templates and checklists (policies, runbooks, tagging schemes, review checklists)

• Real examples from high‑stakes AWS environments (what we actually shipped and why)

If the paid side doesn’t save you more than the subscription in one incident, audit cycle, or bad migration you avoid, you should cancel and keep the playbooks.

Upgrade to the Paid Cloud Playbook

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Share The Cloud Playbook

Until next week — Amrut

Get in touch

You can find me on LinkedIn or X.

If you would like to request a reading topic, please feel free to contact me directly via LinkedIn or X.