Most data lakes do not fail because teams chose the wrong AWS service.
They fail because nobody encoded ownership, schema authority, partitioning, and access rules before the first dataset arrived.
The pattern is predictable:
A team creates an Amazon S3 bucket, points AWS Glue crawlers at it, and queries the data with Amazon Athena.
It works.
Six months later, the catalog contains hundreds of tables. Nobody knows which ones are authoritative. Analysts build private copies. Athena queries scan more data than expected. The platform team becomes the human directory for every dataset question.
The lake has become a data swamp.
The problem is not serverless architecture. The problem is that the structure was left in documentation, tribal knowledge, and manual cleanup instead of being encoded in infrastructure as code.
The cost of an unstructured AWS data lake
An unstructured data lake bills you twice.
The direct cost comes from inefficient queries. Without partitioning conventions enforced at ingestion, analysts can scan entire prefixes or large tables when they only need a small date range.
The second cost is trust.
When teams cannot answer which table is authoritative, they create their own copies. Eventually, the organization pays for duplicate storage and compute while executives receive reports that do not reconcile.
The platform team absorbs the organizational cost. They become the routing layer for questions that the data platform should answer automatically:
Which dataset should I use?
Who owns this table?
Is this data raw, cleaned, or curated?
Who can read customer PII?
Is this table still supported?
Why did this query scan so much data?
The fix is not another dashboard.
The fix is structure at creation.
Why data lakes accumulate hundreds of untrusted tables
The failure usually begins with a reasonable decision:
Let teams land data quickly and organize it later.
AWS Glue crawlers make this easy. Point a crawler at a prefix and it infers a schema, then registers a table in the Glue Data Catalog.
That reduces friction. It also removes accountability.
A crawler-created table may have a schema but no:
Owner
Data classification
Quality contract
Source system
Lifecycle policy
Statement of authority
By month eleven, the catalog may accurately describe what exists while providing almost no signal about what can be trusted.
The same pattern appears in permissions. Teams receive ad hoc S3 or IAM access during onboarding. Exceptions accumulate. A year later, answering “who can read this customer dataset?” requires reconstructing old tickets, policies, and team decisions.
That is not governance. That is archaeology.
The three design decisions that determine whether the lake holds
1. Separate zones with buckets or prefixes
A practical serverless data lake usually has three zones:
Raw: Data as received from source systems
Cleaned: Validated and standardized data
Curated: Business-ready datasets for consumption
Separate S3 buckets provide cleaner IAM boundaries, independent lifecycle policies, and distinct encryption-key scopes.
Prefixes inside one bucket are simpler, but they create a weaker security boundary and make policy conditions easier to misconfigure.
Use separate buckets when you have regulated data, multiple teams writing data, or materially different retention requirements.
Use prefixes when one team owns the entire lake and the operating model is genuinely simple.
The important decision is not which structure sounds more elegant. It is whether the boundary will remain explainable as the organization grows.
2. Use crawlers selectively
Use Glue crawlers for the raw zone when the incoming schema genuinely changes or is outside your control.
For cleaned and curated data, declare table schemas in Terraform with aws_glue_catalog_table.
That gives you:
Reviewed schema changes
Version history through pull requests
Explicit partition keys
Consistent storage descriptors
A clear owner for every table
A schema change should be visible before it reaches production, not discovered after an automated crawler silently changes the catalog.
3. Choose the permission model deliberately
IAM-only access is simpler and may be sufficient for a small analytics organization with limited sensitivity and few consuming teams.
Lake Formation becomes more valuable when you need permissions expressed in data terms:
Database
Table
Column
Data classification
Consuming team
Do not adopt Lake Formation because it sounds more enterprise-ready. Adopt it when the organization needs governance that IAM paths cannot express cleanly.
The mistake is running both systems without deciding which one is authoritative. If you use Lake Formation, define the migration sequence, grant required access first, and remove legacy access only after you have verified the workload path.
The Terraform reference pattern
Build the lake from five modules:
Zone bucket module
Inputs: zone name, retention period, KMS key ARN, versioning setting
Outputs: bucket ARN, read policy, write policyCatalog database module
Defines the logical database for each data domain or zone.Catalog table module
Defines cleaned and curated schemas, partition keys, Parquet configuration, and table properties.Lake Formation grant module
Defines which teams can access which databases, tables, and columns.Ingestion pipeline module
Defines how data moves from raw to cleaned to curated.
Make the module interface fixed. Avoid one-off infrastructure for every new dataset.
Required metadata should be mandatory:
Owner team
Data classification
Source system
Retention category
Zone
A table without an owner should fail the plan, not become a cleanup ticket.
Encode the partition convention in the ingestion and writer controls. For example:
dataset/ingest_date=YYYY-MM-DD/
Do not rely on a wiki to enforce a convention. Use controlled writer roles, validation, and policy checks so incorrectly shaped data cannot quietly enter the lake.
Add lifecycle rules when each zone is created. Retention and storage-class transitions are much cheaper to define at creation than retrofit across years of accumulated data.
The operating metrics
Review these four numbers monthly:
Percentage of catalog tables with an owner tag
Target: 100%.Athena bytes scanned at the 95th percentile
Rising scan volume usually means partitioning or query discipline is degrading.Lake Formation grants compared with consuming teams
Grant growth that dramatically exceeds team growth signals exception accumulation.Time required to answer who can read a table
Target: less than ten minutes.
Review access grants quarterly. Review partitioning and scan behavior monthly with the analytics team.
Build a lake you can still explain in year three
A well-structured data lake may cost more effort in week one and less money, confusion, and risk every week after.
Structure encoded at creation is cheap.
Structure imposed on four hundred existing tables is a reorganization nobody has budgeted for.
If your lake is already in production, compare the current architecture against the five modules above. Identify which module has no clear equivalent. That gap list is your backlog.
The goal is not to build the most sophisticated AWS data platform.
The goal is to build one that remains trusted, governable, and explainable as the organization grows.
That’s it for today!
Did you enjoy this newsletter issue?
Share with your friends, colleagues, and your favorite social media platform.
Until next week — Amrut
Get in touch
You can find me on LinkedIn or X.
If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.