TCP #75: The AWS Cost Killer No One Talks About: Gateway Endpoints

How one misconfigured route table can cost you in unnecessary NAT Gateway charges

You can also read my newsletters from the Substack mobile app and be notified when a new issue is available.

---

As a founding member, you will receive:

• Everything included in paid subscriber benefits + exclusive toolkits and templates.

• High-quality content from my 11+ years of industry experience, where I solve specific business problems in the real world using AWS Cloud. Learn from my actionable insights, strategies, and decision-making process.

• Quarterly report on emerging trends, AWS updates, and cloud innovations with strategic insights.

• Public recognition in the newsletter under the “Founding Member Spotlight” section.

• Early access to deep dives, case studies, and special reports before they’re released to paid subscribers.

---

Last month, I was reviewing our AWS bill. It had exploded from $3,000 to $12,000 overnight.

The culprit? A simple route table misconfiguration sent all S3 traffic through NAT Gateways instead of free Gateway Endpoints.

This isn't an isolated incident.

After auditing over 20 AWS environments, I've discovered that 73% of companies are incurring unnecessary costs on NAT Gateway charges for traffic that AWS routes for free.

Today, I'm sharing the complete playbook that has saved my clients and will save you unnecessary AWS charges.

Image created by the Author

---

The Hidden Cost of "Standard" Architecture

Most AWS architectures follow this pattern:

Private Subnet → NAT Gateway → Internet Gateway → S3/DynamoDB

The monthly damage:

• Base NAT Gateway cost: $45.00/month per AZ

• Data processing: $0.045 per GB

• Cross-AZ data transfer: $0.01-0.02 per GB

• Internet data transfer: $0.09 per GB (first TB)

For a modest 500GB monthly S3 usage across 3 AZs, you're looking at $225+ in unnecessary charges. Scale that to enterprise workloads, and you're burning thousands monthly.

The kicker? This traffic never needed internet access in the first place.

---

Gateway Endpoints: AWS's Best-Kept Secret

VPC Gateway Endpoints create a direct route from your private subnets to S3 and DynamoDB through AWS's private backbone.

No internet routing. No NAT Gateway charges. No data processing fees.

The architecture shift: Private Subnet → Route Table → Gateway Endpoint → S3/DynamoDB

The cost: $0.00 per month. Forever.

Why Gateway vs Interface Endpoints?

Gateway Endpoints (S3 + DynamoDB only):

• Route table entries

• Zero hourly charges

• Zero data processing fees

• Scales automatically

• Regional service access only

Interface Endpoints (100+ services):

• ENI-based with private IPs

• $7.20/month per endpoint per AZ

• $0.01 per GB processed

• Cross-region capable

• Requires security group management

For S3/DynamoDB access, Gateway Endpoints are always the better choice.

---

Implementation Deep Dive

Step 1: Create the Gateway Endpoint

AWS Console Method:

1. Navigate to VPC → Endpoints

2. Create Endpoint → Gateway

3. Select service (s3 or dynamodb)

4. Choose VPC and route tables

5. Configure policy (optional)

Infrastructure as Code:

resource "aws_vpc_endpoint" "s3" {
  vpc_id          = aws_vpc.main.id
  service_name    = "com.amazonaws.${var.region}.s3"
  route_table_ids = [aws_route_table.private.id]
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = "*"
      Action = [
        "s3:GetObject",
        "s3:PutObject"
      ]
      Resource = "arn:aws:s3:::your-bucket/*"
    }]
  })
}

Step 2: Route Table Association Strategy

Critical decision: Which route tables to associate?

• Associate with: Private subnet route tables that need S3/DynamoDB access

• Don't associate with: Public subnet route tables (they use Internet Gateway)

• Edge case: Database subnet route tables if they need S3 backup access

Step 3: Verify Implementation

DNS Resolution Check:

# From EC2 in private subnet
nslookup s3.us-east-1.amazonaws.com
# Gateway Endpoint: 52.x.x.x range
# Internet route: 3.x.x.x range

Route Table Verification:

Look for entries like: pl-63a5400a (com.amazonaws.us-east-1.s3) → vpce-12345678

---

The Route Priority Trap (And How to Avoid It)

The most common implementation failure: Existing 0.0.0.0/0 routes to NAT Gateways can override Gateway Endpoint routes.

Understanding Route Precedence

AWS route tables follow this priority:

1. Local routes (VPC CIDR)

2. Longest prefix match (most specific)

3. Static routes over propagated routes

4. Gateway Endpoints use prefix lists (region-specific IP ranges)

The Fix

Gateway Endpoints should automatically take precedence because they use specific prefix lists (more specific than 0.0.0.0/0). If traffic still routes through NAT:

1. Check route table associations: Ensure the Gateway Endpoint is associated with the correct route tables

2. Verify DNS resolution: Confirm endpoints resolve to 52.x.x.x ranges

3. Review security groups: Ensure HTTPS (443) outbound is allowed

4. Test with VPC Flow Logs: Track traffic patterns

---

Security Deep Dive

IAM Policy Integration

Gateway Endpoints respect existing IAM policies. No changes needed to user/role permissions.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

This policy works identically whether traffic routes through the Internet Gateway or the Gateway Endpoint.

Endpoint Policies: Additional Layer

Add endpoint-level restrictions for defense in depth:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "*",
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "aws:PrincipalVpc": "vpc-12345678"
      }
    }
  }]
}

This ensures only resources in your VPC can use the endpoint.

Bucket Policy Considerations

Existing bucket policies continue working. However, you might want to add VPC-specific restrictions:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::my-bucket",
      "arn:aws:s3:::my-bucket/*"
    ],
    "Condition": {
      "StringNotEquals": {
        "aws:SourceVpc": "vpc-12345678"
      }
    }
  }]
}

---

Advanced Scenarios and Edge Cases

Multi-Region Considerations

Limitation: Gateway Endpoints only work within the same region.

Cross-region S3 access still requires:

• NAT Gateway or Internet Gateway

• Cross-region data transfer charges apply

• Consider S3 Cross-Region Replication to minimize cross-region calls

Hybrid Cloud Integration

AWS Direct Connect: Gateway Endpoints work seamlessly with Direct Connect. Private traffic stays on the AWS backbone.

VPN Connections: On-premises resources accessing S3 through VPN + Gateway Endpoints get improved performance and reduced costs.

Container Orchestration

EKS Considerations:

• Pod subnets need a Gateway Endpoint association

• ALB Ingress Controller benefits from the S3 Gateway Endpoint for access logs

• External-DNS needs Route53 access (Interface Endpoint required)

ECS/Fargate:

• Task subnets require a Gateway Endpoint association

• ECR requires Interface Endpoints (different services)

• CloudWatch Logs may need an Interface Endpoint

Database Backup Scenarios

RDS automated backups to S3: Occur in AWS-managed subnets, don't benefit from your Gateway Endpoints.

Application-level backups: Do benefit from Gateway Endpoints. Significant cost savings for large database exports.

---

Troubleshooting Playbook

Symptom: Traffic Still Goes Through NAT Gateway

Diagnostic Steps:

1. Check VPC Flow Logs:

Look for: NAT Gateway ENI receiving S3 traffic
Should see: Gateway Endpoint prefix list in route tables

1. Verify DNS Resolution:

dig s3.region.amazonaws.com
# Should return 52.x.x.x (vpce) not 3.x.x.x (internet)

1. Route Table Audit:

Confirm: pl-xxxxx entries for S3/DynamoDB
Missing: Gateway Endpoint association with route table

Symptom: Access Denied Errors

Common Causes:

• The endpoint policy is too restrictive

• Missing IAM permissions (unlikely if it worked before)

• Bucket policy blocking VPC access

• Security group blocking HTTPS outbound

Quick Fix: Remove the endpoint policy temporarily to isolate the issue.

Symptom: Intermittent Connectivity

Root Causes:

• Multiple route tables, some missing Gateway Endpoint association

• Cross-AZ issues with route table associations

• DNS caching in applications

Resolution: Ensure all relevant route tables are associated with a Gateway Endpoint.

---

Monitoring and Cost Optimization

CloudWatch Metrics

Monitor Gateway Endpoint usage:

• PacketDropCount: Indicates security group or NACL blocks

• Not available: Usage metrics (they're free, AWS doesn't meter them)

Cost Tracking Strategy

Before Implementation:

• Note NAT Gateway charges in Cost Explorer

• Track data processing fees

• Document cross-AZ transfer costs

After Implementation:

• Verify the NAT Gateway cost reduction

• Monitor for cross-region charges (shouldn't change)

• Set up billing alerts for unexpected increases

VPC Flow Logs Analysis

Enable VPC Flow Logs to track traffic patterns:

{
  "version": "2",
  "account-id": "123456789012",
  "interface-id": "eni-1235b8ca123456789",
  "srcaddr": "10.0.1.4",
  "dstaddr": "52.216.0.1",
  "protocol": "6",
  "action": "ACCEPT"
}

Look for dstaddr in 52.x.x.x range (Gateway Endpoint) vs 3.x.x.x (Internet).

---

Implementation Checklist

Pre-Implementation

• Audit current NAT Gateway usage and costs

• Identify private subnets requiring S3/DynamoDB access

• Document existing route table configurations

• Plan maintenance window (minimal disruption expected)

Implementation

• Create Gateway Endpoints for required services

• Associate with the appropriate route tables

• Configure endpoint policies (if needed)

• [Update Infrastructure as Code templates

Post-Implementation

• Verify DNS resolution from private subnets

• Test application functionality

• Monitor VPC Flow Logs for traffic patterns

• Document new architecture for the team

Ongoing Monitoring

• Set up cost alerts for unexpected charges

• Review Gateway Endpoint associations quarterly

• Update endpoint policies as access patterns change

• Train the team on the Gateway Endpoint concepts

---

Final Thoughts

Gateway Endpoints represent one of AWS's most overlooked cost optimization opportunities. For the vast majority of workloads accessing S3 and DynamoDB from private subnets, they offer:

• Zero-cost alternative to NAT Gateways

• Better performance through AWS private backbone

• Improved security with no internet routing

• Seamless implementation with existing applications

The configuration takes minutes. The savings last forever.

Your next step:

Audit your current NAT Gateway usage. If you're routing S3 or DynamoDB traffic through NAT Gateways, you're paying for something AWS gives you for free.

Want more AWS cost optimization strategies? Reply with your biggest AWS cost challenge, and I'll tackle it in next week's newsletter.

---

SPONSOR US

The Cloud Playbook is now offering sponsorship slots in each issue. If you want to feature your product or service in my newsletter, explore my sponsor page

Become a Proud Sponsor!

---

That’s it for today!

Did you enjoy this newsletter issue?

Share with your friends, colleagues, and your favorite social media platform.

Until next week — Amrut

---

Get in touch

You can find me on LinkedIn or X.

If you would like to request a topic to read, please feel free to contact me directly via LinkedIn or X.