AWS re:Invent 2016: From One to Many: Evolving VPC Design (ARC302)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Rob Alexander, Principal Solutions Architect

December 2, 2016

ARC302

From One to ManyEvolving VPC Design

Disclaimer:

Do Try This at Home!

Assuming you’ve heard of…

Route Table

Elastic

Network

Interface

Amazon VPC

Internet

Gateway

Customer

Gateway Virtual

Private

Gateway

VPN

Connection

VPC subnet

Network ACL

Security group

Enhanced

Networking

VPC

Peering

AWS Direct

Connect

Related Sessions

NET201 – Creating Your Virtual Data Center: VPC

Fundamentals and Connectivity Options

NET305 – Extending Datacenters to the Cloud:

Connectivity Options and Considerations for Hybrid

Environments

From one…

Subnet

Availability Zone A

Subnet

Availability Zone B

VPC

us-east-2

VPCVPC

VPC

VPCTransit VPC

VPC

us-west-2

VPC

VPC

VPC

eu-west-1

VPCVPC

VPC

VPCTransit VPC

VPC

Branch Branch

NA

HQ

VPC

VPC

VPC

VPC

VPC

VPC

Chicago DX

AP

HQ

London DX

ap-northeast-1

VPC

VPC

VPC

VPCTransit VPC

VPC

EU

HQ

Tokyo DX … to many

VPC/16

Choose a CIDR

• CIDR fixed on VPC

creation

• /16 down to /28

• Go Big

VPC IPv4 space design

• Plan for expansion to additional Availability

Zones or regions

• Consider connectivity to corporate networks

• Don’t overlap IP space

• Save space for the future

• IPv4 space is required, but …

IPv6 now supported in VPC

• Optionally enable IPv6 on VPC

• /56 of Amazon’s Global Unicast Address (GUA) per VPC

• /64 CIDR block per subnet

• IPv6 completely independent from IPv4

• Enabled per subnet or per instance (per ENI)

• Supported by Security Groups, Route Tables, NACLs, VPC

Peering, IGW, DX, Flow Logs and DNS Resolution

Availability Zone A

VPC

• Even distribution of IP

space across AZs

• Use at least 2 AZs

• Subnets are AZ

specific

• How big? How many?

Create subnets

Subnet

Availability Zone B

Subnet

Availability Zone C

Subnet

/16

Availability Zone A

Subnet

VPC

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet Subnet Subnet Subnet Subnet Subnet

Subnet Subnet Subnet Subnet Subnet

Subnet Subnet Subnet Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

Subnet

/16

VPC subnet design

• Traditional switching limitations do not apply

• Consider large, mixed use subnets

• Use security groups to enforce isolation

• Use tags for grouping resources

• Use subnets as containers for routing policy

Related Sessions

NET401 – Another Day, Another Billion Packets

Availability Zone A

Public subnet

Private subnet

Availability Zone B

Public subnet

Private subnet

VPC/16

Availability Zone C

Public subnet

Private subnet

/22 /22 /22

/20 /20 /204091 IPs

1019 IPs

VPC/16

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

Availability Zone C

Private subnet

Public subnet

Private subnet

/22 /22 /22

/20

/20

/20

/20

/20

/20

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

.1

VPC

.1

.1 .1

.1 .1

Routing Policy

Main Route Table

Destination Target

10.1.0.0/16 Local

VPC CIDR 10.1.0.0/16

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Public Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 IGW

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Private Route Table

Destination Target

10.1.0.0/16 Local

Corp CIDR VGW

Availability Zone A

Public subnet

Private subnet

Availability Zone B

VPC/54

Availability Zone C

/64

/64

18 MILLION,

Public subnet

Private subnet

Public subnet

Private subnet

What about IPv6?

/64

/64

/64

/64

TRILLIONIPs

Availability Zone A

Private subnet

Public subnet

Availability Zone B

Private subnet

Public subnet

VPC

IPv6

Routing Policy

AWS Region

Internet

Public Route Table

Destination Target

10.1.0.0/16 Local

2001:db8:1234:1a00::/56 Local

0.0.0.0/0 IGW

::/0 IGW

Availability Zone A

Private subnet

Public subnet

Availability Zone B

Private subnet

Public subnet

VPC

IPv6

Routing Policy

AWS Region

Internet

Public Route Table

Destination Target

10.1.0.0/16 Local

2001:db8:1234:1a00::/56 Local

Corp CIDR VGW

::/0 EIGW

Egress-Only IGW

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 ???

Corp CIDR VGW

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Why go outside?

• AWS API endpoints

• Regional services

• Third-party services

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 NAT Instance

Corp CIDR VGW

NAT

Instance

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

Routing Policy

AWS Region

Internet

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 NAT Instance

Corp CIDR VGW

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 Black Hole

Corp CIDR VGW

NAT

Instance

Scalable and Available NAT

Evolving design requirements

• Public subnets for resources reachable from Internet

• Private subnets with egress only access to public network

• Scalable, highly available NAT

• One AWS account

• One VPC

• One region

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

AWS Region

Internet

NAT

Instance

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 Black Hole

Corp CIDR VGW

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

AWS Region

Internet

Deploy a

NAT Gateway

Private Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 NAT Gateway

Corp CIDR VGW

NAT

Gateway

Why a NAT Gateway?

10.1.1.112:54318 52.27.192.88:35678

NAT Instance

Source IP: Port NAT’d Source IP:Port

Security Updates

Package Repos

NTP

VPC

Public Network

Why a NAT Gateway?

10.1.1.112:54318 52.27.192.88:35678

Source IP: Port NAT’d Source IP:Port

VPC

Source IP is the same

Source Port must be

unique

Destination

IP and Port

are the same

NAT Instance

Public Network

52.27.192.88:33622

52.27.192.88:38438

52.27.192.88:48132

52.27.192.88:29754Security Update

Why a NAT Gateway?

10.1.1.112:54318 52.27.192.88:35678

Source IP: Port NAT’d Source IP:Port

VPC

Source IP is the same

Source Port must be

unique

Destination

IP and Port

are the same

Public Network

52.27.192.88:33622

52.27.192.88:38438

52.27.192.88:48132

52.27.192.88:29754

NAT Gateway

Security Update

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

AWS Region

Internet

Deploy a

NAT Gateway

NAT

Gateway

• Still need IGW

• Separate subnets

• Requires EIP

• AZ specific

• Burst to 10 Gbps

1

NAT Gateway: Securing Access

NAT Gateway ENI:

Network ACL

Public subnet

NAT

GatewayNetwork ACLs still apply

NAT Gateway: Securing Access

Use routing

policy to control

access to NAT

Gateway

Private subnet

Public subnet

Private subnet

NAT Enabled

no-NAT

no-NAT Private Route Table

Destination Target

10.1.0.0/16 Local

NAT Enabled Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0/0 NAT Gateway

NAT

Gateway2

NAT Gateway: Securing Access

Outbound Rules

Type Protocol Port Range Destination

All traffic All 0 - 65535 0.0.0.0/0

Use security groups

to restrict outbound

access for instances

Default VPC security group:

3

NAT Gateway: Securing Access

Outbound Rules

Type Protocol Port Range Destination

All traffic All 0 - 65535 10.2.0.0/16

Outbound Rules

Type Protocol Port Range Destination

All traffic All 0 - 65535 0.0.0.0/0

Use security groups

to restrict outbound

access for instances

Default VPC security group:

NAT Enabled VPC security group:

3

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

AWS Region

Internet

Deploy a

NAT Gateway

NAT

Gateway

NAT Enabled

no-NAT

NAT Enabled

no-NAT

Availability Zone A

Private subnet

Public subnet

Private subnet

Availability Zone B

Private subnet

Public subnet

Private subnet

VPC

AWS Region

Internet

Deploy a

NAT Gateway

NAT

Gateway

NAT

Gateway

NAT Enabled

no-NAT

NAT Enabled

no-NAT

• Drop in replacement for NAT instance

• Fully managed

• Highly available and fault tolerant

• Scalable to 10 Gbps burst per gateway

• Supports VPC Flow Logs

• No higher level functions like IPS, UTM,

URL Filtering, packet inspection, etc

• Cannot associate security group to

gateway

Pro & Con: NAT Gateway

AWS

Region

Considering multiple VPCs

Public-facing

web apps

Internal

company

apps

What’s next?

VPN

connection

VPC VPC VPC

Customer

network

One VPC, Two VPC

VPC

Why not 1 big VPC?

Why not 1 AWS Account?

• Blast radius

• Account Limits

• API Limits

Considerations for one or many VPCs

AWS Region

ProdNot

Prod

VPCVPC

Considerations for one or many VPCs

AWS Region

PCI

Apps

VPC VPC

Non

Regulated

Apps

Considerations for one or many VPCs

AWS Region

Prod

VPC

AWS Region

Disaster

Recovery

VPC

Considerations for one or many VPCs

AWS RegionVPC

Audit

Logging &

Analytics

AWS

CloudTrail

AWS

Config

VPC Flow

Logs

VPC

Legal

VPC

Finance

VPC

Sales

App Logs,

S3 Access Logs,

ELB Logs

Amazon

Redshift

Amazon

EMR

S3

AWS Region

Internal application to VPC

Public-facing

web app

Internal

company

app

VPN

connection

VPCVPC

Customer

network

Availability Zone A

Private subnet Private subnet

AWS Region

Virtual

Private

Gateway

VPN

connection

Intranet

app

Intranet

app

Availability Zone B

Internal customers

Private Route Table

Destination Target

10.1.0.0/16 Local

Corp CIDR VGW

VPC

Internal application to VPC

Customer

network

But apps will make heavy use of …

Amazon S3

…as a primary data store

VPC Egress Control

Evolving design requirements

• VPN connectivity to private-only VPC

• No egress in the VPC to public networks

• Private IP access to Amazon S3

• Content-specific access controls

• One AWS account

• One VPC

• One region

Availability Zone A

Private subnet Private subnet

AWS

Region

Virtual

Private

Gateway

VPN

connection

Intranet

app

Intranet

app

Availability Zone B

You really don’t want to do this:

Amazon

S3

Internet

Customer

border router

Customer VPN

Internet

VPC

Customer

network

Availability Zone A

Private subnet Private subnet

AWS

Region

Virtual

Private

Gateway

Intranet

app

Intranet

app

Availability Zone B

So do this instead:

Amazon

S3

VPC

VPN

connection

VPC Endpoints

• No IGW

• No NAT

• No public IPs

• Free

• Robust access

control

Customer

network

Creating S3 VPC endpoint

aws ec2 create-vpc-endpoint

--vpc-id vpc-40f18d25

--service-name com.amazonaws.us-west-2.s3

--route-table-ids rtb-2ae6a24f rtb-61c78704

Private subnet

VPCRoute Table

Destination Target

10.1.0.0/16 Local

Corp CIDR VGW

Prefix List for S3 us-west-2 VPCE

Creating S3 VPC endpoint

aws ec2 create-vpc-endpoint

--vpc-id vpc-40f18d25

--service-name com.amazonaws.us-west-2.s3

--route-table-ids rtb-2ae6a24f rtb-61c78704

Public subnet

VPCRoute Table

Destination Target

10.1.0.0/16 Local

0.0.0.0 IGW

Prefix List for S3 us-west-2 VPCE

Creating S3 VPC endpoint

Private subnet

VPC

Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0 NAT Gateway

Prefix List for S3 us-west-2 VPCE

Public subnet

NAT

Gateway

Prefix lists

aws ec2 describe-prefix-lists

PREFIXLISTS pl-68a54001 com.amazonaws.us-west-2.s3

CIDRS 54.231.160.0/19

CIDRS 52.218.128.0/18

• Logical route destination target

• Dynamically translates to service IPs

• S3 IP ranges change over time

• S3 prefix lists abstract change

Prefix lists

… and use them in your outbound security group rules!

Private subnet

Controlling VPC access to Amazon S3

AWS Identity & Access

Management (IAM) policy

on VPCE:

VPC

{

"Statement": [

{

"Sid": "vpce-restrict-to-backup-bucket",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject”

],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::backups-reinvent",

"arn:aws:s3:::backups-reinvent/*"]

}

]

}

Backups bucket?

Private subnet

Controlling VPC access to Amazon S3

S3 bucket policy:

VPC

From

vpce-bc42a4e5?

{

"Statement": [

{

"Sid": "bucket-restrict-to-specific-vpce",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::backups-reinvent",

"arn:aws:s3:::backups-reinvent/*"],

"Condition": {

"StringNotEquals": {

"aws:sourceVpce": "vpce-bc42a4e5”

}

}

}

]

}

Controlling VPC access to Amazon S3

Recap on security layers:

1. Route table association

2. VPCE policy

3. Bucket policy

4. Security groups with prefix list

Private subnet

VPC

1.

2.

3.

4.

Private subnet Private subnet

AWS

Region

Intranet

apps

Compliance

app

Endpoints in action

VPC

Compliance Backups

VPCE1 VPCE2

Private subnet

Intranet

apps

Private subnet Private subnet

AWS

Region

Intranet

apps

Compliance

app

Endpoints in action

VPC

Compliance Backups

VPCE1 VPCE2

Private subnet

Intranet

appsPrivate subnet Private subnet

Private subnet

Logs Analytics

• Secure, highly scalable and highly

available access to S3

• Fine grained control of access to

content in S3 from VPC

• Control which VPCs/VPCEs can

access which S3 buckets

• No public IPs required, source IPs kept

private

• Bucket policy restricted to specific

VPCs (or VPCEs) will disable S3

Console access

• Requires Amazon DNS enabled on

VPC

Pro & Con: VPC Endpoints

AWS Region

Public-facing

web apps

Internal-

only

apps

What’s next?

VPN

connection

VPC VPC VPC

Customer

networkCustomer Gateway

(CGW)

Shared Service Hubs

AWS

Region

VPC

VPC

VPC

VPC VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPCVPC

Customer

network

Public

apps

Internal

apps

AWS Region

VPC

HA VPN Pair

Availability Zone A

HA VPN

To

VPC

iBGP

eBGP

Customer CIDRs or Default Route

eBGP

AWS ASN 7224

Re-advertise VPC CIDR via IGP

VGW

VPC CIDR

Customer ASN (Public or Private)

CGW1 CGW2

VPN1

Tun1

VPN1

Tun2

Availability Zone A

VPN2

Tun1

VPN2

Tun2

Reuse your CGW Public IP

to connect to more VPCs

Customer

network

MED

MED

AWS

Region

VPCVPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

• DNS

• Directory

• Logging

• Monitoring

• SecurityShared services

Customer

network

Evolving design requirements

• Centralize network connectivity to and from cloud

• Centralize management, security, and common services

• Account owners in control of own VPC resources

• Many AWS accounts

• Many VPCs

• One region

AWS

RegionVPC

VPC

VPC

VPC

VPC

VPC

• DNS

• Directory

• Logging

• Monitoring

• SecurityShared services

Hub and

Spoke

with

Peering

VPC

Shared

services

VPC

VPC

Customer

network

Spoke VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Spoke VPC

VPC

Customer

network

AWS Region

VPC

Hub VPC

Private subnet

VPC

Spoke VPC

Public subnet

10.2.0.0/1610.1.0.0/16

Private subnet

Private Route Table

Destination Target

10.1.0.0/16 Local

10.2.0.0/16 PCX-1

Private Route Table

Destination Target

10.2.0.0/16 Local

10.1.11.0/24 PCX-1

VPC peering

Shared services

10.2.22.0/24

10.1.11.0/24

AWS Region

VPC

Hub VPC

Private subnet

VPC

Spoke VPC

Public subnet

10.2.0.0/1610.1.0.0/16

Private subnet

Private Route Table

Destination Target

10.1.0.0/16 Local

10.2.0.0/16 PCX-1

Private Route Table

Destination Target

10.2.0.0/16 Local

10.1.11.0/24 PCX-1

172.16.0.0/16 PCX-1

Edge-to-edge routing

Shared services

10.2.22.0/24

10.1.11.0/24

172.16.0.0/16Customer

network

AWS Region

VPC

Hub VPC

Private subnet

VPC

Spoke VPC

Proxy

subnets

10.2.0.0/1610.1.0.0/16

Private Route Table

Destination Target

10.2.0.0/16 Local

10.1.0.0/16 PCX-1

Edge-to-edge via proxy

PCX-1 10.2.22.0/24

Internal

ELB

Proxy

fleet

Internet

Public

services

S3VPC

Customer

network

Proxy Route Table

Destination Target

10.1.0.0/16 local

10.2.0.0/16 PCX-1

172.16.0.0/16 VGW

Proxy Route Table

Destination Target

10.1.0.0/16 Local

10.2.0.0/16 PCX-1

172.16.0.0/16 VGW

0.0.0.0/0 IGW

S3 Prefix List VPCE

Customer

network

Availability Zone A

Private subnet

Public subnet

Private subnet

Elastic

Load

Balancer

Shared

services

AWS Region

Internet

VPC

Auto Scaling

proxy

fleet

Public

servicesS3

PCX-1

Availability Zone B

Private subnet

Public subnet

Private subnet

Elastic

Load

Balancer

Shared

services

Auto Scaling

proxy

fleet

Spoke VPC

VPC

Private subnet

Proxy in practice

Hub VPC

Availability Zone A

Private subnet

Public subnet

Private subnet

Elastic

Load

Balancer

Shared

services

AWS Region

Internet

VPC

Auto Scaling

proxy

Fleet

Public

servicesS3

PCX-1

Availability Zone B

Private subnet

Public subnet

Private subnet

Elastic

Load

Balancer

Bastion

host

Auto Scaling

proxy

fleet

Spoke VPC

VPC

Private subnet

Proxy in practice

Hub VPC

Customer

network

Shared Services Hub: To-Do List

• Use IAM to restrict spoke AWS accounts from altering network

• Create a NetOps IAM role in all accounts:https://aws.amazon.com/blogs/security/how-to-assign-permissions-using-new-aws-managed-policies-for-job-functions/

• Enable AWS CloudTrail, AWS Config, and VPC Flow Logs for all accounts

• Integrate CloudTrail with CloudWatch Logs and create alarms:https://aws.amazon.com/blogs/aws/cloudtrail-integration-with-cloudwatch-now-available-in-four-more-regions

• Minimizes on premises network change

• Reduces latency, cost of cloud

applications accessing common services

• Provides spoke accounts control over

own resources

• But controls and secures egress traffic

from spokes

• Security Groups work across peers

• Cost and management of central proxy

layer

• Not a transparent proxy

• Configuring end devices to use proxy

• Restricted to HTTP/S

• No transitive networking

• Peering data transfer cost

Pro & Con: Shared Services Hub and Spoke

AWS Region

VPCVPC

VPC

VPC

VPC

• DNS

• Directory

• Logging

• Monitoring

• Security

VPC

VPC

VPC

VPC

VPC VPC

VPC

VPC

VPC

VPC

Shared services

Customer

network

Dev hubProd hub

Data

services

hub

AWS Region

VPCVPC

VPC

VPC

VPC

• DNS

• Directory

• Logging

• Monitoring

• Security

VPC

VPC

VPC

VPC

VPC VPC

VPC

VPC

VPC

VPC

Shared services

Customer

network

Dev hubProd hub

Data

services

hub

VPC

VPC

Customer

network

AWS Region

Availability Zone A

Private subnet

VPC

Availability Zone B

Private subnet

AWS Lambda

Amazon API Gateway

Elastic

Network

Interface

VPCVPC

VPC

VPC

Prod hub

VPC

Internet

Hybrid

Serverless

Amazon

AuroraReplica

Mobile Application VPC

Legacy

Apps

Customer

network

AWS Region

Availability Zone A

Private subnet

VPC

Availability Zone B

Private subnet

AWS Lambda

Amazon API Gateway

Elastic

Network

Interface

VPCVPC

VPC

VPC

Prod hub

VPC

Internet

Hybrid

Serverless

Amazon

AuroraReplica

Mobile Application VPC

us-east-2 region

VPC VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC

VPC VPC

VPC

VPC

VPC

VPC

eu-west-1 region

VPC VPC

VPC

VPC

VPC

VPC

VPC Mass Transit

Evolving design requirements

• Centralize and minimize network connections

• Allow end to end routing from cloud to existing networks

• Minimal operational overhead

• Leverage AWS network

• Many AWS accounts

• Many VPCs

• Many regions

Availability Zone A

Public subnet

VPC

Transit

VPC

Availability Zone B

Public subnet

AWS

Region

EC2 VPN EC2 VPN

Availability Zone A

Public subnet

VPC

Transit

VPC

Availability Zone B

Public subnet

EC2 VPN EC2 VPN

AWS

RegionVPC

Spoke VPC

Transit VPC

VPCSpoke VPC

VPCSpoke VPC

AWS

RegionVPC

VPC

VPC

VPC

VPC

VPC

VPCTransit VPC

Customer

network

Spoke VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Spoke VPC

Branches

Transit

VPC

https://aws.amazon.com/answers/networking/transit-vpc/

Transit VPC

Transit VPC

Built using Cisco Cloud Services Router (CSR) 1000V

• Available on the AWS Marketplace

• A virtualized ASR with full IOS-XE software stack

• BYOL or Pay-as-you-Go license models

Availability Zone A

Public subnet

VPC

Availability Zone B

Public subnet

CSR1 CSR2

AWS Region

Transit VPC

S3 Bucket

for

VPN Config

Route Table

Destination Target

100.64.127.224/27 Local

0.0.0.0 IGW

Prefix List for S3 VPCE

100.64.127.224 / 27

Transit VPC:

Creation

What is EC2 Auto Recovery?

RECOVER Instance

Instance ID

Instance metadata

Private IP addresses

Elastic IP addresses

EBS volume attachments

Instance retains:

* Supported on C3, C4, M3, M4, P2, R3, T2, and X1 instance types with EBS-only storage

StatusCheckFailed_System

Amazon CloudWatch

per-instance metric alarm:

When alarm triggers?

Availability Zone A

Public subnet

VPC

Availability Zone B

Public subnet

CSR1 CSR2

AWS Region

Transit VPC

S3 Bucket

for

VPN Config

VPCSpoke VPC

AWS Lambda

Cisco

Configurator

AWS Lambda

VGW Poller

transitvpc:spoke = true

Transit VPC:

Add Spoke

SSH Only to CSR Security Group

Availability Zone A

Public subnet

VPC

Availability Zone B

Public subnet

AWS Region

Transit VPC

VPCSpoke VPCTransit VPC:

Preferred

Route

Spoke VPC Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0 VGW

Transit VPC Route Table

Destination Target

100.64.127.224/27 Local

0.0.0.0 IGW

Prefix List for S3 VPCE

Active / Active

Availability Zone A

Public subnet

VPC

Availability Zone B

Public subnet

AWS Region

Transit VPC

VPCSpoke VPC

transitvpc:preferred-path = CSR1

Transit VPC:

Preferred

Route

Spoke VPC Route Table

Destination Target

10.1.0.0/16 Local

0.0.0.0 VGW

Transit VPC Route Table

Destination Target

100.64.127.224/27 Local

0.0.0.0 IGW

Prefix List for S3 VPCE

Spoke VGW Tag

Active / Passive

Transit VPC: Preferred route spoke configuration

From CSR2:

!

address-family ipv4 vrf vpn-8a23d2e3

neighbor 169.254.35.57 remote-as 7224

neighbor 169.254.35.57 timers 10 30 30

neighbor 169.254.35.57 activate

neighbor 169.254.35.57 as-override

neighbor 169.254.35.57 soft-reconfiguration inbound

neighbor 169.254.35.57 route-map rm-vpn-8a23c7e3 out

exit-address-family

!

route-map rm-vpn-8a23c7e3 permit 10

set as-path prepend 64512 64512

!

BGP AS override

configured by default

Availability Zone A

Public subnet

VPC

Availability Zone B

Public subnet

CSR1 CSR2

AWS Region

Transit VPC

S3 Bucket

for

VPN Config

VPCSpoke VPC

AWS Lambda

Cisco

Configurator

AWS Lambda

VGW Poller

transitvpc:spoke = false

Transit VPC:

Remove Spoke

AWS

RegionVPC

VPC

VPC

VPC

VPC

VPC

VPC

Transit VPC

Customer

network

Spoke VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Spoke VPC

Branches

Transit

VPC

Internet

Public

services

Customer

network

VPCTransit VPC

us-east-2

us-west-2

VPC

VPC

Spoke VPC

Spoke

VPC

VPCTransit VPC

eu-west-1

eu-central-1

VPC

VPC

Spoke VPC

Spoke

VPC

AWS Network

Backbone

Internet

• End to End routing between VPCs in all

regions and any other non-AWS network

• Central transit routers can perform higher

level networking and security functions

• Spoke VGWs are HA by default

• Minimizes on premises networking changes

• Can minimize cost if replacing on premises

or colo networking hardware

• Availability and management of transit router

instances

• Licensing costs

• Cost of data transfer between transit, spokes

and other networks

Pro & Con: Transit VPC

AWS

Region VPC

VPC

VPC

VPC

VPCTransit VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Transit VPC

with

AWS Direct

Connect

(DX)

Detached

VGW

transitvpc:spoke = true

Customer

network

AWS Direct Connect

location

Private virtual interface (VIF) to

detached VGW• 1 PVI per VGW

• 1 BGP ASN

• 1 802.1Q VLAN Tag

• 1 BGP MD5 key

Private fiber connectionOne or multiple

50 – 500 Mbps,

1 Gbps or 10 Gbps pipes

AWS

Region VPC

VPC

VPC

VPC

VPCTransit VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Customer

network

AWS Direct Connect

location

Private DX VIF to

dedicated VGW

100.64.127.224 / 27

Private Virtual Interface 1

VLAN Tag 101

BGP ASN 7224

BGP Announce 100.64.127.224/27

Interface IP 169.254.251.5/30

Customer Interface 0/1.101

VLAN Tag 101

BGP ASN 65001

BGP Announce Customer Internal

Interface IP 169.254.251.6/30

AWS

Region VPC

VPC

VPC

VPC

VPCTransit VPC

Spoke VPC

Spoke VPCSpoke VPC

Spoke VPC

Customer

network

AWS Direct Connect

location

Public DX VIF to

dedicated VGW

Public EIPs

Public Virtual Interface 1

VLAN Tag 501

BGP ASN 7224

BGP Announce AWS Regional

Public CIDRs

Interface IP Public /30 Provided

Customer Interface 0/1.501

VLAN Tag 501

BGP ASN 65501 (or Public)

BGP Announce Customer Public

Interface IP Public /30 Provided

NAT + Security layer

Equinix Chicago

Customer

network

us-west-2

VPC

VPC

VPCVPC

Transit VPC

VPC

us-east-2

VPC

VPC

VPCVPC

Transit VPC

VPC

AWS Direct Connect Inter-Region Connectivity

A single DX Public interface can reach all US regions

• Be selective in your public network announcements

• Filtering public prefix announcements if necessary

• Authoritative AWS public IP list available:

https://ip-ranges.amazonaws.com/ip-ranges.json

• For notification of IP changes, subscribe to SNS topic:

arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged

AWS Direct Connect Public Interface

Related Sessions

NET402 – Deep Dive: AWS Direct Connect and VPNs

Leverage corporate network

Headquarters

Branch

Branch

DX Location

Provider Edge (PE)Customer Edge (CE)

eBGP

Provider

MPLS

Network

PECE

PE

CE

eBGP

AWS Region

MPLS / IPVPN

PE DX

eBGP

CE PE

Headquarters

Branch

Branch

Chicago DX Location

eBGP

Provider

MPLS

Network

PECE

PECE

AWS

Ohio

region

Multi-region DX

PE DX

eBGP

CE PE

London DX Location

AWS

Ireland

region

PE DX

eBGP

Going global

AS 7224

AS 7224

100 BGP Route Max

100 BGP Route Max

• Private network, no Internet dependencies

• Predictable latency on DX connections

• Dedicated bandwidth to AWS

• Access to public networks of all US regions

over single US based DX connection

• Public DX BGP announcements may require

filtering

• For large networks, 100 route per VPC limit

may require summarization or default routes

• Cost of provider network and DX connections

Pro & Con: Transit VPC with DX

us-east-2

VPCVPC

VPC

VPCTransit VPC

VPC

us-west-2

VPC

VPC

VPC

eu-west-1

VPCVPC

VPC

VPCTransit VPC

VPC

AWS Network

Backbone

Provider

MPLS

Network

Branch Branch

NA

HQ

VPC

VPC

VPC

VPC

VPC

VPC

Chicago DX

AP

HQ

London DX

ap-northeast-1

VPC

VPC

VPC

VPC

Transit VPC

VPC

EU

HQ

Tokyo DX

Thank you!

Remember to complete

your evaluations!