Another Day, Another Billion Packets

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Author: Eric Brandwine, AWS Security

Speaker: Chris Conner, AWS Solutions Architecture

June 2016

Another Day, Another Billion

Packets

Deja Vu

We have the cloud

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon EC2 Elastic Load

Balancing

Customers have data centers

Whiteboard engineering

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon

EC2

Elastic Load

Balancing

EC2 as it was

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Why that doesn’t work

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 10.44.12.4/32: AWS

• 10.44.92.17/32: AWS

• 10.108.6.4/32: AWS

10.44.0.0/16

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Requirements

Customer selected IP addresses

Route aggregation for external connectivity

Conformance with existing network designs

172.31.0.0/18

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 172.31.0.0/18: AWS

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Amazon Virtual Private Cloud

This is just virtual networking!

Subnet ~= VLAN

VPC ~= VRF (virtual routing and forwarding)

But…

Scaling challenges

VLAN ID space is constrained

• 12 bits => 4096 total VLANs

VRF support is constrained

• Large routers => 1-2 thousand VRFs

Fixed ratio of VLANs:VRFs

Router and capacity dimensions

Big Router

Data Plane

Control

Plane

Big Router

Data Plane

Control

Plane

An example

Average router configuration line: 50 chars

Config per VPC: 10 lines

Subnets per VPC: 4

Config per subnet: 5 lines

Total VPCs: 2,000

Config size: 3 MB

But…

Doesn’t scale

• 12 bit VLAN ID = 4096 VLANs (not enough)

• BIG routers support 4000 VRFs ($200k+)

Large VLANs make NEs cry

Tied to vendor bugfix cycles (6 months +)

We want commodity, fungible network gear

• BIG virtual routers are built by few companies

• Interoperability of advanced features is marginal

Silos of capacity

A

C

B

FE

D

G

A AA

A

B

C

B B

B B

C

D

F FF

D

D

B

G G

/4 /4

/40 /40

0

0

0

0

1324 132

C

G G

3 27

D DD

9910

F F F F F

1815 40

BB B B B

BB B B B

BB B B B

B B

Implementation requirements

Scale to millions of environments the size of Amazon.com

Any server, anywhere in a region can host an instance

attached to any subnet in any VPC

Concepts

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Server:

Physical host in an

Amazon data center

Instance:

Amazon EC2

instance owned by a

customer

VPC:

Amazon Virtual

Private Cloud

owned by a

customer

VPC ID:

Identifier for a VPC

such as vpc-

1a2b3c4d

Mapping Service:

Distributed lookup

service. Maps VPC

+ Instance IP to

server

L2 - Ethernet

10.0.0.2

10.0.0.3

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

The switch floods the

ARP request out all

ports

Ethernet Switch

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

The switch snoops the

ARP response and

learns the port for

MAC(10.0.0.3).

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

L2 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.3?

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.0.3

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.0.3)

10.0.0.2

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue10.0.0.2 is at

192.168.0.3

L2 - VPC

…

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Grey 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Blue 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

192.168.0.4 is not

hosting any instances

in VPC Blue.

Mapping Denied

Alarm Raised

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.4)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.4

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.4

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.4 is at

192.168.0.4


Dst: 192.168.1.4

Mapping invalid!

192.168.1.4 does not

deliver the packet to

the instance.

Alarm Raised.

L3 – IP routing

10.0.0.2

10.0.1.3

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

Ethernet Switch

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

RouterEthernet Switch

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

Src: 192.168.0.3


Query:

Blue 10.0.0.1


Dst: 192.168.0.3

Reply:

Host: Gateway

MAC: MAC(10.0.0.1)

10.0.0.2

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Src: 192.168.0.3


Query:

Blue 10.0.1.3


Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.1.3)

10.0.0.2

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue 10.0.0.2 is at

192.168.0.3

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Caching

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

10.0.0.0/18

172.16.0.0/16

10.0.0.0/24 10.0.1.0/24

10.0.0.7

10.0.0.8

10.0.0.9

10.0.1.12

10.0.1.51

VPC: Blue

Src: 192.168.0.3

Dst: ???

L3 Src: 10.0.0.7

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Getting home (or anywhere, really)

Edges

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

Mapping Service

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

…

Edges (three different ones)

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPN



Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

AWS

Direct Connect



Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Internet

54.148.157.46


VPNEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Direct ConnectEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

InternetEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 54.148.157.46

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Image credit: Wikipedia

https://en.wikipedia.org/wiki/1918_Eighth_Avenue

A brief diversion

VPC pricing

Cost per VPC: $0.00

Cost per subnet: $0.00

Upcharge per instance: $0.00

Nov 10, 2010

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.2.12

172.31.2.51

VPC as a platform

VPC as a platform

VPN and Direct Connect

Security group egress filtering

Network ACLs

Routing tables

Elastic Network Interfaces (ENIs)

Multiple IPs

Amazon S3 endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Amazon S3 endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

EdgesMapping Service

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

S3.us-east-1 Edge 192.168.4.4

…

A new edge


Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC Endpoint 1a2b3c4d

Src: 54.68.100.245

Dst: 54.231.33.89

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

S3 endpoint

Endpoints and policy

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

{

"Statement": [

{

"Sid": "Access-to-specific-bucket-only",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject"

],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"]

}

]

}

{

"Statement": [

{

"Sid": "Access-to-specific-VPC-only",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"],

"Condition": {

"StringNotEquals": {

"aws:sourceVpc": "vpc-111bbb22"

}

}

}

]

}

Simple Complex

Limited Flexible

EC2 VPC

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Default VPC

Simple Complex

Limited Flexible

EC2 - VPC

Remember to complete

your evaluations!

Engineering

Another Day, Another Billion Packets