LISA2017 Big Three Cloud Networking

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

The Ins-and-Outs of Networking in the Big Three Clouds

Chris "mac" McEniry

Introduction

2

Topics• Network Substrates

• Routing, Routing, Routing

• Access Control

3

• Focus on the Big Three Cloud Service Providers (B3CSP)

• Slides are indicated with what Cloud Provider it's talking about

• Amazon Web Services

• Azure

• Google Cloud Platform

4

What about Cloud X?This will not be covered...

5

What's the best cloud?This will be covered... (5 seconds)

6

"It Depends"That's always the answer to questions without requirements...

7

Not Covered: "Legacy" Models• Focus is on current practices. Some legacy practices are not even available to new

accounts.

• Two Main Areas

• Organization: Some providers have a first pass on account or resource organization. Not Covered.

• Networking: Each provider has a first pass on their network offerings. Not Covered:

• EC2 Classic

• Virtual Network Classic

• Legacy Networking

8

Not Covered: IPv6 Addresses• Changes assumptions about Internal/External IP allocation

• Not the majority of use cases (yet)

• Support

• AWS: Yes

• Azure: ?

• GCP: No

9

Beware of Limits/Quotas• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/

VPC_Appendix_Limits.html

• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits

• https://cloud.google.com/compute/docs/vpc/#quotas_and_limits

• https://cloud.google.com/router/quotas

10

Design Exercises• Pen and paper exercises

• Goals

• Better understanding of concepts

• Compare/Contrast implementations in the B3CP

• Best to attempt same goal with all three providers, but one is sufficient for this tutorial.

• Some scattered throughout. More at the end

• Write down assumptions you make when designing

11

Announcements since 9/15

12

• 9/20 VPC Resizing: Adding secondary CIDR

• 11/1: Direct Connect Gateway, 3 more CloudFront POPs, Direct Connect Pricing Update

• (11/27: re:Invent)

• Early 2019: Bahrain Region

13

• 9/21 Availability Zones

• 9/26 (Ignite) VNet Service Endpoints. DDoS Protection for VNets. NSG using Application Security Groups, Service Tags, more rule combinations. 1 Gbps VPN connection. Monitoring ExpressRoute, S2S VPN. ExpressRoute Public access for MS Services, Route Filters. P2S macOS support. P2S AD Authentication. More ExpressRoute Partners.

• 9/27 VNet Integration for Azure Storage and Azure SQL. Storage Firewalls. SQL Endpoints.

• 10/31 Triple capacity in China

14

• 9/19 Sao Paulo Region

• (10/3 Custom IAM Roles)

• 10/7 Multiple (<=8) NICs GA.

• 10/31 Mumbai Region. Dedicate Interconnect GA, more POPs.

• 11/2 Faster SDN

15

Organizing

16

Organizing• Structure of CSP physical and logical/administrative build outs has impacts

on network capabilities and topology

• Examples

• What do you need to account for to build resiliency into the infrastructure?

• How do you maintain separation of administrative domains (e.g. account) while allowing traffic to traverse where needed?

• Does it make sense to do large shared networks, or smaller finely tuned networks?

17

Physical Organization• All B3CSP have Regions

• Geographically separate facilities hosting resources

• All B3CSP have some sub-Region container (separate power, network hardware, etc)

• Availability Zones

• Fault Domains

• Zones

18

Availability Zones, Zones• Separate Data Centers that are close to each other

• AWS: "less than 2 ms latency between each other"

• Directly exposed to customer

• You have to choose where to put resources

19

Fault Domains, Paired Regions• Fault Domain is similar to AZ/Z but is not directly manipulated by customer

• Specify intent by putting resources into Availability Sets

• "I want my 5 web servers to not fail all at the same time"

• Availability Set distributed across Fault Domains*

• Paired Regions: Regions in the same geographic region but are kept separate (> 300 miles, not operated on at the same time, etc)

• East US<->West US, UK West<->UK South

20

Locations

21

Regions

AWS 16 44 Availability Zones

Azure 26(not counting Gov)

60 Fault Domains(not counting Gov)

GCP 12 36 Zones

22

23

24

Logical/Administrative Organization• Each CSP has a way of administering users and resource permissions

• Comparisons:

• Where users are

• Where resources are

• Where permissions are set

• Fundamentally asking "Who can spin up a VM in this network? Who can stop a VM that is running in this network?"

25

(Billing) Accounts• Single unified item

• Hard boundary which contains all resources associated with it - can't share

• Can be in an organization, but only resources are not shared across - only policies

• Can build IAM accounts inside of this (or roles which connect from other accounts)

• All permissions are primarily set for actions (not targets) on the account*

• Must change account/role into another account to operate on its resources

26

Account / Tenant• Single unified item - Microsoft or Organizational Account

• Hard boundary which holds all resources associated with it

• Subdivided into Subscriptions, and then Resource Groups

• Access Control can be place on Subscriptions and/or Resource Groups

• Resources bound to one Resource Group/Subscription at a time

• Can move (with some restrictions) resources across Subscriptions/RGs

27

Cloud Resource Hierarchy• Organization

• Root of hierarchy

• Projects

• Core organizational component

• Associated with Organizations (or stand alone)

• Can have own permissions (and inherit)

• Resources

• Associated with Projects

• Can have own permissions (and inherit)

• In some cases, can be shared with other projects

28

Tags• Instances can be tagged

• Tags are used in selection items

• Firewall

• Routing

29

Network Substrate

30

Network Substrate• Virtual version of the traditional physical networks

• Handles the basic packet forwarding

• Organized into Subnets

• Supports resilience - spread over different resources

• Supports administrative separation - grouping of similar services

31

Not like a physical network• All packet forwarding based on some sort of mapping or hidden

networking layer

• Unicast only: No support for Broadcast or Multicast

• No transit networking through the substrate

32

Virtual Private Cloud (VPC)• A single CIDR allocated to a Region in one Account

• Subnets are smaller contained CIDRs assigned to an Availability Zone

• Subnet CIDRs can't overlap

• VPC CIDR can overlap regardless of Region/Account (as long as you don't want to connect them together)

• Default VPC per Region (172.31.0.0/16) with /20 subnets in each AZ

• Used for resilience and administrative separation

33

VPC Addressing• Can be RFC1918 or other IP space

• AWS will not advertise space out to the Internet

• Each object is a contiguous CIDR

• Allocated at time of object (VPC, Subnet) creation

• Can't change without destroying old/creating new

34

VirtualNet (VNet)• Single CIDR allocated to one Region in one Subscription

• Subnets are smaller contained CIDRs that span Fault Domains for that Region

• Subnet CIDRs can't overlap

• VNET CIDRs can overlap regardless of Region/Subscription (as long as you don't want to connect them together)

• Used for administrative separation

35

VNet Addressing• Can be RFC1918 or other IP space

• Azure will not advertise space out to the Internet

• Single CIDR at time of creation for VNet

• Subnets can change CIDR if it's not in use at all

• Must support a couple of special subnets

• First contiguous CIDR of VNet : Used for Load Balancers

• "Gateway Subnet" : Used for putting gateway devices (i.e. VPN)

36

Virtual Private Cloud Network (VPC Network)

• Global private communication space allocated to a Project

• Can be shared with other Projects

• Subnets are single CIDRs allocated to a Region, and can span Zones

• Two modes of allocation (one way switch from Auto -> Custom)

• Auto Mode: Allocate a subnet to each region. Can manually add your own.

• Custom Mode: Manually build subnets (recommended)

• Each subnet has primary CIDR

• (In Custom Mode) Can allocate secondary CIDR - typically for container networks

37

VPC Network Addressing • RFC1918 space only

• Auto Mode: Predefined /20 to each region. E.g.

• 10.128/20 -> us-central1

• 10.132/20 -> europe-west1

• Instances have primary IP and can have alias IPs

• Alias IPs can come from primary CIDR or secondary CIDR

• Can use CIDR (not just host) for Alias IPs on secondary

• Secondary CIDR does not reserve network/gateway IPs

38

AWS Azure GCP

Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network

IP Addressing

RFC1918 or OtherCarving up CIDR of VPC

RFC1918 or OtherCarving up CIDR of VNet

RFC1918 onlyAccumulation of Subnet CIDRs

Locality One Region One Region Global

Subnet Locality One AZ Region Selection of Zones in a Region

CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR

Account Resource Sharing

NoUsers use multiple Subs.

Resources bound to one VNet inside one Sub at a time

Across Projects - YesAcross Organizations - No

Resiliency• Build out a Network Substrate

• Capable of surviving a failure of one subregion area (AZ/FD/Z)

• Supports a 3 Tier application (web/app/db) with clear delineation

40

Instance Properties

41

Instance Properties• IP Forwarding / Source-Destination Checking

• Checks whether a packet headed to the instance matches the IP(s) of the instance, or whether a packet leaving the instance matches the IP(s) of the instance

• Instance NIC Properties

• Number of IPs per NIC

• Number of NICs per VM

• Locations of NICs

42

AWS Azure GCP

Forwarding Property Source/Destination Check IP Forwarding

(enableIPForwarding) IP Forwarding (can-ip-forward)

Property Default On Off Off

NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface

IPs per NIC 6-50 50Unspecified

(Alias IP not supported with multiple NICs)

NICs per Instance 1-15 2-8 1-8

NIC Locations Same VPC Same VNet Each must be on separate VPC

Networks

Routing - Inside

44

Route Tables• Route Table (rtb) = Multiple sets of routes

• Default or Main Route Table is nothing specific is configured

• Each Subnet is associated with a single Route Table

• Routes are either static (manually configured) or propagated (from BGP connections)

• Priority

• Most specific match

• Static

• Propagated

45

Route Tables• Route in a Route Table = Prefix + Next Hop

• Next Hop Types

• Instance NIC (Src/Dst Check)

• Peering Connection: For connections to other VPCs

• Virtual Private Gateway: For connections to VPNs or Private Circuits

• NAT Gateway: For egress NATing

• VPC Endpoint: For supported AWS Services

• Internet Gateway

46

System Routes + BGP + UDR• All Subnets have System Routes

• Can't be removed, but can be shadowed by custom routes

• Can add additional (custom) User Defined Routes via a Route Table

• Priority

• Most specific match

• User Defined Route

• BGP System Route

• System Internet Route

47

System Routes• 3 Default Route Sets automatically associated with VNet

• Local VNet's Subnets

• Associated Networks: Routes propagated from Peering / VPN / ExpressRoute

• Internet

48

User Defined Routes• Customer configured routes that exist additionally to System Routes

• Prefix + Next Hop. Next Hop one of:

• (Local) VNet: For VNet CIDR Destinations

• Virtual Network Gateway: Site-to-Site Connection

• Virtual Appliance: VMs inside VNet (IP Forwarding)

• Internet

• None: Blackhole

• Can't route back into a subnet

49

Routes• Global Routing Table for a VPC Network

• Custom Static Routes

• Priority

• Most specific match

• Highest priority (lowest by value)

• Multipath (Hash: Protocol, Src IP, Dst IP, Src Port, Dst Port)

50

Routes• Route =

• Name + VPC network + Prefix + (optional) Instance Tags + Priority + Next Hop.

• Next Hop one of

• Instance

• IP - inside of network (must be a primary IP)

• Gateway - The internet gateway (though maybe more options in the future)

• VPN Tunnel

51

AWS Azure GCP

Name Route Tables System Routes +User Defined Routes Routes

Route Selection

Most specific CIDR match,Static

Propagated

Most Specific CIDR match,User Defined Routes,

BGP Routes,System Routes

Most specific CIDR match,then by priority,

then mutlipath hash

Route Sharing

All subnets associated with same Route Table

All subnets associated with same User Defined Route

One shared route table;Specific route rules applied via

instance tag

Default Routes for New

VPC CIDRInternet

Peered ConnectionsVNet CIDR

InternetSubnet CIDRs

Use an Instance as a Gateway• Connect two different network areas together

• Use an instance as a gateway

53

Routing - Internet

54

Ingress/Egress• For Instance to get in/out, it has to have

• Route to Internet ("Route - Inside")

• Policy to permit access to/from Internet ("Access Control")

• Allocated Public IP on the CSP's external network

• Or - something else can do the work for it

55

CSP Public IP Selection• All Internet routed addresses come from the CSP's allocated Internet IP space

• Allocated to a Region

• No BYOIP

• Two types

• Ephemeral Addresses: allocated and released when VM starts/terminates

• Static Addresses: allocated ahead of time and remain attached to the Account even if the VM is terminated (called Elastic IP for AWS)

• Azure/GCP: Can promote an ephemeral public IP to static

56

Public IP == Public NAT No Public IP == Possibly Only Egress Traffic*

• If a VM has a Public IP (ephemeral or static) associate with it, it gets a NAT for Internet traffic

• The remainder of this section talk mostly about the "No Public IP" cases

57

Internet Gateway (IGW)• No default Internet Route

• Have to allocate an IGW to the VPC

• Have to add route to IGW

• Instance has to be configured with a public IP

58

NAT Gateway (NGW)• AWS Managed Service which provides SNAT for egress traffic only

• Must provide AWS with designated Elastic IP (public side) and subnet which has an internet route

• Other subnets must use a different Route Table to send 0/0 to the NAT Gateway

• Assigned to an Availability Zone

• But can support multiple AZs --- impact on resilience

• Common patterns to see one ngw per AZ

• Dependent on an IGW+Route for the NGW's outside access

59

Shared SNAT• Default Internet Route provided

• Default shared SNAT

• Shared == possibly with other accounts

• Prevent this with Access Control (NSG)

60

Ephemeral IP• Default Internet Route provided

• Default ephemeral public IP provided on each instance

• can choose not to allocate

• can be limited with Organization Policy (beta)

61

NAT?• No Managed Service

• You stand up your own instance which is performing NAT

• Add 0/0 to override routes

• Can use tags to decide which Instances get external access

• Can use tags to decide which Instances get NAT access

62

AWS Azure GCP

Default Internet

Route for New Net

None Yes Yes

Default NAT None SNAT Ephermeral IP

DMZ + Private• Want DMZ hosts which can get out, and Internal hosts which have to go

through the DMZ to get out

64

Routing - CSP Networks

65

Routing - CSP Networks• On their networks, CSPs have...

• Other Network Substrates that we may want to privately communicate with

• Other Cloud Services

• These services may exist in local or remote Regions

• CSPs have a lot of similar behaviors - differences highlighted

66

VPC Peering (pcx)• Private connectivity between two VPCs in the same Region

• Same or different accounts

• Latency/bandwidth/cost is same as talking inside a VPC

• Non-overlapping IP space; Non-transitive

• Add routes to the Route Table(s) to be connected

• Can reference Security Groups in foreign VPC

67

VPC Endpoints (vpce)• Private connectivity between VPC and AWS Services

• Limited to same region services only

• Non-transitive

• Only some services provide VPC Endpoints - S3, DynamoDB

• Given an identifier (pl-xxxx) to use in Route Tables

• In some service policies, can use VPC as a permission item (i.e. this VPC can access this service)

• Can reference VPC Endpoint (pl-xxxx) in Security Groups (but not NACL)

68

Virtual Network Peering• Private connectivity between two VNets in the same Region

• Can be different Subscriptions but same Account*

• Non-overlapping IP space; Non-transitive, but can share VPN Gateway

• Latency/bandwidth/cost is same as talking inside a VPC

• System Routes are automatic for Peered VNet

• Can override with UDR to point to instances (on either side) as gateway

69* Being worked on, but in the interim, use VPN peering

VPC Network Peering• Private connectivity between VPC Networks (in all Regions)

• Same or different projects/organizations

• Latency/bandwidth/cost is same as talking inside a VPC

• Non-overlapping IP space; Non-transitive

• Once peered, can't create conflicts (overlapping subnets, routes overlapping, etc) among the full set of VPC Networks that are peered (even between two that aren't direct)

70

Private Google Access• Private connectivity (IP space exposed) to Google Services

• Can be enabled on a Subnet basis

• Must still use internet route to access (can be tag limited)

71

AWS Azure GCP

Peering Name VPC Peering VNet Peering VPC Network Peering

Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations

Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined

CSP Service Peering Name

VPC Endpoint N/A Private Google Access

Peers + Cloud Services• Peer Two Networks in the same CSP (same Account/Subscription/Project)

• Allow (preferably) private connections from both networks to CSP Cloud Services

73

Routing - Private Gateways

74

Private Gateways• Connection points for linking CSP Network to an outside CSP network

• VPN

• Physical

• Can be managed by a third party - Cloud Exchanges

• In all cases, you can run your own VPN software (third party VM appliances)

75

CSP Managed VPN• Use of IPSec devices to connect privately to an on-premise (or other)

network

• Connect VPC/VNet/VPC Network to Corporate Office or Data Center

• Connect VPC/VNet/VPC Network to other CSP Network

• Inter-Region VPC/VNet network connectivity

76

Physical• Private Circuit connecting your network at a CSP POP to an associated

CSP Region

• If not adjacent to POP, have to get tail circuit to connect your site to POP

• Can also be used to peer with Public side of CSP network

• Access to other CSP services

• Access to CSP management interfaces

77

Limits• VPNs and Physical Connections have limits

• In general, can setup parallel connections and rely on multipath

• Multiple VPN tunnels working together

• Multiple Physical connections working together

• Combination of Physical and VPN - preference usually goes to Physical

78

Virtual Private Gateway (VGW)• Logical router sitting outside of VPC

• Associated with single VPC; so limited to a Region

• All foreign networks come in via VGW

• Set routes in the VPC Route Tables to forward traffic to VGW

• Or allow routes to be propagated from the VGW peers

• Only the VPC CIDR is advertised to VGW and its peers

• Can't transit for VPC Peering or VPC Endpoints or Internet traffic - though can proxy

• Will transit for non-VPC peers - "CloudHub"

79

Hardware VPN• Private IPSec connectivity between AWS and non-AWS managed VPN device (aka Customer

Gateway - CGW)

• Can't use between two AWS Regions

• Site-to-Site

• Always builds two tunnels (two devices are allocated on the AWS side) to one CGW

• If you want redundancy on the CGW side, you'll build four tunnels

• IP of AWS VPN side not designated until VPN Connection is configured

• Must supply static IP for CGW (can use same for multiple connections - NAT-T supported)

• Static or BGP

80

Direct Connect (DX)• AWS Physical Connection (CON)

• 1Gbps, 10Gbps options (can do smaller through a Cloud Exchange)

• Can also use multiple (<=4) CON together with LACP

• BGP Peering Only

81

Direct Connect (DX) - Sharing• Can be used with multiple accounts/VPCs.

• Each Direct Connect Connection (CON) is split into virtual interfaces (VIF).

• You assign the VIF information

• VLAN Assignment

• BGP Neighbor

• Hosting Account (one which has the CON) handles creation of VIF

• Guest Account attaches VIF to VGW

• Once attached, can't reattach (have to destroy/recreate)

82

Virtual Network Gateway• Managed Gateway Devices

• Deployed as VMs inside of a dedicated "Gateway Subnet" (/27 or larger)

• Two types

• VPN Gateway

• ExpressRoute

• Can only have one of each type per VNet

• Multiple SKUs for sizing

83

VPN Gateway• Site-to-Site

• IPSec from VPN Gateway to Local Network Gateway

• Allocated to a Region

• Static (Policy list of CIDRs) or BGP Route Based

• Limits/Sizing place preference on Route Based

• Gateway Transit: can extend reachability to Peered VNets

• Use this topology to get around some limits

84

VPN Gateway• VNet-to-VNet

• (special case of Site-to-Site)

• Connect separate Regions and separate Accounts/Subscriptions together

• Point-to-Site

• Access via the in-box Windows VPN SSTP Client

85

ExpressRoute• Private BGP Connections to Microsoft

• Handoff

• IP VPN (i.e. MPLS)

• Ethernet tail to POP

• Cloud Exchange Virtual Cross Connection

• Handoff affects location of on-prem and latency, but not necessarily Region access (depending on Subscription level)

• Offered as redundant pair of connections (no SLA without)

• Provides

• VNet Private Peering

• Azure Public Peering

• (add-on) Microsoft Peering for Office 365 et al

86

ExpressRoute• For each Circuit, can setup one to all of the three peering types

• Peering Information:

• BGP IP/Neighbor IP

• VLAN

• Neighbor ASN - Public restrictions for Public/Microsoft

• For Microsoft: Routing Registry Name

• Restrictions on Peering Type for advertised routes (# and RFC1918)

87

ExpressRoute to VNet• Owner Subscription handles Circuit creation and Peering configuration

• Owner Authorizes Other Subscriptions

• Other Subscription link VNets to ExpressRoute Circuit VNet Peering

• Must have a Virtual Network Gateway (ExpressRoute Type) configured ahead of time

• Standard vs Premium limits on # of VNet linkages and Region

• All VNets linked to the same ExpressRoute Circuit

• ExpressRoute and Point-to-Site are not supported together in the same VNet

88

Cloud Router• Managed Router which handles all dynamic routing via BGP relationships with other

routers

• Makes changes to your Routes

• Scoped to local or global route changes (i.e. advertise remote Regions)

• You assign a private ASN to Cloud Router

• Create multiple interfaces (link-local addressing)

• Create multiple BGP neighbor configurations

• Unlike the other CSPs, even if you setup tunnel devices, the Cloud Router is the BGP neighbor

89

Cloud VPN Gateway• Managed IPSec VPN Endpoint which can run multiple tunnels

• Allocated to a Region, but can forward any traffic on the VPC Network

• Can connect too another Project's or Organization's VPN Gateway

• You specify tunnels to create (so not always in pairs - but that is recommended)

• Typical Tunnel Configuration

• Local IP: Assigned from your pool of static external IPs

• (Static mode) Local subnets, IP ranges - Can't change after creating

• Peer IP, IKE, ESP, PFS, Shared Secret

• NAT-T not supported

90

Cloud VPN Gateway• Static Mode

• Specify traffic selection via --remote-traffic-selector

• Add VPN Gateway as a NextHop in your route table

• Dynamic Mode

• Create an interface on Cloud Router that is associate to the tunnel

• Handle all traffic selection via BGP relationship with Cloud Router

• Cloud Router adds VPN Gateway as NextHop

• BGP Peer Information can be configured or have GCP generate it

91

Dedicated Interconnect• GCP Physical Connection

• 10 Gbps

• Can use multiple with LACP

• BGP Established with Cloud Router (i.e. requires Cloud Router)

• Is divided up into VLAN Attachments

• GCP allocates the VLAN and BGP IPs to use (ASN specified by Cloud Router)

• Use that information to configure Cloud Router and on-prem router

92

Dedicated Interconnect - Sharing• Can be shared among Projects in an Organization

• Even if not using a Shared VPC Network

• Two methods - both using IAM permissions

• Hosting Project grants permissions to Guest Project's Users to update Interconnect

• Guest Project grants permissions to Hosting Project to update Gust Project's Cloud Router

93

AWS Azure GCP

Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router

Gateway Model

Router outside of VPCIs NextHop for all gateway traffic

VM Insides of "Gateway Subnet" which handle routing and

gateway

BGP Talker that updates routes. Separate gateways that forward

traffic.

Substrate Route Mgmt

You add routes to RTB, orsetup to propagate all routes

from VGW to RTBRoutes show up as System

Routes Cloud Router updates Routes

Gateway Peer Transit Yes

VPN: Yes including VNet PeeringExpressRoute: No Yes

AWS Azure (VNet Peering) GCP

VPN Name

VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway

Cloud Interconnect - IPSec VPNaka Cloud VPN

Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site

Deploy Model

Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop

Endpoint associated with VPC Network as NextHop

VPN Transit Yes (CloudHub) No Yes

Size Options

1 3 (4) 1

Advertised Routes

1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)

Received Route limits

100 Prefixes . 100 Prefixes (CR limit)

Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway

Route Selection

Static: What is specifiedBGP: VPC + All VGW BGP Peers

Static: What is specifiedBGP: VNet, Configured Peers

Static: What is specifiedBGP: By Cloud Router

AWS Azure GCP

Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated

Interconnect

Link Options 1Gbps, 10Gbps, LACP

IP VPN50,100,200,500M ; 1,2,5,10Gbps

Cloud Exchange10Gbps, LACP

Routing BGP BGP BGP(to Cloud Router)

VLAN/BGP Allocation You Pick You Pick GCP Picks

Sharing YesCentrally Managed

YesCentrally Managed Configuration

RBAC VNet Linking

YesRBAC Managed

Route Limits 100 Prefixes Depends on Size

VNet: 4000-10000 100 (CR limit)

CSP Network as Data Center Extension• Setup CSP Network which only has connectivity from a Private Data

Center via a physical connection

• Any ingress or egress traffic goes via the Private Data Center

97

Cross Region Network• Want to peer two Networks in the Same VPC

98

Connect Multiple CSP Together• Have two CSP Networks

• One in each of two different CSPs

• "Peer" them together

99

Cloud Storage from Corp Office• Setup protected path from corporate office to Storage offering of CSP

100

Access Control

101

AWS Access Control• Two types

• Security Groups (SG): For instances

• Network Access Control Lists (NACLs): For subnets

102

Security Group (SG)• Stateful Packet Filters

• Separate lists for ingress and egress

• Applied on Network Interfaces

• By "Applied to Instance," it really means "Applied on the Instance's Primary NIC"

• Positive Control: Only ALLOW. No DENY

• Can apply multiple (5) SG on an NIC

• ALLOWs are cumulative

103

Security Group (SG)• Structure:

• Protocol

• Port Range, or ICMP Type

• Source (ingress), or Destination (egress) CIDR, or SG

• New SG has default egress allows anything out

• "default" SG available when VPC created (and always available)

• Ingress: ALLOW from "default" SG to ANY

• Egress: ALLOW to ANY

104

Security Group (SG)• Applied to Instances

• Context is focused on Instance (or groups of Instances) Rules

• But with tagging is hard to tailor to only your instances for modifying SG

105

Network Access Control List (NACL)• Stateless

• Only one allowed per subnet

• Ordered priority

• Positive and Negative control (ALLOW/DENY)

106

Azure Access Control• Just one

• Network Security Groups (NSG)

107

Network Security Group (NSG)• 1 per NIC and/or subnet

• Applied in priority order (ascending)

• default rules for inbound and outbound on both (permit out, permit VNET)

• The NSG on the NIC, if present, and on the Subnet have to permit to permit traffic. One DENY on either will block traffic.

• NSG Tags : canned collections of CIDRs (e.g. VIRTUAL_NETWORK)

• Can do flow logging

108

Network Security Group (NSG)• Structure

• Priority (Low numerically is Higher Priority)

• Name

• Protocol + Source IP/Tag + Source Port + Destination IP/Tag + Destination Port

• Allow/Deny

109

Network Security Group (NSG)• Managing

• Action (Permissions): Microsoft.Network/networkSecurityGroup/*

• Roles: Network Contributor, Owner

• Context is global to VNet

110

Firewall• Single managed firewall for VPC network

• Manages inbound and outbound

• Can limit sources/destinations using Target Tags

• Tag matched against Tags on Instance

111

Firewall• Structure

• Priority (ascending, first match wins)

• (Ingress) Protocol + Destination Port + Source IP

• (Egress) Protocol + Destination Port + Destination IP

• Target Tags

• Allow/Deny

112

Firewall• Managing

• Permissions: compute.firewalls.*

• Role: roles/compute.securityAdmin

• Context is global to Project/Share VPC Project

113

AWS AWS Azure GCP

Name Security Group Network Access Control List Network Security Group Firewall

State Stateful Stateless Stateful Stateful

Additive Positive Only Positive/Negative Postive/Negative Positive/Negative

Target NIC Subnet NIC or Subnet VPC(target tags for instance)

Multiple Yes (5 on instance 250 in account) No No No

Ordered No Yes Yes Yes

Setup Access Control for 3 Tier App• Given 4 VMs - web, app, db, bastion - setup access control such that:

• Anyone is allowed from any tcp port to "web" on 443/tcp

• "web" is allowed from any tcp port to "app" on 8009/tcp

• "app" is allowed from any tcp port to "db" on 3306/tcp

• "bastion" is allowed from any tcp port to all three instances on 22/tcp

• "bastion" is allowed from any udp port to all three instances on 161/udp

115

Summary and Next Steps

116

Locations

117

Regions

AWS 16 44 Availability Zones

Azure 26(not counting Gov)

60 Fault Domains(not counting Gov)

GCP 12 36 Zones

AWS Azure GCP

Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network

IP Addressing

RFC1918 or OtherCarving up CIDR of VPC

RFC1918 or OtherCarving up CIDR of VNet

RFC1918 onlyAccumulation of Subnet CIDRs

Locality One Region One Region Global

Subnet Locality One AZ Region Selection of Zones in a Region

CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR

Account Resource Sharing

NoUsers use multiple Subs.

Resources bound to one VNet inside one Sub at a time

Across Projects - YesAcross Organizations - No

Substrate

AWS Azure GCP

Forwarding Property Source/Destination Check IP Forwarding

(enableIPForwarding) IP Forwarding (can-ip-forward)

Property Default On Off Off

NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface

IPs per NIC 6-50 50Unspecified

(Alias IP not supported with multiple NICs)

NICs per Instance 1-15 2-8 1-8

NIC Locations Same VPC Same VNet Each must be on separate VPC

Networks

Instance Properties

AWS Azure GCP

Name Route Tables System Routes +User Defined Routes Routes

Route Selection

Most specific CIDR match,Static

Propagated

Most Specific CIDR match,User Defined Routes,

BGP Routes,System Routes

Most specific CIDR match,then by priority,

then mutlipath hash

Route Sharing

All subnets associated with same Route Table

All subnets associated with same User Defined Route

One shared route table;Specific route rules applied via

instance tag

Default Routes for New

VPC CIDRInternet

Peered ConnectionsVNet CIDR

InternetSubnet CIDRs

Routing - Inside

AWS Azure GCP

Default Internet

Route for New Net

None Yes Yes

Default NAT None SNAT Ephermeral IP

Routing - Internet

AWS Azure GCP

Peering Name VPC Peering VNet Peering VPC Network Peering

Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations

Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined

CSP Service Peering Name

VPC Endpoint N/A Private Google Access

Routing - CSP Networks

AWS Azure GCP

Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router

Gateway Model

Router outside of VPCIs NextHop for all gateway traffic

VM Insides of "Gateway Subnet" which handle routing and

gateway

BGP Talker that updates routes. Separate gateways that forward

traffic.

Substrate Route Mgmt

You add routes to RTB, orsetup to propagate all routes

from VGW to RTBRoutes show up as System

Routes Cloud Router updates Routes

Gateway Peer Transit Yes

VPN: Yes including VNet PeeringExpressRoute: No Yes

Routing - Private Routes

AWS Azure (VNet Peering) GCP

VPN Name

VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway

Cloud Interconnect - IPSec VPNaka Cloud VPN

Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site

Deploy Model

Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop

Endpoint associated with VPC Network as NextHop

VPN Transit Yes (CloudHub) No Yes

Size Options

1 3 (4) 1

Advertised Routes

1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)

Received Route limits

100 Prefixes . 100 Prefixes (CR limit)

Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway

Route Selection

Static: What is specifiedBGP: VPC + All VGW BGP Peers

Static: What is specifiedBGP: VNet, Configured Peers

Static: What is specifiedBGP: By Cloud Router

Routing - VPN

AWS Azure GCP

Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated

Interconnect

Link Options 1Gbps, 10Gbps, LACP

IP VPN50,100,200,500M ; 1,2,5,10Gbps

Cloud Exchange10Gbps, LACP

Routing BGP BGP BGP(to Cloud Router)

VLAN/BGP Allocation You Pick You Pick GCP Picks

Sharing YesCentrally Managed

YesCentrally Managed Configuration

RBAC VNet Linking

YesRBAC Managed

Route Limits 100 Prefixes Depends on Size

VNet: 4000-10000 100 (CR limit)

Routing - On-Premise

AWS AWS Azure GCP

Name Security Group Network Access Control List Network Security Group Firewall

State Stateful Stateless Stateful Stateful

Additive Positive Only Positive/Negative Postive/Negative Positive/Negative

Target NIC Subnet NIC or Subnet VPC(target tags for instance)

Multiple Yes (5 on instance 250 in account) No No No

Ordered No Yes Yes Yes

Access Control

Additional "Network" Areas• Load Balancers - some affect SNAT and packet forwarding

• DNS - interacts with Substrate DNS view

• CDN

127

Limits!!!• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/

VPC_Appendix_Limits.html

• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits

• https://cloud.google.com/router/quotas

128

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

Remember to fill in yourtutorial evaluation!

Thank You!

R6 - The Ins-and-Outs of Networking in the Big Three CloudsChris "mac" McEniry