Embed Size (px)
Citation preview
October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17
The Ins-and-Outs of Networking in the Big Three Clouds
Chris "mac" McEniry
Introduction
2
Topics• Network Substrates
• Routing, Routing, Routing
• Access Control
3
• Focus on the Big Three Cloud Service Providers (B3CSP)
• Slides are indicated with what Cloud Provider it's talking about
• Amazon Web Services
• Azure
• Google Cloud Platform
4
What about Cloud X?This will not be covered...
5
What's the best cloud?This will be covered... (5 seconds)
6
"It Depends"That's always the answer to questions without requirements...
7
Not Covered: "Legacy" Models• Focus is on current practices. Some legacy practices are not even available to new
accounts.
• Two Main Areas
• Organization: Some providers have a first pass on account or resource organization. Not Covered.
• Networking: Each provider has a first pass on their network offerings. Not Covered:
• EC2 Classic
• Virtual Network Classic
• Legacy Networking
8
Not Covered: IPv6 Addresses• Changes assumptions about Internal/External IP allocation
• Not the majority of use cases (yet)
• Support
• AWS: Yes
• Azure: ?
• GCP: No
9
Beware of Limits/Quotas• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/
VPC_Appendix_Limits.html
• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits
• https://cloud.google.com/compute/docs/vpc/#quotas_and_limits
• https://cloud.google.com/router/quotas
10
Design Exercises• Pen and paper exercises
• Goals
• Better understanding of concepts
• Compare/Contrast implementations in the B3CP
• Best to attempt same goal with all three providers, but one is sufficient for this tutorial.
• Some scattered throughout. More at the end
• Write down assumptions you make when designing
11
Announcements since 9/15
12
• 9/20 VPC Resizing: Adding secondary CIDR
• 11/1: Direct Connect Gateway, 3 more CloudFront POPs, Direct Connect Pricing Update
• (11/27: re:Invent)
• Early 2019: Bahrain Region
13
• 9/21 Availability Zones
• 9/26 (Ignite) VNet Service Endpoints. DDoS Protection for VNets. NSG using Application Security Groups, Service Tags, more rule combinations. 1 Gbps VPN connection. Monitoring ExpressRoute, S2S VPN. ExpressRoute Public access for MS Services, Route Filters. P2S macOS support. P2S AD Authentication. More ExpressRoute Partners.
• 9/27 VNet Integration for Azure Storage and Azure SQL. Storage Firewalls. SQL Endpoints.
• 10/31 Triple capacity in China
14
• 9/19 Sao Paulo Region
• (10/3 Custom IAM Roles)
• 10/7 Multiple (<=8) NICs GA.
• 10/31 Mumbai Region. Dedicate Interconnect GA, more POPs.
• 11/2 Faster SDN
15
Organizing
16
Organizing• Structure of CSP physical and logical/administrative build outs has impacts
on network capabilities and topology
• Examples
• What do you need to account for to build resiliency into the infrastructure?
• How do you maintain separation of administrative domains (e.g. account) while allowing traffic to traverse where needed?
• Does it make sense to do large shared networks, or smaller finely tuned networks?
17
Physical Organization• All B3CSP have Regions
• Geographically separate facilities hosting resources
• All B3CSP have some sub-Region container (separate power, network hardware, etc)
• Availability Zones
• Fault Domains
• Zones
18
Availability Zones, Zones• Separate Data Centers that are close to each other
• AWS: "less than 2 ms latency between each other"
• Directly exposed to customer
• You have to choose where to put resources
19
Fault Domains, Paired Regions• Fault Domain is similar to AZ/Z but is not directly manipulated by customer
• Specify intent by putting resources into Availability Sets
• "I want my 5 web servers to not fail all at the same time"
• Availability Set distributed across Fault Domains*
• Paired Regions: Regions in the same geographic region but are kept separate (> 300 miles, not operated on at the same time, etc)
• East US<->West US, UK West<->UK South
20
Locations
21
Regions
AWS 16 44 Availability Zones
Azure 26(not counting Gov)
60 Fault Domains(not counting Gov)
GCP 12 36 Zones
22
23
24
Logical/Administrative Organization• Each CSP has a way of administering users and resource permissions
• Comparisons:
• Where users are
• Where resources are
• Where permissions are set
• Fundamentally asking "Who can spin up a VM in this network? Who can stop a VM that is running in this network?"
25
(Billing) Accounts• Single unified item
• Hard boundary which contains all resources associated with it - can't share
• Can be in an organization, but only resources are not shared across - only policies
• Can build IAM accounts inside of this (or roles which connect from other accounts)
• All permissions are primarily set for actions (not targets) on the account*
• Must change account/role into another account to operate on its resources
26
Account / Tenant• Single unified item - Microsoft or Organizational Account
• Hard boundary which holds all resources associated with it
• Subdivided into Subscriptions, and then Resource Groups
• Access Control can be place on Subscriptions and/or Resource Groups
• Resources bound to one Resource Group/Subscription at a time
• Can move (with some restrictions) resources across Subscriptions/RGs
27
Cloud Resource Hierarchy• Organization
• Root of hierarchy
• Projects
• Core organizational component
• Associated with Organizations (or stand alone)
• Can have own permissions (and inherit)
• Resources
• Associated with Projects
• Can have own permissions (and inherit)
• In some cases, can be shared with other projects
28
Tags• Instances can be tagged
• Tags are used in selection items
• Firewall
• Routing
29
Network Substrate
30
Network Substrate• Virtual version of the traditional physical networks
• Handles the basic packet forwarding
• Organized into Subnets
• Supports resilience - spread over different resources
• Supports administrative separation - grouping of similar services
31
Not like a physical network• All packet forwarding based on some sort of mapping or hidden
networking layer
• Unicast only: No support for Broadcast or Multicast
• No transit networking through the substrate
32
Virtual Private Cloud (VPC)• A single CIDR allocated to a Region in one Account
• Subnets are smaller contained CIDRs assigned to an Availability Zone
• Subnet CIDRs can't overlap
• VPC CIDR can overlap regardless of Region/Account (as long as you don't want to connect them together)
• Default VPC per Region (172.31.0.0/16) with /20 subnets in each AZ
• Used for resilience and administrative separation
33
VPC Addressing• Can be RFC1918 or other IP space
• AWS will not advertise space out to the Internet
• Each object is a contiguous CIDR
• Allocated at time of object (VPC, Subnet) creation
• Can't change without destroying old/creating new
34
VirtualNet (VNet)• Single CIDR allocated to one Region in one Subscription
• Subnets are smaller contained CIDRs that span Fault Domains for that Region
• Subnet CIDRs can't overlap
• VNET CIDRs can overlap regardless of Region/Subscription (as long as you don't want to connect them together)
• Used for administrative separation
35
VNet Addressing• Can be RFC1918 or other IP space
• Azure will not advertise space out to the Internet
• Single CIDR at time of creation for VNet
• Subnets can change CIDR if it's not in use at all
• Must support a couple of special subnets
• First contiguous CIDR of VNet : Used for Load Balancers
• "Gateway Subnet" : Used for putting gateway devices (i.e. VPN)
36
Virtual Private Cloud Network (VPC Network)
• Global private communication space allocated to a Project
• Can be shared with other Projects
• Subnets are single CIDRs allocated to a Region, and can span Zones
• Two modes of allocation (one way switch from Auto -> Custom)
• Auto Mode: Allocate a subnet to each region. Can manually add your own.
• Custom Mode: Manually build subnets (recommended)
• Each subnet has primary CIDR
• (In Custom Mode) Can allocate secondary CIDR - typically for container networks
37
VPC Network Addressing • RFC1918 space only
• Auto Mode: Predefined /20 to each region. E.g.
• 10.128/20 -> us-central1
• 10.132/20 -> europe-west1
• Instances have primary IP and can have alias IPs
• Alias IPs can come from primary CIDR or secondary CIDR
• Can use CIDR (not just host) for Alias IPs on secondary
• Secondary CIDR does not reserve network/gateway IPs
38
AWS Azure GCP
Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network
IP Addressing
RFC1918 or OtherCarving up CIDR of VPC
RFC1918 or OtherCarving up CIDR of VNet
RFC1918 onlyAccumulation of Subnet CIDRs
Locality One Region One Region Global
Subnet Locality One AZ Region Selection of Zones in a Region
CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR
Account Resource Sharing
NoUsers use multiple Subs.
Resources bound to one VNet inside one Sub at a time
Across Projects - YesAcross Organizations - No
Resiliency• Build out a Network Substrate
• Capable of surviving a failure of one subregion area (AZ/FD/Z)
• Supports a 3 Tier application (web/app/db) with clear delineation
40
Instance Properties
41
Instance Properties• IP Forwarding / Source-Destination Checking
• Checks whether a packet headed to the instance matches the IP(s) of the instance, or whether a packet leaving the instance matches the IP(s) of the instance
• Instance NIC Properties
• Number of IPs per NIC
• Number of NICs per VM
• Locations of NICs
42
AWS Azure GCP
Forwarding Property Source/Destination Check IP Forwarding
(enableIPForwarding) IP Forwarding (can-ip-forward)
Property Default On Off Off
NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface
IPs per NIC 6-50 50Unspecified
(Alias IP not supported with multiple NICs)
NICs per Instance 1-15 2-8 1-8
NIC Locations Same VPC Same VNet Each must be on separate VPC
Networks
Routing - Inside
44
Route Tables• Route Table (rtb) = Multiple sets of routes
• Default or Main Route Table is nothing specific is configured
• Each Subnet is associated with a single Route Table
• Routes are either static (manually configured) or propagated (from BGP connections)
• Priority
• Most specific match
• Static
• Propagated
45
Route Tables• Route in a Route Table = Prefix + Next Hop
• Next Hop Types
• Instance NIC (Src/Dst Check)
• Peering Connection: For connections to other VPCs
• Virtual Private Gateway: For connections to VPNs or Private Circuits
• NAT Gateway: For egress NATing
• VPC Endpoint: For supported AWS Services
• Internet Gateway
46
System Routes + BGP + UDR• All Subnets have System Routes
• Can't be removed, but can be shadowed by custom routes
• Can add additional (custom) User Defined Routes via a Route Table
• Priority
• Most specific match
• User Defined Route
• BGP System Route
• System Internet Route
47
System Routes• 3 Default Route Sets automatically associated with VNet
• Local VNet's Subnets
• Associated Networks: Routes propagated from Peering / VPN / ExpressRoute
• Internet
48
User Defined Routes• Customer configured routes that exist additionally to System Routes
• Prefix + Next Hop. Next Hop one of:
• (Local) VNet: For VNet CIDR Destinations
• Virtual Network Gateway: Site-to-Site Connection
• Virtual Appliance: VMs inside VNet (IP Forwarding)
• Internet
• None: Blackhole
• Can't route back into a subnet
49
Routes• Global Routing Table for a VPC Network
• Custom Static Routes
• Priority
• Most specific match
• Highest priority (lowest by value)
• Multipath (Hash: Protocol, Src IP, Dst IP, Src Port, Dst Port)
50
Routes• Route =
• Name + VPC network + Prefix + (optional) Instance Tags + Priority + Next Hop.
• Next Hop one of
• Instance
• IP - inside of network (must be a primary IP)
• Gateway - The internet gateway (though maybe more options in the future)
• VPN Tunnel
51
AWS Azure GCP
Name Route Tables System Routes +User Defined Routes Routes
Route Selection
Most specific CIDR match,Static
Propagated
Most Specific CIDR match,User Defined Routes,
BGP Routes,System Routes
Most specific CIDR match,then by priority,
then mutlipath hash
Route Sharing
All subnets associated with same Route Table
All subnets associated with same User Defined Route
One shared route table;Specific route rules applied via
instance tag
Default Routes for New
VPC CIDRInternet
Peered ConnectionsVNet CIDR
InternetSubnet CIDRs
Use an Instance as a Gateway• Connect two different network areas together
• Use an instance as a gateway
53
Routing - Internet
54
Ingress/Egress• For Instance to get in/out, it has to have
• Route to Internet ("Route - Inside")
• Policy to permit access to/from Internet ("Access Control")
• Allocated Public IP on the CSP's external network
• Or - something else can do the work for it
55
CSP Public IP Selection• All Internet routed addresses come from the CSP's allocated Internet IP space
• Allocated to a Region
• No BYOIP
• Two types
• Ephemeral Addresses: allocated and released when VM starts/terminates
• Static Addresses: allocated ahead of time and remain attached to the Account even if the VM is terminated (called Elastic IP for AWS)
• Azure/GCP: Can promote an ephemeral public IP to static
56
Public IP == Public NAT No Public IP == Possibly Only Egress Traffic*
• If a VM has a Public IP (ephemeral or static) associate with it, it gets a NAT for Internet traffic
• The remainder of this section talk mostly about the "No Public IP" cases
57
Internet Gateway (IGW)• No default Internet Route
• Have to allocate an IGW to the VPC
• Have to add route to IGW
• Instance has to be configured with a public IP
58
NAT Gateway (NGW)• AWS Managed Service which provides SNAT for egress traffic only
• Must provide AWS with designated Elastic IP (public side) and subnet which has an internet route
• Other subnets must use a different Route Table to send 0/0 to the NAT Gateway
• Assigned to an Availability Zone
• But can support multiple AZs --- impact on resilience
• Common patterns to see one ngw per AZ
• Dependent on an IGW+Route for the NGW's outside access
59
Shared SNAT• Default Internet Route provided
• Default shared SNAT
• Shared == possibly with other accounts
• Prevent this with Access Control (NSG)
60
Ephemeral IP• Default Internet Route provided
• Default ephemeral public IP provided on each instance
• can choose not to allocate
• can be limited with Organization Policy (beta)
61
NAT?• No Managed Service
• You stand up your own instance which is performing NAT
• Add 0/0 to override routes
• Can use tags to decide which Instances get external access
• Can use tags to decide which Instances get NAT access
62
AWS Azure GCP
Default Internet
Route for New Net
None Yes Yes
Default NAT None SNAT Ephermeral IP
DMZ + Private• Want DMZ hosts which can get out, and Internal hosts which have to go
through the DMZ to get out
64
Routing - CSP Networks
65
Routing - CSP Networks• On their networks, CSPs have...
• Other Network Substrates that we may want to privately communicate with
• Other Cloud Services
• These services may exist in local or remote Regions
• CSPs have a lot of similar behaviors - differences highlighted
66
VPC Peering (pcx)• Private connectivity between two VPCs in the same Region
• Same or different accounts
• Latency/bandwidth/cost is same as talking inside a VPC
• Non-overlapping IP space; Non-transitive
• Add routes to the Route Table(s) to be connected
• Can reference Security Groups in foreign VPC
67
VPC Endpoints (vpce)• Private connectivity between VPC and AWS Services
• Limited to same region services only
• Non-transitive
• Only some services provide VPC Endpoints - S3, DynamoDB
• Given an identifier (pl-xxxx) to use in Route Tables
• In some service policies, can use VPC as a permission item (i.e. this VPC can access this service)
• Can reference VPC Endpoint (pl-xxxx) in Security Groups (but not NACL)
68
Virtual Network Peering• Private connectivity between two VNets in the same Region
• Can be different Subscriptions but same Account*
• Non-overlapping IP space; Non-transitive, but can share VPN Gateway
• Latency/bandwidth/cost is same as talking inside a VPC
• System Routes are automatic for Peered VNet
• Can override with UDR to point to instances (on either side) as gateway
69* Being worked on, but in the interim, use VPN peering
VPC Network Peering• Private connectivity between VPC Networks (in all Regions)
• Same or different projects/organizations
• Latency/bandwidth/cost is same as talking inside a VPC
• Non-overlapping IP space; Non-transitive
• Once peered, can't create conflicts (overlapping subnets, routes overlapping, etc) among the full set of VPC Networks that are peered (even between two that aren't direct)
70
Private Google Access• Private connectivity (IP space exposed) to Google Services
• Can be enabled on a Subnet basis
• Must still use internet route to access (can be tag limited)
71
AWS Azure GCP
Peering Name VPC Peering VNet Peering VPC Network Peering
Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations
Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined
CSP Service Peering Name
VPC Endpoint N/A Private Google Access
Peers + Cloud Services• Peer Two Networks in the same CSP (same Account/Subscription/Project)
• Allow (preferably) private connections from both networks to CSP Cloud Services
73
Routing - Private Gateways
74
Private Gateways• Connection points for linking CSP Network to an outside CSP network
• VPN
• Physical
• Can be managed by a third party - Cloud Exchanges
• In all cases, you can run your own VPN software (third party VM appliances)
75
CSP Managed VPN• Use of IPSec devices to connect privately to an on-premise (or other)
network
• Connect VPC/VNet/VPC Network to Corporate Office or Data Center
• Connect VPC/VNet/VPC Network to other CSP Network
• Inter-Region VPC/VNet network connectivity
76
Physical• Private Circuit connecting your network at a CSP POP to an associated
CSP Region
• If not adjacent to POP, have to get tail circuit to connect your site to POP
• Can also be used to peer with Public side of CSP network
• Access to other CSP services
• Access to CSP management interfaces
77
Limits• VPNs and Physical Connections have limits
• In general, can setup parallel connections and rely on multipath
• Multiple VPN tunnels working together
• Multiple Physical connections working together
• Combination of Physical and VPN - preference usually goes to Physical
78
Virtual Private Gateway (VGW)• Logical router sitting outside of VPC
• Associated with single VPC; so limited to a Region
• All foreign networks come in via VGW
• Set routes in the VPC Route Tables to forward traffic to VGW
• Or allow routes to be propagated from the VGW peers
• Only the VPC CIDR is advertised to VGW and its peers
• Can't transit for VPC Peering or VPC Endpoints or Internet traffic - though can proxy
• Will transit for non-VPC peers - "CloudHub"
79
Hardware VPN• Private IPSec connectivity between AWS and non-AWS managed VPN device (aka Customer
Gateway - CGW)
• Can't use between two AWS Regions
• Site-to-Site
• Always builds two tunnels (two devices are allocated on the AWS side) to one CGW
• If you want redundancy on the CGW side, you'll build four tunnels
• IP of AWS VPN side not designated until VPN Connection is configured
• Must supply static IP for CGW (can use same for multiple connections - NAT-T supported)
• Static or BGP
80
Direct Connect (DX)• AWS Physical Connection (CON)
• 1Gbps, 10Gbps options (can do smaller through a Cloud Exchange)
• Can also use multiple (<=4) CON together with LACP
• BGP Peering Only
81
Direct Connect (DX) - Sharing• Can be used with multiple accounts/VPCs.
• Each Direct Connect Connection (CON) is split into virtual interfaces (VIF).
• You assign the VIF information
• VLAN Assignment
• BGP Neighbor
• Hosting Account (one which has the CON) handles creation of VIF
• Guest Account attaches VIF to VGW
• Once attached, can't reattach (have to destroy/recreate)
82
Virtual Network Gateway• Managed Gateway Devices
• Deployed as VMs inside of a dedicated "Gateway Subnet" (/27 or larger)
• Two types
• VPN Gateway
• ExpressRoute
• Can only have one of each type per VNet
• Multiple SKUs for sizing
83
VPN Gateway• Site-to-Site
• IPSec from VPN Gateway to Local Network Gateway
• Allocated to a Region
• Static (Policy list of CIDRs) or BGP Route Based
• Limits/Sizing place preference on Route Based
• Gateway Transit: can extend reachability to Peered VNets
• Use this topology to get around some limits
84
VPN Gateway• VNet-to-VNet
• (special case of Site-to-Site)
• Connect separate Regions and separate Accounts/Subscriptions together
• Point-to-Site
• Access via the in-box Windows VPN SSTP Client
85
ExpressRoute• Private BGP Connections to Microsoft
• Handoff
• IP VPN (i.e. MPLS)
• Ethernet tail to POP
• Cloud Exchange Virtual Cross Connection
• Handoff affects location of on-prem and latency, but not necessarily Region access (depending on Subscription level)
• Offered as redundant pair of connections (no SLA without)
• Provides
• VNet Private Peering
• Azure Public Peering
• (add-on) Microsoft Peering for Office 365 et al
86
ExpressRoute• For each Circuit, can setup one to all of the three peering types
• Peering Information:
• BGP IP/Neighbor IP
• VLAN
• Neighbor ASN - Public restrictions for Public/Microsoft
• For Microsoft: Routing Registry Name
• Restrictions on Peering Type for advertised routes (# and RFC1918)
87
ExpressRoute to VNet• Owner Subscription handles Circuit creation and Peering configuration
• Owner Authorizes Other Subscriptions
• Other Subscription link VNets to ExpressRoute Circuit VNet Peering
• Must have a Virtual Network Gateway (ExpressRoute Type) configured ahead of time
• Standard vs Premium limits on # of VNet linkages and Region
• All VNets linked to the same ExpressRoute Circuit
• ExpressRoute and Point-to-Site are not supported together in the same VNet
88
Cloud Router• Managed Router which handles all dynamic routing via BGP relationships with other
routers
• Makes changes to your Routes
• Scoped to local or global route changes (i.e. advertise remote Regions)
• You assign a private ASN to Cloud Router
• Create multiple interfaces (link-local addressing)
• Create multiple BGP neighbor configurations
• Unlike the other CSPs, even if you setup tunnel devices, the Cloud Router is the BGP neighbor
89
Cloud VPN Gateway• Managed IPSec VPN Endpoint which can run multiple tunnels
• Allocated to a Region, but can forward any traffic on the VPC Network
• Can connect too another Project's or Organization's VPN Gateway
• You specify tunnels to create (so not always in pairs - but that is recommended)
• Typical Tunnel Configuration
• Local IP: Assigned from your pool of static external IPs
• (Static mode) Local subnets, IP ranges - Can't change after creating
• Peer IP, IKE, ESP, PFS, Shared Secret
• NAT-T not supported
90
Cloud VPN Gateway• Static Mode
• Specify traffic selection via --remote-traffic-selector
• Add VPN Gateway as a NextHop in your route table
• Dynamic Mode
• Create an interface on Cloud Router that is associate to the tunnel
• Handle all traffic selection via BGP relationship with Cloud Router
• Cloud Router adds VPN Gateway as NextHop
• BGP Peer Information can be configured or have GCP generate it
91
Dedicated Interconnect• GCP Physical Connection
• 10 Gbps
• Can use multiple with LACP
• BGP Established with Cloud Router (i.e. requires Cloud Router)
• Is divided up into VLAN Attachments
• GCP allocates the VLAN and BGP IPs to use (ASN specified by Cloud Router)
• Use that information to configure Cloud Router and on-prem router
92
Dedicated Interconnect - Sharing• Can be shared among Projects in an Organization
• Even if not using a Shared VPC Network
• Two methods - both using IAM permissions
• Hosting Project grants permissions to Guest Project's Users to update Interconnect
• Guest Project grants permissions to Hosting Project to update Gust Project's Cloud Router
93
AWS Azure GCP
Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router
Gateway Model
Router outside of VPCIs NextHop for all gateway traffic
VM Insides of "Gateway Subnet" which handle routing and
gateway
BGP Talker that updates routes. Separate gateways that forward
traffic.
Substrate Route Mgmt
You add routes to RTB, orsetup to propagate all routes
from VGW to RTBRoutes show up as System
Routes Cloud Router updates Routes
Gateway Peer Transit Yes
VPN: Yes including VNet PeeringExpressRoute: No Yes
AWS Azure (VNet Peering) GCP
VPN Name
VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway
Cloud Interconnect - IPSec VPNaka Cloud VPN
Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site
Deploy Model
Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop
Endpoint associated with VPC Network as NextHop
VPN Transit Yes (CloudHub) No Yes
Size Options
1 3 (4) 1
Advertised Routes
1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)
Received Route limits
100 Prefixes . 100 Prefixes (CR limit)
Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway
Route Selection
Static: What is specifiedBGP: VPC + All VGW BGP Peers
Static: What is specifiedBGP: VNet, Configured Peers
Static: What is specifiedBGP: By Cloud Router
AWS Azure GCP
Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated
Interconnect
Link Options 1Gbps, 10Gbps, LACP
IP VPN50,100,200,500M ; 1,2,5,10Gbps
Cloud Exchange10Gbps, LACP
Routing BGP BGP BGP(to Cloud Router)
VLAN/BGP Allocation You Pick You Pick GCP Picks
Sharing YesCentrally Managed
YesCentrally Managed Configuration
RBAC VNet Linking
YesRBAC Managed
Route Limits 100 Prefixes Depends on Size
VNet: 4000-10000 100 (CR limit)
CSP Network as Data Center Extension• Setup CSP Network which only has connectivity from a Private Data
Center via a physical connection
• Any ingress or egress traffic goes via the Private Data Center
97
Cross Region Network• Want to peer two Networks in the Same VPC
98
Connect Multiple CSP Together• Have two CSP Networks
• One in each of two different CSPs
• "Peer" them together
99
Cloud Storage from Corp Office• Setup protected path from corporate office to Storage offering of CSP
100
Access Control
101
AWS Access Control• Two types
• Security Groups (SG): For instances
• Network Access Control Lists (NACLs): For subnets
102
Security Group (SG)• Stateful Packet Filters
• Separate lists for ingress and egress
• Applied on Network Interfaces
• By "Applied to Instance," it really means "Applied on the Instance's Primary NIC"
• Positive Control: Only ALLOW. No DENY
• Can apply multiple (5) SG on an NIC
• ALLOWs are cumulative
103
Security Group (SG)• Structure:
• Protocol
• Port Range, or ICMP Type
• Source (ingress), or Destination (egress) CIDR, or SG
• New SG has default egress allows anything out
• "default" SG available when VPC created (and always available)
• Ingress: ALLOW from "default" SG to ANY
• Egress: ALLOW to ANY
104
Security Group (SG)• Applied to Instances
• Context is focused on Instance (or groups of Instances) Rules
• But with tagging is hard to tailor to only your instances for modifying SG
105
Network Access Control List (NACL)• Stateless
• Only one allowed per subnet
• Ordered priority
• Positive and Negative control (ALLOW/DENY)
106
Azure Access Control• Just one
• Network Security Groups (NSG)
107
Network Security Group (NSG)• 1 per NIC and/or subnet
• Applied in priority order (ascending)
• default rules for inbound and outbound on both (permit out, permit VNET)
• The NSG on the NIC, if present, and on the Subnet have to permit to permit traffic. One DENY on either will block traffic.
• NSG Tags : canned collections of CIDRs (e.g. VIRTUAL_NETWORK)
• Can do flow logging
108
Network Security Group (NSG)• Structure
• Priority (Low numerically is Higher Priority)
• Name
• Protocol + Source IP/Tag + Source Port + Destination IP/Tag + Destination Port
• Allow/Deny
109
Network Security Group (NSG)• Managing
• Action (Permissions): Microsoft.Network/networkSecurityGroup/*
• Roles: Network Contributor, Owner
• Context is global to VNet
110
Firewall• Single managed firewall for VPC network
• Manages inbound and outbound
• Can limit sources/destinations using Target Tags
• Tag matched against Tags on Instance
111
Firewall• Structure
• Priority (ascending, first match wins)
• (Ingress) Protocol + Destination Port + Source IP
• (Egress) Protocol + Destination Port + Destination IP
• Target Tags
• Allow/Deny
112
Firewall• Managing
• Permissions: compute.firewalls.*
• Role: roles/compute.securityAdmin
• Context is global to Project/Share VPC Project
113
AWS AWS Azure GCP
Name Security Group Network Access Control List Network Security Group Firewall
State Stateful Stateless Stateful Stateful
Additive Positive Only Positive/Negative Postive/Negative Positive/Negative
Target NIC Subnet NIC or Subnet VPC(target tags for instance)
Multiple Yes (5 on instance 250 in account) No No No
Ordered No Yes Yes Yes
Setup Access Control for 3 Tier App• Given 4 VMs - web, app, db, bastion - setup access control such that:
• Anyone is allowed from any tcp port to "web" on 443/tcp
• "web" is allowed from any tcp port to "app" on 8009/tcp
• "app" is allowed from any tcp port to "db" on 3306/tcp
• "bastion" is allowed from any tcp port to all three instances on 22/tcp
• "bastion" is allowed from any udp port to all three instances on 161/udp
115
Summary and Next Steps
116
Locations
117
Regions
AWS 16 44 Availability Zones
Azure 26(not counting Gov)
60 Fault Domains(not counting Gov)
GCP 12 36 Zones
AWS Azure GCP
Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network
IP Addressing
RFC1918 or OtherCarving up CIDR of VPC
RFC1918 or OtherCarving up CIDR of VNet
RFC1918 onlyAccumulation of Subnet CIDRs
Locality One Region One Region Global
Subnet Locality One AZ Region Selection of Zones in a Region
CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR
Account Resource Sharing
NoUsers use multiple Subs.
Resources bound to one VNet inside one Sub at a time
Across Projects - YesAcross Organizations - No
Substrate
AWS Azure GCP
Forwarding Property Source/Destination Check IP Forwarding
(enableIPForwarding) IP Forwarding (can-ip-forward)
Property Default On Off Off
NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface
IPs per NIC 6-50 50Unspecified
(Alias IP not supported with multiple NICs)
NICs per Instance 1-15 2-8 1-8
NIC Locations Same VPC Same VNet Each must be on separate VPC
Networks
Instance Properties
AWS Azure GCP
Name Route Tables System Routes +User Defined Routes Routes
Route Selection
Most specific CIDR match,Static
Propagated
Most Specific CIDR match,User Defined Routes,
BGP Routes,System Routes
Most specific CIDR match,then by priority,
then mutlipath hash
Route Sharing
All subnets associated with same Route Table
All subnets associated with same User Defined Route
One shared route table;Specific route rules applied via
instance tag
Default Routes for New
VPC CIDRInternet
Peered ConnectionsVNet CIDR
InternetSubnet CIDRs
Routing - Inside
AWS Azure GCP
Default Internet
Route for New Net
None Yes Yes
Default NAT None SNAT Ephermeral IP
Routing - Internet
AWS Azure GCP
Peering Name VPC Peering VNet Peering VPC Network Peering
Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations
Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined
CSP Service Peering Name
VPC Endpoint N/A Private Google Access
Routing - CSP Networks
AWS Azure GCP
Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router
Gateway Model
Router outside of VPCIs NextHop for all gateway traffic
VM Insides of "Gateway Subnet" which handle routing and
gateway
BGP Talker that updates routes. Separate gateways that forward
traffic.
Substrate Route Mgmt
You add routes to RTB, orsetup to propagate all routes
from VGW to RTBRoutes show up as System
Routes Cloud Router updates Routes
Gateway Peer Transit Yes
VPN: Yes including VNet PeeringExpressRoute: No Yes
Routing - Private Routes
AWS Azure (VNet Peering) GCP
VPN Name
VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway
Cloud Interconnect - IPSec VPNaka Cloud VPN
Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site
Deploy Model
Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop
Endpoint associated with VPC Network as NextHop
VPN Transit Yes (CloudHub) No Yes
Size Options
1 3 (4) 1
Advertised Routes
1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)
Received Route limits
100 Prefixes . 100 Prefixes (CR limit)
Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway
Route Selection
Static: What is specifiedBGP: VPC + All VGW BGP Peers
Static: What is specifiedBGP: VNet, Configured Peers
Static: What is specifiedBGP: By Cloud Router
Routing - VPN
AWS Azure GCP
Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated
Interconnect
Link Options 1Gbps, 10Gbps, LACP
IP VPN50,100,200,500M ; 1,2,5,10Gbps
Cloud Exchange10Gbps, LACP
Routing BGP BGP BGP(to Cloud Router)
VLAN/BGP Allocation You Pick You Pick GCP Picks
Sharing YesCentrally Managed
YesCentrally Managed Configuration
RBAC VNet Linking
YesRBAC Managed
Route Limits 100 Prefixes Depends on Size
VNet: 4000-10000 100 (CR limit)
Routing - On-Premise
AWS AWS Azure GCP
Name Security Group Network Access Control List Network Security Group Firewall
State Stateful Stateless Stateful Stateful
Additive Positive Only Positive/Negative Postive/Negative Positive/Negative
Target NIC Subnet NIC or Subnet VPC(target tags for instance)
Multiple Yes (5 on instance 250 in account) No No No
Ordered No Yes Yes Yes
Access Control
Additional "Network" Areas• Load Balancers - some affect SNAT and packet forwarding
• DNS - interacts with Substrate DNS view
• CDN
127
Limits!!!• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/
VPC_Appendix_Limits.html
• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits
• https://cloud.google.com/router/quotas
128
October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17
Remember to fill in yourtutorial evaluation!
Thank You!
R6 - The Ins-and-Outs of Networking in the Big Three CloudsChris "mac" McEniry
