Jeff Ostermiller – CCIE #5402e
Technical Solutions Architect
@jostermi
BRKDCT-2334
Real World Data Center Deployments and Best Practices
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Abstract
The seminar will discuss real world Nexus Deployment scenarios to make sure your network will meet the demands for performance and reliability. This session will provide and equip you with the latest information on Cisco® data center network architecture and best practices around those designs. This session will focus on STP, vPC, Fabric Path, QOS, routing and service node insertion from the core of the network to the host. This session will not cover all of the possible options just the best practices to make sure we are all successful.
BRKDCT-2334 3
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Speakers
Jeff Ostermiller CCIE #5402eTechnology Solution ArchitectData Center Architecture@[email protected]
BRKDCT-2334 4
• Data Center Design Elements
• Data Center Design Evolution
• Fundamental Data Center Design
• Services Insertion
• Scalable Data Center
• Overlays
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What do I really need in the Data Center?
• Compute
• IP Based Storage
• Firewalls/Security
• Load Balancers
• WAN Tier
• Internet
• Intranet
• Campus Aggregation
• Visibility
SiSi SiSi
BRKDCT-2334 7
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtual Machine Scalability
• ESX Maximum VMs per Hosts
• 5.5 512 VMs per Hosthttps://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf
• 6.0 1024 VMs per Hosthttps://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf
• Hyper V Maximum VMs per Host
• Windows Server 2012 1024 VMs per Server https://technet.microsoft.com/en-us/library/jj680093.aspx
• Openstack
• Well…
• Containers
• Lots of application containers per OS, few OS’s
BRKDCT-2334 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real Life Virtualization Ratios
• 22 cores per socket and 2 sockets per 1 RU server.
• Memory Sizes
• 128G, 256G, 512G common memory sizes
• 1TB of Memory at Max Memory speed 2133 MHz
• 1.5TB at 1600 MHz
• Lets say 50 VMs per server ( everyone is different )
• 1 client has 100 VMs per server with 1TB per server
• 1 client has 67 VMs per server with 256G
• 1 client has 30 VMs per server with 256G
C220 M4 as Example
BRKDCT-2334 9
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rack Density 47 RU
• Forty 1 RU Servers ( 22 cores x 2 sockets x 40 Servers )
• 2 10/25GE interfaces per server channeled
• 1 ILO Port per server
• 50 VMs per Server – 2000 VMs per Rack
• 1 - 48 port 1 GE out of band management switch
• 2 - 48 Port 10/25GE network switches
• 4 RUs free for Cable Management
Hyper Dense Rack
The switches are only 83% utilized or 17% free
BRKDCT-2334 10
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hyper Dense POD ( Point of Delivery )Hyper Dense Rack with IP Based Storage
• Forty 1 RU Servers ( 22 cores x 2 sockets x 40 Servers )
• 1 - 48 port 1 GE out of band management switch
• 2 - 48 Port 10/25GE network switches
• 1 Rack for IP Based Storage 16 10/25 GE available
• External Storage to the POD may be deployed as shared storage
BRKDCT-2334 11
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rack Density 42 RU
• Twenty 1 RU Servers ( 22 cores x 2 sockets x 20 Servers )
• 2 10/25GE interfaces per server channeled
• 1 ILO Port per server
• 50 VMs per Server – 1000 VMs per Rack
• 1 - 48 port 1 GE out of band management switch
• 2 - 48 Port 10/25GE network switches
• 19 RUs free for cable management
Half Full due to Power/Cooling/Weight Concerns
The switches are only 42% utilized or 58% free
BRKDCT-2334 12
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
28 Free 10/25G ports
• Compute
• IP Based Storage – 4 10/25Ge ports per switch
• Firewalls – 4 10/25Ge ports per switch
• Load Balancers – 4 10Ge ports per switch
• WAN Tier – 4 10Ge ports per switch
• Internet
• Intranet
• Campus Aggregation ???
BRKDCT-2334 13
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
28 12 Free 10/25G ports
• Compute
• IP Based Storage – 4 10/25Ge ports per switch
• Firewalls – 4 10/25Ge ports per switch
• Load Balancers – 4 10Ge ports per switch
• WAN Tier – 4 10Ge ports per switch
• Internet
• Intranet
• Campus Aggregation 12 closets
• what about 96 port switches? Then 60 closets
BRKDCT-2334 14
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
* http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
*
Data Center Design RequirementsThings will fail, so how can we protect ourselves
BRKDCT-2334 15
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example of Constrained Resource
Feature Parameter Verified Limit (Cisco NX-OS 6.2)
ARP/ND
Sup 1 Sup 2 Sup 2E
Number of entries in ARP table 128,000 128,000 128,000
Number of ARP packets per second 1500 1500 5000Number of ARP glean packets per second 1500 1500 5000
Number of IPv6 ND packets per second 1500 1500 5000Number of IPv6 glean packets per second 1500 1500 5000
BRKDCT-2334 16
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Availability ZonesUsing Amazon Web Services terms
PodAvail
Zone
Avail
Zone
Avail
Zone
RegionMulti-Pod
Global Multi-Site Data Center Deployment
PodPod
BRKDCT-2334 17
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connecting the Data Center to the Rest of the Network
• Use routed pt2pt links
• ECMP used to quickly re-routearound failed node/links
• Tune CEF L3/L4 load balancing hash to achieve maximum utilization
• Build triangles not squares for deterministic convergence
• Insure redundant L3 paths to avoid black holes
• Summarize distribution to core to limit event propagation
• Utilized on both Multi-Layer and Routed Access designs
BRKCRS-3036 Advanced Enterprise Campus Design
Layer 3 Equal
Cost Link’sLayer 3 Equal
Cost Link’s SiSiSiSi
SiSiSiSiSiSiSiSiSiSiSiSi
WANInternet
SiSiSiSi SiSi SiSi
Data Center
BRKDCT-2334 18
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Building Block DesignWAN and Campus Connectivity
SiSi SiSi
Created
Isolated
Campus Core
WAN Building
Block
Layer 3
Switch/Router
Segment
Networks with
Routing
Don’t Forget
OOB
Managmenet
DC
Zone
WAN
Zone
Campus
Zone
Site
Border Leaf
switches
common Connect OOB
Management
into Campus
Core!
BRKDCT-2334 19
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Traffic Paths
Source: Cisco Global Cloud Index, 2016 BRKDCT-2334 21
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Oversubscription Ratio
• Large layer 2 domain with collapsed Access and Core
• Worse Case Calculation
• Assume all the traffic is north-south bound
• Assume 100% utilization from the Access Switches
• All the ports operated in dedicated mode
Access to Core/Aggregation
Aggregation
Access
Oversubscription
Ratios
16 10 GEs
48 10 GEs
48:16=3:1
Line Cards 1:1
(3:1)*(1:1)
BRKDCT-2334 22
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center Design Evolution Clos Fabric
• Moving to Spine/Leaf construct
• No Longer Limited to two aggregation boxes
• Created Routed Paths between “access” and “core”
• Routed based on MAC, IP, or VNI
• Layer 2 can be anywhere even with routing
• Automation/Orchestration, removing human error.
Ro
ute
d
Do
ma
in
L2
Domain
Servers and Services
Connected at the Leaves
BRKDCT-2334 23
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Clos Fabric, Fat Trees
Spine
Leaf
8 40 GEs
48 10 GEs
48:32=1.5:1
Line Cards 1:1
(1.5:1)*(1:1)
Changing Traffic Flow
Requirements
Services are deployed
at the leaf nodes
Oversubscription
Ratios defined by
number of spines and
uplink ports
True horizontal scale
BRKDCT-2334 24
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Statistical Probabilities…
• Assume 11 10G source flows, the probability of all 11 flows being able to run at full flow rate (10G) will be almost impossible with 10G (~3%), much better with 40G (~75%) & 100G (~99%)
Intuition: Higher speed links improve ECMP efficiency
1 2 3 4 5
1 2
1 2 20
Prob of 100% throughput ≅ 3%
Prob of 100% throughput ≅ 99%
Prob of 100% throughput ≅ 75%
20×10Gbps
Uplinks2×100Gbps
Uplinks
11×10Gbps flows
(55% load)
5×40Gbps
Uplinks
BRKDCT-2334 25
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why 25 GB Ethernet
• Server IO Doubling every 24 Months
• Core Networking Doubling every 18 Month
• Clients starting to use multiple interfaces per Server again
• Maximize Switch Throughput
• Minimize # of Cables and Tor switches
• SFP-25G Transceivers same form factor at SFP-10G
• 1, 2, 3, 5 meter Twinax
• SR Optics 100m OM4
Ethernet Alliance Introduction-to-25GbE-Webinar_D2p1.pdf
BRKDCT-2334 26
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric OptionsWhich Model is Right for you?
VXLANFabric PathVPCSTP
L3
L3
BRKDCT-2334 27
Application Centric Infrastructure
DB DB
Web Web App Web App
Turnkey integrated solution with
security, centralized management,
compliance and scale
Automated application centric-policy
model with embedded security
Broad and deep ecosystem
Fabric Management Options
Programmable Network
Modern NX-OS with enhanced
NX-APIs
DevOps toolset used for Network
Management(Puppet, Chef, Ansible etc.)
Programmable Fabric
VxLAN-BGP EVPN
standard-based
3rd party controller support
Cisco Controller for software
overlay provisioning and
management across N2K-N9K
VTS
Creation Expansion
Fault MgmtReporting
Connection
Automation, API’s, Controllers and Tool-chain’s
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Programmable Networks
• Standard and Open APIs to the DC Switching Fabric that is common against multiple DC products.
• Ability to script tasks into automated run books; however, no ability to dynamically insert services.
• DevOps Methodology
CLI
Element
Manager
Scripts
ACI
Stand-alone
NX-OSFull ACI mode
https://developer.cisco.com/site/devnet/home/index.gsp
BRKDCT-2334 29
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI
Basic ElementManager
ACI
Scripting to the CLI
and/or API
Fabric Management Options
• Solutions targeted towards different fabric ops models, market segments, and user CLI-proficiencies
Datacenter Network Manager (DCNM)• Graphical network manager + underlay templates
• Switch platform feature breadth and depth
• Cisco Nexus 2K through 9K platforms supported
• CLI/feature knowledge required
Virtual Topology System (VTS)• Fabric overlay focused – VXLAN-based
• Primarily focused on service provider
• Multi-platform support
• Tie-ins to OpenStack and vCenter
Nexus Fabric Manager (NFM)• Simplified interaction– GUI + API
• Build fabric-wide broadcast domains• Via self-managed underlay and overlay
• Ops – minimal CLI requirements
• Point-n-click simplified interface
BRKDCT-2334 30
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Automation ‘and’ OperationsThere are multiple decision vectors
CONTINUOUS
INTEGRATIONORCHESTRATION &
MANAGEMENT (O&M)
Operations involves a full life cycle of infrastructure and
application management?BRKDCT-2334 31
Programmable NetworkProgrammable FabricApplication Centric
Infrastructure
Integrated stack
Or
A-la-carte Automation
Streamlined Workflow Management
Modern NX-OS with enhanced NX-APIs
DevOps toolset used for Network Management(Puppet, Chef, Ansible etc.)
Customer Script based Operations and Workflows
Turnkey integrated solution with security, centralized
management, compliance and scale
Automated application centric-policy model with
embedded security
Broad and deep ecosystem
FCAPS ‘and’ Automation
Fault
Accounting
Performance
Security
Configuration
External
ToolsIntegrated
Tools
VTSCreation Expansion
Fault MgmtReporting
Connection
External
Tools
Integrated
Tools
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Learning Python via the API
BRKDCT-2334 34
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dynamic Python Programimport requests
import json
url='http://YOURIP/ins'
switchuser='USERID'
switchpassword='PASSWORD’
myheaders={'content-type':'application/json-rpc’}
payload=[
{
"jsonrpc": "2.0",
"method": "cli",
"params": {
"cmd": "config t",
"version": 1
},
"id": 1
},
{
"jsonrpc": "2.0",
"method": "cli",
"params": {
"cmd": "vlan 1234",
"version": 1
},
"id": 2
}
]
response = requests.post(url,data=json.dumps(payload), headers=myheaders,auth=(switchuser,switchpassword)).json()
BRKDCT-2334 35
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Python Consolidation
payload=[
{
"jsonrpc": "2.0",
"method": "cli",
"params": {
"cmd": "config t",
"version": 1
},
"id": 1
},
{
"jsonrpc": "2.0",
"method": "cli",
"params": {
….
payload=[
{"jsonrpc": "2.0","method": "cli","params": {"cmd": "config t","version": 1},"id": 1},
{"jsonrpc": "2.0","method": "cli","params": {"cmd": "vlan 1234","version": 1},"id": 2},
{"jsonrpc": "2.0","method": "cli","params": {"cmd": "exit","version": 1},"id": 3}
]
BRKDCT-2334 36
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Adding a Loop to PythonPython Programming for Networking Engineers
@kirkbyers
http://pynet.twb-tech.com
Network Programmability User Group
https://netprog.atlassian.net/wiki/display/NPUG/NPUG
DevNet
https://developer.cisco.com/
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Device Orchestration Support:Technologies
Integrate network devices with
configuration management &
orchestration stacks
Customization
https://dcloud-cms.cisco.com/demo_news/cisco-nexus-9000-open-nx-os-programmability-and-automation-v1
For YourReference
BRKDCT-2334 38
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
A few more sophisticated options
• General Ansible
• Ansible Intro - http://docs.ansible.com/intro.html
• Ansible Modules - http://docs.ansible.com/modules.html
• YAML Syntax - http://docs.ansible.com/YAMLSyntax.html#yaml-basics
• Ansible Videos - http://www.ansible.com/resources
• Jason Edelman’s Blog - http://www.jedelman.com/home/network-automation-
with-cisco-nexus-switches-ansible/
• Jason Edelman’s Github - https://github.com/jedelman8/nxos-
ansible/blob/master/README.md
For YourReference
BRKDCT-2334 39
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Classic Customer vPC DesignCommon Building Blocks
vPC peer-link
vPC Keepalive
vPC
FEX
vPC
vPC
vPC Deployments Started in 2009BRKDCT-2334 41
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Classic Customer vPC Design
• vPC Peer Link
• Use Multiple Ports on Multiple Port ASICs
• Server Connections DUAL attach always single attach only when forced
• No switches connected below access tier
• Breaks ISSU.
• VLANs must be synchronized on all 4 boxes
• With FEX remember Oversubscription
Layer 2 only at fixed configuration Top of Rack switch
FEX
vPC peer-link
vPC Keepalive
vPC
BRKDCT-2334 42
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Out of Band Management Network
• Build a true out of band management network.
• Several Client use older existing L2 campus switches
• Connect all Management interfaces into network
• Connect the Management switches back into the Campus CORE, Not the DC Core
• vPC Keepalive Link
• On Fixed Config switches use MGNT0 to avoid issues with ISSU and STP
Layer 2 only at fixed configuration Top of Rack switch
BRKDCT-2334 43
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
vPC Best Practice Summary
• Use LACP Protocol when connecting devices
• Use multiple interfaces for Peer-Link
• Enable Auto Recovery
• IP Arp sync
• Use Peer-Switch with appropriate spanning tree priorities set
• IPv6 ND synchronization
• Peer Gateway
• with exclude VLAN where required
• Fabric Path Multicast Loadbalance.
http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf
Layer 2
BRKDCT-2334 44
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multicast ExampleAlmost all Designs moving forward will have a Multicast requirement BRKIPM-2264
Anycast-RP 2:
feature pim
feature eigrp
interface loopback0
ip address 10.1.1.4/32
ip router eigrp 10
ip pim sparse-mode
interface loopback1
ip address 10.10.10.50/32
ip router eigrp 10
ip pim sparse-mode
router eigrp 10
ip pim rp-address 10.10.10.50 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.10.10.50 10.1.1.4
ip pim anycast-rp 10.10.10.50 10.1.1.6
Anycast-RP 1:
feature pim
feature eigrp
interface loopback0
ip address 10.1.1.6/32
ip router eigrp 10
ip pim sparse-mode
interface loopback1
ip address 10.10.10.50/32
ip router eigrp 10
ip pim sparse-mode
router eigrp 10
ip pim rp-address 10.10.10.50 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.10.10.50 10.1.1.4
ip pim anycast-rp 10.10.10.50 10.1.1.6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center Building Blocks
• Control Plane Scale
• ARP learning
• MAC addresses
• CPU Traffic Level, any packets that get punted to the CPU
• Spanning Tree Scale
• RSTP -> 16k Logical Ports, logical port limit is equal (# of ports)*(Vlans per ports)
• MST -> 25K Logical Ports, logical port limit is equal (# of ports)*(# of MST instances allowed per port)
• Port Channel Scaling Numbers
• Buffer Oversubscription
• Failure Domain Size ( Availability Zones )
• ISSU
Scaling ConcernsPOD
PODPOD
Multi-Pod
BRKDCT-2334 46
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example of Constrained Resource
Feature Parameter Verified Limit (Cisco NX-OS 6.2)
ARP/ND
Sup 1 Sup 2 Sup 2E
Number of entries in ARP table 128,000 128,000 128,000
Number of ARP packets per second 1500 1500 5000Number of ARP glean packets for second 1500 1500 5000
Number of IPv6 ND packets per second 1500 1500 5000Number of IPv6 glean packets per second 1500 1500 5000
128,000ARPs/1500(ARPs Per Second)=85.3 seconds
BRKDCT-2334 47
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
COPP Policy Monitoring
Customer 1 ( 5.2.9 code)
show policy-map interface control-pla class copp-system-p-class-normal | inc violate prev
4 | inc module|violated
module 3
violated 0 bytes; action: drop
module 8
violated 1152074225 bytes; action: drop (approximately 18 Million ARPs)
module 9
violated 2879379238 bytes; action: drop (approximately 45 Million ARPs)
Customer 2 (6.2.12 code)
show policy-map interface control-plane class copp-system-p-class-normal | inc violate
violate action: drop
violated 8241085736 bytes, (approximately 128 Million ARPs in 123 Days)
5-min violate rate 0 bytes/sec
violated 0 bytes,
5-min violate rate 0 bytes/sec
violated 0 bytes,
Control Plane Policy Exceeded
BRKDCT-2334 48
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Effects of an ARP flood
2000 ARPs/Second
Arp Time out 4 Hours Arp Time out 4 Hours
25 Minute ARP Time out
30 Minute CAM Time out
25 Minute ARP Time out
30 Minute CAM Time out
Which ARP is better?
Server ARP
Storage ARP
Router ARP
Switch ARP
Routing PeersRouting Peers
BRKDCT-2334 49
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Policing
show copp diff profile strict profile moderate
'+' Line appears only in profile strict, version 6.2(6a)
'-' Line appears only in profile moderate, version 6.2(6a)
-policy-map type control-plane copp-system-p-policy-moderate
reduced
- class copp-system-p-class-normal
- set cos 1
- police cir 680 kbps bc 310 ms conform transmit violate drop
reduced
+ class copp-system-p-class-normal
+ set cos 1
+ police cir 680 kbps bc 250 ms conform transmit violate drop
• 680 kbps / (64 byte Arp Frames * 8 bits ) = 1328 ARPs per second
• BC = TC * CIR or 310 msec *680,000 = 204000 this means approximately another 400 ARPs per second are allowed for burst.
show policy-map interface control-plane class copp-system-p-class-normal | inc violate
BRKDCT-2334 50
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Protection
• Configure a syslog message threshold for CoPP
• in order to monitor drops enforced by CoPP.
• The logging threshold and level can be customized within each traffic class with use of the logging drop threshold <packet-count> level <level> command.
logging drop threshold 100 level 5
Example syslog output
%COPP-5-COPP_DROPS5: CoPP drops exceed threshold in class:
copp-system-class-critical,
check show policy-map interface control-plane for more info.
Notification about Drops
BRKDCT-2334 51
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Tuning
• Do not disable CoPP. Tune the default CoPP, as needed.
• Create Custom Policy to match your environment
• Monitor exceptions to see your tuning results
monitor session 1
source exception all
destination interface Eth1/3
no shut
nexus7k(config-monitor)# show monitor session 1
source exception : fabricpath, layer3, other
filter VLANs : filter not specified
destination ports : Eth1/3
Feature Enabled Value Modules Supported
--------------------------------------------------
L3-TX - - 1 6 8
ExSP-L3 - - 1
ExSP-FP - - 8
ExSP-OTHER - - 1
RB span No
BRKDCT-2334 52
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Protection
Nexus# copp copy profile strict prefix LAB
Nexus(config)# arp access-list LAB-copp-arp-critical
Nexus(config-arp-acl)# 10 permit ip 10.1.2.1 255.255.255.255 mac any
Nexus(config-arp-acl)# 20 permit ip 10.1.2.5 255.255.255.255 mac any
Nexus(config-arp-acl)# class-map type control-plane match-any LAB-copp-class-arp-critical
Nexus(config-cmap)# match access-group name LAB-copp-arp-critical
Nexus(config-cmap)# policy-map type control-plane LAB-copp-policy-strict
Nexus(config-pmap)# class LAB-copp-class-arp-critical insert-befor LAB-copp-class-normal
Nexus(config-pmap-c)# set cos 6
Nexus(config-pmap-c)# police cir 100 kbps bc 250 ms conform transmit violate drop
Nexus(config)# control-plane
Nexus(config-cp)# service-policy input LAB-copp-policy-strict
Good ARPs versus Bad ARPs
BRKDCT-2334 53
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dedicated Solutions have ability to use Hardware Acceleration Resources like SSL Offload
Centralized Firewalls Transparent or Routed
Virtualized Services Layer 3
Services Deployment Models
Centralized ADCs Routed or One Armed
BRKDCT-2334 55
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
1001
0001011
1100010
1110
1001
0001011
1100010
1110
1001
0001011
1100010
1110
1001
0001011
1100010
1110
1001
0001011
1100010
1110
Legacy Security: Siloed, Inefficient & Expensive
Data
Packet
1001
0001011
1100010
1110
/
1001
0001011
1100010
1110
FW Platform
DDOS Platform SSL Platform
ADC Platform
WAF Platform
IPS Platform
SSLDDoS WAF
FW IPSADC
Reduced Effectiveness Increased Latency Slows Network Static & Manual
1001
0001011
1100010
1110
Trusted
Core
BRKDCT-2334 56
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Security and Network Services Inside the DCNetworking with Excel
inside
outside
inside
Back end VLAN
outside
Front end VLAN
• Multiple types of State-full services
• Firewalls
• Application Delivery Controllers
• Inspection Prevention/Detection
Systems
• State-full implies one-way symmetrical
establishments
• State-full Devices HA and Scalability:
• Active-Standby mode for state-full
convergences & recovery
• Active Active mode for Redundancy
and Scalability
BRKDCT-2334 57
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Security and Network Services Inside the DC
inside
outside
inside
Back end VLAN
outside
Front end VLAN
IGP to route
with Firewall eBGP to
avoid Firewall
VIP Pulls
Traffic
NAT on
Firewall
FW NAT 200.1.1.1-> 10.1.1.1.1
Application VIP 10.1.1.1
RHI to
inject VIP
Conditional
Routing to
announce VIP
Servers Default
Gateway?
Static Routes?
BRKDCT-2334 58
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Insert Firewall at Aggregation Point
• vPC connect ADCs and Firewalls
• Load balancer configured in One Armed, Routed
• Source NAT used to direct traffic back to LB
Services with a vPC based designAdding in ADC, Application Delivery Controllers and Firewalls
Layer 2 Trunks
Enterprise Network
Layer 2 Trunks
If Routing on Services nodes, use standard Etherchannel not vPCs
BRKDCT-2334 59
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Securing the Data Center vPC based design
Enterprise Network
Layer 2 Trunks
VLAN 110-150
Layer 2 Trunks
VLANs 10-50
Web App DBClients
ACLs
BRKDCT-2334 60
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 3 Firewalls InsertionServers Default Gateway located on Firewall
Layer 2 Trunks
Enterprise Network
Layer 2 Trunks
• vPC connect firewalls
• Server Default Gateway on Firewall
• If Clustering or L2 Heartbeats
required you need to handle igmp( One Option )
N5k# configure terminal
N5k(config)# vlan 5
N5k(config-vlan)# no ip igmp snooping
• Look at Moving Layer 3 back to Switch
with VRFs to create isolation to allow
for more flexibility
*Does not show the Value of Cisco Fabrics at allBRKDCT-2334 61
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EtherChannel on the ASA
• Supports 802.3ad and LACP/cLACP standards
• Direct support for vPC/VSS - CVD
• No issues with traffic normalization or asymmetry
• Up to 8 active and 8 standby links*
• 100Mb, 1Gb, 10Gb are all supported – must match
• Supported in all modes (transparent, routed, multi-context)
• Configurable hash algorithm (default is src/dest IP)
• SHOULD match the peer device for most deterministic flows
• Redundant interface feature and LAG on ASA are mutually exclusive
• Not supported on 4GE SSM (5540/50) or 5505
• ASA 9.2+ cluster allows 32 port active EtherChannel
BRKSEC-2020 Intermediate Firewall Deployment
*Non-clustered ASA allows 16 active and 16 standby links supported with cLACP
For YourReference
BRKDCT-2334 62
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why Deploy Transparent Mode?
• Very popular architecture in data center environments
• Existing Nexus/DC Network Fabric does not need to be modified to employ L2 Firewall!
• Simple as changing host(s) VLAN ID
• Firewall does not need to run routing protocols / become a segment gateway
• Firewalls are more suited to flow-based inspection (not packet forwarding like a router)
• Routing protocols can establish adjacencies through the firewall
• Protocols such as HSRP, VRRP, GLBP can cross the firewall
• Multicast streams can traverse the firewall
• Non-IP traffic can be allowed (IPX, MPLS, BPDUs)
• (CVD) most internal DC zoning scenarios recommend Transparent FW (L2) deployed versus Routed Firewall (L3)
• L3 Use-cases still valid, especially in Multi-tenant and Secure Enclave architectures
BRKDCT-2334 63
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What Happens when you Exceed the Performance of a Single Pair?
• Clustering if the appliance supports this
• How do you scale without native clustering support?
• Smart-Channeling
Active Standby Performance Limitation
Eth 1/1
N7k1
Smart-
channel
Eth 4/4
Smart-
channel
For YourReference
BRKDCT-2334 64
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Traffic Distribution to Transparent Services/Appliances in HW at line rate
• Health Monitoring and Robust Failure Handling
• Traffic Persistence – ingress and egress Flow persistent on the same service
• Selective Traffic Distribution (include/exclude certain traffic)
Benefits of Smart-channel, Partial List For YourReference
BRKDCT-2334 65
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco ASA Firewall Clustering Basics
• Designed to solve two critical issues with firewall HA:
1. Aggregate firewall capacities for DC environments (BW, CPS, etc.)
2. Provide dynamic N+1 stateful redundancy with zero packet loss
• Supported in routed (L3) and transparent (L2) firewall modes, both single and multi-context - Mixed Mode supported as well
• (NG)IPS module is fully supported in clustered firewall deployment
• This adds NGIPS (FirePOWER) / NGFW / Device Context (FireSIGHT), etc. to ASA• Manages Asymmetric flows
• For ASA Clustering Deep-Dive watch recording of BRKSEC-3032 –Advanced - ASA Clustering Deep Dive
BRKDCT-2334 66
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is cLACP and What Does it Do?
• The challenge for clustering is that LACP is defined to run between two devices only according to IEEE specification and may only have 8 interfaces forwarding data
• Requirement to support LACP over multiple ASA units in a cluster and make clustered ASAs able to interoperate with standard LACP devices as one ASA
• Provide Etherchannel re-configuration with traffic black-hole avoidance and load balancing at both link and device level during link failure or device failure
• Provide cLACP API to cluster CP to notify Etherchannel link status change and provide health monitoring
• cLACP recovery/redundancy between ASA units in the case of Master unit leaves cluster
• Extend the maximum number of active forwarding interfaces to 16 (or potentially greater)
• 32-links Now Supported (16 active/16 standby in ASA 9.2)
BRKDCT-2334 67
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CL MASTER CL SLAVE CL SLAVE CL SLAVE ASA x Node Cluster
ASA Port-Channel 32
ASA Port-Channel 40
CL MASTER CL SLAVE CL SLAVE CL SLAVE ASA x Node Cluster
Correct Use of EtherChannels When Clustering with VPCs
VPC PEER LINK
N7K VPC 32
Cluster Data Plane
Data Plane of Cluster MUST use cLACP (Spanned Port-Channel) VPC Identifier on N7K must be the same for channel consistency
– ASA uses the ‘span-cluster’ command on channel
Control Plane [Cluster Control Link] of Cluster MUST use standard LACP (Local Port-Channel)
Each VPC Identifier on Nexus 7K is unique
Port Channel Identifier on ASA is arbitrary
– (max number 48)
Cluster Control Plane
LACP – Local Port Channels
1 2 3 4
cLACP – Spanned Port Channel
N7K VPC 41N7K VPC 40 N7K VPC 43
N7K VPC 42
1 2 3 4
http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/VMDC/ASA_Cluster/ASA_Cluster/ASA_Cluster.html
BRKDCT-2334 68
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Fabric w/FabricPath
• Externally the Fabric looks like a single switch
• Internally, ISIS adds Fabric-wide intelligence and ties the elements together.
• Provides in a plug-and-play fashion:• Optimal, low latency connectivity any to any
• High bandwidth, high resiliency
• Open management and troubleshooting
• ISIS for multipathing and reachability
FabricPath FabricPath
BRKDCT-2334 70
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 3 locations with Fabric Path
Layer 3 at Spine
• Overload Bit does not delay emulated switch id advertisement currently
• MAC scale is based off of the F2 or F3 modules being used
• Reduced points of configuration
Distributed Layer 3 at each Leaf
• Overload Bit provides fast failover no startup
• MAC scale at edge
• Management application to synchronize configurations for Layer 3
Layer 3 attached to Border Leaf
• Overload Bit provide fast failover on startup
• MAC scale can we scaled horizontally by adding in multiple GWs
• Common point of configuration for Layer 3
BRKDCT-2334 71
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FabricPath: Design with Routing at Spine
FabricPath
Default-Gateway
Nx7k FP Spine (F3)
Anycast-HSRP
Nexus 5600
FP leaf
UCS-FI
F3 mac-scale (ARP)
BRKDCT-2334 72
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Routing at FabricPath SpineAnycast HSRP L3
SVISVISVISVI
Anycast HSRP
GWY IP X
GWY MAC A
GWY IP X
GWY MAC A
GWY IP X
GWY MAC A
GWY IP X
GWY MAC A
GWY MAC A→L1,L2,L3,L4
FabricPath
L3
L2/L3 boundary
All Anycast HSRP forwarders
share same VIP and VMAC
Hosts resolve shared
VIP to shared VMACRouted traffic spread
over spines based on
ECMP
Anycast HSRP
between agg switches
Layer 3 LinkLayer 2 CELayer 2 FabricPath
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FabricPath: External / WAN Connectivity
- Default-Gateway
- Nx7k FP Spine (F3)
- MPLS PE Layer
• Spine/leaf architecture
• FabricPath for L2 multi-pathing
• MPLS Integration to WAN
• No spanning-tree
• Default gateway at spine layer
• ASA for firewall layer
• Nexus 5600 DC Access
ASR1K/9K
MPLS / LISP
MPLS, WAN
, Internet, Campus
Note:
- F3 simplifies the deploy with MPLS and FabricPath Support.
- Previously we leveraged F2 for FabricPath (VDC)
- M2 for MPLS Connectivity (VDC)
BRKDCT-2334 74
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• DC Fabric with a FabricPath based data plane and MP-iBGP control plane.
• Use MP-iBGP on the leaf nodes to distribute internal host/subnet routes and external reachability
information.
• Introduced Segment ID to increase name space to 16M identifier in the fabric.
Standalone FabricHost and Subnet Route Distribution
MAN/WAN
N1KV/OVS
External Subnet
Route Injection
MP-iBGP AdjacenciesRR RR
Fabric Host/Subnet
Route Injection
MP-iBGP Control Plane
FabricPath DataPlane
Route-Reflectors deployed for scaling purposes
BRKDCT-2334 75
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Optimized NetworkingDistributed Gateway Mode
• Distributed Gateway exists on all Leafs where VLAN/Segment-ID is active
• No HSRP
• There are different Forwarding Modes for the Distributed Gateway:
• Proxy-Gateway (Enhanced Forwarding)• Leverages local proxy-ARP • Intra and Inter-Subnet forwarding based on Routing• Contain floods and failure domains to the Leaf
• Anycast-Gateway (Traditional Forwarding)• Intra-Subnet forwarding based on Bridging• Data-plane based conversational learning for
endpoints MAC addresses• ARP is flooded across the fabric
vlan 123vn-segment 30000
!interface vlan 123
vrf member OrgA:PartAfabric forwarding mode proxy-gatewayip address 10.10.10.1/24no shutdownno ip redirects
vlan 145vn-segment 31000
!interface vlan 145
vrf member OrgA:PartAfabric forwarding mode anycast-gatewayip address 20.20.20.1/24no shutdown
BRKDCT-2334 76
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technologies Intra-DC and Inter-DC
Requirement Intra-DC Inter-DC
Layer 2 connectivity FabricPath, VXLAN OTV, VPLS
IP Mobility LISP, FP, AnyCast Gateway LISP, OTV
Secure Segmentation VXLAN / Segment-ID LISP, MPLS-IP-VPNs
DC-eastDC-west
POD POD POD POD
IP Network
Fabric Path, VXLAN
(Intra-DC L2)
OTV, VPLS
(Inter-DC L2-x-L3)
App
OS
App
OS
App
OS
App
OS
EF, LISP, VXLAN
(Intra-DC x-L3)
LISP
IP mobility
(Inter-DC)
Fabric Path,VXLAN
(Intra-DC L2)
App
OS
App
OS
EF, LISP
(Intra-DC mobility)
BRKDCT-2334 78
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is a Virtual Overlay Technology ?
• Servers perform data encapsulation and forwarding
• SW based virtual switches instantiate customer topologies
V
M
1
V
M
2
V
M
3
Virtual
Switch
Hypervisor
V
M
4
V
M
5
V
M
6
Virtual
Switch
Hypervisor
IP Network
Ethernet Frames
IP/UDP Packets
IP Addr
2.2.2.2
IP Addr
1.1.1.1
BRKDCT-2334 79
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtual Overlay Encapsulations and Forwarding
• Ethernet Frames are encapsulated into an IP frame format
• New control logic for learning and mapping VM identity (MAC address) to Host
identity (IP address)
• Two main Hypervisor based Overlays
• VXLAN Virtual Extensible Local Area Newtork
• NVGRE, Network Virtualization Generic Router Encapsulation
• GENEVE Generic Network Virtualization Encapsulation ( draft )
• Network Based Overlays
• OTV, Overlay Transport Virtualization
• VPLS, EVPN
• FabricPath
• VXLAN and NVGRE
BRKDCT-2334 80
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MTU and VXLAN
• VXLAN adds 50 Bytes to the Original Ethernet Frame
• Avoid Fragmentation by adjusting the IP Networks MTU
• Data Centers often require Jumbo MTU; most Server NIC do support up to 9000 Bytes
• Using a MTU of 9216* Bytes accommodates VXLAN Overhead plus Server max. MTU
Underlay
Un
de
rla
y
Outer IP Header
Outer MAC Header
UDP Header
VXLAN Header
Original Layer-2 Frame Ove
rlay
50 (
54)
Byte
s o
f O
verh
ead
*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic
For YourReference
BRKDCT-2334 81
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN TerminologyGateway, Bridging, Routing
VXLAN (VNI) to VLAN (VLAN) Bridging (L2 Gateway)
VXLANORANGE
Ingress VXLAN packet on
Orange segment
Egress interface chosen
(bridge may .1Q tag the packet)
VXLAN L2
Gateway
Egress is a tagged interface.
Packet is routed to the new VLAN
VXLAN (VNI) to VLAN Routing (SVI)(L3 Gateway)
VXLANORANGE
Ingress VXLAN packet on
Orange segmentVXLAN
Router
VLANBLUE
VLANORANGE
Tunnel Decap, then VLAN Route
BRKDCT-2334 82
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Underlay
Local LAN
Segment
Physical
Host
Local LAN
Segment
Physical
Host
Virtual
Hosts
Local LAN
Segment
Virtual
Switch
Edge
Device
Edge
Device
Edge
Device
IP Interface
BRKDCT-2334 83
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Underlay
Local LAN
Segment
Physical
Host
Local LAN
Segment
Physical
Host
Virtual
Hosts
Local LAN
Segment
Virtual
Switch
Edge
Device
Edge
Device
Edge
Device
VTEP
VTEP
VTEP
VV
V
Encapsulation
BRKDCT-2334 84
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Control/Data Plane Learning
Flood and Learn
• No Control plane. Data plane learning is only option
• Data Plane Learning similar to Ethernet. Packets are flooded out all ports and over a Multicast address to find destination device.
BGP Based Control Plane
• Control plane uses standards-based BGP
• Layer 2 MAC and Layer 3 IP info distribution by BGP
• Forwarding decision based on control plane to minimize flooding
• IETF Draft L2VPN-EVPN evolved to RFC 7432
BRKDCT-2334 85
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN – Ethernet VPNVXLAN Evolution
Control-
PlaneEVPN MP-BGP
draft-ietf-l2vpn-evpn
Data-
Plane
Multi-Protocol Label Switching
(MPLS)draft-ietf-l2vpn-evpn
Provider Backbone Bridges
(PBB)draft-ietf-l2vpn-pbb-evpn
Network Virtualization Overlay
(NVO)draft-sd-l2vpn-evpn-overlay
EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for Data Center Fabric encapsulations
Provides Layer-2 and Layer-3 Overlays over simple IP Networks
For YourReference
BRKDCT-2334 86
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Additional Functions of VXLAN/EVPN
Early ARP
Termination
Distributed Anycast Gateway
Suppresses flooding for Unknown Unicast ARP
Authenticate VTEPs through BGP peer authentication
Seamless and Optimal vm-mobility
Active/Active MultipathingActive/Active and Resilient Multipathing using vPC on
Nexus
Ingress Replication Unicast Alternative to Multicast underlay
Security & Authentication
BRKDCT-2334 87
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Host and Subnet Route Distribution
• Host Route Distribution decoupled from the Underlay protocol
• Use MultiProtocol-BGP (MP-BGP) on the Leaf nodes to distribute internal Host/Subnet Routes and external reachability information
• Route-Reflectors deployed for scaling purposes
VXLAN/EVPN
BGP Route-ReflectorRR
iBGP Adjacency
Edge
Device
Edge
Device
VTEP
VTEP
VTEP
VV
V
Encapsulation
RR RR
BRKDCT-2334 88
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Protocol Learning & Distribution VXLAN/EVPN
Host A
MAC_A / IP_AHost B
MAC_B / IP_B
Virtual Switch
Host C
MAC_C / IP_C
Host Y
MAC_Y / IP_Y
1VTEPs advertise Host Routes (IP+MAC)for the Host within the Control-Plane
Edge
Device
Edge
Device
VTEP
VTEP
VTEP
VV
V
Encapsulation
RR RR
11
1
BRKDCT-2334 89
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Protocol Learning & DistributionVXLAN/EVPN
Host A
MAC_A / IP_AHost B
MAC_B / IP_B
Virtual Switch
Host C
MAC_C / IP_C
Host Y
MAC_Y / IP_Y
Edge
Device
Edge
Device
VTEP
VTEP
VTEP
VV
V
Encapsulation
RR RR
2
2BGP propagates routes forThe Host to all other VTEPs
MAC, IP VNI NH
MAC_A, IP_A 30000 IP_L1
MAC_B, IP_B 30000 IP_L2
MAC, IP VNI NH
MAC_A, IP_A 30000 IP_L1
MAC_C, IP_C 30000 IP_L3
MAC_Y, IP_Y 30001 IP_L3
3VTEPs obtain host routes for
remote hosts and install in RIB/FIB
3 3
3
MAC, IP VNI NH
MAC_B, IP_B 30000 IP_L2
MAC_C, IP_C 30000 IP_L3
MAC_Y, IP_Y 30001 IP_L32
2
BRKDCT-2334 90
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Host MovesVXLAN/EVPN
BGP Route-ReflectorRR
iBGP Adjacency
Edge
Device
Edge
Device
VTEP
VTEP
VTEP
VV
V
RR RR
• Host Moves to Leaf 3
• Leaf 3 detects Host A and advertises it with Seq #1
• Leaf 1 sees more recent route and withdraws its advertisement
L1# sh bgp l2vpn evpn 192.168.101.101
BGP routing table information for VRF default, address family
L2VPN EVPN
Route Distinguisher: 10.254.254.102:32868 (L2VNI 30001)
BGP routing table entry for
[2]:[0]:[0]:[48]:[0050.56ac.0773]:[32]:[192.168.101.101]/272,
version 30
Paths: (1 available, best #1)
Flags: (0x00030a) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled
nexthop
AS-Path: NONE, path locally originated
10.254.254.102 (metric 0) from 0.0.0.0 (10.254.254.102)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 30001 50000
Extcommunity: RT:65501:30001 RT:65501:50000 MAC Mobility
Sequence:00:1
Path-id 1 advertised to peers:
10.254.254.101
MAC, IP VNI
(L2)
VNI
(L3)
NH Encap Seq
0050.56ac.0773, 192.168.101.101 30001 50000 0.0.0.0 8:VXLAN 1
BRKDCT-2334 91
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Symmetric IRB
• Symmetric
• Similar to creating a Transit Segment
• Regardless of where Source or Destination VNI exists
• Post Routing traffic uses different VNI than Bridged traffic
• Additional VNI for Routing traffic (per VRF)• From Host A via VLAN “blue” routed at L1 to
VNI “purple” reaching destination VLAN “red”
• From Host Y via VLAN “red” routed at L3 to VNI “purple” reaching destination VLAN “blue”
• Used in Cisco VXLAN/EVPN
VXLAN Routing
Host Y
VNI 30001
Host A
VNI 30000
L3
L2
L1
BRKDCT-2334 92
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multicast Enabled Underlay
• PIM-ASM or PIM-BiDir (Different hardware has different capabilities)
• Spine and Aggregation Switches make good Rendezvous-Point (RP); much lile RR
• PIM-ASM (sparse-mode)
• Source-trees, build a couple of unidirectional trees from RP; (s,g)
• Every VTEP is Source and Destination
• PIM-Anycast RP vs MSDP for example
• PIM-BiDir
• No Sources tree use a bi-directional shared tree
• No (S,G), we have (*,G)
• Phanton RP (Leverages Unicast for convergence)
• Each VNI does not need the same a different Multicast Group
Underlay
Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000ASR 1000
CSR 1000ASR 9000
Multicast
Mode
IGMP L2/L3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir
BRKDCT-2334 93
For YourReference
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multiple Loopback AddressesUse Unique Loopback based on Purpose
SPINE(non-VTEP)
Anycast-RP 1:
feature pim
feature eigrp
interface loopback0
ip address 10.1.1.6/32
ip router eigrp 10
ip pim sparse-mode
interface loopback1
ip address 10.10.10.50/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
LEAF (VTEP)
Anycast-RP 1:
feature pim
feature ospf
interface loopback0
description for VTEP
ip address 1.1.1.2/32
ip address 1.1.1.1/32 secondary
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description for routing protocol
ip address 10.10.10.201/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Configuration – Mapping VLANs to VNIsLayer 2 Gateway on Multicast Enabled Fabric
Feature interface-vlanfeature vn-segment-vlanfeature nv overlayfeature pim
Vlan 102vn-segment 10102
interface nve1no shutdownsource-interface loopback0member vni 10102 mcast-group 239.1.1.102
interface <phy if>switchport mode accessswitch port access vlan 102
Used for the VTEP
VXLAN Identifier
IP Multicast Group for Multi-
Destination Traffic
Locally Significant VLAN
Tunnel Interface
BRKDCT-2334 95
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ip pim rp-address 10.50.255.254 group-list 224.0.0.0/4ip pim ssm range 232.0.0.0/8
interface Ethernet1/49-50no switchportmtu 9200ip ospf network point-to-pointip router ospf 10 area 0.0.0.0ip pim sparse-mode
interface Ethernet1/49ip address 1.1.1.14/30
interface Ethernet1/50ip address 1.1.1.18/30
interface loopback 0ip address 1.1.1.2/32ip address 1.1.1.1/32 secondaryip router ospf 1 area 0.0.0.0ip pim sparse-mode
Enabling Multicast on TOR for VXLAN
Multicast RP Definition
Multicast RP DefinitionLinks to SpinesLinks to Spines
Used for the VTEP
BRKDCT-2334 96
vrf context TENANTvni 23002rd autoaddress-family ipv4 unicastroute-target both autoroute-target both auto evpn
router bgp 65123router-id 10.10.10.201address-family ipv4 unicastaddress-family l2vpn evpn
neighbor 10.50.255.3 remote-as 65123update-source loopback1send-community both
vrf TENANTaddress-family ipv4 unicastadvertise l2vpn evpnredistribute direct route-map FABRIC
interface loopback 1ip address 10.10.10.201/32ip router ospf 1 area 0.0.0.0ip pim sparse-mode
VXLAN EVPN
Pointing to BGP
Route Reflectors
Used for the Router IDs
VRF Definition
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
WAN
TIER
VXLAN Fabric Design PlaneEVPN Control Plane
D
S1 S2
BL1 BL2 L44
L3 Fabric
B
OSPF AREA 0
VPC Peer
LInk
Routed
Links
Appliances
PC L2 Trunks
Multicast
Anycast RPs
Multicast
Anycast RPs
C
L42
A
*Caveat: Behavior with vPCs and routing BRKDCT-2334 98
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
WAN
TIER
VXLAN Fabric Design Plane
• vPC uses secondary address as virtual VTEP
• By default all connections to either vPC attached TOR uses virtual VTEP
• Routing Failover faster if not vPCconnected switch.
Routed
Links
VPC Peer
LInk
router bgp 65123
address-family ipv4 unicast
address-family l2vpn evpn
advertise-pip
BRKDCT-2334 99*Caveat: Behavior with vPCs and routing
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Networks Layer 2 and or Layer 3
Web
App DB
Clients
ACLs
Layer 3
ACLs
Layer 2
ACLs
VM Mobility ManagementStorage
VLAN 10
VNI 10
VLAN 12
VNI 12
L3 VNI 3001VLAN 11
VNI 11
BRKDCT-2334 100
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Networks Layer 2 and or Layer 3
Web
App DB
Clients
ACLs
Layer 3
ACLs
Layer 2
ACLs
Description VLAN Gateway VNId Multicast Group L3 VNI TenantWAN to FW 100 FW 100100 239.1.1.100 - none
FW to ADC 101 FW 100101 239.1.1.100 - none
ADC to WEB 102 FW 100102 239.1.1.100 - none
FW to APP/DB 103 Fabric 100103 239.1.1.100 - none
APP 104 Fabric 100104 239.1.1.100 300100 CISCOLIVE
DB 105 Fabric 100105 239.1.1.100 300100 CISCOLIVE
VM Mobility 10 No 10 239.1.1.1 - none
Storage 11 No 11 239.1.1.1 - none
Management 12 Fabric 12 239.1.1.1 300000 CommonServices
BRKDCT-2334 101
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ESX 6.0 Multiple TCP/IP StacksStarted in ESX 5.5
DEFAULT
vmk0 1.1.0.0/24
vmk1 1.1.1.0/24
DG 1.1.0.254
DNS 4.2.2.1
vMotion
vmk2 1.1.2.0/24
DG 1.1.2.254
DNS 8.8.8.8
Provisioning
vmk3 1.1.3.0/24
DG 1.1.3.254
DNS 8.8.8.8
BRKDCT-2334 102
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN and Layer 2 Loop avoidance
• VXLAN doesn’t implement a native L2 loop detection and protection
• BPDU’s are not forwarded across the VXLAN domain
• A backdoor link can be established between two or more TORs
Leaf 1 Leaf 2Leaf 3 Leaf 5
Spine 1 Spine 2
Leaf 6Leaf 4
Srv 1VLAN 100
VM-1VM-4
Srv 5VLAN 100
Srv 2VLAN 100
Srv 3VLAN 100
Enable BPDUGuard
BRKDCT-2334 103
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN NVE Show Commands
Nexus# show nve peer
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.50.255.251 Up CP 01:18:41 64f6.9dee.b6f5
Nexus# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 100100 239.1.1.100 Up CP L2 [100] SA
nve1 300100 n/a Up CP L3 [CISCOLIVE]
nve1 100101 239.1.1.100 Up CP L2 [101] SA
nve1 100102 239.1.1.100 Up CP L2 [102]
nve1 100103 239.1.1.100 Up CP L2 [103]
nve1 300000 n/a Up CP L3 [CommonService]
nve1 10 239.1.1.1 Up CP L2 [10]
nve1 11 239.1.1.1 Up CP L2 [11]
BRKDCT-2334 104
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN NVE Show CommandsNexus# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 1 b838.6146.a9fd dynamic 0 F F Po23
+ 1 b838.6146.a9fe dynamic 0 F F Po23
* 100 0010.9400.0034 dynamic 0 F F Eth1/45
* 100 0010.9400.0035 dynamic 0 F F nve1(10.50.255.251)
+ 100 0010.9400.0002 dynamic 0 F F vPC Peer-Link
show bgp l2vpn evpn ( Filtered )
BGP routing table information for VRF default, address family L2VPN EVPN
…
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.50.255.6:32867 (L2VNI 10100)
* i[2]:[0]:[0]:[48]:[0010.9400.0035]:[32]:[172.16.1.110]/272
10.50.255.251 100 0 i
*>i 10.50.255.251 100 0 i
BRKDCT-2334 105
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VNI Scalability Per Platform
• Reference the VXLAN Verified Scalability Limits (Unidimensional) at a high level
• Focus on the Validated Deployment Case studies
• Can you support 750, 900, 1000, 1500, or 1600 VNIs?
• How Many TORs can communicate? Can I use Ingress replication or does my
design require Multicast?
• Routes
• Underlay Routes
• Overlay Routes
• Host Routes
• MAC addresses
5600, 7000, 9300, and 9500 have different scalability numbers
BRKDCT-2334 106
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
L2 Network Overlays for Data Center Interconnect
• OTV/VPLS resilient geo-extension of segments
• Preserve failure isolation between locations
• Network resiliency and multi-pathing
• Built in loop handling
• Optimal traffic handling
• Streamlined operations
• Egress routing optimization
• HW Accelerated high performance connectivity
OTV/VPLS
North
Data
CenterFault
Domain
Fault
Domain
Fault
Domain
Fault
Domain
LAN Extension
BRKDCT-2334 108
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Server-ServerEgress Routing Localization:
Server-Client
Egress Routing Localization:
Server-Client
Hypervisor Hypervisor
Ingress Routing
Localization:
Clients-Server
Path Optimization
Layer 2 extensions represent a challenge for optimal routing
Challenging placement of gateway and advertisement of routing prefix/subnet
Optimal Routing Challenges
BRKDCT-2334 109
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Path OptimizationEgress Routing with LAN Extension
HSRP
Active
HSRP
Standb
y
HSRP
Listen
HSRP
Listen
HSRP Hellos
VLAN
20
VLAN
10
• Extended VLANs typically have associated HSRP groups
• By default, only one HSRP router elected active, with all servers pointing to HSRP VIP as default gateway
• Result: sub-optimal (trombone) routing
BRKDCT-2334 110
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
HSRP
Active
HSRP
Standb
y
HSRP
Listen
HSRP
Listen
ARP
reply
ARP for
HSRP VIP
VLAN
20
VLAN
10
Path Optimization
• Extended VLANs typically have associated HSRP groups
• By default, only one HSRP router elected active, with all servers pointing to HSRP VIP as default gateway
• Result: sub-optimal (trombone) routing
Egress Routing with LAN Extension
111
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Path Optimization
• Extended VLANs typically have associated HSRP groups
• By default, only one HSRP router elected active, with all servers pointing to HSRP VIP as default gateway
• Result: sub-optimal (trombone) routing
Egress Routing with LAN Extension
HSRP
Active
HSRP
Standb
y
HSRP
Listen
HSRP
Listen
VLAN
20
VLAN
10
Packet from
Vlan 10 to Vlan 20
DMAC = DGW
Routing
Packet from
Vlan 10 to Vlan 20
DMAC = Host Vlan 20
112
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Egress Routing Localization
• Filter FHRP with combination of VACL and MAC route filter
• Result: Still have one HSRP group with one VIP, but now have active router at each site for optimal first-hop routing
FHRP Filtering Solution
HSRP
ActiveHSRP
Standb
y
ARP for
HSRP VIP
ARP
reply
HSRP FilteringHSRP
ActiveHSRP
Standb
y
HSRP Hellos HSRP Hellos
VLAN
20
VLAN
10
no ip arp gratuitous hsrp duplicate
113
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Sample Cluster - Primary Service in Left DCFHRP Localization – Egress Path Optimization
HA cluster Node B
Layer 3 CoreISP A ISP B
HA cluster Node A
Access
Agg
Cluster VIP = 10.1.1.100 Preempt
Default GW = 10.1.1.1
Node A
Data Center
AData Center
B
VLAN A
Public Network
Asymmetrical flows No Stateful device
Low ingress traffic
HSRP
Active
HSRP
StandbyHSRP
Active
HSRP
StandbyHSRP Filtering
BRKDCT-2334 114
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Recommended Reading
BRKDCT-2334 116
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
BRKDCT-2334 117
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCT-2334 118
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com