Real World Data Centerd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKDCT-2334.pdf · Real World Data Center ... however, no ability to dynamically insert services. ... • Via self-managed

Jeff Ostermiller – CCIE #5402e

[email protected]

Technical Solutions Architect

@jostermi

BRKDCT-2334

Real World Data Center Deployments and Best Practices

mailto:[email protected]

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Abstract

The seminar will discuss real world Nexus Deployment scenarios to make sure your network will meet the demands for performance and reliability. This session will provide and equip you with the latest information on Cisco® data center network architecture and best practices around those designs. This session will focus on STP, vPC, Fabric Path, QOS, routing and service node insertion from the core of the network to the host. This session will not cover all of the possible options just the best practices to make sure we are all successful.

BRKDCT-2334 3


Speakers

Jeff Ostermiller CCIE #5402eTechnology Solution ArchitectData Center Architecture@[email protected]

BRKDCT-2334 4

mailto:[email protected]

• Data Center Design Elements

• Data Center Design Evolution

• Fundamental Data Center Design

• Services Insertion

• Scalable Data Center

• Overlays

Agenda

Data Center Design Elements


What do I really need in the Data Center?

• Compute

• IP Based Storage

• Firewalls/Security

• Load Balancers

• WAN Tier

• Internet

• Intranet

• Campus Aggregation

• Visibility

SiSi SiSi

BRKDCT-2334 7


Virtual Machine Scalability

• ESX Maximum VMs per Hosts

• 5.5 512 VMs per Hosthttps://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf

• 6.0 1024 VMs per Hosthttps://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf

• Hyper V Maximum VMs per Host

• Windows Server 2012 1024 VMs per Server https://technet.microsoft.com/en-us/library/jj680093.aspx

• Openstack

• Well…

• Containers

• Lots of application containers per OS, few OS’s

BRKDCT-2334 8

https://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf

https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf

https://technet.microsoft.com/en-us/library/jj680093.aspx


Real Life Virtualization Ratios

• 22 cores per socket and 2 sockets per 1 RU server.

• Memory Sizes

• 128G, 256G, 512G common memory sizes

• 1TB of Memory at Max Memory speed 2133 MHz

• 1.5TB at 1600 MHz

• Lets say 50 VMs per server ( everyone is different )

• 1 client has 100 VMs per server with 1TB per server

• 1 client has 67 VMs per server with 256G

• 1 client has 30 VMs per server with 256G

C220 M4 as Example

BRKDCT-2334 9


Rack Density 47 RU

• Forty 1 RU Servers ( 22 cores x 2 sockets x 40 Servers )

• 2 10/25GE interfaces per server channeled

• 1 ILO Port per server

• 50 VMs per Server – 2000 VMs per Rack

• 1 - 48 port 1 GE out of band management switch

• 2 - 48 Port 10/25GE network switches

• 4 RUs free for Cable Management

Hyper Dense Rack

The switches are only 83% utilized or 17% free

BRKDCT-2334 10


Hyper Dense POD ( Point of Delivery )Hyper Dense Rack with IP Based Storage

• Forty 1 RU Servers ( 22 cores x 2 sockets x 40 Servers )



• 1 Rack for IP Based Storage 16 10/25 GE available

• External Storage to the POD may be deployed as shared storage

BRKDCT-2334 11


Rack Density 42 RU

• Twenty 1 RU Servers ( 22 cores x 2 sockets x 20 Servers )

• 2 10/25GE interfaces per server channeled

• 1 ILO Port per server

• 50 VMs per Server – 1000 VMs per Rack



• 19 RUs free for cable management

Half Full due to Power/Cooling/Weight Concerns

The switches are only 42% utilized or 58% free

BRKDCT-2334 12


28 Free 10/25G ports

• Compute

• IP Based Storage – 4 10/25Ge ports per switch

• Firewalls – 4 10/25Ge ports per switch

• Load Balancers – 4 10Ge ports per switch

• WAN Tier – 4 10Ge ports per switch

• Internet

• Intranet

• Campus Aggregation ???

BRKDCT-2334 13


28 12 Free 10/25G ports

• Compute

• IP Based Storage – 4 10/25Ge ports per switch

• Firewalls – 4 10/25Ge ports per switch

• Load Balancers – 4 10Ge ports per switch

• WAN Tier – 4 10Ge ports per switch

• Internet

• Intranet

• Campus Aggregation 12 closets

• what about 96 port switches? Then 60 closets

BRKDCT-2334 14


* http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

*

Data Center Design RequirementsThings will fail, so how can we protect ourselves

BRKDCT-2334 15

http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html


Example of Constrained Resource

Feature Parameter Verified Limit (Cisco NX-OS 6.2)

ARP/ND

Sup 1 Sup 2 Sup 2E

Number of entries in ARP table 128,000 128,000 128,000

Number of ARP packets per second 1500 1500 5000Number of ARP glean packets per second 1500 1500 5000

Number of IPv6 ND packets per second 1500 1500 5000Number of IPv6 glean packets per second 1500 1500 5000

BRKDCT-2334 16


Availability ZonesUsing Amazon Web Services terms

PodAvail

Zone

Avail

Zone

Avail

Zone

RegionMulti-Pod

Global Multi-Site Data Center Deployment

PodPod

BRKDCT-2334 17


Connecting the Data Center to the Rest of the Network

• Use routed pt2pt links

• ECMP used to quickly re-routearound failed node/links

• Tune CEF L3/L4 load balancing hash to achieve maximum utilization

• Build triangles not squares for deterministic convergence

• Insure redundant L3 paths to avoid black holes

• Summarize distribution to core to limit event propagation

• Utilized on both Multi-Layer and Routed Access designs

BRKCRS-3036 Advanced Enterprise Campus Design

Layer 3 Equal

Cost Link’sLayer 3 Equal

Cost Link’s SiSiSiSi

SiSiSiSiSiSiSiSiSiSiSiSi

WANInternet

SiSiSiSi SiSi SiSi

Data Center

BRKDCT-2334 18


Building Block DesignWAN and Campus Connectivity

SiSi SiSi

Created

Isolated

Campus Core

WAN Building

Block

Layer 3

Switch/Router

Segment

Networks with

Routing

Don’t Forget

OOB

Managmenet

DC

Zone

WAN

Zone

Campus

Zone

Site

Border Leaf

switches

common Connect OOB

Management

into Campus

Core!

BRKDCT-2334 19

Data Center Design Evolution


Traffic Paths

Source: Cisco Global Cloud Index, 2016 BRKDCT-2334 21


Oversubscription Ratio

• Large layer 2 domain with collapsed Access and Core

• Worse Case Calculation

• Assume all the traffic is north-south bound

• Assume 100% utilization from the Access Switches

• All the ports operated in dedicated mode

Access to Core/Aggregation

Aggregation

Access

Oversubscription

Ratios

16 10 GEs

48 10 GEs

48:16=3:1

Line Cards 1:1

(3:1)*(1:1)

BRKDCT-2334 22


Data Center Design Evolution Clos Fabric

• Moving to Spine/Leaf construct

• No Longer Limited to two aggregation boxes

• Created Routed Paths between “access” and “core”

• Routed based on MAC, IP, or VNI

• Layer 2 can be anywhere even with routing

• Automation/Orchestration, removing human error.

Ro

ute

d

Do

ma

in

L2

Domain

Servers and Services

Connected at the Leaves

BRKDCT-2334 23


Clos Fabric, Fat Trees

Spine

Leaf

8 40 GEs

48 10 GEs

48:32=1.5:1

Line Cards 1:1

(1.5:1)*(1:1)

Changing Traffic Flow

Requirements

Services are deployed

at the leaf nodes

Oversubscription

Ratios defined by

number of spines and

uplink ports

True horizontal scale

BRKDCT-2334 24


Statistical Probabilities…

• Assume 11 10G source flows, the probability of all 11 flows being able to run at full flow rate (10G) will be almost impossible with 10G (~3%), much better with 40G (~75%) & 100G (~99%)

Intuition: Higher speed links improve ECMP efficiency

1 2 3 4 5

1 2

1 2 20

Prob of 100% throughput ≅ 3%



20×10Gbps

Uplinks2×100Gbps

Uplinks

11×10Gbps flows

(55% load)

5×40Gbps

Uplinks

BRKDCT-2334 25


Why 25 GB Ethernet

• Server IO Doubling every 24 Months

• Core Networking Doubling every 18 Month

• Clients starting to use multiple interfaces per Server again

• Maximize Switch Throughput

• Minimize # of Cables and Tor switches

• SFP-25G Transceivers same form factor at SFP-10G

• 1, 2, 3, 5 meter Twinax

• SR Optics 100m OM4

Ethernet Alliance Introduction-to-25GbE-Webinar_D2p1.pdf

BRKDCT-2334 26


Fabric OptionsWhich Model is Right for you?

VXLANFabric PathVPCSTP

L3

L3

BRKDCT-2334 27

Application Centric Infrastructure

DB DB

Web Web App Web App

Turnkey integrated solution with

security, centralized management,

compliance and scale

Automated application centric-policy

model with embedded security

Broad and deep ecosystem

Fabric Management Options

Programmable Network

Modern NX-OS with enhanced

NX-APIs

DevOps toolset used for Network

Management(Puppet, Chef, Ansible etc.)

Programmable Fabric

VxLAN-BGP EVPN

standard-based

3rd party controller support

Cisco Controller for software

overlay provisioning and

management across N2K-N9K

VTS

Creation Expansion

Fault MgmtReporting

Connection

Automation, API’s, Controllers and Tool-chain’s


Programmable Networks

• Standard and Open APIs to the DC Switching Fabric that is common against multiple DC products.

• Ability to script tasks into automated run books; however, no ability to dynamically insert services.

• DevOps Methodology

CLI

Element

Manager

Scripts

ACI

Stand-alone

NX-OSFull ACI mode

https://developer.cisco.com/site/devnet/home/index.gsp

BRKDCT-2334 29

https://developer.cisco.com/site/devnet/home/index.gsp


CLI

Basic ElementManager

ACI

Scripting to the CLI

and/or API

Fabric Management Options

• Solutions targeted towards different fabric ops models, market segments, and user CLI-proficiencies

Datacenter Network Manager (DCNM)• Graphical network manager + underlay templates

• Switch platform feature breadth and depth

• Cisco Nexus 2K through 9K platforms supported

• CLI/feature knowledge required

Virtual Topology System (VTS)• Fabric overlay focused – VXLAN-based

• Primarily focused on service provider

• Multi-platform support

• Tie-ins to OpenStack and vCenter

Nexus Fabric Manager (NFM)• Simplified interaction– GUI + API

• Build fabric-wide broadcast domains• Via self-managed underlay and overlay

• Ops – minimal CLI requirements

• Point-n-click simplified interface

BRKDCT-2334 30


Automation ‘and’ OperationsThere are multiple decision vectors

CONTINUOUS

INTEGRATIONORCHESTRATION &

MANAGEMENT (O&M)

Operations involves a full life cycle of infrastructure and

application management?BRKDCT-2334 31

Programmable NetworkProgrammable FabricApplication Centric

Infrastructure

Integrated stack

Or

A-la-carte Automation

Streamlined Workflow Management

Modern NX-OS with enhanced NX-APIs

DevOps toolset used for Network Management(Puppet, Chef, Ansible etc.)

Customer Script based Operations and Workflows

Turnkey integrated solution with security, centralized

management, compliance and scale

Automated application centric-policy model with

embedded security

Broad and deep ecosystem

FCAPS ‘and’ Automation

Fault

Accounting

Performance

Security

Configuration

External

ToolsIntegrated

Tools

VTSCreation Expansion

Fault MgmtReporting

Connection

External

Tools

Integrated

Tools


NX-API Developer Sandbox


Learning Python via the API

BRKDCT-2334 34


Dynamic Python Programimport requests

import json

url='http://YOURIP/ins'

switchuser='USERID'

switchpassword='PASSWORD’

myheaders={'content-type':'application/json-rpc’}

payload=[

{

"jsonrpc": "2.0",

"method": "cli",

"params": {

"cmd": "config t",

"version": 1

},

"id": 1

},

{

"jsonrpc": "2.0",

"method": "cli",

"params": {

"cmd": "vlan 1234",

"version": 1

},

"id": 2

}

]

response = requests.post(url,data=json.dumps(payload), headers=myheaders,auth=(switchuser,switchpassword)).json()

BRKDCT-2334 35


Python Consolidation

payload=[

{

"jsonrpc": "2.0",

"method": "cli",

"params": {

"cmd": "config t",

"version": 1

},

"id": 1

},

{

"jsonrpc": "2.0",

"method": "cli",

"params": {

….

payload=[

{"jsonrpc": "2.0","method": "cli","params": {"cmd": "config t","version": 1},"id": 1},

{"jsonrpc": "2.0","method": "cli","params": {"cmd": "vlan 1234","version": 1},"id": 2},

{"jsonrpc": "2.0","method": "cli","params": {"cmd": "exit","version": 1},"id": 3}

]

BRKDCT-2334 36


Adding a Loop to PythonPython Programming for Networking Engineers

@kirkbyers

http://pynet.twb-tech.com

Network Programmability User Group

https://netprog.atlassian.net/wiki/display/NPUG/NPUG

DevNet

https://developer.cisco.com/

http://pynet.twb-tech.com

https://netprog.atlassian.net/wiki/display/NPUG/NPUG

https://developer.cisco.com/


Network Device Orchestration Support:Technologies

Integrate network devices with

configuration management &

orchestration stacks

Customization

https://dcloud-cms.cisco.com/demo_news/cisco-nexus-9000-open-nx-os-programmability-and-automation-v1

For YourReference

BRKDCT-2334 38

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&docid=T6k6Ozx7rXrlkM&tbnid=gsoEKEIzV8aSFM:&ved=0CAUQjRw&url=http://www.enterprisetech.com/2013/12/04/dell-grabs-puppet-master-competitor-systems/&ei=xY3iU5v4JMqcyASAmYCYCQ&bvm=bv.72197243,d.aWw&psig=AFQjCNFvkZhwg4ndTW93BvjqMEe2zfpFfg&ust=1407442751306451


http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAQQjRw&url=http://www.getchef.com/&ei=-43iU6XxH4aTyASb2oKADw&bvm=bv.72197243,d.aWw&psig=AFQjCNEkpvgedcv0hTYPw_Cm3WSN7pBygA&ust=1407442811574777


https://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAQQjRw&url=https://www.normation.com/societe/partners&ei=MY7iU5C8KZW1yATfooGwDw&bvm=bv.72197243,d.aWw&psig=AFQjCNF11yLExp-cyfl9pkitp_lh_iBjUw&ust=1407442865736439


https://dcloud-cms.cisco.com/demo_news/cisco-nexus-9000-open-nx-os-programmability-and-automation-v1


A few more sophisticated options

• General Ansible

• Ansible Intro - http://docs.ansible.com/intro.html

• Ansible Modules - http://docs.ansible.com/modules.html

• YAML Syntax - http://docs.ansible.com/YAMLSyntax.html#yaml-basics

• Ansible Videos - http://www.ansible.com/resources

• Jason Edelman’s Blog - http://www.jedelman.com/home/network-automation-

with-cisco-nexus-switches-ansible/

• Jason Edelman’s Github - https://github.com/jedelman8/nxos-

ansible/blob/master/README.md

For YourReference

BRKDCT-2334 39

http://docs.ansible.com/intro.html

http://docs.ansible.com/modules.html

http://docs.ansible.com/YAMLSyntax.html

http://www.ansible.com/resources

http://www.jedelman.com/home/network-automation-with-cisco-nexus-switches-ansible/

https://github.com/jedelman8/nxos-ansible/blob/master/README.md







Fundamental Data Center Design


Classic Customer vPC DesignCommon Building Blocks

vPC peer-link

vPC Keepalive

vPC

FEX

vPC

vPC

vPC Deployments Started in 2009BRKDCT-2334 41


Classic Customer vPC Design

• vPC Peer Link

• Use Multiple Ports on Multiple Port ASICs

• Server Connections DUAL attach always single attach only when forced

• No switches connected below access tier

• Breaks ISSU.

• VLANs must be synchronized on all 4 boxes

• With FEX remember Oversubscription

Layer 2 only at fixed configuration Top of Rack switch

FEX

vPC peer-link

vPC Keepalive

vPC

BRKDCT-2334 42


Out of Band Management Network

• Build a true out of band management network.

• Several Client use older existing L2 campus switches

• Connect all Management interfaces into network

• Connect the Management switches back into the Campus CORE, Not the DC Core

• vPC Keepalive Link

• On Fixed Config switches use MGNT0 to avoid issues with ISSU and STP

Layer 2 only at fixed configuration Top of Rack switch

BRKDCT-2334 43


vPC Best Practice Summary

• Use LACP Protocol when connecting devices

• Use multiple interfaces for Peer-Link

• Enable Auto Recovery

• IP Arp sync

• Use Peer-Switch with appropriate spanning tree priorities set

• IPv6 ND synchronization

• Peer Gateway

• with exclude VLAN where required

• Fabric Path Multicast Loadbalance.

http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

Layer 2

BRKDCT-2334 44

http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf


Multicast ExampleAlmost all Designs moving forward will have a Multicast requirement BRKIPM-2264

Anycast-RP 2:

feature pim

feature eigrp

interface loopback0

ip address 10.1.1.4/32

ip router eigrp 10

ip pim sparse-mode

interface loopback1

ip address 10.10.10.50/32

ip router eigrp 10

ip pim sparse-mode

router eigrp 10

ip pim rp-address 10.10.10.50 group-list 224.0.0.0/4

ip pim ssm range 232.0.0.0/8

ip pim anycast-rp 10.10.10.50 10.1.1.4

ip pim anycast-rp 10.10.10.50 10.1.1.6

Anycast-RP 1:

feature pim

feature eigrp

interface loopback0


ip router eigrp 10

ip pim sparse-mode

interface loopback1

ip address 10.10.10.50/32

ip router eigrp 10

ip pim sparse-mode

router eigrp 10

ip pim rp-address 10.10.10.50 group-list 224.0.0.0/4

ip pim ssm range 232.0.0.0/8

ip pim anycast-rp 10.10.10.50 10.1.1.4

ip pim anycast-rp 10.10.10.50 10.1.1.6


Data Center Building Blocks

• Control Plane Scale

• ARP learning

• MAC addresses

• CPU Traffic Level, any packets that get punted to the CPU

• Spanning Tree Scale

• RSTP -> 16k Logical Ports, logical port limit is equal (# of ports)*(Vlans per ports)

• MST -> 25K Logical Ports, logical port limit is equal (# of ports)*(# of MST instances allowed per port)

• Port Channel Scaling Numbers

• Buffer Oversubscription

• Failure Domain Size ( Availability Zones )

• ISSU

Scaling ConcernsPOD

PODPOD

Multi-Pod

BRKDCT-2334 46


Example of Constrained Resource

Feature Parameter Verified Limit (Cisco NX-OS 6.2)

ARP/ND

Sup 1 Sup 2 Sup 2E

Number of entries in ARP table 128,000 128,000 128,000

Number of ARP packets per second 1500 1500 5000Number of ARP glean packets for second 1500 1500 5000

Number of IPv6 ND packets per second 1500 1500 5000Number of IPv6 glean packets per second 1500 1500 5000

128,000ARPs/1500(ARPs Per Second)=85.3 seconds

BRKDCT-2334 47


COPP Policy Monitoring

Customer 1 ( 5.2.9 code)

show policy-map interface control-pla class copp-system-p-class-normal | inc violate prev

4 | inc module|violated

module 3

violated 0 bytes; action: drop

module 8

violated 1152074225 bytes; action: drop (approximately 18 Million ARPs)

module 9

violated 2879379238 bytes; action: drop (approximately 45 Million ARPs)

Customer 2 (6.2.12 code)

show policy-map interface control-plane class copp-system-p-class-normal | inc violate

violate action: drop

violated 8241085736 bytes, (approximately 128 Million ARPs in 123 Days)

5-min violate rate 0 bytes/sec

violated 0 bytes,

5-min violate rate 0 bytes/sec

violated 0 bytes,

Control Plane Policy Exceeded

BRKDCT-2334 48


Effects of an ARP flood

2000 ARPs/Second

Arp Time out 4 Hours Arp Time out 4 Hours

25 Minute ARP Time out

30 Minute CAM Time out

25 Minute ARP Time out

30 Minute CAM Time out

Which ARP is better?

Server ARP

Storage ARP

Router ARP

Switch ARP

Routing PeersRouting Peers

BRKDCT-2334 49


Control Plane Policing

show copp diff profile strict profile moderate

'+' Line appears only in profile strict, version 6.2(6a)

'-' Line appears only in profile moderate, version 6.2(6a)

-policy-map type control-plane copp-system-p-policy-moderate

reduced

- class copp-system-p-class-normal

- set cos 1

- police cir 680 kbps bc 310 ms conform transmit violate drop

reduced

+ class copp-system-p-class-normal

+ set cos 1

+ police cir 680 kbps bc 250 ms conform transmit violate drop

• 680 kbps / (64 byte Arp Frames * 8 bits ) = 1328 ARPs per second

• BC = TC * CIR or 310 msec *680,000 = 204000 this means approximately another 400 ARPs per second are allowed for burst.

show policy-map interface control-plane class copp-system-p-class-normal | inc violate

BRKDCT-2334 50


Control Plane Protection

• Configure a syslog message threshold for CoPP

• in order to monitor drops enforced by CoPP.

• The logging threshold and level can be customized within each traffic class with use of the logging drop threshold <packet-count> level <level> command.

logging drop threshold 100 level 5

Example syslog output

%COPP-5-COPP_DROPS5: CoPP drops exceed threshold in class:

copp-system-class-critical,

check show policy-map interface control-plane for more info.

Notification about Drops

BRKDCT-2334 51


Control Plane Tuning

• Do not disable CoPP. Tune the default CoPP, as needed.

• Create Custom Policy to match your environment

• Monitor exceptions to see your tuning results

monitor session 1

source exception all

destination interface Eth1/3

no shut

nexus7k(config-monitor)# show monitor session 1

source exception : fabricpath, layer3, other

filter VLANs : filter not specified

destination ports : Eth1/3

Feature Enabled Value Modules Supported

--------------------------------------------------

L3-TX - - 1 6 8

ExSP-L3 - - 1

ExSP-FP - - 8

ExSP-OTHER - - 1

RB span No

BRKDCT-2334 52


Control Plane Protection

Nexus# copp copy profile strict prefix LAB

Nexus(config)# arp access-list LAB-copp-arp-critical

Nexus(config-arp-acl)# 10 permit ip 10.1.2.1 255.255.255.255 mac any

Nexus(config-arp-acl)# 20 permit ip 10.1.2.5 255.255.255.255 mac any

Nexus(config-arp-acl)# class-map type control-plane match-any LAB-copp-class-arp-critical

Nexus(config-cmap)# match access-group name LAB-copp-arp-critical

Nexus(config-cmap)# policy-map type control-plane LAB-copp-policy-strict

Nexus(config-pmap)# class LAB-copp-class-arp-critical insert-befor LAB-copp-class-normal

Nexus(config-pmap-c)# set cos 6

Nexus(config-pmap-c)# police cir 100 kbps bc 250 ms conform transmit violate drop

Nexus(config)# control-plane

Nexus(config-cp)# service-policy input LAB-copp-policy-strict

Good ARPs versus Bad ARPs

BRKDCT-2334 53

Services Insertion


Dedicated Solutions have ability to use Hardware Acceleration Resources like SSL Offload

Centralized Firewalls Transparent or Routed

Virtualized Services Layer 3

Services Deployment Models

Centralized ADCs Routed or One Armed

BRKDCT-2334 55


1001

0001011

1100010

1110

1001

0001011

1100010

1110

1001

0001011

1100010

1110

1001

0001011

1100010

1110

1001

0001011

1100010

1110

Legacy Security: Siloed, Inefficient & Expensive

Data

Packet

1001

0001011

1100010

1110

/

1001

0001011

1100010

1110

FW Platform

DDOS Platform SSL Platform

ADC Platform

WAF Platform

IPS Platform

SSLDDoS WAF

FW IPSADC

Reduced Effectiveness Increased Latency Slows Network Static & Manual

1001

0001011

1100010

1110

Trusted

Core

BRKDCT-2334 56


Security and Network Services Inside the DCNetworking with Excel

inside

outside

inside

Back end VLAN

outside

Front end VLAN

• Multiple types of State-full services

• Firewalls

• Application Delivery Controllers

• Inspection Prevention/Detection

Systems

• State-full implies one-way symmetrical

establishments

• State-full Devices HA and Scalability:

• Active-Standby mode for state-full

convergences & recovery

• Active Active mode for Redundancy

and Scalability

BRKDCT-2334 57


Security and Network Services Inside the DC

inside

outside

inside

Back end VLAN

outside

Front end VLAN

IGP to route

with Firewall eBGP to

avoid Firewall

VIP Pulls

Traffic

NAT on

Firewall

FW NAT 200.1.1.1-> 10.1.1.1.1

Application VIP 10.1.1.1

RHI to

inject VIP

Conditional

Routing to

announce VIP

Servers Default

Gateway?

Static Routes?

BRKDCT-2334 58


• Insert Firewall at Aggregation Point

• vPC connect ADCs and Firewalls

• Load balancer configured in One Armed, Routed

• Source NAT used to direct traffic back to LB

Services with a vPC based designAdding in ADC, Application Delivery Controllers and Firewalls

Layer 2 Trunks

Enterprise Network

Layer 2 Trunks

If Routing on Services nodes, use standard Etherchannel not vPCs

BRKDCT-2334 59


Securing the Data Center vPC based design

Enterprise Network

Layer 2 Trunks

VLAN 110-150

Layer 2 Trunks

VLANs 10-50

Web App DBClients

ACLs

BRKDCT-2334 60


Layer 3 Firewalls InsertionServers Default Gateway located on Firewall

Layer 2 Trunks

Enterprise Network

Layer 2 Trunks

• vPC connect firewalls

• Server Default Gateway on Firewall

• If Clustering or L2 Heartbeats

required you need to handle igmp( One Option )

N5k# configure terminal

N5k(config)# vlan 5

N5k(config-vlan)# no ip igmp snooping

• Look at Moving Layer 3 back to Switch

with VRFs to create isolation to allow

for more flexibility

*Does not show the Value of Cisco Fabrics at allBRKDCT-2334 61


EtherChannel on the ASA

• Supports 802.3ad and LACP/cLACP standards

• Direct support for vPC/VSS - CVD

• No issues with traffic normalization or asymmetry

• Up to 8 active and 8 standby links*

• 100Mb, 1Gb, 10Gb are all supported – must match

• Supported in all modes (transparent, routed, multi-context)

• Configurable hash algorithm (default is src/dest IP)

• SHOULD match the peer device for most deterministic flows

• Redundant interface feature and LAG on ASA are mutually exclusive

• Not supported on 4GE SSM (5540/50) or 5505

• ASA 9.2+ cluster allows 32 port active EtherChannel

BRKSEC-2020 Intermediate Firewall Deployment

*Non-clustered ASA allows 16 active and 16 standby links supported with cLACP

For YourReference

BRKDCT-2334 62


Why Deploy Transparent Mode?

• Very popular architecture in data center environments

• Existing Nexus/DC Network Fabric does not need to be modified to employ L2 Firewall!

• Simple as changing host(s) VLAN ID

• Firewall does not need to run routing protocols / become a segment gateway

• Firewalls are more suited to flow-based inspection (not packet forwarding like a router)

• Routing protocols can establish adjacencies through the firewall

• Protocols such as HSRP, VRRP, GLBP can cross the firewall

• Multicast streams can traverse the firewall

• Non-IP traffic can be allowed (IPX, MPLS, BPDUs)

• (CVD) most internal DC zoning scenarios recommend Transparent FW (L2) deployed versus Routed Firewall (L3)

• L3 Use-cases still valid, especially in Multi-tenant and Secure Enclave architectures

BRKDCT-2334 63


What Happens when you Exceed the Performance of a Single Pair?

• Clustering if the appliance supports this

• How do you scale without native clustering support?

• Smart-Channeling

Active Standby Performance Limitation

Eth 1/1

N7k1

Smart-

channel

Eth 4/4

Smart-

channel

For YourReference

BRKDCT-2334 64


• Traffic Distribution to Transparent Services/Appliances in HW at line rate

• Health Monitoring and Robust Failure Handling

• Traffic Persistence – ingress and egress Flow persistent on the same service

• Selective Traffic Distribution (include/exclude certain traffic)

Benefits of Smart-channel, Partial List For YourReference

BRKDCT-2334 65


Cisco ASA Firewall Clustering Basics

• Designed to solve two critical issues with firewall HA:

1. Aggregate firewall capacities for DC environments (BW, CPS, etc.)

2. Provide dynamic N+1 stateful redundancy with zero packet loss

• Supported in routed (L3) and transparent (L2) firewall modes, both single and multi-context - Mixed Mode supported as well

• (NG)IPS module is fully supported in clustered firewall deployment

• This adds NGIPS (FirePOWER) / NGFW / Device Context (FireSIGHT), etc. to ASA• Manages Asymmetric flows

• For ASA Clustering Deep-Dive watch recording of BRKSEC-3032 –Advanced - ASA Clustering Deep Dive

BRKDCT-2334 66


What is cLACP and What Does it Do?

• The challenge for clustering is that LACP is defined to run between two devices only according to IEEE specification and may only have 8 interfaces forwarding data

• Requirement to support LACP over multiple ASA units in a cluster and make clustered ASAs able to interoperate with standard LACP devices as one ASA

• Provide Etherchannel re-configuration with traffic black-hole avoidance and load balancing at both link and device level during link failure or device failure

• Provide cLACP API to cluster CP to notify Etherchannel link status change and provide health monitoring

• cLACP recovery/redundancy between ASA units in the case of Master unit leaves cluster

• Extend the maximum number of active forwarding interfaces to 16 (or potentially greater)

• 32-links Now Supported (16 active/16 standby in ASA 9.2)

BRKDCT-2334 67


CL MASTER CL SLAVE CL SLAVE CL SLAVE ASA x Node Cluster

ASA Port-Channel 32

ASA Port-Channel 40

CL MASTER CL SLAVE CL SLAVE CL SLAVE ASA x Node Cluster

Correct Use of EtherChannels When Clustering with VPCs

VPC PEER LINK

N7K VPC 32

Cluster Data Plane

Data Plane of Cluster MUST use cLACP (Spanned Port-Channel) VPC Identifier on N7K must be the same for channel consistency

– ASA uses the ‘span-cluster’ command on channel

Control Plane [Cluster Control Link] of Cluster MUST use standard LACP (Local Port-Channel)

Each VPC Identifier on Nexus 7K is unique

Port Channel Identifier on ASA is arbitrary

– (max number 48)

Cluster Control Plane

LACP – Local Port Channels

1 2 3 4

cLACP – Spanned Port Channel

N7K VPC 41N7K VPC 40 N7K VPC 43

N7K VPC 42

1 2 3 4

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/VMDC/ASA_Cluster/ASA_Cluster/ASA_Cluster.html

BRKDCT-2334 68

Scalable Data Center


DC Fabric w/FabricPath

• Externally the Fabric looks like a single switch

• Internally, ISIS adds Fabric-wide intelligence and ties the elements together.

• Provides in a plug-and-play fashion:• Optimal, low latency connectivity any to any

• High bandwidth, high resiliency

• Open management and troubleshooting

• ISIS for multipathing and reachability

FabricPath FabricPath

BRKDCT-2334 70


Layer 3 locations with Fabric Path

Layer 3 at Spine

• Overload Bit does not delay emulated switch id advertisement currently

• MAC scale is based off of the F2 or F3 modules being used

• Reduced points of configuration

Distributed Layer 3 at each Leaf

• Overload Bit provides fast failover no startup

• MAC scale at edge

• Management application to synchronize configurations for Layer 3

Layer 3 attached to Border Leaf

• Overload Bit provide fast failover on startup

• MAC scale can we scaled horizontally by adding in multiple GWs

• Common point of configuration for Layer 3

BRKDCT-2334 71


FabricPath: Design with Routing at Spine

FabricPath

Default-Gateway

Nx7k FP Spine (F3)

Anycast-HSRP

Nexus 5600

FP leaf

UCS-FI

F3 mac-scale (ARP)

BRKDCT-2334 72


Routing at FabricPath SpineAnycast HSRP L3

SVISVISVISVI

Anycast HSRP

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY MAC A→L1,L2,L3,L4

FabricPath

L3

L2/L3 boundary

All Anycast HSRP forwarders

share same VIP and VMAC

Hosts resolve shared

VIP to shared VMACRouted traffic spread

over spines based on

ECMP

Anycast HSRP

between agg switches

Layer 3 LinkLayer 2 CELayer 2 FabricPath


FabricPath: External / WAN Connectivity

- Default-Gateway

- Nx7k FP Spine (F3)

- MPLS PE Layer

• Spine/leaf architecture

• FabricPath for L2 multi-pathing

• MPLS Integration to WAN

• No spanning-tree

• Default gateway at spine layer

• ASA for firewall layer

• Nexus 5600 DC Access

ASR1K/9K

MPLS / LISP

MPLS, WAN

, Internet, Campus

Note:

- F3 simplifies the deploy with MPLS and FabricPath Support.

- Previously we leveraged F2 for FabricPath (VDC)

- M2 for MPLS Connectivity (VDC)

BRKDCT-2334 74


• DC Fabric with a FabricPath based data plane and MP-iBGP control plane.

• Use MP-iBGP on the leaf nodes to distribute internal host/subnet routes and external reachability

information.

• Introduced Segment ID to increase name space to 16M identifier in the fabric.

Standalone FabricHost and Subnet Route Distribution

MAN/WAN

N1KV/OVS

External Subnet

Route Injection

MP-iBGP AdjacenciesRR RR

Fabric Host/Subnet

Route Injection

MP-iBGP Control Plane

FabricPath DataPlane

Route-Reflectors deployed for scaling purposes

BRKDCT-2334 75


Optimized NetworkingDistributed Gateway Mode

• Distributed Gateway exists on all Leafs where VLAN/Segment-ID is active

• No HSRP

• There are different Forwarding Modes for the Distributed Gateway:

• Proxy-Gateway (Enhanced Forwarding)• Leverages local proxy-ARP • Intra and Inter-Subnet forwarding based on Routing• Contain floods and failure domains to the Leaf

• Anycast-Gateway (Traditional Forwarding)• Intra-Subnet forwarding based on Bridging• Data-plane based conversational learning for

endpoints MAC addresses• ARP is flooded across the fabric

vlan 123vn-segment 30000

!interface vlan 123

vrf member OrgA:PartAfabric forwarding mode proxy-gatewayip address 10.10.10.1/24no shutdownno ip redirects

vlan 145vn-segment 31000

!interface vlan 145

vrf member OrgA:PartAfabric forwarding mode anycast-gatewayip address 20.20.20.1/24no shutdown

BRKDCT-2334 76

Overlays


Technologies Intra-DC and Inter-DC

Requirement Intra-DC Inter-DC

Layer 2 connectivity FabricPath, VXLAN OTV, VPLS

IP Mobility LISP, FP, AnyCast Gateway LISP, OTV

Secure Segmentation VXLAN / Segment-ID LISP, MPLS-IP-VPNs

DC-eastDC-west

POD POD POD POD

IP Network

Fabric Path, VXLAN

(Intra-DC L2)

OTV, VPLS

(Inter-DC L2-x-L3)

App

OS

App

OS

App

OS

App

OS

EF, LISP, VXLAN

(Intra-DC x-L3)

LISP

IP mobility

(Inter-DC)

Fabric Path,VXLAN

(Intra-DC L2)

App

OS

App

OS

EF, LISP

(Intra-DC mobility)

BRKDCT-2334 78


What is a Virtual Overlay Technology ?

• Servers perform data encapsulation and forwarding

• SW based virtual switches instantiate customer topologies

V

M

1

V

M

2

V

M

3

Virtual

Switch

Hypervisor

V

M

4

V

M

5

V

M

6

Virtual

Switch

Hypervisor

IP Network

Ethernet Frames

IP/UDP Packets

IP Addr

2.2.2.2

IP Addr

1.1.1.1

BRKDCT-2334 79


Virtual Overlay Encapsulations and Forwarding

• Ethernet Frames are encapsulated into an IP frame format

• New control logic for learning and mapping VM identity (MAC address) to Host

identity (IP address)

• Two main Hypervisor based Overlays

• VXLAN Virtual Extensible Local Area Newtork

• NVGRE, Network Virtualization Generic Router Encapsulation

• GENEVE Generic Network Virtualization Encapsulation ( draft )

• Network Based Overlays

• OTV, Overlay Transport Virtualization

• VPLS, EVPN

• FabricPath

• VXLAN and NVGRE

BRKDCT-2334 80


MTU and VXLAN

• VXLAN adds 50 Bytes to the Original Ethernet Frame

• Avoid Fragmentation by adjusting the IP Networks MTU

• Data Centers often require Jumbo MTU; most Server NIC do support up to 9000 Bytes

• Using a MTU of 9216* Bytes accommodates VXLAN Overhead plus Server max. MTU

Underlay

Un

de

rla

y

Outer IP Header

Outer MAC Header

UDP Header

VXLAN Header

Original Layer-2 Frame Ove

rlay

50 (

54)

Byte

s o

f O

verh

ead

*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic

For YourReference

BRKDCT-2334 81


VXLAN TerminologyGateway, Bridging, Routing

VXLAN (VNI) to VLAN (VLAN) Bridging (L2 Gateway)

VXLANORANGE

Ingress VXLAN packet on

Orange segment

Egress interface chosen

(bridge may .1Q tag the packet)

VXLAN L2

Gateway

Egress is a tagged interface.

Packet is routed to the new VLAN

VXLAN (VNI) to VLAN Routing (SVI)(L3 Gateway)

VXLANORANGE

Ingress VXLAN packet on

Orange segmentVXLAN

Router

VLANBLUE

VLANORANGE

Tunnel Decap, then VLAN Route

BRKDCT-2334 82


VXLAN Underlay

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

Virtual

Hosts

Local LAN

Segment

Virtual

Switch

Edge

Device

Edge

Device

Edge

Device

IP Interface

BRKDCT-2334 83


VXLAN Underlay

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

Virtual

Hosts

Local LAN

Segment

Virtual

Switch

Edge

Device

Edge

Device

Edge

Device

VTEP

VTEP

VTEP

VV

V

Encapsulation

BRKDCT-2334 84


VXLAN Control/Data Plane Learning

Flood and Learn

• No Control plane. Data plane learning is only option

• Data Plane Learning similar to Ethernet. Packets are flooded out all ports and over a Multicast address to find destination device.

BGP Based Control Plane

• Control plane uses standards-based BGP

• Layer 2 MAC and Layer 3 IP info distribution by BGP

• Forwarding decision based on control plane to minimize flooding

• IETF Draft L2VPN-EVPN evolved to RFC 7432

BRKDCT-2334 85


EVPN – Ethernet VPNVXLAN Evolution

Control-

PlaneEVPN MP-BGP

draft-ietf-l2vpn-evpn

Data-

Plane

Multi-Protocol Label Switching

(MPLS)draft-ietf-l2vpn-evpn

Provider Backbone Bridges

(PBB)draft-ietf-l2vpn-pbb-evpn

Network Virtualization Overlay

(NVO)draft-sd-l2vpn-evpn-overlay

EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for Data Center Fabric encapsulations

Provides Layer-2 and Layer-3 Overlays over simple IP Networks

For YourReference

BRKDCT-2334 86


Additional Functions of VXLAN/EVPN

Early ARP

Termination

Distributed Anycast Gateway

Suppresses flooding for Unknown Unicast ARP

Authenticate VTEPs through BGP peer authentication

Seamless and Optimal vm-mobility

Active/Active MultipathingActive/Active and Resilient Multipathing using vPC on

Nexus

Ingress Replication Unicast Alternative to Multicast underlay

Security & Authentication

BRKDCT-2334 87


Host and Subnet Route Distribution

• Host Route Distribution decoupled from the Underlay protocol

• Use MultiProtocol-BGP (MP-BGP) on the Leaf nodes to distribute internal Host/Subnet Routes and external reachability information

• Route-Reflectors deployed for scaling purposes

VXLAN/EVPN

BGP Route-ReflectorRR

iBGP Adjacency

Edge

Device

Edge

Device

VTEP

VTEP

VTEP

VV

V

Encapsulation

RR RR

BRKDCT-2334 88


Protocol Learning & Distribution VXLAN/EVPN

Host A

MAC_A / IP_AHost B

MAC_B / IP_B

Virtual Switch

Host C

MAC_C / IP_C

Host Y

MAC_Y / IP_Y

1VTEPs advertise Host Routes (IP+MAC)for the Host within the Control-Plane

Edge

Device

Edge

Device

VTEP

VTEP

VTEP

VV

V

Encapsulation

RR RR

11

1

BRKDCT-2334 89


Protocol Learning & DistributionVXLAN/EVPN

Host A

MAC_A / IP_AHost B

MAC_B / IP_B

Virtual Switch

Host C

MAC_C / IP_C

Host Y

MAC_Y / IP_Y

Edge

Device

Edge

Device

VTEP

VTEP

VTEP

VV

V

Encapsulation

RR RR

2

2BGP propagates routes forThe Host to all other VTEPs

MAC, IP VNI NH

MAC_A, IP_A 30000 IP_L1

MAC_B, IP_B 30000 IP_L2

MAC, IP VNI NH

MAC_A, IP_A 30000 IP_L1

MAC_C, IP_C 30000 IP_L3

MAC_Y, IP_Y 30001 IP_L3

3VTEPs obtain host routes for

remote hosts and install in RIB/FIB

3 3

3

MAC, IP VNI NH

MAC_B, IP_B 30000 IP_L2

MAC_C, IP_C 30000 IP_L3

MAC_Y, IP_Y 30001 IP_L32

2

BRKDCT-2334 90


Host MovesVXLAN/EVPN

BGP Route-ReflectorRR

iBGP Adjacency

Edge

Device

Edge

Device

VTEP

VTEP

VTEP

VV

V

RR RR

• Host Moves to Leaf 3

• Leaf 3 detects Host A and advertises it with Seq #1

• Leaf 1 sees more recent route and withdraws its advertisement

L1# sh bgp l2vpn evpn 192.168.101.101

BGP routing table information for VRF default, address family

L2VPN EVPN

Route Distinguisher: 10.254.254.102:32868 (L2VNI 30001)

BGP routing table entry for

[2]:[0]:[0]:[48]:[0050.56ac.0773]:[32]:[192.168.101.101]/272,

version 30

Paths: (1 available, best #1)

Flags: (0x00030a) on xmit-list, is not in l2rib/evpn

Advertised path-id 1

Path type: local, path is valid, is best path, no labeled

nexthop

AS-Path: NONE, path locally originated

10.254.254.102 (metric 0) from 0.0.0.0 (10.254.254.102)

Origin IGP, MED not set, localpref 100, weight 32768

Received label 30001 50000

Extcommunity: RT:65501:30001 RT:65501:50000 MAC Mobility

Sequence:00:1

Path-id 1 advertised to peers:

10.254.254.101

MAC, IP VNI

(L2)

VNI

(L3)

NH Encap Seq

0050.56ac.0773, 192.168.101.101 30001 50000 0.0.0.0 8:VXLAN 1

BRKDCT-2334 91


Symmetric IRB

• Symmetric

• Similar to creating a Transit Segment

• Regardless of where Source or Destination VNI exists

• Post Routing traffic uses different VNI than Bridged traffic

• Additional VNI for Routing traffic (per VRF)• From Host A via VLAN “blue” routed at L1 to

VNI “purple” reaching destination VLAN “red”

• From Host Y via VLAN “red” routed at L3 to VNI “purple” reaching destination VLAN “blue”

• Used in Cisco VXLAN/EVPN

VXLAN Routing

Host Y

VNI 30001

Host A

VNI 30000

L3

L2

L1

BRKDCT-2334 92


Multicast Enabled Underlay

• PIM-ASM or PIM-BiDir (Different hardware has different capabilities)

• Spine and Aggregation Switches make good Rendezvous-Point (RP); much lile RR

• PIM-ASM (sparse-mode)

• Source-trees, build a couple of unidirectional trees from RP; (s,g)

• Every VTEP is Source and Destination

• PIM-Anycast RP vs MSDP for example

• PIM-BiDir

• No Sources tree use a bi-directional shared tree

• No (S,G), we have (*,G)

• Phanton RP (Leverages Unicast for convergence)

• Each VNI does not need the same a different Multicast Group

Underlay

Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000ASR 1000

CSR 1000ASR 9000

Multicast

Mode

IGMP L2/L3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir

BRKDCT-2334 93

For YourReference


Multiple Loopback AddressesUse Unique Loopback based on Purpose

SPINE(non-VTEP)

Anycast-RP 1:

feature pim

feature eigrp

interface loopback0


ip router eigrp 10

ip pim sparse-mode

interface loopback1

ip address 10.10.10.50/32

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

LEAF (VTEP)

Anycast-RP 1:

feature pim

feature ospf

interface loopback0

description for VTEP


ip address 1.1.1.1/32 secondary


ip pim sparse-mode

interface loopback1

description for routing protocol

ip address 10.10.10.201/32


ip pim sparse-mode


VXLAN Configuration – Mapping VLANs to VNIsLayer 2 Gateway on Multicast Enabled Fabric

Feature interface-vlanfeature vn-segment-vlanfeature nv overlayfeature pim

Vlan 102vn-segment 10102

interface nve1no shutdownsource-interface loopback0member vni 10102 mcast-group 239.1.1.102

interface <phy if>switchport mode accessswitch port access vlan 102

Used for the VTEP

VXLAN Identifier

IP Multicast Group for Multi-

Destination Traffic

Locally Significant VLAN

Tunnel Interface

BRKDCT-2334 95


ip pim rp-address 10.50.255.254 group-list 224.0.0.0/4ip pim ssm range 232.0.0.0/8

interface Ethernet1/49-50no switchportmtu 9200ip ospf network point-to-pointip router ospf 10 area 0.0.0.0ip pim sparse-mode

interface Ethernet1/49ip address 1.1.1.14/30

interface Ethernet1/50ip address 1.1.1.18/30

interface loopback 0ip address 1.1.1.2/32ip address 1.1.1.1/32 secondaryip router ospf 1 area 0.0.0.0ip pim sparse-mode

Enabling Multicast on TOR for VXLAN

Multicast RP Definition

Multicast RP DefinitionLinks to SpinesLinks to Spines

Used for the VTEP

BRKDCT-2334 96

vrf context TENANTvni 23002rd autoaddress-family ipv4 unicastroute-target both autoroute-target both auto evpn

router bgp 65123router-id 10.10.10.201address-family ipv4 unicastaddress-family l2vpn evpn

neighbor 10.50.255.3 remote-as 65123update-source loopback1send-community both

vrf TENANTaddress-family ipv4 unicastadvertise l2vpn evpnredistribute direct route-map FABRIC

interface loopback 1ip address 10.10.10.201/32ip router ospf 1 area 0.0.0.0ip pim sparse-mode

VXLAN EVPN

Pointing to BGP

Route Reflectors

Used for the Router IDs

VRF Definition


WAN

TIER

VXLAN Fabric Design PlaneEVPN Control Plane

D

S1 S2

BL1 BL2 L44

L3 Fabric

B

OSPF AREA 0

VPC Peer

LInk

Routed

Links

Appliances

PC L2 Trunks

Multicast

Anycast RPs

Multicast

Anycast RPs

C

L42

A

*Caveat: Behavior with vPCs and routing BRKDCT-2334 98


WAN

TIER

VXLAN Fabric Design Plane

• vPC uses secondary address as virtual VTEP

• By default all connections to either vPC attached TOR uses virtual VTEP

• Routing Failover faster if not vPCconnected switch.

Routed

Links

VPC Peer

LInk

router bgp 65123

address-family ipv4 unicast

address-family l2vpn evpn

advertise-pip

BRKDCT-2334 99*Caveat: Behavior with vPCs and routing


VXLAN Networks Layer 2 and or Layer 3

Web

App DB

Clients

ACLs

Layer 3

ACLs

Layer 2

ACLs

VM Mobility ManagementStorage

VLAN 10

VNI 10

VLAN 12

VNI 12

L3 VNI 3001VLAN 11

VNI 11

BRKDCT-2334 100


VXLAN Networks Layer 2 and or Layer 3

Web

App DB

Clients

ACLs

Layer 3

ACLs

Layer 2

ACLs

Description VLAN Gateway VNId Multicast Group L3 VNI TenantWAN to FW 100 FW 100100 239.1.1.100 - none

FW to ADC 101 FW 100101 239.1.1.100 - none

ADC to WEB 102 FW 100102 239.1.1.100 - none

FW to APP/DB 103 Fabric 100103 239.1.1.100 - none

APP 104 Fabric 100104 239.1.1.100 300100 CISCOLIVE

DB 105 Fabric 100105 239.1.1.100 300100 CISCOLIVE

VM Mobility 10 No 10 239.1.1.1 - none

Storage 11 No 11 239.1.1.1 - none

Management 12 Fabric 12 239.1.1.1 300000 CommonServices

BRKDCT-2334 101


ESX 6.0 Multiple TCP/IP StacksStarted in ESX 5.5

DEFAULT

vmk0 1.1.0.0/24

vmk1 1.1.1.0/24

DG 1.1.0.254

DNS 4.2.2.1

vMotion

vmk2 1.1.2.0/24

DG 1.1.2.254

DNS 8.8.8.8

Provisioning

vmk3 1.1.3.0/24

DG 1.1.3.254

DNS 8.8.8.8

BRKDCT-2334 102


VXLAN and Layer 2 Loop avoidance

• VXLAN doesn’t implement a native L2 loop detection and protection

• BPDU’s are not forwarded across the VXLAN domain

• A backdoor link can be established between two or more TORs

Leaf 1 Leaf 2Leaf 3 Leaf 5

Spine 1 Spine 2

Leaf 6Leaf 4

Srv 1VLAN 100

VM-1VM-4

Srv 5VLAN 100

Srv 2VLAN 100

Srv 3VLAN 100

Enable BPDUGuard

BRKDCT-2334 103


VXLAN NVE Show Commands

Nexus# show nve peer

Interface Peer-IP State LearnType Uptime Router-Mac

--------- --------------- ----- --------- -------- -----------------

nve1 10.50.255.251 Up CP 01:18:41 64f6.9dee.b6f5

Nexus# show nve vni

Codes: CP - Control Plane DP - Data Plane

UC - Unconfigured SA - Suppress ARP

Interface VNI Multicast-group State Mode Type [BD/VRF] Flags

--------- -------- ----------------- ----- ---- ------------------ -----

nve1 100100 239.1.1.100 Up CP L2 [100] SA

nve1 300100 n/a Up CP L3 [CISCOLIVE]

nve1 100101 239.1.1.100 Up CP L2 [101] SA

nve1 100102 239.1.1.100 Up CP L2 [102]

nve1 100103 239.1.1.100 Up CP L2 [103]

nve1 300000 n/a Up CP L3 [CommonService]

nve1 10 239.1.1.1 Up CP L2 [10]

nve1 11 239.1.1.1 Up CP L2 [11]

BRKDCT-2334 104


VXLAN NVE Show CommandsNexus# show mac address-table dynamic

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

* 1 b838.6146.a9fd dynamic 0 F F Po23

+ 1 b838.6146.a9fe dynamic 0 F F Po23

* 100 0010.9400.0034 dynamic 0 F F Eth1/45

* 100 0010.9400.0035 dynamic 0 F F nve1(10.50.255.251)

+ 100 0010.9400.0002 dynamic 0 F F vPC Peer-Link

show bgp l2vpn evpn ( Filtered )

BGP routing table information for VRF default, address family L2VPN EVPN

…

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 10.50.255.6:32867 (L2VNI 10100)

* i[2]:[0]:[0]:[48]:[0010.9400.0035]:[32]:[172.16.1.110]/272

10.50.255.251 100 0 i

*>i 10.50.255.251 100 0 i

BRKDCT-2334 105


VNI Scalability Per Platform

• Reference the VXLAN Verified Scalability Limits (Unidimensional) at a high level

• Focus on the Validated Deployment Case studies

• Can you support 750, 900, 1000, 1500, or 1600 VNIs?

• How Many TORs can communicate? Can I use Ingress replication or does my

design require Multicast?

• Routes

• Underlay Routes

• Overlay Routes

• Host Routes

• MAC addresses

5600, 7000, 9300, and 9500 have different scalability numbers

BRKDCT-2334 106

OTV


L2 Network Overlays for Data Center Interconnect

• OTV/VPLS resilient geo-extension of segments

• Preserve failure isolation between locations

• Network resiliency and multi-pathing

• Built in loop handling

• Optimal traffic handling

• Streamlined operations

• Egress routing optimization

• HW Accelerated high performance connectivity

OTV/VPLS

North

Data

CenterFault

Domain

Fault

Domain

Fault

Domain

Fault

Domain

LAN Extension

BRKDCT-2334 108


Server-ServerEgress Routing Localization:

Server-Client

Egress Routing Localization:

Server-Client

Hypervisor Hypervisor

Ingress Routing

Localization:

Clients-Server

Path Optimization

Layer 2 extensions represent a challenge for optimal routing

Challenging placement of gateway and advertisement of routing prefix/subnet

Optimal Routing Challenges

BRKDCT-2334 109


Path OptimizationEgress Routing with LAN Extension

HSRP

Active

HSRP

Standb

y

HSRP

Listen

HSRP

Listen

HSRP Hellos

VLAN

20

VLAN

10

• Extended VLANs typically have associated HSRP groups

• By default, only one HSRP router elected active, with all servers pointing to HSRP VIP as default gateway

• Result: sub-optimal (trombone) routing

BRKDCT-2334 110


HSRP

Active

HSRP

Standb

y

HSRP

Listen

HSRP

Listen

ARP

reply

ARP for

HSRP VIP

VLAN

20

VLAN

10

Path Optimization




Egress Routing with LAN Extension

111


Path Optimization




Egress Routing with LAN Extension

HSRP

Active

HSRP

Standb

y

HSRP

Listen

HSRP

Listen

VLAN

20

VLAN

10

Packet from

Vlan 10 to Vlan 20

DMAC = DGW

Routing

Packet from

Vlan 10 to Vlan 20

DMAC = Host Vlan 20

112


Egress Routing Localization

• Filter FHRP with combination of VACL and MAC route filter

• Result: Still have one HSRP group with one VIP, but now have active router at each site for optimal first-hop routing

FHRP Filtering Solution

HSRP

ActiveHSRP

Standb

y

ARP for

HSRP VIP

ARP

reply

HSRP FilteringHSRP

ActiveHSRP

Standb

y

HSRP Hellos HSRP Hellos

VLAN

20

VLAN

10

no ip arp gratuitous hsrp duplicate

113


Sample Cluster - Primary Service in Left DCFHRP Localization – Egress Path Optimization

HA cluster Node B

Layer 3 CoreISP A ISP B

HA cluster Node A

Access

Agg

Cluster VIP = 10.1.1.100 Preempt

Default GW = 10.1.1.1

Node A

Data Center

AData Center

B

VLAN A

Public Network

Asymmetrical flows No Stateful device

Low ingress traffic

HSRP

Active

HSRP

StandbyHSRP

Active

HSRP

StandbyHSRP Filtering

BRKDCT-2334 114

Summary


Recommended Reading

BRKDCT-2334 116


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKDCT-2334 117

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKDCT-2334 118

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you

Documents

Real World Data Centerd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKDCT-2334.pdf · Real World Data Center ... however, no ability to dynamically insert services. ... • Via self-managed