Datacenter Virtualisation - Cisco · In this example ESX1 and ESX2 are under VSM1 and share the green and red Port-Profile ESX3 and ESX4 are under VSM2 and share the Blue and Yellow

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Datacenter Virtualisation

Maurizio PortolaniDatacenter Solution Architect

© 2009 Cisco Systems, Inc. All rights reserved. Cisco PublicPresentation_ID 2

Agenda

Virtual Switches and the Nexus1000v

Why using the Nexus1kv?

VEM Forwarding: NIC Teaming and Etherchannels

Scalability Considerations

Switching Infrastructure Requirements

Designs with Blade Servers


MAC1

VM1

Ethernet1/1

MAC2

VM2

Destination MAC Port

MAC1 1/1

MAC2 1/1

Forwarding Table

?

DMAC = MAC2DMAC = MAC2

Why is a Virtual Switch needed in the first place?


Destination MAC Port

MAC1 1/1

MAC2 1/1

Forwarding Table

VM1

Ethernet1/1

MAC2

VM2

vSwitch or Nexus 1000v

Virtual SwitchingVirtualized Servers Need “VN-link” Technology

MAC1


ESX Server Components

VMWare ESX Server

vSwitch

vmnics

vnics

Virtual Machine

Software virtual switch

VMware ESX is a “bare‐metal” hypervisor that partitions physical servers in multiple virtual machines


Nexus 1000vDistributed Virtual Switch

Fabric Function

Linecards Equivalent

vCenter

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

N1kN1k--VSM# sh moduleVSM# sh module

Mod Ports ModuleMod Ports Module--Type Model StatusType Model Status1 1 Supervisor Module Cisco Nexus 1000V active *1 1 Supervisor Module Cisco Nexus 1000V active *2 1 Supervisor Module Cisco Nexus 1000V standby2 1 Supervisor Module Cisco Nexus 1000V standby3 48 Virtual Ethernet Module ok3 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok


Nexus 1000VSystem Module

VMWare ESX Server

VEM - Module 3

VMWare ESX Server

VEM – Module 4

N1kN1k--VSM# sh moduleVSM# sh module

Mod Ports ModuleMod Ports Module--Type Model StatusType Model Status1 1 Supervisor Module Cisco Nexus 1000V active *1 1 Supervisor Module Cisco Nexus 1000V active *2 1 Supervisor Module Cisco Nexus 1000V standby2 1 Supervisor Module Cisco Nexus 1000V standby3 48 Virtual Ethernet Module ok3 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok


Nexus 1000VVirtual Interface

VMWare ESX Server

veth = Virtual Machine port (vnic)veth3 veth7 veth68

N1kN1k--VSM# sh interface virtual VSM# sh interface virtual Port Adapter Owner Mod HostPort Adapter Owner Mod Host

Veth3 Net Adapter 1 Ubuntu VM 1 peVeth3 Net Adapter 1 Ubuntu VM 1 pe--esx1esx1Veth7 Net Adapter 1 Ubuntu VM 2 peVeth7 Net Adapter 1 Ubuntu VM 2 pe--esx1esx1Veth68 Net Adapter 1 Ubuntu VM 3 peVeth68 Net Adapter 1 Ubuntu VM 3 pe--esx1esx1

VEM - Module 3


Nexus 1000v Interface

VMWare ESX Server

VEM - Module 3

VMWare ESX Server

VEM – Module 4

WSWS--C6504EC6504E--VSS#sh cdp neighborsVSS#sh cdp neighborsDevice ID Local Intrfce Platform Port IDDevice ID Local Intrfce Platform Port ID

N1kN1k--VSM Gig 1/1/1 Nexus1000 Eth 3/1VSM Gig 1/1/1 Nexus1000 Eth 3/1N1kN1k--VSM Gig 2/1/2 Nexus1000 Eth 3/2VSM Gig 2/1/2 Nexus1000 Eth 3/2N1kN1k--VSM Gig 1/8/1 Nexus1000 Eth 4/1VSM Gig 1/8/1 Nexus1000 Eth 4/1N1kN1k--VSM Gig 2/8/2 Nexus1000 Eth 4/2VSM Gig 2/8/2 Nexus1000 Eth 4/2

eth3/1

eth3/2

eth4/1

eth4/2

eth = uplink port on the ESX Server


Definition of Port-profile

switchportswitchport access vlan 10switchport mode access

switchportswitchport access vlan 11switchport mode access


VM #4VM VM #4#4

VMW ESXVMW ESXVMW ESX

ServerServer

What is a Policy or Port Profile?A Collection of Networking Configurations

Nexus 1000vNexus 1000v

Nexus 1000 DVSNexus 1000 DVSNexus 1000 DVS

VM #1VM VM #1#1

VM #4VM VM #4#4

VM #3VM VM #3#3

VM #2VM VM #2#2

VMVM

vCenter


Network Administrator viewN1kN1k--VSM# sh portVSM# sh port--profile name Ubuntuprofile name Ubuntu--VMVM

portport--profile Ubuntuprofile Ubuntu--VMVM

description:description:

status: enabledstatus: enabled

capability uplink: nocapability uplink: no

capability l3control: nocapability l3control: no

system vlans: nonesystem vlans: none

portport--group: Ubuntugroup: Ubuntu--VMVM

maxmax--ports: 32ports: 32

inherit:inherit:

config attributes:config attributes:

switchport mode accessswitchport mode access

switchport access vlan 95switchport access vlan 95

no shutdownno shutdown

assigned interfaces:assigned interfaces:

Vethernet2Vethernet2

Vethernet4Vethernet4

Port-Profile as viewed from the Network and Server Administrator

Server Administrator view


What makes the Virtual Switch “Distributed”?

ESX servers that are under the same Nexus 1kv VSM share the same Port-Profile ConfigurationWhen a new Port-Profile is defined it gets automatically propagated to all the ESX servers (VEMs) that are the VSMIn this example ESX1 and ESX2 are under VSM1 and share the green and red Port-ProfileESX3 and ESX4 are under VSM2 and share the Blue and Yellow Port Profile

3 41 2

VSM1VSM1 VSM2VSM2

Port ProfilesPort Profiles Port ProfilesPort Profiles


Prior to DVS Ensuring Port-Group Consistency was a Manual Process

Each ESX host is configured individually for Networking


VMotion Requires the Destination vSwitch to have the same Port-Groups/Port-Profiles as the originating ESX host

Prior to DVS you had to manually ensure that the same Port-Group existed on ESX Host 1 as ESX Host 2

VM4

vmnic0

VM5

ESX Host 2

VM6VM1 VM2

ESX Host 1

VM3

vSwitch

Rack10Rack1

vmnic1

vSwitch

vmnic0 vmnic1


“Distributed” Virtual Switching facilitates VMotion Migration

VMW ESX

Server 2

VMW ESX

Server 1

VEM

VM #4

VM #3

VM #2

VM #1

VM #4

VM #3

VM #2

VM #1

VMs Need to MoveVMotionDRSSW Upgrade/PatchHardware Failure

VEM

Port Profiles


Agenda








vNetwork Distributed SwitchBoth Cisco and VMWARE provide DVS functionalities

http://www.vmware.com/products/vnetwork-distributed-switch/


Cisco Nexus1kv Provide Separation of Network and Server Roles

Server AdministratorServer Administrator Network AdministratorNetwork Administrator


Configuring Access-Lists, Port Security, SPAN, etc… without Nexus1kv is Complicated

Is VM#1 on Server 1? Or on which server, on which switch do I put the ACL?

ACL need to be specify the IP address of the VM else you risk to drop both VM1 and VM3 traffic

SPAN will get all traffic from VM1, VM2, VM3, VM4!! You need to filter that!!

Port Security CAN’T be used

VMW ESX

Server 1

VM #4

VM #3

VM #2

VM #1

ACLs (complicated)

SPAN (realistically can’t be used)

Port Security needs to be disabled

vSwitch


You can use Access-Lists, Port Security, SPAN, etc… WITH Nexus1kv

Is VM#1 on Server 1? It doesn’t matter ACL “follows” the VM

SPAN will get only the traffic from the virtual Ethernet Port

Port Security ensures that VMs won’t generate fake make addresses

VMW ESX

Server 1

VEM

VM #4

VM #3

VM #2

VM #1

ACLs specific to a Port-Group

SPAN on a virtual ethernet port

Port Security


Nexus 1000 DVSNexus 1000 DVSNexus 1000 DVS

vNIC Security

VMs can be secured in multiple ways:

VLANs

ACLs

Private VLANs

Port-Security

VM #4

VM #3

Server

VM #2

VM #1

vnics

vmnic

IEEE 802.1q trunk


Hypervisor Hypervisor

PromiscuousPort

PromiscuousPort

Community‘A’

Community‘B’

IsolatedPorts

Primary VLAN

Community VLAN

Community VLAN

Isolated VLAN

Only One Subnet

xx

Private VLANs can be extended across ESX servers by using the Nexus1kv

Promiscuous ports receive and transmit to all hosts

Communities allow communications between groups

Isolated ports talk to promiscuous ports only

xx

xx

.11 .12 .13 .14 .15 .16 .17 .18OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App


With Nexus1kv Troubleshooting is easier

VMWare ESX Server

veth = Virtual Machine NICveth3 veth7 veth68

N1kN1k--VSM# sh interface virtual VSM# sh interface virtual Port Adapter Owner Mod HostPort Adapter Owner Mod Host

Veth3 Net Adapter 1 Ubuntu VM 1 peVeth3 Net Adapter 1 Ubuntu VM 1 pe--esx1esx1Veth7 Net Adapter 1 Ubuntu VM 2 peVeth7 Net Adapter 1 Ubuntu VM 2 pe--esx1esx1Veth68 Net Adapter 1 Ubuntu VM 3 peVeth68 Net Adapter 1 Ubuntu VM 3 pe--esx1esx1

VEM - Module 3


Tracing Virtual Ethernet Portsshow interface VEthernet

Vethernet2 is upHardware is Virtual, address is 0050.5675.26c5Owner is VMware VM1, adapter is vethernet1Active on module 8, host tc-esx05.cisco.comVMware DVS port 16777215Port-Profile is MyApplicationPort mode is accessRx444385 Input Packets 444384 Unicast Packets0 Multicast Packets 1 Broadcast Packets572675241 BytesTx687655 Output Packets 687654 Unicast Packets0 Multicast Packets 1 Broadcast Packets 1 Flood Packets592295257 Bytes0 Input Packet Drops 0 Output Packet Drops


SPAN traffic to a Catalyst 6500 or a Nexus 7k where you have a sniffer attached

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Capture here


Ease of ProvisioningPlug-and-play designs with VBS

1 Add or replace a VBS Switch to the Cluster

2 Switch config and code automatically propagated

3 Add a blade Server

4 It’s always booted from the same LUN


Ease of ProvisioningMaking Blade Servers Deployment Faster

1 Physically Add a new blade (or replace an old one)

2 Go to vCenter, add host to cluster

3 Done:

the new blade is in production

All port-groups appear

Nexus 1000vNexus 1000v


Agenda








ESX Server NIC Teaming

VMWare ESX Server

vSwitch – Module 3

VMWare ESX Server

vSwitch – Module 4

NIC team load balancing algorithms based on either/or, not AND


ESX Server NIC Teaming

Source MAC

MAC address is pin to a particular interface

Virtual Port ID

vSwitch use the virtualNIC to select the

outgoing interface

Load Balancing is a matter of choosing between HA or load sharing

ESX Server

vSwitch

App

OS

App

OS

ESX Server

vSwitch

App

OS

App

OS

IP Hashing

IP address load balance across different NIC

Explicit

Manually configure a path through a specific

physical NIC


VEM Forwarding Behavior

VMWare ESX Server

VEM - Module 3

No Spanning tree

BPDU are dropped

MAC A MAC B MAC C

VEM MAC Table

DMAC : C

MAC A

MAC B

MAC C

Local MAC Adress are switched locally

Everything else send to upstream switch

BPDU BPDU

DMAC : X


Do I need multiple VEMs? No

Virtual Machines

VLANs“A”

VLANs“B”

ESX Server Host

VMNIC1VMNIC0 VMNIC2 VMNIC3

1 2

Port Profile AVEM - Module 3

30 31 32

Port Profile B


Nexus 1000VNIC Teaming and Load-Balancing

VMWare ESX Server

VSM

The Nexus 1000V load balance based on 16 different parameters

N1kN1k--VSM(config)# portVSM(config)# port--channel loadchannel load--balance ethernet ?balance ethernet ?destdest--ipip--port Destination IP address and L4 portport Destination IP address and L4 portdestdest--ipip--portport--vlan Destination IP address, L4 port and VLANvlan Destination IP address, L4 port and VLANdestinationdestination--ipip--vlan Destination IP address and VLANvlan Destination IP address and VLANdestinationdestination--mac Destination MAC addressmac Destination MAC addressdestinationdestination--port Destination L4 portport Destination L4 portsourcesource--destdest--ipip--port Source & Destination IP address and L4 portport Source & Destination IP address and L4 portsourcesource--destdest--ipip--portport--vlan Source & Destination IP address, L4 port and VLANvlan Source & Destination IP address, L4 port and VLANsourcesource--destdest--ipip--vlan Source & Destination IP address and VLANvlan Source & Destination IP address and VLANsourcesource--destdest--mac Source & Destination MAC addressmac Source & Destination MAC addresssourcesource--destdest--port Source & Destination L4 portport Source & Destination L4 portsourcesource--ipip--port Source IP address and L4 portport Source IP address and L4 portsourcesource--ipip--portport--vlan Source IP address, L4 port and VLANvlan Source IP address, L4 port and VLANsourcesource--ipip--vlan Source IP address and VLANvlan Source IP address and VLANsourcesource--mac Source MAC addressmac Source MAC addresssourcesource--port Source L4 portport Source L4 portvlanvlan--only VLAN onlyonly VLAN only

Virtual Supervisor Module (VSM)


Loop Avoidance without Spanning-Tree

Border interface

Server interface


Nexus 1000VVPC Host Mode

VMWare ESX Server

VEM

The Nexus 1000V detect the upstream switch and create automatically using CDP a port‐channel bundling all the links to the same switch

Virtual Supervisor Module (VSM)

N1kN1k--VSM#sh cdp neighborsVSM#sh cdp neighborsDevice ID Local Intrfce Platform Port IDDevice ID Local Intrfce Platform Port ID

N1kN1k--VSM Eth 3/1 WSVSM Eth 3/1 WS--49004900--1 Gig 1/1 1 Gig 1/1 N1kN1k--VSM Eth 3/2 WSVSM Eth 3/2 WS--49004900--1 Gig 1/21 Gig 1/2

N1kN1k--VSM Eth 3/3 WSVSM Eth 3/3 WS--49004900--2 Gig 1/1 2 Gig 1/1 N1kN1k--VSM Eth 3/4 WSVSM Eth 3/4 WS--49004900--2 Gig 1/22 Gig 1/2


Nexus 1000v with VSS or vPC

VMWare ESX Server

VEM - Module 3

VMWare ESX Server

VEM - Module 4

Nexus 1000v leverage cdp to create automagically an etherchannel as soon as the same upstream switch is seen on the VEM uplink.


Agenda








Manageability and Scalability Details

RBAC

Wireshark

ERSPAN

LLDP, CDP

EEM

Rollback

Cisco Nexus 1000V Virtual Supervisor Module: Virtual appliance in VMDK or ISO image, supports up to 64VMware ESX or ESXi

Cisco Nexus 1000V Virtual Ethernet Module: maximum 256 ports


Cisco Nexus 1000V Scalability

A single Nexus 1000V• 66 modules (2x Supervisors and 64x Ethernet Modules)

Virtual Ethernet Module: • 32 physical NICs

• 256 virtual NICs

Limit Per Nexus 1000V• 512 Port Profiles

• 2048 physical ports

• 8,192 virtual ports (vmknic, vswif, vnic)

Virtual Supervisor Virtual Supervisor -- StandbyStandby

VEMVEM

VEMVEM

VEMVEM

VEMVEM

VEMVEM

VEM VEM

VEMVEM

VEMVEM

VEMVEM

VEMVEM

Virtual Supervisor Virtual Supervisor -- ActiveActive

Nexus 1000V


Agenda









Nexus7k02

Nexus7k01VDC2

core

VDC2

Possible Server Design

tc-mds01

Serv

ice

Mod

ules

Nexus 5k02

Nexus 5k01

VSAN A

VSAN B

Low latency 10 GigETeaming with 10 GigEFcoE possible

Nexus 2k

Nexus 2k


Virtual Machine Considerations

Hardware MAC learning

Large HW-based MAC address Tables (128k entries on the Nexus 7k)

Control plane policing

Broadcast suppression

Layer 2 trace

Broadcast and Storm Control

Private VLAN integration

Unified I/O ready

Virtual Servers


VDCs

http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_1/nx-os/virtual_device_context/configuration/guide/vdc_overview.html#wp1073104

Using VDC it is possible to move servers seamlessly from one environment to a different one without having to recable the Network Infrastructure

1 3 52 4 6

11 13 1512 14 1610

978

21 23 2522 24 26

191718 20

292728 30

3132


10 Gigabit Server Connectivity

10 Gigabit EthernetFCoE

DCE

VNTAG / Nexus 1000v

Class-Based Bandwidth Allocation


Redundant Datacenters TopologyvPC, VDC, Link Layer Encryption

VDCs create multiple environment with the same pair of 7ksCTS encrypts traffic between the DatacentersvPC ensures that all links are forwarding

DC1 DC2VDC2,3,4 VDC2,3,4

Eth7/3

VDC2,3,4 VDC2,3,4

FC ports FC ports

VSAN A VSAN B VSAN A VSAN B

TRANSIT VSAN


Agenda








With Nexus1kv the Switch just a plug-and-play “Fabric”

With the Nexus1kv the Profiles are defined on the Nexus1kv

The Mapping is performed on the Virtual Center

The Switch provides simply the Switching Fabric and trunks all necessary VLANs.

Nexus1kv

Mapping of “servers” to VLANs/Port Profiles

vCenter

Profile Definition Nexus1kv CLI


Switching Fabric with Virtualized Servers

Cisco VBS

Network Management Model

Equivalent to a 3750 stackable: plug-and-play

Stacking Capability Up to 8 Blade Switches, i.e. single config point

Etherchanneling Across switches in the stack

Server Identity Flexattach

You have Virtualized Servers on the Blades

You are better off using clustered Cisco VBS


Nexus 1000v with Blade Enclosures

Fabric Function

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Hypervisor

OS

App

OS

App

OS

App

OS

App

Port-Profile Definition

10 Gigabit Uplinks


Q and A


Documents

Datacenter Virtualisation - Cisco · In this example ESX1 and ESX2 are under VSM1 and share the green and red Port-Profile ESX3 and ESX4 are under VSM2 and share the Blue and Yellow