Hardware accelerated switching with Linux @ SWLUG Talks May 2014

v

Hardware accelerated switching with Linux

Nat Morris

26th April 2014 @ South Wales Linux User Group

cumulusnetworks.com 2

About me

Nat Morris• Based in Haverfordwest (beyond the M4)• Team lead, Cumulus Networks• Director & Board Member, UK Network Operators

Forum (UKNOF)• Feeder of dogs• Attended first SWLUG meeting in 2001

Twitter• @natmorris


About Cumulus Networks

Team JR Rivers, co-founder and CEO Nolan Leake, co-founder and CTO Shrijeet Mukherjee, VP Engineering Reza Malekzadeh, VP Business Jason Martin, VP Customer

Experience

Investors Andreessen Horowitz Battery Ventures Sequoia Capital Wing. VC (Peter Wagner) Ed Bugnion, Diane Greene and

Mendel Rosenblum (VMware founders)



IP Fabric Networking Landscape

Netw

ork

H

ard

ware

Netw

ork

O

S

Open Closed


The Expanding Landscape

hardware

operating system

appapp

hardware

operating system

app app

Single Vendor Blob

Multi-Vendor Ecosystem

app app

Expanding Ecosystem

The missing piece: Cumulus® Linux®, bringing the Linux revolution to networking


Understanding Characteristics of a Leaf Switch

8cumulusnetworks.com

10/40 Gigabit spine uplink ports

Serial console port

Ethernet Out-of-Band

Management Port

1/10 Gigabit Ethernetnetwork access ports

* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options

Understanding Characteristics of a Spine Switch


Serial console port

Ethernet Out-of-Band Management Port

* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options

40G / 10G Fabric ports

Add leaf switches incrementally

Connecting 40G Uplinks to Spine Layer


Spine Switch 1

Leaf Switch 1

uplink 1 uplink 2 uplink 3 uplink 4

Spine Switch 2 Spine Switch 3 Spine Switch 4

Anatomy of a Network Switch


( Management Interfaces ) ( Data Plane )

CPUSoC

DRAMBootFlash

MassStorage

SwitchingASIC

SerialConsole

EthernetMgmt Port

10GbPort

40GbPort…10Gb

Port40GbPort

…

PCIe


Bare Metal Switch Provisioning

Similar approach to installing OS on server

BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)

Supported hardware (HCL) preloaded with ONIE

ONIE available on GitHub• http://onie.github.io/onie/

bare metal server

operating system

app

app

app

BIOS and PXEbare metal

switch

operating system

app

app

app

U-Boot and ONIE

http://onie.github.io/onie/

http://onie.github.io/onie/

13

Hardware Vendors

cumulusnetworks.com


Operating System Vendors


Hardware Compatibility List (HCL)

Switch Model Number DescriptionMerchant Silicon

Cumulus Linux Release

Dell S6000-ON 32 x 40G-QSFP+ Trident II 2.1 or later

Edge-CoreAS6700-32X with ONIE

32 x 40G-QSFP+ Trident II 2.0.1 or later

Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later

Quanta QCTQuantaMesh T5032-LY6

32 x 40G-QSFP+ Trident II 2.0.1 or laterAgema AG-7448CU48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 1.5.0 or later

Dell S4810-ON48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 2.0.2 or later

Edge-Core AS5600-52X with ONIE48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident+ 1.5.0 or later




Trident II 2.1.x or later

Penguin Computing Arctica 4804X48 x 10G-SFP+ and 4 x 40G-QSFP+


Quanta QCTQuantaMesh T-3048-LY2

48 x 10G-SFP+ and 4 x 40G-QSFP+


Quanta QCTQuantaMesh T-3048-LY2R



Quanta QCTQuantaMesh T5048-LY8


Trident II 2.1.x or later*

Edge-Core AS4600-54T with ONIE 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later

Penguin Computing Arctica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later

Quanta QCTQuantaMesh T1048-LB9

48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later

40

G1

0G

1G

http://edge-core.com/prodCat.asp?c=1

http://www.penguincomputing.com/products/network-switches

http://www.quantaqct.com/en/01_product/01_list.php?mid=30&sid=114











16

Choice

cumulusnetworks.com

17

Choice

cumulusnetworks.com

ONIE: Bare Metal Install – First Time Boot Up


Boot Loader(HW Vendor Supplied)

ONIE(HW Vendor Supplied)

Installer(OS Vendor)

Boot Loader• Low Level boot loader, configures CPU complex• Loads and boots ONIE

ONIE• Linux Kernel with Busybox• Configures management Ethernet interface• Locates and executes an OS installer• Provides tools and environment for installer

OS Installer• Available from network or USB• Linux executable• Installs vendor OS into mass storage

Network OS(OS Vendor Supplied)

Fetches

Installs

ONIE: Network OS Installer Discovery and Install Behavior


Configure Network Interface

Locate Installer

Run Installer

• Uses DHCPv4, DHCPv6• Configures Ethernet interface for IPv4 / IPv6• Configures DNS and hostname

• Determines the location of an installer executable• Examines local file systems, e.g. USB flash drives• Uses DHCP options, DNS Service Discovery, Multicast DNS

and IPv6 Neighbors

• Downloads installer via URL• Passes various environment variables to installer• Launches installer


Networking Interfaces in Linux

Interface Description

eth0 Physical interface for out-of-band management

lo Loopback (logical interface redirecting to switch)127.0.0.1 in /etc/hostsDebian lists secondary 127.0.1.1

swpN Physical interface for data plane trafficN corresponds to port number

bridge Logical interface creating a single Layer 2 broadcast domainTraffic on sub-interfaces can be untagged or taggedCommonly called “VLAN”

bond Logical interface aggregating two or more interfacesCommonly called “LAG” or “port channel”


Pushing Changes Down

CPU, RAM, Flash, etc. Switch Silicon

Front Panel Ports

lldpd

Routing Tables

ARP Table

Devices

Bridge FDB Filter Tables

Bonds VLANs

Lin

ux

Kern

el

Virtual Kernel Ports

Bridging

mstpd

ACLRouting Suite

Quaggasnmpd

vconfig

iptable

ebtable

ip6tableiproute

2

VXLAN Bridges

Switch HAL

brctl

SwitchDriver

Use

r Space

Quagga daemon, Quagga.conf, and vtysh

CLI and /etc/network/interfaces

switchd


Show Interface Statistics

High level statistics for an interfacecumulus@switch:~$ ip -s link show dev swp13: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500 link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 21780 242 0 0 0 242 TX: bytes packets errors dropped carrier collsns 1145554 11325 0 0 0 0

Low level statistics for an interfacecumulus@switch:~$ sudo ethtool -S swp1


Deconstructing /etc/network/interfaces

auto swp1 iface swp1 inet static address 192.168.0.11/30 gateway 192.168.0.1 up ip link set $IFACE up down ip link set $IFACE down

Bring up interface during boot up or service network

reloadInterface

nameMethod: manual, static,

dhcp

ifup verse to bring up interface

ifdown verse to bring down interface

IP address settings for interface,

only if using static

Method

Action

manual

No IP address configured by default

static

IP address configured using address and gateway options

dhcp Obtain IP address using DHCP server


Bridging

Bridge = single isolated Layer 2 broadcast domain

Allows hosts connected to bridge ports (members) to discover each other without having to define routes

Traffic on ports is tagged (802.1q VLAN ID) or untagged (native)

• Tagging involves using sub-interfaces, e.g. swpN.ID

Commonly called “VLAN” in traditional networking


Defining a Bridge

auto br-vlan100 iface br-vlan100 inet manual bridge_ports swp4.100 swp5.100

up ip link set $IFACE up down ip link set $IFACE down

Bring up interface during boot up or service network

reloadInterface

nameMethod: manual, static,

dhcp

ifup verse to bring up interface

ifdown verse to bring down interface

Bridge members.swp4, swp4.100, swp5, and swp5.100 must be

defined first.100 creates sub-interface

(turning swp into trunk port)


Show Bridge

Show bridges

Show bridge MAC addressescumulus@switch:~$ brctl showmacs br-redport name mac addr is local? ageing timerswp4 06:90:70:22:a6:2e no 19.47swp1 12:12:36:43:6f:9d no 40.50swp1 44:38:39:00:12:9b yes 0.00swp2 44:38:39:00:12:9c yes 0.00

cumulus@switch:~$ brctl showbridge name bridge id STP enabled interfacesbr-vlan100 8000.089e01f89511 no swp5 swp6


Cumulus Linux Packaging and Support

main

updates

security-updates

addons

testing

250 packages

~ 20 Cumulus Linux packages

Examples:

Ruby, Perl, Python, Bash, IPtables, LLDP

Updates: packages revised Security: known concerns, CVEs

User-identified utilities + libraries

Puppet, Factor, Chef, collectd

Early access utilities and libraries

Bird (CL 1.5)

40K+ packagesDebian.org

Fully Supporte

d

Fully Supported

*

Best Effort

Best Effort*

Matu

rity

Level of

QA

*packages not controlled by Cumulus

28

Traditional Hierarchical Network Topology

L3

L2

Access

Aggregation

Core

Legacy and limitations Not designed for today’s

data center running modern workloads

• Server density • Increased server-to-server

traffic Numerous proprietary

protocols• STP/RSTP/PVSTP, VTP, HSRP,

MLAG, LACP “This is what we’ve been

taught”

29

L3 Is the Future

L3

L2

ECMP

Clos network (“spine/leaf”)

1. Simpler network1. Fewer protocols

2. Standards-based1. Fewer proprietary

features

3. Predictable latency1. Every leaf is 1 hop

away

4. Horizontally scalable

Leaf

Spine

Core

Basic Clos Architecture (2-Tier Spine/Leaf)


Optimized for high bandwidth East to West traffic patterns

compute and storage

network services

Core or WAN

Spine Layer

Leaf Layer

Basic Clos Architecture (3-Tier or 5-Stage)


Leaf

Spine

InterPod Spine

Network Services Leaf

32

Ansible demo

spine 1

swp1 - 4

swp1 - 4

swp1 - 4

swp1 - 4

leaf 2

swp17 - 20

swp17 - 20

swp17 - 20

swp17 - 20

wbench

leaf 1

spine 2eth0 eth0

eth0 eth0

eth1eth0

192.168.0.0/24

10.2.1.3/32

10.2.1.4/32

10.2.1.1/32

10.2.1.2/32

swp30-33

swp34-37

10.4.1.1/25

10.4.1.129/25

swp30-33

swp34-37

10.4.2.1/25

10.4.2.129/25

192.168.0.1

33

Questions

34

© 2014 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.

Thank You!

Bringing the Linux Revolution to Networking

Technology

Hardware accelerated switching with Linux @ SWLUG Talks May 2014