OPEN NETWORKING CUMULUS LINUX - DFN

Andreas la QuianteCNX1999 , …, CCIE , … CCONP2019::9

[email protected], GermanyDate: 2020-SEP-16

OPEN NETWORKINGCUMULUS LINUX

73. DFN-Betriebstagung

mailto:[email protected]

2

ANDREAS

乐珼安安德烈亚斯

Danke für die Einladung!

3

AGENDA


16-September-2020Slot: 10:00 - 10:30 Uhr

Cumulus – Hardware offene Switch Betriebssysteme

Andreas la Quiante, Senior Systems Engineer, NVIDIA (Cumulus Networks)Cell: (+49 (0) 151535 76676)

4

01 - MOTIVATION

5

CONTROL AND DATA PLANE

Control-Plane

Data-Plane


Often:

48 x 1/10/25G

32 x 100G/200G/400G

6

OPTIONS … MANY OPTIONS


today…and we have only limited time…we need to focus!

7

Implementation Option 1


Open == Linux as a Bootloader / Open == Shell Access

Vendor

SDK

e.g. Vendor

Routing

Linux

Kernel

User Space

Kernel Space

Hardware

Offload

8



Network Processor Vendor SDK

Vendor

SDK

Community

Routing

Linux

Kernel

Netlink

User Space

Kernel Space

Hardware

Offload

Use of the Linux Infrastructure

9

Implementation Option 2 (gute Virtualisierung)


NetworkProcessor Vendor SDK

Vendor

SDK

Community

Routing

Linux

Kernel

Netlink

Community

Routing

Linux

Kernel

VM

10



Native Linux with Driver

Community

Routing

Linux

Kernel

Netlink

Features

https://github.com/Mellanox/mlxsw/wiki

User Space

Kernel Space

Hardware

Offload

driver

11

WE COVER TODAY: CUMULUS LINUX


Cumulus Linux

12

NETWORKING IS SIMILAR TO COOKING/DINING


Use Case 1 Use Case 2 Use Case 3

You need spices to

prepare different dishes

Like Cumulus Linux offering

different switch configurations

13

WHAT HAVE YOU OR COULD YOU CREATE?

Three examples:


Use Case 1 Use Case 2 Use Case 3

Research

HPC or Storage

(…vSphere 7.0)

RDMA over Converged Ethernet

Enterprise

Automation focus

DevOps

Linux Shop

Service Provider

K8S as a Service

Multi-Tenant with RoH

(EVPN-VXLAN)

14

MOTIVATION

Cumulus/Linux networking

offers for many,

especially Linux and DevOps customers,

Consistency and a positive community!


And allows extremely simple and native automation via common open source tools

efficient == clever

15

02 - PRODUCT

16

Products and Offerings backed by a great team

▪ Cumulus Linux 1.x


▪ Cumulus Linux 3.x ESR*


▪ NetQ 1.x

▪ NetQ 2.x

▪ NetQ 3.x

▪ Consulting Services

▪ Global Support Services

▪ Training and Education

▪ Documentation and an

active Community

*ESR == Extended Support Release

18

03 - ARCHITECTURE

19

Architecture – a personal view


www.onie.org

ifupdown2

vrf

vlan aware bridge

vxlan / evpn

neighbor mgr

quagga / frr

acl tool

portwd

switchd

NetQ Agent

PTM

…

ZTP

Image

(ZTP script

runs once)

20

Architecture - ideal for simulations


Control-Plane

Data-Plane

FRR

iproute2

mstpd

brctl

iptable

ebtable

ip6table

ifupdown2

Routing Tables ARP Table Bridge FDB Filter Tables

ethtool

cl-acltoollldpd

snmpd

swp1

Linux Kernel

NetQ

Agent

Soft

ware

only

part

21

simple simulation example


Server01 Server02

Leaf01 Leaf02

Spine01

22

Creation via .dot file

graph „Linux02“ {

"server01" [function="host" os="generic/ubuntu1804" ssh_user="cumulus" config="tc_files/cl.sh"]

"server02" [function="host" os="generic/ubuntu1804" ssh_user="cumulus" config="tc_files/cl.sh"]

"leaf01" [function="leaf" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]

"leaf02" [function="leaf" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]

"spine01" [function="spine" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]

"server01":"eth1" -- "leaf01":"swp1"

"server02":"eth1" -- "leaf02":"spw1"

"spine01":"swp11" -- "leaf01":"swp11"

"spine01":"spw12" -- "leaf02":"swp12"

}

TopologyConverter.py

Ergebnis: Vagrant Datei

Provider: VB or KVM

air.cumulusnetworks.com

$ ssh -p xxxxx [email protected]


23

Single DC Design with OOB (Cumulus AIR)

https://air.cumulusnetworks.com/Login


Ansible EVPN Playbooks

(Production Ready Automation)

L2 EVPN

L3 Centralized Routing

L3 Decentralized Sym. Routing

NetQ

https://air.cumulusnetworks.com/Login

24

Architecture, physical Switch


Control-Plane

Data-Plane

FRR

iproute2

mstpd

brctl

iptable

ebtable

ip6table

ifupdown2

Routing Tables ARP Table Bridge FDB Filter Tables

ethtool

switchdcl-acltoollldpd

snmpd

swp1

swp1 (xe0)

Linux Kernel

Network

Prozessor

NetQ

Agent

25

Excurs Netlink Manager


NLMGR

Your SW

https://github.com/CumulusNetworks/python-nlmanager

Routing Tables

31

04 – USE-CASE EXAMPLES

32

DevOps, Linux and Automation


Automation focus

DevOps

Linux Shop • Open Source affine

Linux NOS

• Standard Orchestration Tools

in use or planned

• Leaf-Spine Fabric

with Overlays (EVPN-VxLAN)

33

DevOps, Linux and Automation

Automation ZTP

DHCP

(eth0)

OOB

Network

ZTP (runs once)

• User/Password

• Management VRF

• SSH Keys

• License File

Orchestration

• Ansible etc.

[t]

n Minutes

Up

and

Running


AM

SE

1 to 1000 nodes

34

Standard – Native - Automation


ServerSwitch

e/n/I

ifreload -aifupdown2

$ ifreload -a

+ an interface manager for scale, ease and non-disruption

35

The consistent way to manage

Many compute nodes (bare-metal, VM, …)

Some networking nodes, one open tool

or similar


36

Spices for Use-Cases


HPC or Storage

(…vSphere 7.0)

RDMA over

Converged Ethernet• BGP unnumbered

• Pause

• ECN

• ECN with VxLAN

37

BGP UNNUMBERED

Initiator Target

spine01

leaf01 leaf02 leaf03 leaf04

eth1 eth1


Layer 3 Data Center Fabric

Without the need to explicitly

Configure Layer 3 Addresses,

remote Layer 3 Addresses or

Remote ASNs!

BGP Configuration on leaf01

Router bgp 42000011

bgp router-it 10.0.0.11

neighbor swp51 interface remote-as external

39

LEAF-SPINE + OVERLAY + ECN

RoCEv1

RoCEv2

RoCEv2 & VXLAN

Initiator Target

spine02 spine03

leaf01 leaf02 leaf03 leaf04

eth1

Congestion

leads to setting

ECN in the (outer) IP Header

Mapping ECN from outer

to inner Header

Reacting on

ECN marking

Marking return traffic

eth1

VxLAN


spine01 spine04

40

Spices for Use-Cases


K8S as a Service

Multi-Tenant with RoH

(EVPN-VXLAN) • FRR, VRF, maybe even VTEPs

on switches and server

• Layer 2, Layer MLAG or

Layer 3 node connectivity

• Container running on CL

41

K8S


Compute (4.19.32)

Compute (4.19.32)

VxLAN

42

K8S as a Service (Financial Vertical)


Tenant A

K8S Maste/Minions

Tenant A FW

VxLAN VNIs:

Red outside

Green inside

Address-family l2vpn evpn

…

neighbor swp2 route-map FW out

route-map FW permit 1

match evpn vni 10020

match evpn vni 10021

C

C

C Container for

Provisioning

Partner IP

43

05 – SUMMARY

44

A DC FABRIC FOR ALL USE CASES


Compute ComputeStorage StorageAI AI

ONE FABRICAll use-cases All use-cases

45

EVPN – Layer 2 (CL 4.2 EVPN-MH) and Layer 3 (centralized, decentralized (both asym and sym)Multi-Tenant

Server01

VID: 10

VRF: LinuxAdmin

IP 172.31.10.1/24

swp1

bridge

VLAN 10

svi10

swp1

bridge

VLAN 10

Server02

VID: 10

VRF: LinuxAdmin

Server02‘

VID: 20

VRF: LinuxAdmin

swp

bridge

VLAN 20

svi20

VNI 10

VNI10010

Swp n

bridge

VLAN 30

svi30

VRF UnixAdmin

Server06

VID: 30

VRF: UnixAdmin

VNI 30

VNI 10030

cumulus@leaf01:mgmt:~$ net show evpn vni

VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF

10 L2 vxlan10 2 4 0 LinuxAdmin

10010 L3 L3VNI_Linux 1 2 n/a LinuxAdmin

VNI 3020


IP 172.31.10.111/24

172.31.10.110

IP 172.31.10.2/24 IP 172.31.20.2/24

Vlan 1001 Vlan 1001

46

If you like one uniform data center and tool set

EVPN-VxLAN

THANK YOU

48

NEXT STEPS

• Test Drive (theory chapters + hands-on lab, free of charge)

• Try it yourself (download VX or whole Vagrant lab)

• Use Cumulus Air with 3 pre-build EVPN setups

• You need Linux networking for your project?

• You need a SME for your meeting?

• A future proof/reliable infrastructure is key?

• You like or need to automate?


Call !

49

READING SUGGESTIONS


50

Reference Material

乐珼安安德烈亚斯 https://cumulusnetworks.com/learn/web-scale-networking-resources/cheatsheets/

One skill-set to rule them all

Documents

OPEN NETWORKING CUMULUS LINUX - DFN