Bachelor Informatica

Open-source network operat-ing systems: feature evalua-tion of SONiC

Erik Puijk, 11017651

June 7, 2018

Supervisor(s): Dr. Paola Grosso, Lukasz Makowski MSc

Signed:

Informatica—

Universiteit

vanAmst

erdam

2

Abstract

Open network switches are increasing in popularity and allow the deployment of dif-ferent open-source network operating systems (NOS). In contrast to locked-in switches,open switches with an open-source NOS have been tested less extensively as they are quitenew phenomena. This thesis examines whether open switches with an open-source networkoperating system, namely SONiC, can be deployed successfully to perform fundamentalnetworking operations. Furthermore, it examines for what use cases SONiC is suitable andbeneficial. We experiment with open switches in various topologies to examine the deploy-ment of fundamental OSI Layer 2 and Layer 3 networking features. We conclude that allSONiC-supported features we tested can be deployed successfully. Moreover, we examineseveral use cases of open switches with SONiC and conclude that SONiC is most suitablefor use cases in large-scale data centers and enterprise networks.

3

4

Contents

1 Introduction 7

2 Open networking background 92.1 Open switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Network operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.1 ASIC communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.2 ASIC control module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 SONiC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3.1 Quagga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Networking features 153.1 Layer 2 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Layer 3 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Experiments 194.1 Preparatory phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1.1 Mellanox SN2100 and ONIE . . . . . . . . . . . . . . . . . . . . . . . . . 204.1.2 Arista 7050QX-32S and Aboot . . . . . . . . . . . . . . . . . . . . . . . . 20

4.2 Feature tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2.1 Layer 2 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2.2 Layer 3 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.2.3 Result overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5 Discussion 295.1 Ease of use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6 Use case scenarios for open switches 316.1 Current use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.2 Example use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

7 Conclusions 357.1 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

8 Acknowledgements 37

9 Bibliography 39

10 Appendix 41

5

6

CHAPTER 1

Introduction

Network switches are essential in computer networks, to connect devices and forward framesbetween them on the OSI data link layer (Layer 2)[1]. Some switches are also capable of OSILayer 3 features, such as the routing of segments among networks using IP. In this thesis, wediscuss exactly this category of devices. Traditionally, switches are sold with locked-in hardwarewith a pre-installed network operating system (NOS) on them, without the possibility for anetwork administrator to install third-party NOS’s or other software on them.

An open switch, in contrast, does allow the user to install another operating system on thedevice. Open switches thus give network administrators more possibilities to customize theswitch to their own needs, possibly explaining their rising popularity. Another advantage ofthis category of switches is the reduction of expenses, due to the possibility to install low-costsoftware. In the past, this cost reduction would be compensated by an increase in operationalcosts, because using the switches would require hiring external Linux-expertise to configure theswitches [2]. However, as Linux expertise has grown over the past few years, this barrier forusing open switches has become much thinner. Also, large manufacturers such as Dell andHP have been developing open switches on which NOS’s like Cumulus Networks or Pica8 arealready installed [3]. This removed another obstacle from companies using the switches becauseno manual installation is required.

The increasing popularity raises questions about the suitability for use in real networks, theease of use and how their feature sets compare to those of traditional switches. Considering thatopen switches are relatively new compared to locked-in switches, there is still need for testingand evaluation to assess whether open switches can replace traditional switches without loss offunctionality, performance or ease of use, which the reduction in costs might not weigh up to.

In this context, we examine the functioning of open switches running an open-source networkoperating system, namely SONiC, in a network. We study whether open switches with SONiCare able to deploy several fundamental networking features. In addition, we examine which usecases benefit from the open and flexible nature of open switches with SONiC. We therefore setout to answer the following two research questions:

1. Which networking features can be successfully deployed on open switches withSONiC?

2. Which use cases are (more) easily supported by open switches with SONiC?

Chapter 2 provides background information about open networking, open switches, networkoperating systems (SONiC specifically) and routing suites. Chapter 3 will briefly discuss sev-eral networking features that will later be used in our experiments. Chapter 4 contains theexperiments we performed to answer our first research question. In chapter 5, we discuss theexperimental methods and results and other findings obtained during this research. In chapter6, we examine several use cases of open switches with SONiC. Lastly, in chapter 7 we return toour research questions, formulate a conclusion and suggest possibilities for future research.

7

8

CHAPTER 2

Open networking background

2.1 Open switches

Generally, a switch can be represented as a stack of four layered components. Figure 2.1 showsthese components.

Control and management plane

Network operating system

Hardware

Silicon/ASIC

Figure 2.1: Layered component stack of a switch.

The silicon, or ASIC (application-specific integrated circuit), is a specified hardware elementdesigned for a specific task. In the case of switches, this task is to quickly send packets throughthe network [4]. The hardware-layer includes all other physical components of the switch, like theinterfaces, the input/output-ports, the LEDs and the power supply [5]. The network operatingsystem (NOS) controls the hardware and the underlying ASIC for networking purposes and allowscontrol and management plane applications to use the hardware. Control plane and managementplane applications provide particular features to the user of the switch, in addition to those ofthe underlying operating system [6].

To understand the difference between open switches and traditional switches, one needs toconsider the manner in which the above components interact with each other. Switches in whichthe NOS and the underlying hardware are disintegrated, meaning that they can be changedindependently of each other, are called open switches. In traditional (locked-in) switches, thisis not possible, for the switch is delivered with pre-installed software that cannot be changed.Open switches thus give the user more choice in what NOS to run.

Open switches can be separated into subcategories. Bare metal switches provide the hardwarerequired to run, and allow the user to load the NOS of choice. The manufacturers of bare metalswitches are original design manufacturers (ODMs) for well-known switch merchants, whichmeans that the ODMs products are re-branded and sold by other companies. A boot-loaderallows the user to boot an NOS of choice on the device [7].

9

2.2 Network operating systems

A network operating system is a key component in the aforementioned component stack of openswitches. The NOS controls the hardware of the device and provides applications on the switchwith the hardware and software resources they need. These resources might for example bememory allocation or input and output resources.

2.2.1 ASIC communication

The aforementioned application-specific integrated circuits (ASICs) are designed to handle aspecific task. In networking switches, the ASIC is designed and optimized for quickly processingincoming packets according to the routing table. In order for an NOS to be able to program theASIC, several APIs have been developed to communicate with the ASIC. The Switch AbstractionInterface1 (SAI) is a well-known method. It is an open source framework that aims to abstractaway from the ASIC, which differs for each vendor, so software can be programmed for use inmultiple different switches without any changes. This allows for more freedom in the use ofsoftware independently from the hardware choice [8].

Another, less adopted method for ASIC communication is the use of SDKs developed bythe ASIC vendor. In practice, this approach is not included in open-source software for openswitches considering changes in the SDKs would need the application to be modified. Examplesof these SDKs are SwitchX SDK by Mellanox [9] and OpenNSL by Broadcom [10].

2.2.2 ASIC control module

On top of the ASIC API, the ASIC control module provides an interface for control plane ap-plications to communicate with the hardware. It also presents the current state of the hardwareto the user of the switch. Control plane apps can use the ASIC control module to read fromor write to data stored in the hardware. These applications therefore are independent from thehardware in the machine they are running on. Figure 2.2 shows the role of the ASIC controlmodule and the ASIC API in the component stack illustrated before.

Control and management plane

Network operating system

Hardware

Silicon/ASIC

ASIC control module

ASIC API

Figure 2.2: ASIC control module and ASIC API in the layered component stack.

1https://github.com/opencomputeproject/SAI

10

2.3 SONiC

SONiC2 (Software for Open Networking in the Cloud) is an open-source network operatingsystem that claims to include all features to have a fully functional Layer 3 network device.It is under constant development by Microsoft. The latest release, SONiC.201803, supportsfeatures including BGP, LLDP, link aggregation/LACP and VLAN (trunking) [11]. SONiC isLinux-based and runs on Debian Jessie. The SONiC architecture is depicted in figure 2.3.

Figure 2.3: An overview of SONiC components [12].

SONiC uses SAI for programming the ASIC, which makes SONiC compatible with all ASICsthat are supported by SAI. The SONiC Object Library3 allows external applications to interactwith each other and with SONiC applications.

Switch State Service

The Switch State Service (SwSS) acts as the ASIC control module for SONiC, using a databaseto interface between the network applications running on the switch and the switch hardware.Figure 2.4 shows the SwSS architecture. Network applications use a database named APP DB forreading and writing. Orchestration agents are responsible for synchronization between APP DBand another database, namely ASIC DB. To provide universality, the databases are set up as key-value storage. SwSS allows network applications running on SONiC to be completely independentfrom the hardware they are running on.

2https://github.com/Azure/SONiC/wiki3https://github.com/Azure/sonic-object-library

11

Figure 2.4: The design of Switch State Service, showing the relation between APP DB, ASIC DBand the orchestration agents [12].

Since SONiC is open-sourced, the networking community can contribute to the improvement ofSONiC. In addition, SONiC can be extended with other open-source software, be it third-partyor proprietary. Figure 2.5 shows how SONiC is a part of the Open Compute Project, which isan organization that has as its main goal to design and deploy the most efficient and scalablehardware for use in data centers. OCP certified switches can use SONiC through SAI to run aLayer 2 or Layer 3 switch, and in addition allows the user to extend the NOS with third-partyor OCP software components [13].

Figure 2.5: SONiC as an open switching platform for OCP users [13].

2.3.1 Quagga

Routing suites are control plane software collections that offer a variety of routing protocolsto run on top of a network operating system. A routing suite exchanges routing informationwith other routers and updates routing information in the kernel. Quagga4 is an example of arouting suite and provides BGP routing functionality for SONiC. It is open-source and currentlysupports GNU/Linux and BSD.

There are two main Quagga processes present on a SONiC device: bgpd and zebra. Figure2.6 shows how Quagga and SONiC interact when a BGP route advertisement is received. First,this advertisement is passed to bgpd, short for BGP daemon. The BGP daemon processes thisroute and determines whether it should be placed in the kernel routing table. If so, the route ispassed to zebra, which after additional filtering updates the routing table in APP DB. This is

4https://www.quagga.net/

12

done with fpmsyncd, which is a Forwarding Plane Manager and programs the forwarding plane.Fpmsyncd uses the SONiC Object Library API. SONiC handles updating the kernel routingtable. Zebra is also capable of selecting the best route across different routing protocols, but asSONiC supports only BGP this is not applicable to a SONiC device.

Figure 2.6: The interaction between Quagga and SONiC when a new BGP route is received[12]. Quagga determines whether a new route should be placed in the routing table, after whichSONiC takes care of updating the kernel routing table.

13

14

CHAPTER 3

Networking features

This chapter briefly covers several networking features or protocols that are relevant to thisresearch. We selected both OSI Layer 2 and 3 features that we consider fundamental to a datacenter switch.

3.1 Layer 2 features

Layer 2 of the OSI model is the data link layer. It provides mechanisms to allow communicationbetween devices within the same local area network (LAN). Data elements transferred on thedata link layer are called (link-layer) frames. A frame is transferred between two devices on aphysical link, and MAC addresses are used to address link-layer frames. The Ethernet protocolis a well-known example of a Layer 2 protocol and is used often in local area networks and widearea networks. WiFi is an alternative technology, although it is used only in local area networks[14].

LLDP

The Link Layer Discovery Protocol (LLDP) allows networking devices to advertise informationabout their identity and capabilities. LLDP is useful for managing network resources and providesa vendor-independent mechanism for devices to exchange device information within a network.Each interface of a device sends out a frame consisting of an LLDP Data Unit (LLDPDU). AnLLDPDU consists of a sequence of type-length-value (TLV) structures, each containing specificinformation. Each LLDPDU contains the following mandatory TLVs: Chassis ID, Port ID andTime To Live. Furthermore, optional TLVs can be included as well, such as system name,system capabilities and port description. LLDPDU frames are sent at a fixed interval, to keepthe information up-to-date [15].

LACP

The Link Aggregation Control Protocol (LACP) can be used to combine multiple physical portsand form a single logical channel. This process is also called link aggregation. To higher layer pro-tocols, the aggregated links form a single channel. Link aggregation provides increased through-put due to the use of multiple physical links. It also adds redundancy because even with aphysical link-failure, traffic is still capable of flowing through the logical link by making use ofthe other (operative) physical links [16].

STP

The Spanning Tree Protocol (STP) aims at constructing and managing a loop-free topology ina bridged network. STP is supposed to prevent loops in a topology and guarantees that thereare unique paths to all destinations within a topology. This is critical in local area networks, for

15

loops can cause Ethernet frames to be switched around endlessly. For instance, when a switchreceives an ARP packet, it broadcasts this packet through all of its interfaces (except the onethe ARP packet was received on). If a loop exists in the network of the switch, it will eventuallyreceive its own broadcast ARP packet, which will be broadcast again. This endless process isknown as a broadcast storm and it is one of the phenomena STP should prevent [17]. It does so byselecting a Root Bridge, and then creates a loop-free topology by blocking particular interfacessuch that all other bridges have a shortest path to the Root Bridge [18].

Figure 3.1: Example of how STP creates a loop-free topology with shortest paths to the RootBridge [19]

Figure 3.1 shows an example of a topology with a loop. Switch A is selected as Root Bridge,thus all other switches should have a shortest path to switch A. By blocking the link betweenswitch B and switch C, the loop in the topology is removed and all switches have a shortest pathto switch A.

VLAN (trunking)

A Virtual Local Area Network (VLAN) can be described as an isolated and partitioned broadcastdomain at OSI Layer 2. A Local Area Network (LAN) can be partitioned into several VirtualLANs to create multiple (logically) separate networks despite the (possible) presence of a physicalconnection to other devices that are not in the VLAN. This is achieved by configuring VLANson a switch and specifying which switch interface belongs to which VLAN. The ability to defineisolated broadcast domains using software makes VLAN a well-known networking feature. Itallows network administrators to create multiple separated broadcast domains without havingto change the physical wiring of the network [20].

A trunk link can be used to transport frames from multiple VLANs, for instance to inter-connect two switches that have configured the same VLANs. This is achieved by using tags inlink-layer frames to specify what VLAN a frame belongs to. These trunk links are thereforecalled ”tagged” links. Links that can only transport frames from one specific VLAN are called”untagged” links and thus the frames that are transported on these links do not contain a VLANtag. Figure 3.2 shows an example. The entirely blue or entirely red links are untagged links forVLAN 100 and 200 respectively, and the frames transported on these links do not contain aVLAN tag. The link between switch 1 and 2 is a tagged link, for it must transport frames fromboth VLAN 100 and 200 and thus a tag must be used to indicate the corresponding VLAN fora particular packet.

16

Figure 3.2: Example of a VLAN configuration with two switches and two VLANs [21]. Thetrunk link between switch 1 and 2 transports traffic from both VLAN 100 and VLAN 200.

3.2 Layer 3 features

Layer 3 of the OSI model, the network layer, is responsible for the routing of data betweendifferent local area networks (LANs). Network-layer packets are addressed using unique andhierarchical IP addresses [14]. Routers are devices that forward packets to their destination bydetermining the route to be taken by the packet to reach its destination. In a network, routerscommunicate with each other to exchange information about the availability of subnets. Thisis done by routing protocols, which in addition define policies or algorithms that determine thebest route to a destination when multiple routes are available.

Inter-VLAN routing

Inter-VLAN routing is used when traffic must flow between different VLANs, which in somecases may be necessary. Typically, this is accomplished by specifying a VLAN interface for eachrelevant VLAN, allowing the switch to route the traffic between VLANs through these VLANinterfaces. Figure 3.3 shows an example of a router (R1) that routes between VLANs. WhenPC A sends traffic destined for PC B, the traffic is first tagged with VLAN 20. When this trafficreaches router R1, it routes this traffic to VLAN 30 by using the configured VLAN interfaces.Then, this traffic can proceed to PC B with the VLAN 30 tag.

Figure 3.3: Example of a topology in which inter-VLAN routing is needed [22]. Router R1 usesthe VLAN interface addresses to route the traffic from VLAN 20 to VLAN 30.

17

BGP

An autonomous system (AS) is a selection of IP prefixes that is managed by a certain networkadministrator. Commonly, such autonomous systems are managed by internet providers or largecompanies. The Border Gateway Protocol (BGP) is a well-known routing protocol used forrouting between different autonomous systems, but can also be used to route within an AS.

Routers running BGP can set up sessions with each other using TCP to exchange routinginformation, for instance subnet availability. BGP provides several mechanisms to control thepropagation of routes, such as route-maps. Route maps allow the network administrator to definerules for certain prefixes [23]. For instance, there may be rules to drop an incoming route orchange it before placing it in the routing table. BGP is used commonly in large-scale data centersbecause it scales well compared to other routing protocols [24].

18

CHAPTER 4

Experiments

4.1 Preparatory phase

To examine whether open switches with SONiC can be deployed successfully in a network, wedecided to perform experiments on the deployment of the fundamental networking features thatwere described in chapter 3. The SNE OpenLab has provided two open switches for this research.

• Mellanox SN21001

• Arista 7050QX-32S2

The ASICs in these switches are manufactured by different vendors: Mellanox and Broadcom,respectively. This allows us to test two different ASIC types at once, and shows whether SONiCcan run on two different ASICs. Both switches are capable of operating on OSI Layer 2 and Layer3. Figures 4.1 and 4.2 show photos of Mellanox SN2100 and Arista 7050QX-32S, respectively.More details of the switches can be found in Appendix A.

Figure 4.1: Mellanox SN2100.

Figure 4.2: Arista 7050QX-32S.

1http://www.mellanox.com/related-docs/prod_eth_switches/PB_SN2100.pdf2https://www.arista.com/assets/data/pdf/Datasheets/7050QX-32_32S_Datasheet.pdf

19

The open-source NOS used in the experiments is SONiC. Section 2.3 has briefly explained thearchitecture of SONiC and what features it supports. SONiC is under constant development andnew features are added regularly. In addition, the absence of other network operating systemsthat support a wide variety of open switches made SONiC an obvious NOS to experiment with.Aside from SONiC, there are various open-source network operating systems, such as OPX.SONiC, however, supports a wide variety of switches and thus allows for the straightforwardextension of this research to other supported devices. Details about the version of SONiC usedin this research are provided in Appendix B1.

4.1.1 Mellanox SN2100 and ONIE

Mellanox SN2100 comes with Mellanox Onyx as NOS. Other operating systems can be installedover ONIE3 (Open Network Install Environment). It is an open-source project by CumulusNetworks that allows creating an open networking environment and is present by default onvarious open switches [25]. Figure 4.3 shows how ONIE can be used to boot a network operatingsystem. When a device boots for the first time, a low-level boot loader boots ONIE from flash

Figure 4.3: Process of first time boot using ONIE [25].

memory. ONIE then fetches the image of the NOS supplied by the switch vendor and installsthis image. The next time the device boots, ONIE is not used by default and the device goesstraight in the NOS. ONIE can, however, still be used to uninstall the vendor NOS and installanother (open source) NOS from an image provided over the network or via USB for instance[25].

4.1.2 Arista 7050QX-32S and Aboot

Arista 7050QX-32S runs Arista EOS by default. A component of this operating system is Aboot4,which can be used to boot image files to run other operating systems. Boot parameters can beconfigured in files which are stored in flash memory. In addition, Aboot provides a shell whichcan be used to modify boot configuration [26].

3https://opencomputeproject.github.io/onie/4https://www.arista.com/ko/um-eos/eos-6-1-boot-loader--aboot

20

4.2 Feature tests

In the last chapter, we discussed several networking features we consider crucial to a networkswitch. The objective was to test whether each of the features worked correctly on Mellanox andArista running SONiC. The results of the experiments possibly provide an insight into whetherthe switches can be deployed in a data center network as a replacement of traditional switches.For most experiments, we used SONiC’s config db.json configuration file to set up our testingenvironment. Appendix C contains the most relevant configurations. Complete configurationfiles can be found in the GitHub repository5.

4.2.1 Layer 2 features

LLDP

MethodsThe first feature to be tested was LLDP. To verify whether LLDP correctly exchanged deviceinformation, we considered the topology depicted in figure 4.4. We wanted to verify whetherall mandatory LLDP TLVs were exchanged between Mellanox and Arista. This mandatoryinformation is as follows [15]:

• Chassis ID

• Port ID

• Time To Live

Furthermore, we examined whether any additional information, such as system information anddevice capabilities, is exchanged through LLDP as well.

Mellanox SN2100Ethernet32

Ethernet36 Arista 7050QX-32S

Ethernet16

Ethernet20

Figure 4.4: Topology to analyze LLDP with Mellanox and Arista.

ResultsSONiC provides the command show lldp neighbors to view the LLDP neighbor informationregistered by SONiC. Appendix D1 presents the complete output of this command. Both devicespresented the mandatory LLDP information in their LLDP tables. SONiC runs LLDP so thatthe Chassis ID that was included is taken from the MAC address of the eth0 interfaces on bothswitches.

In addition, LLDP exchanged correct system name and system description information, andalso included device capabilities and which of these capabilities were currently in use.

LACP

MethodsIn addition, we tested LACP/link aggregation behaviour. With the topology depicted in figure4.5, we tried to configure link aggregation combining the double links between Mellanox andArista. We considered both Layer 2 link aggregation and Layer 3 link aggregation. The latterhas an IP address configured for each link aggregation interface, while the former does not.

A working port channel should provide redundancy in the sense that when one of the linksfail, communication is still possible through the port channel by making use of the other link.Thus, to test an aggregated link, one can simulate a physical breakdown of one of the two linksand examine whether communication is still possible between the two devices.

5https://github.com/erikpu/open-switch-configurations

21

Mellanox SN2100Ethernet32

Ethernet36 Arista 7050QX-32S

Ethernet16

Ethernet20

PortChannel1

Figure 4.5: Topology to analyze LACP with a port channel consisting of two physical linksbetween Mellanox and Arista.

ResultsAppendix C1.1 shows the configuration we set for a Layer 2 port channel between Mellanox andArista. We were unable to configure a successful Layer 2 port channel. That is, SONiC couldnot successfully set up a port channel with no interface addresses. PortChannel1 failed to go UP

on both devices. Listing 4.1 shows that PortChannel1 was stuck in DOWN state.

admin@sonic -mellanox :~$ ip a show dev PortChannel1

4: PortChannel1: <NO-CARRIER ,BROADCAST ,MULTICAST ,UP> mtu 9100 qdisc noqueue state

DOWN group default

link/ether ec:0d:9a:8d:f1:c0 brd ff:ff:ff:ff:ff:ff

Listing 4.1: PortChannel1 failed to go UP.

Moreover, with teamdctl we were able to view the state of PortChannel1 within the SONiCteamd docker, which is responsible for link aggregation. On both Mellanox and Arista, no portswere present as slave of PortChannel1. Listing 4.2 shows this for Mellanox. It was confirmed bya SONiC collaborator that port channel functionality is currently aimed at Layer 3 port channelsand not at Layer 2 port channels6.

admin@sonic -mellanox :~$ docker exec -it 7e504bdc03e0 bash

root@sonic -mellanox :/# teamdctl PortChannel1 state view

setup:

runner: lacp

runner:

active: yes

fast rate: no

Listing 4.2: No slave ports present for PortChannel1.

Appendix C1.2 shows the configuration of the Layer 3 port channel. Layer 3 link aggregationdid work correctly. In this case, PortChannel1 did go UP and allowed us to communicate betweenthe interface IP addresses configured on PortChannel1 (172.16.0.10/24 on the Mellanox sideand 172.16.0.20/24 on the Arista side). Also, teamdctl now showed us the correct slave portsfor PortChannel1.

In addition, Layer 3 link aggregation passed our redundancy tests. Listing 4.3 shows anexample of this (some output has been omitted for brevity). Appendix D2 shows more completeoutput of the example. In listing 4.3, it can be seen on lines 4-14 that when both links areoperational, communication through the port channel is possible. In lines 20-21, one can seethat one of the links is not operational anymore. Line 27 shows that despite the fact that onelink broke down, PortChannel1 is still operational. Next, lines 32-36 show that communicationis possible through the port channel even when one of the two links is not operational. Thisindicates that Layer 3 link aggregation provides a redundant connection between Mellanox andArista.

6https://github.com/Azure/SONiC/issues/186

22

1 # (both links up)

23 # (...)

4 Ethernet32 32,33,34,35 N/A 9100 Ethernet32 up up

5 Ethernet36 36,37,38,39 N/A 9100 Ethernet36 up up

6 # (...)

78 # (communication is fine)

910 admin@sonic -mellanox :~$ sudo ping 172.16.0.20

11 PING 172.16.0.20 (172.16.0.20) 56(84) bytes of data.

12 64 bytes from 172.16.0.20: icmp_seq =1 ttl=64 time =0.362 ms

13 64 bytes from 172.16.0.20: icmp_seq =2 ttl=64 time =0.220 ms

14 64 bytes from 172.16.0.20: icmp_seq =3 ttl=64 time =0.320 ms

15 # (...)

1617 # (link failure Ethernet32 !)

1819 # (...)

20 Ethernet32 32,33,34,35 N/A 9100 Ethernet32 down up

21 Ethernet36 36,37,38,39 N/A 9100 Ethernet36 up up

22 # (...)

2324 # (PortChannel1 still UP)

2526 # (...)

27 4: PortChannel1: <BROADCAST ,MULTICAST ,UP,LOWER_UP > mtu 9100 qdisc noqueue state UP

group default

28 # (...)

2930 # (communication still working)

3132 admin@sonic -mellanox :~$ sudo ping 172.16.0.20

33 PING 172.16.0.20 (172.16.0.20) 56(84) bytes of data.

34 64 bytes from 172.16.0.20: icmp_seq =1 ttl=64 time =0.294 ms

35 64 bytes from 172.16.0.20: icmp_seq =2 ttl=64 time =0.310 ms

36 64 bytes from 172.16.0.20: icmp_seq =3 ttl=64 time =0.296 ms

37 # (...)

Listing 4.3: Behaviour of PortChannel1 when link Ethernet32 (Mellanox)-Ethernet16 (Arista)fails (some output has been omitted for brevity).

STP

MethodsWe also examined the Spanning Tree Protocol. Considering the topology in figure 4.6, if no STPis configured at all, a broadcast storm may occur because there is a network loop. Broadcastmessages sent from Mellanox to Arista out of both Ethernet32 and Ethernet36 will be circulatedbecause Arista will broadcast these back to Mellanox through Ethernet20 and Ethernet16, re-spectively, leading to an endless broadcast storm. STP must prevent this by selecting a RootBridge and blocking one of the links between Mellanox and Arista.

Mellanox SN2100Ethernet32

Ethernet36 Arista 7050QX-32S

Ethernet16

Ethernet20 VLAN 100

Figure 4.6: Topology to analyze STP.

The current version of SONiC does not have support for STP, but we decided to try to con-figure STP nevertheless using brctl. We placed the relevant ports in VLAN 100 and we firstset the priority of the Mellanox bridge to 100 and of the Arista bridge to 200, meaning thatMellanox should be selected as Root Bridge and thus one of the Arista ports should be set inblocking state (lower priority indicates higher chance to be selected as Root Bridge). Also, we

23

reversed the priorities so that Arista should be selected as Root Bridge and one of the Mellanoxports should be in blocking state. The relevant SONiC configurations (of the VLANs) can befound in Appendix C2.

ResultsIn both our tests the devices selected themselves as Root Bridge, meaning that all ports wereset in forwarding state and the loop in our topology remained existent. Listings 4.4 and 4.5show the results of the STP status on Mellanox and Arista (some output has been omitted forbrevity). The full output can be found in Appendix D3. One can notice on lines 8 and 12 ofboth listings that all four ports are in forwarding state and in both cases the devices selectedthemselves as Root Bridge, which can be concluded from lines 3 and 4 in both listings.

1 admin@sonic -mellanox :~$ sudo brctl showstp Bridge

2 Bridge

3 bridge id 0064. ec0d9a8df1c0

4 designated root 0064. ec0d9a8df1c0

5 # (...)

67 Ethernet32 (1)

8 port id 8001 state forwarding

9 # (...)

1011 Ethernet36 (2)

12 port id 8002 state forwarding

13 # (...)

Listing 4.4: Mellanox selected itself as Root Bridge (some output was omitted for brevity).

1 admin@sonic -arista :~$ sudo brctl showstp Bridge

2 Bridge

3 bridge id 00c8.001 c737bf75c

4 designated root 00c8.001 c737bf75c

5 # (...)

67 Ethernet16 (1)

8 port id 8001 state forwarding

9 # (...)

1011 Ethernet20 (2)

12 port id 8002 state forwarding

13 # (...)

Listing 4.5: Arista selected itself as Root Bridge (some output was omitted for brevity).

To investigate this behaviour, we used tcpdump to capture STP messages between Mellanox andArista. We found that both devices were sending STP messages to each other, but no receivingSTP messages were passed to the control plane. It was confirmed by a SONiC contributor thatin the current version of SONiC, no interface trap is configured for STP messages, meaning thatincoming STP messages are not passed to the control plane and thus STP is unable to operatecorrectly on SONiC7.

VLAN (trunking)

MethodsAdditionally, we examined VLAN (trunking). We decided that for this experiment, and theother experiments following this one, it would be interesting to use several hosts in our testingtopologies. In order to do so, we attached two physical servers to our topology and set up twovirtual machines (VMs) to run on each host server. We used the same operating system on allfour VMs (Linux 4.9.82-1+deb9u3).

We used Vagrant8 to build and manage our virtual machine environment, and VirtualBox9

for the virtual machine itself. Appendix B2 and B3 specify the Vagrant and VirtualBox versions

7https://github.com/Azure/SONiC/issues/184#issuecomment-3879425268https://www.vagrantup.com/9https://www.virtualbox.org/

24

used in the experiments, respectively. To allow the VMs to participate in the network, weconfigured a bridge between the physical interfaces of the servers and the virtual interfaces ofthe VMs using the Vagrantfiles. The virtual interfaces could then be configured as if they werephysical interfaces on four separate machines. The Vagrantfiles used in this experiment can alsobe found in the previously mentioned GitHub repository.

To verify whether the switches are able to perform correct VLAN (trunking) functionality,we used the topology depicted in figure 4.7. For the SONiC configuration, Appendix C3 can beconsulted. The link between Mellanox and Arista should be configured as a ”tagged” (trunk)link, for it must carry frames that can belong to either VLAN 100 or VLAN 200. We examinedwhether the open switches allowed us to configure such ”tagged” ports and whether packets canbe delivered correctly within the same VLANs using the trunk link. For instance, VM A1 mustbe able to communicate with VM B1, because they are configured to be in the same VLAN. VMA1 should not be able to communicate with VM B2, because they are in different VLANs andshould therefore be completely isolated from each other.

Ethernet56

Phys. server A

VM A1 VM A2

VLAN trunk

Ethernet60

172.16.100.1/24 172.16.200.1/24

VLAN 200 VLAN 100

Arista 7050QX-32S

Ethernet40

Ethernet16

Phys. server B

VM B1 VM B2

Ethernet44

172.16.100.2/24 172.16.200.2/24

VLAN 200 VLAN 100

Mellanox SN2100 Ethernet32

VLAN 100: 172.16.100.3/24VLAN 200: 172.16.200.3/24

VLAN interfacesVLAN 100: 172.16.100.4/24VLAN 200: 172.16.200.4/24

VLAN interfaces

Figure 4.7: Topology to analyze VLAN (trunking) and inter-VLAN routing. Two VLANs areconfigured with two virtual machines in each VLAN. A trunk link is present between Mellanoxand Arista.

ResultsWe were able to communicate within VLANs, but not between VLANs, which is the correct andexpected behaviour. As an example, listing 4.6 shows with ping that communication is possiblefrom VM A1 to VM B1, which are in the same VLAN.

vagrant@vmA1 :~$ ping 172.16.100.2

PING 172.16.100.2 (172.16.100.2) 56(84) bytes of data.

64 bytes from 172.16.100.2: icmp_seq =1 ttl =64 time =0.602 ms

64 bytes from 172.16.100.2: icmp_seq =2 ttl =64 time =0.580 ms

64 bytes from 172.16.100.2: icmp_seq =3 ttl =64 time =0.591 ms

Listing 4.6: ping from VM A1 to VM B1.

In contrast, communication between devices configured in different VLANs is not possible. List-ing 4.7 displays this behaviour. The example shows that we cannot communicate between VLAN

25

100 to VLAN 200.

vagrant@vmA1 :~$ ping 172.16.200.2

PING 172.16.200.2 (172.16.200.2) 56(84) bytes of data.

From 172.16.100.1 icmp_seq =1 Destination Host Unreachable

From 172.16.100.1 icmp_seq =2 Destination Host Unreachable

From 172.16.100.1 icmp_seq =3 Destination Host Unreachable

Listing 4.7: ping from VM A1 to VM B2.

This indicates that the VLAN trunk succeeded in tagging frames with the correct VLAN number,allowing for complete VLAN isolation shown in the above listings.

4.2.2 Layer 3 features

Inter-VLAN routing

MethodsBesides VLAN trunking functionality, we also tested inter-VLAN routing using the same topol-ogy as depicted in figure 4.7. Figure 4.7 shows the VLAN interface addresses we used for bothMellanox and Arista. For the relevant sections of the SONiC configuration, Appendix C4 canbe consulted. If inter-VLAN routing works correctly, the switches will route packets from oneVLAN to another and thus we should be able to communicate between VLANs using the VLANinterfaces. For instance, VM A1 should be able to ping VM B2 and vice versa.

ResultsUsing inter-VLAN routing we are able to communicate between different VLANs. Listing 4.8shows an example of a traceroute from VM A1 to VM B2. It shows on line 3 that the VLANinterface of VLAN 100 on Mellanox (172.16.100.3) is used to route the packet to VLAN 200,after which Arista delivers the packet to VM B2.

1 vagrant@vmA1 :~$ traceroute 172.16.200.2

2 traceroute to 172.16.200.2 (172.16.200.2) , 30 hops max , 60 byte packets

3 1 172.16.100.3 (172.16.100.3) 0.399 ms 0.307 ms 0.227 ms

4 2 172.16.200.2 (172.16.200.2) 0.696 ms 0.619 ms 0.526 ms

Listing 4.8: traceroute from VM A1 to VM B2.

BGP

MethodsTo verify whether BGP operates correctly on the switches, we set up the topology in figure 4.8.SONiCs BGP configuration can be found in Appendix C5. We have configured the switches tobe in different autonomous systems, so as to create a BGP session between Mellanox and Arista.We expect this BGP session to exchange routes between Mellanox and Arista. Specifically, Mel-lanox has knowledge of subnets 10.0.0.0/24 and 10.0.1.0/24 and Arista does not. BGP mustpropagate the routes to these subnets to Arista. Similarly, BGP must propagate routes to thesubnets 10.0.4.0/24 and 10.0.5.0/24 to Mellanox. In the end, we want to be able to commu-nicate between the virtual machines in hosts A and B using these propagated routes.

26

Mellanox SN2100

10.0.0.10

10.0.2.10

10.0.3.10

Phys. server A

AS 65000

subnet 10.0.3.0/24

VM A1 VM A2

subnet 10.0.2.0/24

10.0.1.10

10.0.0.1 10.0.1.1

subnet 10.0.1.0/24

subnet 10.0.0.0/24

Arista 7050QX-32S

10.0.4.20

10.0.2.20

10.0.3.20

Phys. server B

AS 65100

VM B1 VM B2

10.0.5.20

10.0.4.2 10.0.5.2

subnet 10.0.5.0/24

subnet 10.0.4.0/24

Figure 4.8: Topology to analyze BGP. Mellanox and Arista are placed in two separate au-tonomous systems, each containing two subnets.

ResultsSONiC succesfully set up the two BGP sessions we configured. For example, listing 4.9 showsthe current BGP sessions Mellanox has with Arista on lines 8 and 9.

1 admin@sonic -mellanox :~$ show ip bgp summary

2 Command: sudo vtysh -c "show ip bgp summary"

3 BGP router identifier 10.1.0.10 , local AS number 65000

4 RIB entries 21, using 2352 bytes of memory

5 Peers 2, using 9312 bytes of memory

67 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/

PfxRcd

8 10.0.2.20 4 65100 34 37 0 0 0 00:30:30 8

9 10.0.3.20 4 65100 38 41 0 0 0 00:30:27 8

1011 Total number of neighbors 2

Listing 4.9: BGP neighbors on Mellanox, showing the two sessions with Arista.

In addition, Mellanox and Arista successfully exchanged routing information regarding the sub-nets that were directly connected to them. Mellanox propagated the subnets 10.0.0.0/24 and10.0.1.0/24 to Arista and Arista propagated the subnets 10.0.4.0/24 and 10.0.5.0/24 toMellanox. For instance, listing 4.10 shows that subnets 10.0.4.0/24 (line 11) and 10.0.5.0/24

(line 13) are present in the routing table on Mellanox, and that these subnets can be reachedthrough both 10.0.2.20 (via interface Ethernet32) and 10.0.3.20 (via interface Ethernet36)and that this reachability was sourced from BGP. Similarly, in the Arista routing table, subnets10.0.0.0/24 and 10.0.1.0/24 are present and can be reached through 10.0.2.10 (via inter-face Ethernet16) and 10.0.3.10 (via interface Ethernet20). Again, these routes were learnedthrough BGP.

27

1 admin@sonic -mellanox :~$ show ip route

2 Command: sudo vtysh -c "show ip route"

3 Codes: K - kernel route , C - connected , S - static , R - RIP ,

4 O - OSPF , I - IS-IS , B - BGP , P - PIM , A - Babel ,

5 > - selected route , * - FIB route

67 C>* 10.0.0.0/24 is directly connected , Ethernet56

8 C>* 10.0.1.0/24 is directly connected , Ethernet60

9 C>* 10.0.2.0/24 is directly connected , Ethernet32

10 C>* 10.0.3.0/24 is directly connected , Ethernet36

11 B>* 10.0.4.0/24 [20/0] via 10.0.2.20 , Ethernet32 , src 10.1.0.10 , 00:30:05

12 * via 10.0.3.20 , Ethernet36 , src 10.1.0.10 , 00:30:05

13 B>* 10.0.5.0/24 [20/0] via 10.0.2.20 , Ethernet32 , src 10.1.0.10 , 00:30:05

14 * via 10.0.3.20 , Ethernet36 , src 10.1.0.10 , 00:30:05

15 C>* 10.1.0.10/32 is directly connected , lo

16 B>* 10.1.0.20/32 [20/0] via 10.0.2.20 , Ethernet32 , src 10.1.0.10 , 00:30:05

17 * via 10.0.3.20 , Ethernet36 , src 10.1.0.10 , 00:30:05

18 # (...)

Listing 4.10: Routing table on Mellanox, showing the routes to the subnets we configured before(some output was omitted for brevity).

Ultimately, to complete the verification of BGP, we used traceroute between the virtual ma-chines to determine whether communication is possible and if yes, what routes are taken. Listing4.11 shows that VM A1 and VM B2 are able to communicate through Mellanox and Arista. First,a packet goes through the connected interface on the Mellanox device (10.0.0.10) (line 3). Then,it is routed to the Arista device (10.1.0.20) (line 4), which routes it via its interface in subnet10.0.5.0/24. SONiC uses the loopback address configured in config db.json as Router ID inthe BGP sessions, which for Mellanox is 10.1.0.10/32 and for Arista 10.1.0.20/32.

1 vagrant@vmA1 :~$ sudo traceroute 10.0.5.2

2 traceroute to 10.0.5.2 (10.0.5.2) , 30 hops max , 60 byte packets

3 1 10.0.0.10 (10.0.0.10) 0.333 ms 0.229 ms 0.258 ms

4 2 10.1.0.20 (10.1.0.20) 0.381 ms 0.264 ms 0.250 ms

5 3 10.0.5.2 (10.0.5.2) 0.523 ms 0.416 ms 0.312 ms

Listing 4.11: traceroute from VM A1 to VM B2.

4.2.3 Result overview

Table 4.1 presents an overview of the obtained results, including several remarks.

Feature Results CommentsLLDP Pass -LACP Pass L2 link aggregation not workingSTP Fail Not supported by SONiC; packets droppedVLAN (trunk) Pass -Inter-VLAN routing Pass -BGP Pass -

Table 4.1: An overview of the obtained results.

28

CHAPTER 5

Discussion

The performed experiments provided insights into the shortcomings of several fundamental net-working features when deployed on open switches with SONiC. Aside from commenting on someof the obtained results, this chapter will provide general insights into the ease of use of openswitches with SONiC.

LACP

The first notable observation in our LACP experiments was that none of our port channels wereshowing up when issuing the command show interfaces portchannel. This command shouldshow all configured port channels in SONiC. It turns out SONiC takes the configuration ofminigraph.xml, which is a deprecated configuration method of SONiC, for the command show

interfaces portchannel1 instead of the main configuration file config db.json.The Layer 3 port channel we set up in the experiments has shown that it is possible to

deploy a Layer 3 port channel that provides redundant communication between the devicesrunning SONiC. In order to get this working, aside from configuring the port channel in SONiCsconfiguration file, we had to edit SONiCs teamd template2. By default, SONiC sets the fieldmin ports, which is the minimum amount of active ports in a port channel for the port channel tobe operational, to ceil(0.75∗number of port channel members), which was 2 in our topology.In order to test the redundancy of our port channel, we set min ports to 1.

BGP

Similarly, successful BGP functionality was implied by the experiments we performed. Configur-ing BGP neighbors and AS numbers was done in config db.json. Nonetheless, we had to editthe BGP configuration template3 of SONiC to specify that BGP should be configured such thatit also advertises directly connected networks (which was the case in our topology) and not onlynetworks obtained from other BGP neighbors.

5.1 Ease of use

SONiC allowed for straightforward configuration in the config db.json configuration file. Sev-eral examples of configuration have been shown in this thesis already. Moreover, SONiC providedsome CLI configuration commands but not for all features we tested. For instance, configurationcommands could be used to shut down interfaces and BGP sessions, but not to set interfaceaddresses or configure BGP peers. Command-line configuration also allowed the user to set upVLANs and add or remove interfaces. In general, the command-line configuration provided asmall subset of configuration possibilities compared to configuration in config db.json. For

1https://github.com/Azure/sonic-utilities/blob/master/scripts/teamshow2https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-teamd/teamd.j23https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-fpm-quagga/bgpd.conf.j2

29

users that prefer command-line configuration this might make SONiC less straightforward touse.

SONiC did provide a wide variety of commands to view current configuration or device sta-tus. Several examples have been provided in this research. For all supported (thus not STP)features we tested there existed one or more corresponding show commands to view configu-ration or state information. One exception, as mentioned before, was that show interfaces

portchannel takes its information from a deprecated configuration method to show the config-ured port channels.

In short, the ease of use of SONiC depends on the preferences of the user. Users that preferworking in a configuration file will probably experience SONiC as a user-friendly NOS, whileusers that prefer CLI configuration might be less comfortable with SONiCs configuration.

30

CHAPTER 6

Use case scenarios for open switches

The use of open switches with SONiC can be beneficial in various use cases. For example, SONiCallows the user to look into the source code and then use or extend it for a certain objective. Inthis chapter, we briefly examine a current use case that exploits the properties of open switcheswith SONiC, and we provide two example use cases ourselves.

6.1 Current use cases

Microsoft Global Cloud

Microsoft developed SONiC for use in their Microsoft Global Cloud. Since Microsoft runs one ofthe largest networks in the world, Microsoft sets out strict requirements for the technology thatit deploys. One of these requirements is that new features can be implemented without havingan impact on the end users. Since SONiC consists of several containers, each containing theresources to deploy a certain networking feature (e.g. BGP, LAG), only one container needs tobe updated when there is an update or a bug fix for a certain feature, instead of replacing thewhole switch image (which will result in data plane downtime). This makes SONiC suitable foruse cases in which no downtime is permitted while deploying updates [13].

Another requirement is that SONiC can be used on the newest and most innovated hardwareplatforms. Because SONiC uses SAI, a data center can constantly innovate with newer switchhardware without having to change the software stack. Regardless of what switch hardware isused, as long as SAI is supported, all switches can be configured the same way, for their softwarestacks can be identical. Operators are thus able to preserve their software investments whilekeeping up with hardware-wise innovation. Thus, SONiC is suitable for use cases in which theremay be regular changes in hardware [13].

In addition, Microsoft states the requirement that cloud-scale deep telemetry and fully au-tomated failure mitigation has to be utilized in their cloud. Since SONiC has innovations suchas NetBouncer and Everflow available, SONiC meets these requirements. NetBouncer can bedeployed to automatically detect faulty devices or links within a large data center accurately[27]. Everflow debugs several network faults such as packet drops or loops. Furthermore, it canquickly identify devices that cause high latency in a network [28]. In short, SONiC is suitablefor use cases in which large networks have to be monitored constantly to automatically respondto potential problems.

6.2 Example use cases

In addition to the above use case of SONiC in the Microsoft Global Cloud, we provide twoexample use cases in which the open nature of SONiC can be deployed for useful applications.

31

Plug-and-play VLAN

The first use case relates to companies that want to provide flexible working spots to theiremployees, who in turn own a laptop provided by the company. On different days the sameemployee might want to work on different physical locations within the company office. To gainaccess to the company’s network, the employees can use Ethernet cables to connect their laptopswith the network. Since generally a company consists of several departments with partitionedand isolated computer networks (VLANs) and an employee from any department can sit andconnect anywhere, how does the network decide to which VLAN the newly connected laptopshould be added? Employees want to start working right away after connecting to the network,so it is preferred their laptops are added to the correct VLAN without intervention by the networkadministrator.

If the company network consists of open switches running SONiC, the network administratorcould decide to develop an application that automatically places newly connected hosts in aparticular VLAN, depending on the MAC address of the host, so that the VLAN functionalityon the switch is essentially plug-and-play. That is, no manual configuration is needed every timea new host connects.

This can be implemented as follows. Via SONiCs logs or LLDP entries, newly connected hosts(including their MAC address) can be discovered by the application. The network administratorcan define a policy that states the relation between the host MAC address and the VLANthat host will be placed in, if at all. This is illustrated in the right side of figure 6.1. Forinstance, the application can decide to block hosts with unknown MAC addresses, or place hostsmanufactured by the same vendor in the same VLAN (MAC addresses are vendor-specific). Next,the application can simply edit SONiCs config db.json configuration file to change the VLANsettings.

We suppose that the network administrator has a list of all company laptops in use by theemployees, including their MAC addresses. The network administrator can easily define a policythat maps each laptop to a certain department (VLAN). This way, no matter where an employeeconnects its laptop to the network, the SONiC switch will detect the MAC address of this laptopand determine in which VLAN the switch interface this new host is connected to should beplaced. Similarly, if a laptop not belonging to one of the employees uses one of the Ethernetcables, its MAC address is not recognized and can be blocked by the switch.

Host detection oninterface y

SONiC Policy table

Laptop with MAC x

y, MAC x

Add interface y toVLAN z

Block

y, VLAN z

Block

Other laptops

Office

Switch

Figure 6.1: An overview of the plug-and-play VLAN application. The switch detects a new hostwith MAC address x on switch interface y. The policy table maps from MAC address x to VLANnumber z, after which the application places interface y in VLAN z.

32

Automatic port channel

Our second use case focuses on data centers that need to provide a redundant network, becausethey handle important data that must reach its destination. The network administrator candecide to aggregate all connections between each switch in the data center, to provide redundancy.Such a data center may have a large number of switches, and configuring link aggregation forthe connections between these switches can by time-consuming.

LACP provides a way to do this, because it is able to automatically negotiate between twodevices to set up a port channel. However, before this can happen, the network administratorneeds to specify which switch interfaces belong to which port channel, if at all. The applicationin the next paragraph proposes a more flexible alternative for this LACP feature by allowingthe network administrator to connect neighbors to any interface on a switch without the needof specifying that these interfaces will be used to create a port channel. The application willdynamically determine whether a port channel should be set up, and if so, with which interfaces.

If the data center contains open switches with SONiC, the network administrator can developan application that automatically sets up a port channel if it discovers that the switch has multiplephysical links with the same neighbor (for instance by comparing MAC addresses or Chassis IDsof all connected devices). This way, when the network administrator connects two switches withmultiple physical links, no further configuration is necessary to set up the port channel.

y, PC z

y, ID xNeighbor detectionon interface y

SONiC Policy table

Switch with ID x

Create new portchannel z

Add to existing portchannel z

Other switches

Datacenter

Switch

No port channelmodifications

Figure 6.2: An overview of the automatic port channel application. The switch detects a newneighbor with Chassis ID x on switch interface y. Using neighbor and port channel information,the application decides whether 1) there should be no port channel modifications because thenew link is the only link with the neighbor device, or 2) a new port channel z should be createdwith interface y and the other interface already connected to the same neighbor, or 3) the newinterface y should be added to port channel z because there already is a port channel with theneighbor.

When the switch detects a newly connected neighbor, the application uses neighbor and portchannel information to determine whether there is already a port channel with this neighbor andif not, whether there is already a single link with this neighbor, for instance by comparing theChassis ID of all connected neighbors. Figure 6.2 shows how this determination process. Thereare three possible cases:

1. There is no other link with the neighbor device.

2. There is one other link with the neighbor device.

33

3. There is already a port channel with the neighbor device.

In case 1, no port channel needs to be created since it would not provide redundancy. In case 2,a port channel needs to be created with the interface of the already present link and the interfaceof the new link. In case 3, the interface of the new link needs to be added to the port channelthat is already present with the neighbor. Editing port channels can be done by editing SONiCsconfiguration file. To set up a working port channel, the neighbor device will also have to gothrough this same process to set up the port channel on the neighbors side. Thus, the applicationmust run on both devices.

34

CHAPTER 7

Conclusions

Open switches play an increasingly significant role in innovative networks. We have examinedthe deployment of several networking features that we believe are essential for a network switch,and did so using multiple topologies with open switches to simulate different scenarios. In thediscussion, we provided remarks about some experimental results and shed some light on theease of use of a SONiC switch. In the last chapter, we briefly examined a current use case ofopen switches with SONiC and provided two possible use cases ourselves.

We can conclude that the networking features that we tested and SONiC states to support,LLDP, LACP (Layer 3), VLAN trunking, inter-VLAN routing and BGP, can in fact be success-fully deployed on open switches with SONiC. In the discussion, we stated that some of thesefeatures required additional configuration, but the features were deployed successfully after all.

In addition, we can conclude that open switches with SONiC are suitable for use cases in largedata centers as well as in enterprise networks, because SONiC allows the network administratorto update features without data plane downtime. In addition, the network administrator is ableto keep up with hardware innovations while no changes to the software stack need to be made.Also, SONiC provides innovative applications to monitor the network and automatically detectnetwork failures. The openness of SONiC gives the network administrator more possibilities tocustomize and extend software on the switch.

7.1 Future research

In future research, more (different) open switches can be used to experiment with. Examplesare several Dell and Edgecore devices, which are supported by SONiC. This would allow usto set up more interesting and realistic network topologies to test features with. In addition,since SONiC develops constantly, retesting with SONiC after a certain amount of time could inprinciple provide different results.

Furthermore, other open-source network operating systems such as OPX can be tested as well,to allow for a comparison with SONiC. Interoperability between SONiC and traditional, closednetwork operating systems can also be performed in future research, since a network generallyconsists of devices with different operating systems.

Lastly, future research can be done on the use cases of SONiC. Additional use cases can beexamined and the described example use cases can be implemented and tested.

35

36

CHAPTER 8

Acknowledgements

I would like to thank my supervisors, Paola Grosso and Lukasz Makowski, for their support andguidance throughout this research and the writing of this thesis. Moreover, I would like to thankthe SNE OpenLab for providing the devices that were required in this research.

37

38

CHAPTER 9

Bibliography

[1] Neil Briscoe. “Understanding the OSI 7-layer model”. In: PC Network Advisor 120.2 (2000).

[2] Zeus Kerravala. White boxes are now ready for prime time. Consulted on 03-04-2018. 2016.url: https://www.networkworld.com/article/3100927/network- switch/white-boxes-are-now-ready-for-prime-time.html.

[3] Mike Sheldon. The future of networking: Its in a white box. Consulted on 03-04-2018. 2017.url: https://www.networkworld.com/article/3182138/hardware/the-future-of-networking-its-in-a-white-box.html.

[4] Margaret Rouse. ASIC (application-specific integrated circuit). Consulted on 09-04-2018.2005. url: https://whatis.techtarget.com/definition/ASIC-application-specific-integrated-circuit.

[5] Microsoft Azure. Software for Open Networking in the Cloud SONiC. Consulted on 03-04-2018. 2018. url: http://azure.github.io/SONiC/.

[6] Ivan Pepelnjak. Management, Control and Data Planes in Network Devices and Systems.Consulted on 09-04-2018. 2013. url: http://blog.ipspace.net/2013/08/management-control-and-data-planes-in.html.

[7] Jeff Doyle. Clearing the fog around open switching terminology. Consulted on 09-04-2018.2015. url: https : / / www . networkworld . com / article / 2919599 / cisco - subnet /

clearing-the-fog-around-open-switching-terminology.html.

[8] Kamala Subramaniam. Switch Abstraction Interface (SAI) officially accepted by the OpenCompute Project (OCP). Consulted on 12-04-2018. 2015. url: https://azure.microsoft.com/en-us/blog/switch-abstraction-interface-sai-officially-accepted-by-

the-open-compute-project-ocp/.

[9] Mellanox Technologies. Switch Software Development Kit. Consulted on 14-04-2018. 2018.url: http://www.mellanox.com/page/products_dyn?product_family=124&mtag=switchx_sdk.

[10] Broadcom. OpenNSL. Consulted on 14-04-2018. 2018. url: https://www.broadcom.com/products/ethernet-connectivity/software/opennsl/.

[11] Lihua Yuan. Features and Roadmap. Consulted on 16-04-2018. 2018. url: https : / /

github.com/Azure/SONiC/wiki/Features-and-Roadmap.

[12] Microsoft Azure. Architecture. Consulted on 16-04-2018. 2017. url: https://github.com/Azure/SONiC/wiki/Architecture.

[13] Yousef Khalidi. SONiC: The networking switch software that powers the Microsoft GlobalCloud. Consulted on 01-06-2018. 2017. url: https://azure.microsoft.com/en-us/blog / sonic - the - networking - switch - software - that - powers - the - microsoft -

global-cloud/.

39

[14] James F. Kurose and Keith W. Ross. Computer Networking: A Top-Down Approach. Sixthedition. Pearson, 2013.

[15] “IEEE Standard for Local and metropolitan area networks - Station and Media Access Con-trol Connectivity Discovery”. In: IEEE Std 802.1AB-2016 (Revision of IEEE Std 802.1AB-2009) (2016), pp. 1–146. doi: 10.1109/IEEESTD.2016.7433915.

[16] Mick Seaman. “Link aggregation control protocol”. In: IEEE http://grouper. ieee. org/-groups/802/3/ad/public/mar99/seaman 1 (1999), p. 0399.

[17] Alan Elder and Jonathan Harrison. Spanning tree protocol. US Patent App. 14/673,652.2015.

[18] “IEEE Standard for Local and metropolitan area networks: Media Access Control (MAC)Bridges”. In: IEEE Std 802.1D-2004 (Revision of IEEE Std 802.1D-1998) (2004), pp. 1–281. doi: 10.1109/IEEESTD.2004.94569.

[19] Spanning Tree Concepts. Consulted on 06-06-2018. 2016. url: https://seeseenayy.

blogspot.com/2016/09/ccnav3-chapter-2-notes-spanning-tree.html.

[20] David J Husak. Direct addressing between VLAN subnets. US Patent 6,157,647. 2000.

[21] ExamCollection. How to verify and configure VLANs trunking. Consulted on 03-05-2018.2018. url: https : / / www . examcollection . com / certification - training / ccnp -

configure-and-verify-vlans-and-trunking.html.

[22] CNNA Blog. Introduction to inter-vlan routing. Consulted on 06-06-2018. 2018. url: http://www.ccnablog.com/inter-vlan-routing/.

[23] Kunihiro Ishiguro et al. System Architecture. Consulted on 16-04-2018. 2005. url: https://www.quagga.net/docs/quagga.html#System-Architecture.

[24] Onsel Kuluk. BGP in the Data Center: Part One. Consulted on 07-06-2018. 2018. url:https://www.packetdesign.com/blog/bgp-in-the-data-center-part-1/.

[25] Cumulus Networks. Open Network Install Environment. Consulted on 19-04-2018. 2018.url: https://opencomputeproject.github.io/onie/.

[26] Arista Networks. Boot Loader Aboot. Consulted on 20-04-2018. 2018. url: https://www.arista.com/ko/um-eos/eos-6-1-boot-loader--aboot.

[27] Microsoft. CloudBrain for Automatic Troubleshooting for the Cloud. Consulted on 05-06-2018. 2016. url: https://www.microsoft.com/en-us/research/project/cloudbrain/.

[28] et al. Yibo Zhu. “Packet-level telemetry in large datacenter networks”. In: ACM SIGCOMMComputer Communication Review. Vol. 45. 4. ACM. 2015, pp. 479–491.

40

CHAPTER 10

Appendix

A: Platform summary

Listings 10.1 and 10.2 show a platform summary of Mellanox SN2100 and Arista 7050QX-32S,respectively.

Mellanox

admin@sonic−mellanox : ˜ $ show plat form summaryPlatform : x86 64−mlnx msn2100−r0HwSKU: ACS−MSN2100ASIC : mellanox

Listing 10.1: Mellanox SN2100 device information.

Arista

admin@sonic−a r i s t a : ˜ $ show plat form summaryPlatform : x86 64−a r i s t a 7 0 5 0 q x 3 2 sHwSKU: Arista −7050−QX−32SASIC : broadcom

Listing 10.2: Arista 7050QX-32S device information.

41

B: Software information

B1: SONiC version

Listings 10.3 and 10.4 show information about the versions of SONiC that were used on MellanoxSN2100 and Arista 7050QX-32S, respectively.

Mellanox

admin@sonic−mellanox : ˜ $ show v e r s i o nSONiC Software Vers ion : SONiC .HEAD.574− ed915e3D i s t r i b u t i o n : Debian 8 .10Kernel : 3.16.0−5−amd64Build commit : ed915e3Build date : Wed Apr 4 06 : 5 8 : 48 UTC 2018Bu i l t by : johnar@jenkins−worker−3

Docker images :REPOSITORY TAG IMAGE ID SIZEdocker−orchagent−mlnx HEAD.574− ed915e3 32203598 a5bc 287 MBdocker−orchagent−mlnx l a t e s t 32203598 a5bc 287 MBdocker−syncd−mlnx HEAD.574− ed915e3 b7ab299d2758 350 .9 MBdocker−syncd−mlnx l a t e s t b7ab299d2758 350 .9 MBdocker−l ldp−sv2 HEAD.574− ed915e3 d8f65d12b406 297 .2 MBdocker−l ldp−sv2 l a t e s t d8f65d12b406 297 .2 MBdocker−dhcp−r e l a y HEAD.574− ed915e3 46 c2a839b5ba 280 .1 MBdocker−dhcp−r e l a y l a t e s t 46 c2a839b5ba 280 .1 MBdocker−database HEAD.574− ed915e3 622 f0 f354847 278 .8 MBdocker−database l a t e s t 622 f0 f354847 278 .8 MBdocker−teamd HEAD.574− ed915e3 9dd25e367798 284 .1 MBdocker−teamd l a t e s t 9dd25e367798 284 .1 MBdocker−snmp−sv2 HEAD.574− ed915e3 4 b4277d6cc32 319 .3 MBdocker−snmp−sv2 l a t e s t 4 b4277d6cc32 319 .3 MBdocker−router−a d v e r t i s e r HEAD.574− ed915e3 6 c2adf7743ec 276 .4 MBdocker−router−a d v e r t i s e r l a t e s t 6 c2adf7743ec 276 .4 MBdocker−platform−monitor HEAD.574− ed915e3 9 b31194f f812 298 .3 MBdocker−platform−monitor l a t e s t 9 b31194f f812 298 .3 MBdocker−fpm−quagga HEAD.574− ed915e3 67 c77e fc3d4a 290 .6 MBdocker−fpm−quagga l a t e s t 67 c77e fc3d4a 290 .6 MB

Listing 10.3: SONiC version information on Mellanox SN2100.

42

Arista

admin@sonic−a r i s t a : ˜ $ show ve r s i onSONiC Software Vers ion : SONiC .HEAD.547−4754b43D i s t r i b u t i o n : Debian 8 .10Kernel : 3.16.0−5−amd64Build commit : 4754 b43Build date : Fr i Apr 6 07 : 38 : 17 UTC 2018Bu i l t by : johnar@jenkins−worker−4

Docker images :REPOSITORY TAG IMAGE ID SIZEdocker−syncd−brcm HEAD.547−4754b43 145 a93bf2613 358 .1 MBdocker−syncd−brcm l a t e s t 145 a93bf2613 358 .1 MBdocker−orchagent−brcm HEAD.547−4754b43 32 fdab0a0a85 287 MBdocker−orchagent−brcm l a t e s t 32 fdab0a0a85 287 MBdocker−l ldp−sv2 HEAD.547−4754b43 9 ce7dc0 f55 f6 297 .2 MBdocker−l ldp−sv2 l a t e s t 9 ce7dc0 f55 f6 297 .2 MBdocker−dhcp−r e l a y HEAD.547−4754b43 17 fd00cd2091 280 .1 MBdocker−dhcp−r e l a y l a t e s t 17 fd00cd2091 280 .1 MBdocker−database HEAD.547−4754b43 5 af52a038baf 278 .8 MBdocker−database l a t e s t 5 a f52a038baf 278 .8 MBdocker−teamd HEAD.547−4754b43 24 d8a02873a7 284 .1 MBdocker−teamd l a t e s t 24 d8a02873a7 284 .1 MBdocker−snmp−sv2 HEAD.547−4754b43 129 e2b96b2c4 319 .3 MBdocker−snmp−sv2 l a t e s t 129 e2b96b2c4 319 .3 MBdocker−router−a d v e r t i s e r HEAD.547−4754b43 58 be616fcdb1 276 .4 MBdocker−router−a d v e r t i s e r l a t e s t 58 be616fcdb1 276 .4 MBdocker−platform−monitor HEAD.547−4754b43 6b68d76ae87b 298 .3 MBdocker−platform−monitor l a t e s t 6b68d76ae87b 298 .3 MBdocker−fpm−quagga HEAD.547−4754b43 0 f f 2 1 6 0 2 1 f e a 290 .6 MBdocker−fpm−quagga l a t e s t 0 f f 2 1 6 0 2 1 f e a 290 .6 MB

Listing 10.4: SONiC version information on Arista 7050QX-32S.

43

B2: Vagrant version

Listing 10.5 specifies the version of Vagrant used in the experiments.

root@hosta :˜/ v a g r a n t f i l e s# vagrant ve r s i onI n s t a l l e d Vers ion : 2 . 0 . 4Lates t Vers ion : 2 . 1 . 1

root@hostb :˜/ v a g r a n t f i l e s# vagrant ve r s i onI n s t a l l e d Vers ion : 2 . 0 . 4Lates t Vers ion : 2 . 1 . 1

Listing 10.5: Vagrant version used on servers A and B.

B3: VirtualBox version

Listing 10.6 specifies the version of VirtualBox used in the experiments.

root@hosta :˜/ v a g r a n t f i l e s# vboxmanage −−ve r s i on5 . 2 . 1 0 r122088

root@hostb :˜/ v a g r a n t f i l e s# vboxmanage −−ve r s i on5 . 2 . 1 0 Debianr121806

Listing 10.6: VirtualBox version used on servers A and B.

44

C: Configurations

This section lists the relevant configurations for each experiment in config db.json. Completeconfiguration files can be found in the GitHub repository1.

C1.1: LACP (L2)

Mellanox

"PORTCHANNEL_INTERFACE ": {

"PortChannel1 |": {}

},

"PORTCHANNEL ": {

"PortChannel1 ": {

"members ": [

"Ethernet32",

"Ethernet36"

]

}

},

Arista

"PORTCHANNEL_INTERFACE ": {

"PortChannel1 |": {}

},

"PORTCHANNEL ": {

"PortChannel1 ": {

"members ": [

"Ethernet16",

"Ethernet20"

]

}

},

1https://github.com/erikpu/white-label-configurations

45

C1.2: LACP (L3)

Mellanox

"PORTCHANNEL_INTERFACE ": {

"PortChannel1 |172.16.0.10/24": {}

},

"PORTCHANNEL ": {

"PortChannel1 ": {

"members ": [

"Ethernet32",

"Ethernet36"

]

}

},

Arista

"PORTCHANNEL_INTERFACE ": {

"PortChannel1 |172.16.0.20/24": {}

},

"PORTCHANNEL ": {

"PortChannel1 ": {

"members ": [

"Ethernet16",

"Ethernet20"

]

}

},

46

C2: STP

Mellanox

"VLAN": {

"Vlan100 ": {

"members ": [

"Ethernet32",

"Ethernet36"

],

"vlanid ": "100"

}

},

"VLAN_MEMBER ": {

"Vlan100|Ethernet32 ": {

"tagging_mode ": "untagged"

},

"Vlan100|Ethernet36 ": {

"tagging_mode ": "untagged"

}

},

Arista

"VLAN": {

"Vlan100 ": {

"members ": [

"Ethernet16",

"Ethernet20"

],

"vlanid ": "100"

}

},

"VLAN_MEMBER ": {

"Vlan100|Ethernet16 ": {

"tagging_mode ": "untagged"

},

"Vlan100|Ethernet20 ": {

"tagging_mode ": "untagged"

}

},

47

C3: VLAN (trunking)

Mellanox

"VLAN": {

"Vlan100 ": {

"members ": [

"Ethernet56",

"Ethernet32"

],

"vlanid ": "100"

},

"Vlan200 ": {

"members ": [

"Ethernet60",

"Ethernet32"

],

"vlanid ": "200"

}

},

"VLAN_MEMBER ": {

"Vlan100|Ethernet32 ": {

"tagging_mode ": "tagged"

},

"Vlan100|Ethernet56 ": {

"tagging_mode ": "untagged"

},

"Vlan200|Ethernet32 ": {

"tagging_mode ": "tagged"

},

"Vlan200|Ethernet60 ": {

"tagging_mode ": "untagged"

}

},

48

Arista

"VLAN": {

"Vlan100 ": {

"members ": [

"Ethernet40",

"Ethernet16"

],

"vlanid ": "100"

},

"Vlan200 ": {

"members ": [

"Ethernet44",

"Ethernet16"

],

"vlanid ": "200"

}

},

"VLAN_MEMBER ": {

"Vlan100|Ethernet40 ": {

"tagging_mode ": "untagged"

},

"Vlan100|Ethernet16 ": {

"tagging_mode ": "tagged"

},

"Vlan200|Ethernet44 ": {

"tagging_mode ": "untagged"

},

"Vlan200|Ethernet16 ": {

"tagging_mode ": "tagged"

}

},

C4: Inter-VLAN routing

Mellanox

"VLAN_INTERFACE ": {

"Vlan100 |172.16.100.3/24": {},

"Vlan200 |172.16.200.3/24": {}

},

Arista

"VLAN_INTERFACE ": {

"Vlan100 |172.16.100.4/24": {},

"Vlan200 |172.16.200.4/24": {}

},

49

C5: BGP

Mellanox

"BGP_NEIGHBOR ": {

"10.0.2.20": {

"name": "Arista",

"local_addr ": "10.0.2.10" ,

"asn": "65100"

},

"10.0.3.20": {

"name": "Arista",

"local_addr ": "10.0.3.10" ,

"asn": "65100"

}

},

Arista

"BGP_NEIGHBOR ": {

"10.0.2.10": {

"name": "Mellanox",

"local_addr ": "10.0.2.20" ,

"asn": "65000"

},

"10.0.3.10": {

"name": "Mellanox",

"local_addr ": "10.0.3.20" ,

"asn": "65000"

}

},

50

D: Output

This section contains complete output of commands used in the experiments. Output that wastoo extensive to be placed in the report itself can be found in this section.

D1: LLDP

Mellanox

1 admin@sonic -mellanox :~$ show lldp neighbors

2 Command: sudo lldpctl

3 -------------------------------------------------------------------------------

4 LLDP neighbors:

5 -------------------------------------------------------------------------------

6 Interface: Ethernet32 , via: LLDP , RID: 1, Time: 0 day , 00:04:48

7 Chassis:

8 ChassisID: mac 00:1c:73:7b:f7:5c

9 SysName: sonic -arista

10 SysDescr: Debian GNU/Linux 8 (jessie) Linux 3.16.0-5- amd64 #1 SMP Debian

3.16.51 -3+ deb8u1 (2018 -01 -08) x86_64

11 TTL: 120

12 MgmtIP: 10.1.0.32

13 MgmtIP: fe80 ::21c:73ff:fe7b:f75c

14 Capability: Bridge , on

15 Capability: Router , on

16 Capability: Wlan , off

17 Capability: Station , off

18 Port:

19 PortID: local Ethernet16

20 PortDescr: Ethernet16

21 -------------------------------------------------------------------------------

22 Interface: Ethernet36 , via: LLDP , RID: 1, Time: 0 day , 00:04:47

23 Chassis:

24 ChassisID: mac 00:1c:73:7b:f7:5c

25 SysName: sonic -arista

26 SysDescr: Debian GNU/Linux 8 (jessie) Linux 3.16.0-5- amd64 #1 SMP Debian

3.16.51 -3+ deb8u1 (2018 -01 -08) x86_64

27 TTL: 120

28 MgmtIP: 10.1.0.32

29 MgmtIP: fe80 ::21c:73ff:fe7b:f75c

30 Capability: Bridge , on

31 Capability: Router , on

32 Capability: Wlan , off

33 Capability: Station , off

34 Port:

35 PortID: local Ethernet20

36 PortDescr: Ethernet20

37 -------------------------------------------------------------------------------

LLDP neighbors on Mellanox.

51

Arista

1 admin@sonic -arista :~$ show lldp neighbors

2 Command: sudo lldpctl

3 -------------------------------------------------------------------------------

4 LLDP neighbors:

5 -------------------------------------------------------------------------------

6 Interface: Ethernet20 , via: LLDP , RID: 1, Time: 0 day , 00:05:42

7 Chassis:

8 ChassisID: mac ec:0d:9a:8d:f1:e4

9 SysName: sonic -mellanox

10 SysDescr: Debian GNU/Linux 8 (jessie) Linux 3.16.0-5- amd64 #1 SMP Debian

3.16.51 -3+ deb8u1 (2018 -01 -08) x86_64

11 TTL: 120

12 MgmtIP: 10.1.0.32

13 MgmtIP: fe80::ee0d:9aff:fe8d:f1e4

14 Capability: Bridge , on

15 Capability: Router , on

16 Capability: Wlan , off

17 Capability: Station , off

18 Port:

19 PortID: local Ethernet36

20 PortDescr: Ethernet36

21 -------------------------------------------------------------------------------

22 Interface: Ethernet16 , via: LLDP , RID: 1, Time: 0 day , 00:05:44

23 Chassis:

24 ChassisID: mac ec:0d:9a:8d:f1:e4

25 SysName: sonic -mellanox

26 SysDescr: Debian GNU/Linux 8 (jessie) Linux 3.16.0-5- amd64 #1 SMP Debian

3.16.51 -3+ deb8u1 (2018 -01 -08) x86_64

27 TTL: 120

28 MgmtIP: 10.1.0.32

29 MgmtIP: fe80::ee0d:9aff:fe8d:f1e4

30 Capability: Bridge , on

31 Capability: Router , on

32 Capability: Wlan , off

33 Capability: Station , off

34 Port:

35 PortID: local Ethernet32

36 PortDescr: Ethernet32

37 -------------------------------------------------------------------------------

LLDP neighbors on Arista.

52

D2: LACP (L3) redundancy

1 # (both links up)

23 admin@sonic -mellanox :~$ show interfaces status

4 Command: intfutil status

5 Interface Lanes Speed MTU Alias Oper Admin

6 ----------- ----------- ------- ----- ---------- ------ -------

7 # (...)

8 Ethernet32 32,33,34,35 N/A 9100 Ethernet32 up up

9 Ethernet36 36,37,38,39 N/A 9100 Ethernet36 up up

10 # (...)

1112 # (communication is fine)

1314 admin@sonic -mellanox :~$ sudo ping 172.16.0.20

15 PING 172.16.0.20 (172.16.0.20) 56(84) bytes of data.

16 64 bytes from 172.16.0.20: icmp_seq =1 ttl=64 time =0.362 ms

17 64 bytes from 172.16.0.20: icmp_seq =2 ttl=64 time =0.220 ms

18 64 bytes from 172.16.0.20: icmp_seq =3 ttl=64 time =0.320 ms

19 ^C

20 --- 172.16.0.20 ping statistics ---

21 3 packets transmitted , 3 received , 0% packet loss , time 1998ms

22 rtt min/avg/max/mdev = 0.220/0.300/0.362/0.062 ms

2324 # (link failure Ethernet32 !)

2526 admin@sonic -mellanox :~$ show interfaces status

27 Command: intfutil status

28 Interface Lanes Speed MTU Alias Oper Admin

29 ----------- ----------- ------- ----- ---------- ------ -------

30 # (...)

31 Ethernet32 32,33,34,35 N/A 9100 Ethernet32 down up

32 Ethernet36 36,37,38,39 N/A 9100 Ethernet36 up up

33 # (...)

3435 # (PortChannel1 still UP)

3637 admin@sonic -mellanox :~$ ip a show dev PortChannel1

38 4: PortChannel1: <BROADCAST ,MULTICAST ,UP,LOWER_UP > mtu 9100 qdisc noqueue state UP

group default

39 link/ether ec:0d:9a:8d:f1:c0 brd ff:ff:ff:ff:ff:ff

40 inet 172.16.0.10/24 brd 172.16.0.255 scope global PortChannel1

41 valid_lft forever preferred_lft forever

42 inet6 fe80::ee0d:9aff:fe8d:f1c0 /64 scope link

43 valid_lft forever preferred_lft forever

4445 # (communication still working)

4647 admin@sonic -mellanox :~$ sudo ping 172.16.0.20

48 PING 172.16.0.20 (172.16.0.20) 56(84) bytes of data.

49 64 bytes from 172.16.0.20: icmp_seq =1 ttl=64 time =0.294 ms

50 64 bytes from 172.16.0.20: icmp_seq =2 ttl=64 time =0.310 ms

51 64 bytes from 172.16.0.20: icmp_seq =3 ttl=64 time =0.296 ms

52 ^C

53 --- 172.16.0.20 ping statistics ---

54 3 packets transmitted , 3 received , 0% packet loss , time 1998ms

55 rtt min/avg/max/mdev = 0.294/0.300/0.310/0.007 ms

Behaviour of PortChannel1 when link Ethernet32 (Mellanox)-Ethernet16 (Arista) fails(comments are added with ’#’ for clarity). Some output was omitted for brevity.

53

D3: STP

Mellanox

admin@sonic -mellanox :~$ sudo brctl showstp Bridge

Bridge

bridge id 0064. ec0d9a8df1c0

designated root 0064. ec0d9a8df1c0

root port 0 path cost 0

max age 20.00 bridge max age 20.00

hello time 2.00 bridge hello time 2.00

forward delay 15.00 bridge forward delay 15.00

ageing time 300.00

hello timer 0.98 tcn timer 0.00

topology change timer 0.00 gc timer 57.79

flags

Ethernet32 (1)

port id 8001 state forwarding

designated root 0064. ec0d9a8df1c0 path cost 100

designated bridge 0064. ec0d9a8df1c0 message age timer 0.00

designated port 8001 forward delay timer 0.00

designated cost 0 hold timer 0.00

flags

Ethernet36 (2)

port id 8002 state forwarding

designated root 0064. ec0d9a8df1c0 path cost 100

designated bridge 0064. ec0d9a8df1c0 message age timer 0.00

designated port 8002 forward delay timer 0.00

designated cost 0 hold timer 0.00

flags

STP state on Mellanox: both ports in forwarding state.

Arista

admin@sonic -arista :~$ sudo brctl showstp Bridge

Bridge

bridge id 00c8.001 c737bf75c

designated root 00c8.001 c737bf75c

root port 0 path cost 0

max age 20.00 bridge max age 20.00

hello time 2.00 bridge hello time 2.00

forward delay 15.00 bridge forward delay 15.00

ageing time 300.00

hello timer 1.52 tcn timer 0.00

topology change timer 0.00 gc timer 161.18

flags

Ethernet16 (1)

port id 8001 state forwarding

designated root 00c8.001 c737bf75c path cost 100

designated bridge 00c8.001 c737bf75c message age timer 0.00

designated port 8001 forward delay timer 0.00

designated cost 0 hold timer 0.52

flags

Ethernet20 (2)

port id 8002 state forwarding

designated root 00c8.001 c737bf75c path cost 100

designated bridge 00c8.001 c737bf75c message age timer 0.00

designated port 8002 forward delay timer 0.00

designated cost 0 hold timer 0.52

flags

STP state on Arista: both ports in forwarding state.

54