ROUTEFLOW &

SOFTWARE-DEFINED EXCHANGES

Raphael Vicente Rosa

Christian Esteve Rothenberg

University of Campinas (UNICAMP), Brazil

Agenda

• Background:

• Software-Defined Networking (SDN)

• Separating Routing from Routers (BGP-based pre-SDN)

• RouteFlow: Software-Defined IP Routing

• Basics

• Cardigan deployment at live IXP (>1 year)

• Software-Defined Exchanges

• Research perspectives and open innovation

2

Background on SDN [5]

3

[5] "Software-Defined Networking: A Comprehensive Survey."

SDN AND OPEN SOURCE ARE EVOLVING AND

ACCELERATING THE PATH OF STANDARDIZATION

Components of the New Frontier

"We’d love to see OpenDaylight do for networking what Linux

has done for the computing industry."

SDN & NFV: The Frontier of Networking

Existing

• CLIs

• Closed Source

• Vendor Lead

• Classic Network

Appliances

New

• APIs

• Open Source

• Customer Lead

• Network Function

Virtualization (NFV)

Adapted from: Kyle Mestery, Next Generation Network Developer Skills

Separating Routing From Routers

• Routing Control Platform (2004) [1]

• Path computation based on a consistent view of network state

• Controlled interactions between routing protocol layers

• Expressive specification of routing policies

6

Figure 1 – Routing Control Platforms (RCPs) [1]

Redefinition of Inter-AS Routing

• Inter-AS Routing Issues

• Routing tables scalability (route entries)

• BGP policy definitions (communities, filter tags)

• Dynamic negotiation of network resources (e.g., bandwidth prices)

• …

• Complexity can be better managed

• Simpler routing configuration

• Network-wide traffic engineering

• Better network diagnostics and troubleshooting

• New interdomain routing protocols

7

RCP Challenges

• Robustness

• Consistent view: loop-free paths

• Scalability

• Session establishment, path computation, management

• Convergence speed

• Intra and Inter AS route computation

• Configuration languages

• Correctness and security

• Troubleshooting and diagnostics

• Detach control and data planes

8

RouteFlow: Basics

Data Plane

Control Plane

Linux

RouteFlow: Basics

RouteFlow [2]

• Background

• Glue of IP routing stacks with OpenFlow

• Controller-centric hybrid IP networks

• Migration path to SDN (e.g., Cardigan)

• Benefits:

• Cheap network gear with minimal embedded software

• Provide interoperability with legacy network elements

• Avoid re-writing legacy protocols in a centralized fashion

• Innovation power to stakeholders

• network operators, service providers

11

[2] “Revisiting routing control platforms with the eyes and muscles of software-defined networking”

RouteFlow: High Level Architecture

• Modular

• 3 components

• Hierarchical

• Distributed

• Multi-controller

• POX, Ryu, ODL

• Routing stacks

• Quagga, BIRD, XORP

12

Figure 2 – RouteFlow Architecture [2]

RouteFlow

13

Figure 3 – RouteFlow General Architecture

RouteFlow: SDX for Layer 3

• RouteFlow – an approach to build BGP in the controller • Create a virtual model of the router in the controller

• Forward BGP update packets to the virtual interface

• Compute BGP route tables (RIB and FIB) based on virtual model

• Translate FIB into Flow Tables based on real switch model

• Install Flow Tables in the data plane

• Build a SDN-based Router • vandevecken code fork on RouteFlow (higher performance, simpler, 1.3 capable)

• Build peering fabric based on dynamic SDN policies • rather than static network constructs

• Use SDN rules to scrub bad behavior* • Deal with Router hygiene issues (default deny instead of default allow).

• Deal with “Capacity Theft” issues (as well as “noise” traffic)

(Adapted from – Inder Monga NVS'14)

14

Abstracting SDX:

what are the common functions?

• A place where peering domains come together

• Peering domains need not be physically connected

• Supports establishment of inter-domain connectivity/routing

• A broker of inter-domain and inter-SRP trust

• Each SDX defines its rules of engagement

• Enforcer of individual domain policies

• Marketplace for Software Defined Services (SDS) can be built

as a SDX overlay

• For example, an application-specific exchange (a video marketplace)

• Resources may be virtual or physical

(Adapted from – Inder Monga NVS'14)

15

Internet eXchange Points (IXPs)

• It's a network fabric that participants connect to

• They advertise their networks

• Any packet you place into the exchange finds the appropriate

destination

• It is not a: router, switch, layer 3 switch…

• It's just a place you poke two things...

• Advertisements for things you'll accept

• and packets you want to get rid of

16

IXPs

17

Figure 4 – Typical IXP Architecture [3]

Anatomy of a large European IXP [3]

• A particular Internet ecosystem (2012) • Agnostic of any tier structure (flatenning 400 members)

• Peer-Peer links (more than 50,000)

• High-densely connected (visible / invisible / cannot tell P-P links)

• Prefix exchange ratio

• Traffic matrix (10 PB of IP traffic daily)

• Misconceptions reassessment • Tier-1 ISPs do not peer at IXP (they do)

• IXPs are not used for transit (they are)

• The number of peer-peer links on the internet is small (larger than assumed)

• IXP links are mostly used for back-up (they are not)

18

Cardigan [4]

• SDN-based distributed router in a live IXP

• Reduces operational complexity

• Hybrid SDN-IP network side-by-side

• New approach to the router abstraction model

• RouteFlow Vandervecken branch

19

Figure 5 – Cardigan initial 2-switch deployment [4]

Initial Limitations and Easy Fixes

• Extensible message formats

• RFProtocol flexibility

• Set of matches, actions, options...

• IPv6, MPLS on Ethernet

• Inneficient gateway resolution

• RFClient: Reception of its Netlink announcement and discovery of the associated gateway MAC address

• Cache IPv4 and IPv6 routes until ARP resolutions

• Scalable router abstraction

• Router abstraction dependent on the physical topology

• No arbitrary paths inside the ISP network

• Traffic classification using VLAN tags per static inter-switch links (ISLs) configuration

• Fine granular traffic control introducing MPLS paths

20

Cardigan 2.0

• MPLS Label Switched Path (LSP)

• A prefix to a path (set of paths)

• Ingress node through a set of transit nodes

• Operator freely defines packet circuits over the network (arbitrary

level of detail)

• Forwarding Path Manager (FPM) component On RouteFlow

architecture

• Feeds all calculated routes (even if initially discarded)

21

Cardigan 2.0

22

Figure 6 – Cardigan with MPLS design [4]

Deployment

• Pronto Switches (PicOs) - 1G SFPs

• Out-of-band VM controller by layer 2 VLANs

• Traffic forwarded directly by OpenFlow switches

• In production for 9 months

• 90 organizations - forwarding customer traffic and sharing routes

• 1134 flows on each swicth (1028 layer 3 routes)

23

Discussion Itens (1/2)

• Protocol compliance

• OF 1.0 TTL decrement

• MAC addressing

• Scalability of flow tables

• OF agent implementation

• Vendor switches memory leaks and flow counters

• Encapsulation Hazards

• MTU size for Ethernet, VLAN, MPLS, etc

24

Discussion Itens (2/2)

• Gateway Address Resolution - Increased performance

• Separation of gateway resolution and route processing

• Scalability

• Distribution of the FIB accross multiple devices, different data plane

technologies (NPU/FPGA)

• Resilience

• High-available non-stop forwarding solution and systematical SDN

troubleshooting

• Policy enforcements at IXPs – tedious tasks

• Manual time-of-the-day routing, dynamic traffic engineering, route

preferences, etc)’

25

Related Work

• IXP: an interesting networking landscape

• SDX: A Software Defined Internet Exchange • Arpit Gupta (Georgia Institute of Technology), Laurent Vanbever (Princeton

University), Muhammad Shahbaz (Georgia Institute of Technology), Sean P. Donovan (Georgia Institute of Technology), Brandon Schlinker (University of Southern California), Nick Feamster (Georgia Institute of Technology), Jennifer Rexford (Princeton University), Scott Shenker (UC Berkeley), Russ Clark (Georgia Institute of Technology), Ethan Katz-Bassett (University of Southern California)

• ACM SIGCOMM, Chicago, IL. August 2014.

• Enforcing RPKI-Based Routing Policy on the Data Plane at an Internet Exchange

• HotSDN ’14

26

Future Work

• Rethink peering between SDN domains

• Strictly enforce routing and security policy

• Prevent crafted BGP packets from exploiting policy

• Prevent misdirected traffic

• Simplify operations

• Increase reliability

27

Research & Open Innovation

• Open Innovation

• Google Summer of Code [ 2013, 2014, … new proposals]

• Github Community [ forks, pull requests ]

• Code Contributions [5K – 10K LOC, testing hours, bug reports ]

• Mailing Lists [ +200 Members, 160 Topics ]

• Community contributions & collaborations

• Web-based UI & Internet 2 HW pilot [C. Small, Indiana]

• Aggregated BGP Routing Service [C. Corrêa, Unirio]

• SNMP plugin [J. Stringer, Google]

• IPv6, MPLS, IXP [REANZ, Google]

• Optimal BGP best path reflection [R. Raszuk, NTT-MCL]

• Intra data center BGP routing [R. V. Rosa, Unicamp]

• MPLS LDP support [Vikram Dham]

• ….

28

RouteFlow

• Ongoing Work

• OpenFlow hardware upfront: Pica8, NoviFlow, Corsa (OF 1.3)

• Open Issues

• Replicate data plane events in the virtual environment

• Extensive evaluation on realistic networking conditions

• OpenFlow table abstractions

• Towards a routing control PaaS

• IXP Perspective

• Scale to large networks

• Future implementation of policies

• Load balancing, closest exit usage, complex setups

https://github.com/routeflow/RouteFlow/wiki/Projects-and-ideas-for-RouteFlow

29

Summary

• Routing Control Platforms possible through SDN

• IXP is a particular microcosm of the Internet ecosystem

• SDN may solve IXP challenges

• Software Defined Exchanges

• RouteFlow as a SDX layer 3

• Rethink peering between IP and SDN domains (and SDN-SDN)

• More related work at below L3 and at application-layer

• Cardigan RouteFlow pilot deployment is a real SDN-

based legacy IXP implementation

30

References • [1] Nick Feamster, Hari Balakrishnan, Jennifer Rexford, Aman Shaikh, and Jacobus van der

Merwe. 2004. The case for separating routing from routers. In Proceedings of the ACM SIGCOMM workshop on Future directions in network architecture (FDNA '04). ACM, New York, NY, USA, 5-12.

• [2] Christian Esteve Rothenberg, Marcelo Ribeiro Nascimento, Marcos Rogerio Salvador, Carlos Nilton Araujo Corrêa, Sidney Cunha de Lucena, and Robert Raszuk. 2012. Revisiting routing control platforms with the eyes and muscles of software-defined networking. In Proceedings of the first workshop on Hot topics in software defined networks (HotSDN '12). ACM, New York, NY, USA, 13-18.

• [3] Bernhard Ager, Nikolaos Chatzis, Anja Feldmann, Nadi Sarrar, Steve Uhlig, and Walter Willinger. 2012. Anatomy of a large european IXP. SIGCOMM Comput. Commun. Rev. 42, 4 (August 2012).

• [4] Jonathan Philip Stringer, Qiang Fu, Christopher Lorier, Richard Nelson, and Christian Esteve Rothenberg. 2013. Cardigan: deploying a distributed routing fabric. In Proceedings of the second ACM SIGCOMM workshop on Hot topics in software defined networking (HotSDN '13). ACM, New York, NY, USA, 169-170

• [5] Diego Kreutz, Fernando M. V. Ramos, Paulo Verissimo, Christian Esteve Rothenberg, Siamak Azodolmolky, Steve Uhlig. "Software-Defined Networking: A Comprehensive Survey." To appear in Proceedings of the IEEE, 2015.

• https://github.com/SDN-Survey/latex/wiki

• http://arxiv.org/abs/1406.0440

31

32

Thank you!

Questions?

• Raphael Vicente Rosa

• raphaelvrosa [at] gmail [dot] com

Ask and contribute! routeflow-discuss@googlegroups.com

Get the code! https://github.com/routeflow/RouteFlow/

BACKUP

33

RouteFlow Project History 34

• Start Msc. Thesis

work by Marcelo

N. • First Prototype

• QuagFlow Poster

@ SIGCOMM

• Open-Source

Release

• Demos @ ONS11

• Demo @

SuperComputing 11

• Tutorial & Demo @

OFELIA/CHANGE SS • First Short-Paper

@ WPEIF

• Evaluation on

NetFPGA testbed

• Indiana University

- Pronto OF

switches + BGP

peering with

Juniper MX • Demos @ ONS12

• Running on FIBRE /

OFELIA testbed

• HotSDN Paper

• Collaboraion with

NTT

RouteFlow 2013 / 2014

• Significant contributions from CARDIGAN and the

VANDERWECKEN fork

• Live in production network in New Zealand

• Inter-VLAN routing at UNESP, Brazil.

• Google Summer of Code (GSOC 2013 / 2014)

• …

35