Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
Real-World Insights from an
SDN Lab
Ron MilfordManager, InCNTRE SDN Lab
Indiana University
A bit about IU, the GlobalNOC, and
InCNTRE....
Indiana University’s Network History
• 1998 University Corporation for Advanced Internet Development (aka
Internet2) selects IU to build and operate the premier network
supporting US research and education. The GlobalNOC is created.
• 1998 National Science Foundation funds IU to provide high-
performance networking between Asia Pacific and US.
• 2005 National Lambda Rail (a facilities-based R&E network) selects IU
to provide NOC services
• 2011 IU announces the formation of Indiana Center for Network
Translational Research and Education (InCNTRE)
• 2013 InCNTRE under contract with ONF to assist with the
development of OpenFlow conformance certification program
The IU GlobalNOC operates this 15,000+ mile network
and....
...these networks internationally...plus
many more domestically
The IU GlobalNOC
• 27 Research and Education Networks
o Architecture, Engineering, Monitoring, Installation
o Exchanges, Regional, National, International
o Optical, Layer2, Layer3
• 3 SDN/OpenFlow Networks
o GENI - 2010, Deployed 1st Nationwide R&E Testbed OpenFlow
Network
o NDDI Prototype - 2011
� Nationwide R&E Production Prototype OpenFlow Network
� IBM G8264
o NDDI AL2S - 2012
� I2 100Gb Advanced Layer 2 Services Production OpenFlow
Network
� Juniper MX-80, Brocade MLXe-4
InCNTRE Mission
• Seeks to advance development, increase knowledge, and encourage
adoption of OpenFlow and other standards-based Software-Defined
Networking (SDN) technologies through:
• A Neutral Testing Facility
• Educational programs
• Training
• Internships
• Direct networking research
• Collaboration between faculty, students, and networking
professionals
InCNTRE Initiatives
• Summer of Networking Internship Program
• Network Professional Training
• SDN Network Research
o FlowScale, RouteFlow, XSP, GENI...
• SDN Lab
o Interoperability Testing
o OpenFlow Conformance Testing
o Internships
o Open Networking Foundation Research Associates
InCNTRE SDN Lab
• Research/Demo Networks
o InteropNet OpenFlow Lab
� Interop Las Vegas 2011
� Interop Las Vegas 2012
o SuperComputing 2011 SCinet Research Sandbox
• ONF Collaboration
o Co - Chair Testing & Interoperability Working Group
o Conformance Testing Program Development
o Conformance Test Specifications
� OpenFlow Switch 1.0 - In Review Phase
� OpenFlow Switch 1.3 - Under development
ONF PlugFests
ONF PlugFests
• March 2012• Controllers - Big Switch, NEC, NTT Data, NOX
• Switches - Big Switch(Indigo), Broadcom, HP, IBM, Intel/WindRiver, Juniper, NEC
• https://www.opennetworking.org/images/stories/downloads/white-papers/onf-testing-
interop-march-2012-whitepaper-v1.0.pdf
• https://www.opennetworking.org/images/stories/downloads/technical-reports/onf-testing-
interop-march-2012-tech-doc-v1.0.pdf
• October 2012• 19 Companies, ~40 Switches, 4 testbeds (OF 1.0, 1.2, Test Frameworks)
• Controllers – Big Switch, HP, Huawei, Ixia, NEC, NTT Data, NTT MCL, NOX
• https://www.opennetworking.org/images/stories/downloads/technical-reports/onf-testing-
interop-oct-2012-tech-doc-v0-4.pdf
• June 2013• Adding OpenFlow 1.3 & OF-Config Testing
What is OpenFlow and Software Defined
Networking in 60 seconds...
OpenFlow is SDN, but SDN is not alway OpenFlow
Software Defined Networking (SDN) is an architecture that
separates and abstracts elements of the network. SDN
centralizes the control plane from the high-speed data plane in
network switching equipment. SDN allows programmatic control
of a network instead of static configuration.
OpenFlow is a protocol that allows the centralized control plane
to control the behavior of distributed switching equipment and
the API that exposes and abstracts the capabilities of the
OpenFlow controlled equipment.
Ethernet Switch
Table-based (e.g., TCAM/CAM) high-speed forwarding engine
Embedded Operating System
Data Plane
Control Plane
Features Value AddCLI, SNMP, TFTP
OpenFlow Controller
Table-based (e.g., TCAM/CAM) high-speed forwarding engine
Embedded Operating System implements OpenFlow
Data Plane
Control Plane
FeaturesValue Add
OpenFlow Protocol
OpenFlow ControllerOpenFlow Controller
OpenFlow Controller
FeaturesValue Add
OpenFlow ProtocolEach switch
connects directly with OF Controller
OpenFlow Abstraction Permits "Slicing" of
the Network into virtual resources
• Allows "Slicing" network into virtual administrative domains
• Each Slice controlled by it's own controller and admin
• Network can be "Sliced" based on many criteriao Switch or Data Path Identifier (DPID)
o Port
o Src or Dst MAC Addresses
o Src or Dst IP Addresses
o Ethertype
o VLAN ID
o TOS Bits
o Rest of 12-tuple (OpenFlow 1.0)
• Flowvisor
• Coming soon in controllers near you
InCNTRE's experience with some interesting
demo networks...
InteropNet LV 2011 OpenFlow Lab
• 14 controller and switch vendors• ~40 hardware and software Switches
• 5 Demos, each with their own controller and administrative “slice” of the network.
InteropNet LV 2011 OpenFlow Lab
• Worked with Stanford's Open Labs
• 3 Months planning, 3 weeks staging, 5 days in production
• OpenFlow 1.0 Switch Vendorso Broadcom, Brocade, Dell, Extreme, Fulcrum, HP, IBM, Juniper, NEC, Netgear, Pronto
o Open vSwitch on XenServer
• OpenFlow 1.0 Controllerso Big Switch, NEC, NOX
• Flowvisor Virtual Slicingo 6 Slices
� Switch & Port
� Source & Destination IP Subnet
� ~60,000 flowspace rules
• With right IP Subnet configured, could connect to almost any port and be in your slice
Supercomputing 2011
• Distributed through show floor
• IBM, HP, NEC, Pronto Switches
• Beacon, NOX, NEC Controllers
• Inter-Connected to GENI and NDDI Prototype OpenFlow Networks
• Inter-Connected to ESnet, Internet 2, NLR, SCinet non-OpenFlow
Networks
• 9 Slices/Administrative Domainso Switch & Port
o Source & Destination MAC Address
o Source & Destination IP Subnet
o VLAN ID
o Ethertype
• 216 Flowspace Ruleso Strict Port Assignments
Supercomputing 2011
IP Subnet: 192.168.140.1/24
MAC Address: 00:02:C9:10:F1:AC
Ethertype: 0x8902
IP Subnets: GENI Subnet, 10.0.0.0/24, 20.0.0.0/24 30.0.0.0/24, 40.0.0.0/24,
50.0.0.0/24
IP Subnets: GENI Subnet, 10.0.0.0/24, 20.0.0.0/24 30.0.0.0/24, 40.0.0.0/24,
50.0.0.0/24
VLAN IDs: 1327,2908-2912
IP Subnet: GENI Subnet
Ethertype: 0x27A0
VLAN: 233, 3715, 3716
Issues from SC11
• Switch CPU limitations• Too many messages to/from the controller
• Traffic flowing before controller can prepopulate tables
• No isolation of switch resources• One heavily active or misbehaving slice can overrun the switch CPU, TCAM or memory
• Switches failing “Open”• Floods traffic everywhere
• Can easily create loops & broadcast storms
• LLDP Behavior Different Across Vendors• Controllers use LLDP for topology discovery
• Hybrid switches send LLDP
• FlowVisor• Can’t change controller IP without restarting FlowVisor
• Rule generation script unable to insert/remove individual flowspace rules
• Some switches not reconnecting correctly through FlowVisor. Required OpenFlow Process
Restarts
Deep Dive into a our nationwide
SDN/OpenFlow Network...
Network Development and Deployment
Initiative (NDDI)
• Partnership among Internet2, Indiana University and the Clean Slate
Program at Stanford University
• Building an OpenFlow-based network substrate to support Internet2’s
new 100Gb/s services
• Services known as I2’s Advanced Layer 2 Service
• Software suite, developed by the GlobalNOC @ IU, known as Open
Exchange Software Suite (OE-SS)
– , © 2013 Internet2
Internet2 Network – Layer 2 Services
A bit about the software IU developed to
provision services on AL2S...
OS3E Service
• Open Science, Scholarship, and Services Exchange
• First service to be built on top of the NDDI substrate.
• A nationwide distributed Layer-2 exchange
• persistent, on-demand & scheduled L2 VLANs
• Supports open inter-domain standards (IDCP)
OS3E Use Cases
• High bandwidth paths for large file transfers
• Layer-2 connectivity between testbeds
• Distributed exchange for establishing IP peering
OESS Key Features
• Develop once, implement on many different types of
switches
• Uses D-Bus to communicate with the OpenFlow
controller allowing easy migration to other OpenFlow
Controllers in the future.
• VLAN Provisioning time: <1 Second typically
• Automated fail-over to backup path
• Controller redundancy
• Automated measurement
• Easy management
Inter-domain Support
• Inter-domain support enables peering with other
dynamic circuit networks.
• Integraded OSCARS 0.6 with OESS for inter-domain
component. Uses IDC (Interdomain Controller) protocol
• See a demo on the real network! http://www.youtube.com/watch?v=C6Nfg6DZqvI&list=UUuFMyVNxNVeTQrKg
km2M_4Q&index=16
• Initial peering with Internet2 at Los Angeles, Chicago, and
Seattle
• Interconnecting with GEANT in New York
• Many more to come
You too can play with OESS
• Apache 2.0 license
• 3 deployment packages• RPMs for RedHat/CentOS 6
• “.tar.gz” distribution
• VM image (virtualbox)
• http://code.google.com/p/nddi
A bit about a production load-balancer
built using OpenFlow...
• Traffic load balancer as a service using OpenFlow
• Use existing switch hardware instead of a dedicated
load balancer
• Built on Beacon
• Integration with other OpenFlow tools• L2 Provisioning, VM Migration
• Hash based on OpenFlow fields (IP src/dst)
• Divides IU Address Space into IP prefixes
• Uses Round Robin to Pre-Distribute Flows across sensors
• Monitors sensor utilization and dynamically adjusts
• Redirects specific types of traffic to specialized
sensors (e.g. http or tcp/udp port)
• Application layer monitoring and failover
• Web UI for admin
Deployment
• IDS Cluster
• 2 Campuses
• 12 Sensor hosts per campus
• 10Gb/sec on all links
• 480Gb/sec total capacity
• https://github.com/InCNTRE/FlowScale
A bit about Internet2's "Innovation Platform"
(i.e., 100G/s connection, SDN support, and
Science DMZ)...
100 GigE Layer 2 Connection
50 – 4/11/2013, © 2012 Internet2
www.internet2.edu
SDN Control Server
Performance Node
Dedicated scienceswitches/servers in
labswith high-speed
storageand network access
Traditional L3 Campus Border Security
High-PerformanceLayer 2/3
Switch/Router
TraditionalCampus
Border Router
Traditionalregional andcommodityproviders
CampusEnterpriseNetwork
Internet2innovationbackbonedelivered
as 100G L1Nx100G
Nx100G
Nx10G
Nx10G
Nx10G
Nx10G
Science DMZ
fasterdata.es.net
Software Defined NetworkingGENI
Experiments
Dark Fiber
Optical System
GENI ?Dynamic Layer
2
IP NetworkLayer 3
StaticLayer 2
R&E IP
TR-CPS
Innovation ServicesTraditional Services
Software Defined NetworkingSubstrate
Traditional SwitchSubstrate
Your Research
Innovation Platform
51 – 4/11/2013, © 2012 Internet2
Long-term Architectural Aspiration
52 – 4/11/2013, © 2012 Internet2
Software
Stack —
Today
Vendor Switch
Vendor Software
Flowvisor
Controller(Currently NOX)
OESS
OSCARS
OESS UI
OpenFlow
OpenFlow
IDC
IDC
OSCARS API
OSCARS UI
53 – 4/11/2013, © 2012 Internet2
Software
Stack —
Future
Vendor Switch
Vendor Software
Flowvisor
Controller(Currently NOX)
OESS
OSCARS
OESS UI
OpenFlow
OpenFlow
IDC
IDC
OSCARS API
NSI Extension
OSCARS API
OSCARS UI
54 – 4/11/2013, © 2012 Internet2
Software
Stack —
Not fully
baked
Vendor Switch
Vendor Software
Flowvisor
Controller(Currently NOX)
OESS
OSCARS
OESS UI
Aggregate Manager
Aggregate Manager (FOAM)
OpenFlow
OpenFlow
IDC
IDC
OSCARS API
NSI Extension
Experimenter OpenFlow Controller
OSCARS UI
• http://globalnoc.iu.edu/
• http://www.incntre.iu.edu/
• http://internationalnetworking.iu.edu/