Internet2 and SDN

November 21, 2012Terena Network Architects WorkshopEric Boyd

1 – 11/22/2012, © 2012 Internet2

3 – © 2012 Internet2

Optical Backbone Capacity  3.2 Tb / 32  100G Waves

New community route miles 9,284 miles

Nationwide Distribution Network 10G

Software Defined Network Platform No

17,834

Yes

Federal Funds / Matching Funds 62.5M / 34.1M

8.8

Project Summary

/ 88

• Advanced Layer3 Service Network– 10 Juniper routers– Interconnected via bundles of 10G

• Advanced Layer2 Service Network– 14 nodes deployed (growing to ~35)– Heterogeneous (Brocade, Juniper, Cisco, …)– Interconnected via 100G– 5 node advanced test network (NEC)

• Advanced Layer1 Service Network– Only 10/40/100G capable Layer 1 Service in the US– 88 channels of 100G capability– 56 Add/drop and flexibility to add more

• TR‐CPS Network– 5 Juniper MX‐960 routers

• Interconnects– 4x 100GigE and 6x 10GigE between L2 and L3– Expect 100G interconnects between L2 and Exchange Points

5 – 11/22/2012, © 2011 Internet2

Internet2 Service Layers

• Data Intensive Science requires the ability to serve the needs of scientific communities seeking to transfer massive files amongst a set of sites– Astronomy, Climatology, Genomics, High Energy Physics, etc.

• Internet2’s innovation strategy directly addresses this use case– Globally interconnected 100G Network– Support for Science DMZs– Open up the network stack through Software-Defined Networking

• Recent Developments– LHCONE Multipoint Service is operational– DYNES deployments are accelerating

Data Intensive Science

6 – 11/22/2012, © 2011 Internet2

• The network research community needs access to an at-scale, sliceable network research infrastructure

• Internet2 is expanding upon open source software (“GENI API”) to provide a production, sliceable network environment

• GENI is a suite of network research infrastructure now in its design and prototyping phase sponsored by the NSF to support experimental research in network science and engineering.

• Internet2 contributes a wave and operates part of the GENI Mesoscale Infrastructure

• Recent Developments– Internet2 is interconnecting 4 GENI Mesoscale Nodes at 10G– Internet2 is actively working to deploy the GENI software environment

atop the Advanced Layer 2 Service

Services for Network Research

7 – © 2012 Internet2

• 3 components of the Innovation Program:– Ubiquitous bandwidth and limited aggregation (100G)– Open up the network stack (support Software-Defined Networking)– Enable data intensive science (support for Science DMZs)

• Matches NSF/Research community call for investment– CC-NIE– Big Data

• Matches NSF, GPO, OSTP in the application space:– GENI RACKS– US Ignite

8 – 11/22/2012, © 2012 Internet2

Components of Internet2 Innovation Program

9 – 11/22/2012, © 2012 Internet2

Partnership leads to innovation

Community

Community & Staff

Innovation Campus Pilot Program

100GEScience DMZSoftware Defined Networking

• 3 components of the Internet2 Innovation Program:– Ubiquitous bandwidth and limited aggregation 

(100G)– Open up the network stack (support Software‐

Defined Networking)– Enable data intensive science (support for Science 

DMZs)• Matches NSF/Research community call for 

investment– CC‐NIE– Big Data

• Matches NSF, GPO, OSTP in the application space:– GENI RACKS– US Ignite 

11 – 11/22/2012, © 2012 Internet2

Components of Internet2 Innovation Program

• GENI Mesoscale grant begins (Fall, 2009)– Internet2 wave contribution, operating Mesoscale backbone nodes

• Community involvement in BTOP Topology (Spring, 2010)• NTAC/AOAC whitepaper calling on Internet2 to build an advanced

Layer 2 network with OpenFlow/SDN support (Spring 2011)• Demonstration of 10G, SDN-enabled, NEC-based Layer 2 service

(October, 2011)• Community leaders call to “get out in front” (Fall 2011)• AOAC discussions on innovation program (Fall/Winter 2011/2012)• Board support to combine multiple threads and push aggressively

forward on a new “innovation program” (Winter 2012)• RFP issued to challenge the vendor community (Winter 2012)• Announcement of planned 100G Advanced Layer 2 service (April,

2012)• Launch of 100G, SDN-enabled, heterogeneous Advanced Layer 2

service (October, 2012)

A bit of history around recent network innovation

12 – 11/22/2012, © 2011 Internet2

• Expand on Internet2 Advanced Layer 2 Service– Open exchange concept open

to all interested parties– 100G add/drop nodes will be

deployed where connections are requested

– Existing Layer 3 services can be plumbed through this network, but are provided under separate agreements

– Focus in 2012 on virtualization and GENI support

– Fee structure to encourage utilization and adoption

Initial  Innovation  Program Approach

13 – 11/22/2012, © 2009 Internet2

InnovationPlatformCore Node

CampusInnovation

DMZ

L2PERF

SDN

SDN/Perf

Early Adopter Campus

InnovationLab / Net

TraditionalCampus

Net

HPC L3

InnovationPlatformCore Node

CampusInnovation

DMZ

L2PERF

SDN

SDN/Perf

Regional Network 100G DWDM

Regionally Connected Campus

InnovationLab / Net

TraditionalCampus

Net

HPC L3

Dar

k Fi

ber U

p to

25

Km

Regional NetworkServices

Regional NetworkServices

OR

TR-CPS Peering Service

TraditionalInternet2 IP

Network

NETPLUSLayer-2

ConnectedProvider

Examples of other Internet2 services that could be carriedover new 100G transport network (existing service fees apply)

Access to InternationalExchange Points

(Included)Interoperable

with Esnet, Dynes, Autobahn

• Leading Campuses quickly positioned for OpenFlow / SDN application innovation and data intensive science delivery

• Start with SDN updates in major science buildings

• Support pervasive 100G• Support Science‐DMZ• Begin “GENI‐enabling” campus

Innovation  Investment on  the  Campus

14 – 11/22/2012, © 2012 Internet2

Engineering, Biology, Computer Science Buildings

Building Network Replacementsto support SDN

Traditional Campus NetworkSDN Science DMZ100G BorderLink at 100G to

Internet2 AL2S

(via CENIC)

100G/SDN

CommodityInternet 10G’s

GENI Rack(s)

For more on Science DMZ, see: http://fasterdata.es.net

• Heterogeneity is good• If it doesn’t exist, build it• If its open source, improve it• If its in the marketplace, stress it and engage vendor• Build it on the Internet2 network / push it out to the R&E community

Innovation Themes

15 – 11/22/2012, © 2011 Internet2

• Hardware• Virtualization• Controller• OESS• Applications• Interoperability (at all levels)

Innovation Components

16 – 11/22/2012, © 2011 Internet2

• Today we are rolling out 100G Advanced Layer 2 Service network …

• How did we get there?• April - October, 2012

– Virtualization demo’d on NEC network (July, 2012)– Weekly calls with Brocade, Cisco, and Juniper (July, 2012 onwards)– Went through 3 revs of Brocade code (July, 2012 onwards)– Testing Juniper code (September, 2012 onwards)– About to test Cisco, Dell code (planned October, 2012)– 14 Brocade nodes deployed (September, 2012)– 18 100 G backbone circuits turned up (September, 2012)– First participant connected (September, 2012)

Innovation Stories

17 – 11/22/2012, © 2011 Internet2

This is what we want to be ARE able to say: The 100Gtestbed of innovat ion for tomorrow’s Internet is avai lable nat ionwide, r ight now. The playground is open.

19 – 11/22/2012, © 2012 Internet2

Long‐term Architectural Aspiration

20 – 11/22/2012, © 2012 Internet2

Software Stack —Today

Vendor Switch

Vendor Software

Controller(Currently NOX)

OESS

OSCARS

OESS UI

OpenFlow

IDC

IDC

OSCARS API

OSCARS UI

21 – 11/22/2012, © 2012 Internet2

Software Stack —Q1 2013

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

OpenFlow

OpenFlow

IDC

IDC

OSCARS API

OSCARS UI

22 – 11/22/2012, © 2012 Internet2

Software Stack —Future

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

OpenFlow

OpenFlow

IDC

IDC

OSCARS APINSI Extension

OSCARS API

OSCARS UI

23 – 11/22/2012, © 2012 Internet2

Software Stack —Not fullybaked

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

Aggregate Manager

Aggregate Manager (FOAM)

OpenFlow

OpenFlow

IDC

IDC

OSCARS APINSI Extension

Experimenter OpenFlow Controller

OSCARS UI

• Internet2 is committed to extending a policy‐free approach within the Internet2 Advanced Layer 2 Service

• Each individual node should function like an “exchange point” in terms of policy, cost, capabilities

• Internet2 would like to position this service on the forefront of pushing “open” approaches in distributed networks.

24 – © 2012 Internet2

Advanced Layer 2 Service Policies

CORE• 10G & 100G dedicated ports• Heterogenous

– Brocade, Juniper (underway)• 7x24x365 production‐quality support• Robust evolution and regressive lab 

testing planVLAN PROVISIONING• User (& backup) VLAN provisioning 

through GUI• Ability to reach Internet2 Layer 3 

Services• Ability to reach International 

Exchange Points at 100G• Interdomain provisioning to IDC 

DomainsAPPLICATION INTERFACE• IDC API• OESS API• Sherpa API

25 – 11/22/2012, © 2011 Internet2

Advanced Layer 2 Service Features (Now)

CORE• Heterogenous

– Juniper (Q1 2013)– Cisco (TBD)– Dell (DYNES sites, TBD)

• Virtualization via Flowvisor(Q1 2013)

• OpenFlow 1.3 Support (TBD)

VLAN PROVISIONING• Multipoint VLANs (Q4 2012)• Interdomain provisioning to 

NSI Domains (TBD)

26 – 11/22/2012, © 2011 Internet2

Advanced Layer 2 Service Features (Planned)

APPLICATION INTERFACE• NSI API (Planned)• GENI API (Q2 2013)• User OpenFlow Slice 

Support (Q2 2013)

INNOVATION TESTING ENVIRONMENT• InCentre Participant (Q4 

2012)• ONS Participant (TBD)• SDN Production Test 

Platform– NEC (Q4 2012)

27 – 11/22/2012, © 2011 Internet2

Internet2 and GENI Today

RegionalR&E Network

RegionalR&E Network

Today, the campus/regional/national backbone is used to get to the GENI infrastructure

28 – 11/22/2012, © 2011 Internet2

Internet2 and GENI Tomorrow

RegionalR&E Network

RegionalR&E Network

Tomorrow, the campus/regional/national backbone implements the GENI infrastructure

29 – 11/22/2012, © 2012 Internet2

Current GENI Mesoscale infrastructure

30 – 11/22/2012, © 2012 Internet2

Current GENI Mesoscale infrastructure

GENI interconnectivityin a representative city, for example, Washington, D.C.

31 – 11/22/2012, © 2012 Internet2

Current GENI Mesoscale infrastructure

• GENI environment implemented on the Internet2 backbone

– Today: The campus/regional/national backbone is used to get to the GENI infrastructure

– Tomorrow: The campus/regional/national backbone implements the GENI infrastructure

• Software and operational environment of GENI is supported in a production environment

– Tomorrow: Aggregate manager API, dynamic stitching,  instrumentation and measurement API

– Next Day: GENI environment continues to evolve to meet needs of network research community

32 – 11/22/2012, © 2011 Internet2

Internet2 and GENI

Internet2 and SDN

November 21, 2012Terena Network Architects WorkshopEric Boyd

33 – 11/22/2012, © 2012 Internet2