Software Defined Network Exchanges(SDXs)For Data Intensive Sciences: BioInformatics SDX

Jim ChenAssociate Director

International Center for Advanced Internet Research(iCAIR)Northwestern University

APAN41Manila, PhilippinesJanuary 25-29, 2016

• Creation and Early Implementation of Advanced Networking Technologies - The Next Generation Internet All Optical Networks, Terascale Networks, Networks for Petascale Science

• Advanced Applications, Middleware, Large-Scale Infrastructure, NG Optical Networks and Testbeds, Public Policy Studies and Forums Related to NG Networks

• Three Major Areas of Activity: a) Basic Research b) Design and Implementation of Prototypes c) Operations of Specialized Communication Facilities (e.g., StarLight)

Accelerating Leading Edge Innovation

and Enhanced Global Communications

through Advanced Internet Technologies,

in Partnership with the Global Community

Introduction to iCAIR:

Open Innovation Platforms

• GENI: Global Environment for Networking Innovations, “a virtual laboratory for exploring future internet technologies and protocols at scale”.

• NSF13602 NSFCloud: “enable the academic research community to develop and experiment with novel cloud and cloud-like architectures that can support a diversity of innovative applications”.

• NSF14554 IRNC RXP: “platforms of network innovation supporting the research and education needs of global scientific collaborations”.

An Experimental Testbed For

Computer Science Research

www. chameleoncloud.org

CHAMELEON HARDWARE

SCUs connect to core and fully connected to each other

Heterogeneous Cloud Units

Alternate Processors and Networks

SwitchStandard

Cloud Unit42 compute

4 storage

x10

Chicago

To UTSA, GENI, Future Partners

AustinChameleon Core Network

100Gbps uplink public network(each site)

Core Services3.6 PB Central File Systems, Front End and Data Movers

Core ServicesFront End and Data

Mover Nodes 504 x86 Compute Servers48 Dist. Storage Servers102 Heterogeneous Servers16 Mgt and Storage Nodes

SwitchStandard

Cloud Unit42 compute

4 storage

x2

www. chameleoncloud.org

CAPABILITIES AND SUPPORTED RESEARCH

Virtualization technology (e.g., SR-IOV, accelerators), systems, networking, infrastructure-level resource management, etc.

Repeatable experiments in new models, algorithms, platforms, auto-scaling, high-availability, cloud federation, etc.

Development of new models, algorithms, platforms, auto-scaling HA, etc., innovative application and educational uses

Isolated partition, bare metal reconfiguration: OpenStack and Grid’5000

Isolated partition, Chameleon Appliances

Persistent, reliable, shared clouds

Software Defined Networking Exchanges (SDXs)

• With the Increasing Deployment of SDN In Production Networks, the Need for an SDN Exchange (SDX) Has Been Recognized.

• Current SDN Architecture Is Single Domain Centralized Controller Oriented

• Required Capabilities for Multi-Domain Distributed SDN Resource Discovery, Signaling Provisioning, Operations, and Fault Detection and Recovery Are Fairly Challenging.

• Nonetheless – Many Motivations Exist for SDXs

StarLight International Software Defined Networking Exchange (SDX)

• The National Science Foundation-Funded StarLight International SDX Will Provide The Services, Architecture, and Technologies Designed To Provide Scientists, Engineers, and Educators With Highly Advanced, Diverse, Reliable, Persistent, and Secure Networking Services, Enabling Them to Optimally Access Resources in North America, South America, Asia, South Asia (including India), Australia, New Zealand, Europe, the Middle East, North Africa, And Other Sites Around the World.

• The StarLight SDX Initiative Undertakes Continued Innovation and Development of Advanced Networking Services and Technologies.

Sloan Digital Sky

Survey

www.sdss.org

Globus Alliance

www.globus.org

LIGO

www.ligo.org TeraGrid

www.teragrid.org

ALMA: Atacama

Large Millimeter

Array

www.alma.nrao.edu

CAMERA

metagenomics

camera.calit2.net

Comprehensive

Large-Array

Stewardship System

www.class.noaa.gov

DØ (DZero)

www-d0.fnal.gov

ISS: International

Space Station

www.nasa.gov/statio

n

IVOA:

International

Virtual

Observatory

www.ivoa.net

BIRN: Biomedical

Informatics Research

Network

www.nbirn.net

GEON: Geosciences

Network

www.geongrid.org

ANDRILL:

Antarctic

Geological

Drilling

www.andrill.org

GLEON: Global Lake

Ecological

Observatory

Network

www.gleon.orgPacific Rim

Applications and

Grid Middleware

Assembly

www.pragma-

grid.net

CineGrid

www.cinegrid.orgCarbon Tracker

www.esrl.noaa.gov/

gmd/ccgg/carbontrack

er

XSEDE

www.xsede.org

LHCONE

www.lhcone.net

WLCG

lcg.web.cern.ch/LCG/publi

c/

OOI-CI

ci.oceanobservatories.org

OSG

www.opensciencegrid.org

SKA

www.skatelescope.o

rg

NG Digital

Sky Survey

ATLAS

Compilation By Maxine Brown

Motivations for SDXs• White House Office of Science and Technology Policy –

Large Scale Science Instrumentation

• Large Scale Ultra High Resolution Digital Media Services

• Multi-Domain Networks Interconnecting Data Centers (SDN Is Already in Production Within Large Scale Data Centers)

• Multi-Domain SDN Services

• Providing Capabilities for Edge Control

• Democratization Of Exchange Facilities

• Many New Types of Services and Capabilities

• Etc.

Software Defined Networking Exchanges (SDXs)

• With Support From the GENI Project, the International Center for Advanced Internet Research (iCAIR) and Its Research Partners Have Designed and Implemented a Prototype SDX at the StarLight International/National Communications Exchange Facility

• This SDX Is a Multi-Domain Service Enabling Federated Controllers To Exchange Signaling and Provisioning Information.

Selected SDX Architectural Attributes

• Control and Network Resource APIs

• Multi Domain Integrated Path Controller

• Controller Signaling, Including Edge Signaling

• SDN/OF Multi Layer Traffic Exchange

• Multi Domain Resource Advertisement/Discovery

• Topology Exchange

• Multiple Service Levels At All Layers

• Granulated Resource Access (Policy Based), Including Through Edge Processes

• Foundation Resource Programmability

• Various Types of Gateways To Other Network Environments

• Integration of OF and Non-OF Paths, Including 3rd Party Integration

• Programmability for Large Scale Large Capacity Streams

Architectural Components• Hybrid Networking Services (Multi-Service, Multi-

Layer, Multi-Domain)

• Network Programming Languages (e.g., P4, Frenetic)

• Abstraction Definitions

• APIs

• Service Signaling and Policy Bundling

• Policy Bundle Distribution

• Primitives

• BGP Extensions and Substitutes

• NDL Schema

• Orchestration Processes

Other Architectural Components 2• Northbound Interfaces

• Network OSs

• Network Hypervisors

• State Information Data Bases

• Data Modeling Languages (e.g., YANG)

• Controller Federation Processes

• Hybrid Services/Services Federation/Services Chaining

• Southbound Interfaces

• East-west bound Interfaces

Other Architectural Components 3

• Data Plane acceleration

• Network Function Virtualization (NFV)

• Measurements

• Real Time Analytics

• Distributed Virtual NOC Operations

GLIF is a consortium of institutions, organizations, consortia and country National Research & Education Networks who voluntarily share optical networking resources and expertise to develop the Global LambdaGrid for the advancement of scientific collaboration and discovery –a Federation!.

StarLight – “By Researchers For Researchers”

Abbott Hall, Northwestern University’s

Chicago CampusView from StarLight

StarLight is an experimental optical infrastructure andproving ground for network services optimized forhigh-performance applicationsMultiple10GE+100 GbpsStarWaveMultiple 10GEsOver Optics –World’s “Largest”10G/100G ExchangeFirst of a KindEnabling InteroperabilityAt L1, L2, L3

GLIF 2011

Multiple HPC Cloud Computing Testbeds

Specifically Designed for Science Research

At Scale Experimentation

Integerated With High Performance Networks

=> Open Commons Consortium

The OCC And Next Generation Networking

• The OCC Has Formed a Partnership With The Advanced Networking Community To Created Large Scale (Global) Distributed Environments For Data Intensive Scientific Research – Based On a Foundation of Programmable, Deterministic, Edge Customizable High Performance Large Capacity Networks (e.g., 100 G and 100 G +)

Creating A Data Commons

• Clouds and Data Commons For Global Data Intensive Science

• Vast Amounts Of Data Are Being Created By Many New Powerful Sophisticated Instruments – Zettabytes!

• Few Individual Scientists Have the Resources To Manage, Store, Analyze, and Transport This Data

• A Data Commons Is A Facility Designed To Co-locate Data, Storage, Computing, and Networking Infrastructure, Along With Commonly Used Tools For Analyzing and Sharing Data As A Research Resource.

Core Data Commons Services

• Policy Based Access Procedures, Digital IDs

• Metadata Services

• Storage Services

• High Performance Transport, Including Specialized Transport Services For Large Scale Streams

• Policy Based Data Export Tools/Services

• Compute Services with Images/Containers Integrating Commonly Used Tools, Applications and Services, Specialized for Individual Science Domain Research Communities

GENI Engineering Conference (GEC 19) SDX Demonstration Atlanta March 18-20, 2014

• Initial SDX Capability Between GENI Sites (StarLight and SOX) Was Demonstrated

• Motivation: To Share a Vision of Interconnected US Nationwide SDN Infrastructure, With Multiple SDN Capable Networks and Domains

• SDX Benefits Was Showcased Through a Compelling Application –Nowcast – Developed By Mike Zink and His Colleagues at University of Massachusetts, Amherst

• (GEC 19 Was Co-Located With the GLIF Tech Workshop, March 19-20)

Comparison With Existing SystemSlide by Mike Zink, UMass Amherst

Source: Mike Zink, UMass Amherst

GENI SDX Demo Scenario 1

Starlight SDX

ATLSDX

RENCI

GT(GEC)

R2 R3R1 R4

MiddleBox

Radars

MergeGrid

Nowcast

Publish on web

Starlight SDX

ATLSDX

OSF

GT(GEC)

R2 R3R1 R4

MiddleBox

Radars

MergeGrid

Nowcast

Publish on web

Slide by Mike Zink, UMass Amherst

ESnet

ORNL

AL2S

Georgia

Tech SDX

StarLight

SDX

R4R3R2R1

GENI SDX Demo Scenario 2

Simulated

Radar (4)

GEC 19

Mid

Box

Global LambdaGrid Workshop (GLIF)

• International SDX (iSDX) Demonstrations Showcase a World-wide Prototype Environment That Could Be Used for Modeling Major Weather Systems, Including The Depiction of Severe Weather Patterns.

• Application Based on Nowcast System Being Developed by the NSF Engineering Research Center for Collaborative Adaptive Sensing of the Atmosphere (Led By University of Massachusetts at Amherst), Which Is Being Designed for Next Generation Weather Prediction and Visualization Systems.

Global LambdaGrid Workshop Sept 30-Oct 1 New Zealand

• Demonstrations of A Prototype Implemented Across the Globe Using the GLIF To Interconnect Sites World-Wide, Supported By Interoperable International SDXs, Including Prototype SDXs at – a) the StarLight International/National Communications Exchange Facility, Designed by iCAIR

– b) Auckland, New Zealand, Designed by REANNZ and Google,

– c) Taiwan, Designed by High Performance Computing Center/TWAREN

– d) SOX in Atlanta, Designed by Georgia Tech

– d) NetherLight in Amsterdam, Designed by SURFnet,

– e) Ottawa, Designed by CANARIE and Cybera,

f) Tokyo, Designed by the University of Tokyo

• Each Site Had Its Own SDN/OpenFlow Controllers, Which Were Federated To Enable Cross-Domain Interoperability.

• Controllers Used Distributed Control Plane To Directly Address and Dynamically Manage Multiple Paths Among Sites Via Distributed Data Plane - Transporting

Nowcast Instrumentation Traffic Among Sites.

Will Be Contiguous To

the StarLight SDX

Global Research Platform

• Currently – A Concept

• A Specialized Globally Distributed Platform For Science Discovery and Innovation

• Based On State-Of-the-Art-Clouds

• Interconnected With Computational Grids, Supercomputing Centers, Specialized Instruments, et al

• Also, Based On World-Wide 100 Gbps Networks

• Leveraging Advanced Architectural Concepts, e.g., SDN/SDX/SDI – Science DMZs

• Ref: Demonstrations @ SC15, Austin Texas November 2015

Pipe-

lines

Proc-

essing

100G

Crypto

SCInet

Oakland, CA

NERSC

Chicago StarLight

Austin, TX

SC15

Controller

Pipelines

Manager

Software Defined Exchange

Selective Data

Display

& User Interaction

100G

Crypto X

X

NERSC-

TB1

AOFA-

CR5

Source

Data

N x 4K

Video

Dynamic Remote I/O Network

Remote Data Source

(Gigapixels/sec)

Network on

demandProcessing on

demand

Pipe-

lines

Proc-

essing

Washington, DC

NRL

CTL

STAR-

TB1

40

G40

G40

G

Live Video

4K x 60 fps

Pipe-

lines

Proc-

essing

Source

Data

N x 4K

Video

Pipe-

lines

Proc-

essing

>95% BW

Efficiency

100G Encryption

(MACSEC)

Processing on

demand

Network/cry

pto on

demand

Dynamic Remote I/O Demonstration of large-scale remote data access, a dynamic pipelined distributed processing framework and software defined networking enabled automation between distant operating locations. Live production quality 4K video workflows across a nationally distributed set of storage and computing resources - relevant to emerging data processing challenges.

CTL

ExoGENI @ UvA

SAGE2 VMServer

Tiled Display@ iCAIR

SAGE2 Display

Tiled Display@ Ottawa

SAGE2 Display

PublicInternet

Google Maps, Weather Data, Etc

Starlight

10G10G

10G

1G1G

1G

SAGE2 10GImagesVideo

Desktop SharingApplications

Desktop Sharing

ONOS framework for resource discovery, allocation and reservation

On demand allocation of multi-domain (SDX) network resources for collaborative sessions

On demand exploration of large remote data setsLeverage high speed network for high quality

video streaming (QoS) Secure access through ScienceDMZSupport for VM deployments

Network connectivity/performance visualization

:Remote Resource Plans

Automated GOLE Fabric

JGN-X

NetherLight

CERNCaltech

UvACzechLight

KRLight

AIST

KDDI Labs

StarLightGLORIAD

GÉANT

ANA & EEX

NORDUnetANA

ESnet

Pionier

Caltech

AMPATH

SouthernLight

MAN LAN

50

AutoGOLE Dashboardhttp://dashboard.lab.uvalight.net:8000/dpm

Arista

Controller Controller

Controller Controller

BDDP

BDDP

StarLight

NCHC

iCAIR

KISTI

CANARIE

Juniper

Arista(vlan Translation)

vlan2782

vlan2782

vlan2780

vlan2780

vlan2780

vlan2780

SC14-SC15 NSI-SDN-Topology

NSA

NSA

NSA

NSA

NSA

A Peer-to-Peer Distribution Protocol

• Once an NSA comes up, it contacts its neighbors to request the topology information

iCAIR

NCHCKISTI

CANARIE

StarLight

1. Request

2. Response

controller

NSA

update

NSA ANSA B

NSA C

NSA D

Topology Exchange (from NSA A)

NSA A Update topology & link cost computation (to Domain B) from Controller A

Contacts neighbor NSA peer B with AA

By policy, NSA B replies info of its peer NSA C and D with link costs

NSA A Contacts remote NSA peer C and D with AA and obtain the link between C and D by policy

A DB

C

Ctrl ACtrl B

Ctrl C

Ctrl D

NBI

PetaTrans: Petascale Sciences Data Transfer

SDXs100G

Switches

PetaTrans@100G

ExoGENI@40G

100GSwitch

100GSwitch

PacificWave100G

Switches

100GSwitch

ExoGENI@100G

ExoGENI@40G

HECN 100G Node

Seattle/California

Washington D.C.

Chicago

Ottawa

Amsterdam

PRP SitesESnet/ANA200

@100GPacificWave

/GRPNet@ 2 X 100G

ESnet/MAX @ 100G

@ 100G

Persistence 100G Services Beyond SC15

100GE End to End Services based on Open Architecture (E2SOA) for Peta

Scale Sciences: Prototype Set Up

StarLightPetaTrans1

StarLightPetaTrans

2

InventecOCP 100G

Switch

Ciena8700

MREN8700

ANA200Nether Light8700

InventecOCP 100G

Switch

SURFNet100GENode

StarLight SDXService Hub

OpenFlow NSI/openNSACANARIE

CANARIE

6500CIENA 8700

ExoGENI40G

Nodes

Hospitals,

Doctors

Cloud Computation

Genomic Data Commons

Patients

Output: Data-

Aware,

Analytics-Informed

Diagnosis,

Prognosis,

Optimal Treatment

Future Vision:

A Nationwide Virtual Comprehensive Cancer Center

Biomedical Data Commons:Flow Orchestration: Control Plane + Data Plane

Data Plane

Control Plane

Data Repository A (West Coast)

Data Repository C (Asia) Data Repository D (Europe)

Data Repository B

(South)

Visualization Engines

North AmericaCompute Engines

(Midwest)

BI Data Flow Visualization (Inbound-Outbound) From SDSC To UoC

Genomic Data Commons Data Transfer

DiCOS

Distributed Cloud Operating System

Based on Grid Computing’s middleware

(Large Hadron Collider experiment, CERN)

The middleware links more than 260,000 processing cores

and 180 PB disk storage

Got a successful experience from high energy Physics field

Promote the system to biological/genetic field

Provide Web-UI and CLI

Other field’s applications are also welcome

Academia Sinica Grid Computing Center,

Taiwan

RNA Seq Data Queue

Software Repository

DJM

Initiate aProcess Job

DDM Supported

http, xml

Pilots

WNWN

SESESE

DiCOS DDM

Pilot Factory

Cloud Storage

ASGC

Platform and Software

Configuration Management

LocalCache

NCU, NCHC

Pull

BDCVDB

Parcel/UDPOver SDN

VisualizationR/ROOT

CERN

DiCOS for BDC-TP

Xrootd/GridFTP

U of Chicago/StarLight FASTQ/

BAM

Data Bundle

Slide source: Eric Yen/ASGC

www.startap.net/starlight

Thanks to the NSF, DOE, NIH, USGS, DARPA

NOAA, Universities, National Labs,

International Partners,

and Other Supporters

Jim-chen@northwestern.edu