Internet2 DCN Architecture, Status, and Community Development

Activities

Tom Lehman (USC/ISI) John Vollbrecht (Internet2)

Joint Techs University of Nebraska-Lincoln

Nebraska Student Union Lincoln, NE

July 23rd, 2008

Hybrid Networking

•  There has been interest from many communities for the development of network architectures and mechanisms that utilize lower layers of the protocol stack along with IP at layer 3

•  This has become known as “hybrid networking” •  It is motivated by applications from the research

and education community that require greater capabilities –  High bandwidth flows (for example, flows that come

close to saturating links in the shared IP backbone) –  Flows with special requirements related to quality

of service, for example jitter requirements –  Network and Application Virtualization

•  Dynamic and Automated traffic engineering of the IP Routed network - router to router circuits, router bypass

Hybrid Networks - Motivating Factors •  Hybrid networks are intended to provide a

flexible mix of IP routed service and “lower layer services” –  “flexible” means the network can respond quickly to user/

application/connector requirements and requests to access both the IP Routed and/or lower layer services

–  “lower layer services” means access to layer 2 and below paths which can be utilized in a multitude of ways by creative users.

•  Typical user requirements for these lower layer services are based on: –  critical, large bandwidth flows which may require one of more

of the following: deterministic network performance, dedicated network resources, guaranteed network capacity, freedom to use protocols other than (congestion control friendly) TCP, privacy/security requirements, scheduled services

–  User/application communities which desire to build entire topologies which integrate domain specific resources along with dedicated network resources (which have one or more of the above mentioned characteristics)

Hybrid Networks Heterogeneous By Nature •  Hybrid networks are extremely heterogeneous at

several levels •  DataPlane can be constructed from

–  router based Multiprotocol Label Switching (MPLS) tunnels –  Ethernet VLAN based Circuits –  Synchronous Optical Network / Synchronous Digital Hierarchy

(SONET/SDH) circuits –  Wavelength Division Multiplexing (WDM) connections –  Combinations of the above

Hybrid Networks Heterogeneous By Nature

•  Control Planes can be based on –  Multiprotocol Label Switching (MPLS) –  Generalized Multiprotocol Label Switching (GMPLS) –  Web Services –  Management Systems –  Combinations of the above

•  Client (user) services or attachment points could be –  Ethernet –  SONET –  IP Router –  InfiniBand

Multi-Domain, Multi-Layer Control Planes Key Requirements

•  The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following: –  Multi-Technology - MPLS, Ethernet, Ethernet PBB-TE,

SONET, NG-SONET, T-MPLS, WDM –  Multi-Level - domains or network regions may operate

in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries

•  Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains

•  And of course all this implies that this will be a Multi-Vendor environment.

•  Multi-Control – mpls, gmpls, management, vendor proprietary

DCN Services - topologies

•  Individual circuits are the “atomic” service provided by the DCN and control plane

• These circuits could be intra or inter domain •  It is envisioned that higher level “services”

may be developed which coordinate the instantiation of multiple individual circuits to develop entire “topologies” –  co-scheduling/allocation of other resources

(compute, data storage) may also be desired –  Probably a task for individual science/application

domains or someone developing middleware on their behalf

Dynamic Network Services IntraDomain

• Source Address • Destination Address • Bandwidth • VLAN TAG (untagged | any | tagged | tunnel) • User Identification (certificate) • Schedule

Client A

Client B

Circuit Request

Ethernet Mapped SONET or

SONET Circuits

Dynamically Provisioned Dedicated Resource Path (“Circuit”)

Internet2 DCN Service

Internet2 IDC

• api can run on the client, or in a separate machine, or from a web browser

XML USER API

Actual Network Path

DRAGON Enabled Control Plane

Dynamic Network Services InterDomain

•  No difference from a client (user) perspective for InterDomain vs IntraDomain

RON Dynamic Infrastructure Ethernet VLAN

RON Dynamic Infrastructure Ethernet VLAN

Internet2 DCN Ethernet Mapped SONET

1. Client Service Request 2. Resource Scheduling 5. Service Instantiation (as a result of Signaling)

A. Abstracted topology exchange

A A 2

2 1

USER API

XML

Multi-Domain Dynamically Provisioned Circuit

DCN Control Plane

InterDomain Controller (IDC) Protocol (IDCP)

•  Developed via collaboration with multiple organizations –  Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others

•  The following organizations have implemented/deployed systems which are compatible with this IDCP –  Internet2 Dynamic Circuit Network (DCN) –  ESNet Science Data Network (SDN) –  GÉANT2 AutoBahn System –  Nortel (via a wrapper on top of their commercial DRAC System) –  Surfnet (via use of above Nortel solution) –  LHCNet (use of I2 DCN Software Suite) –  Nysernet (use of I2 DCN Software Suite) –  University of Amsterdam (use of I2 DCN Software Suite) –  DRAGON Network

•  The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: –  LambdaStation (FermiLab) –  TeraPaths (Brookhaven) –  Phoebus

DCN Web Services • Web Service Definitions • wsdl - web service definition of message types

and formats • xsd – definition of schemas used for network

topology descriptions and path definitions • Ongoing work with OGF Working Group(s),

PerfSonar, and GLIF with the goal to achieve interoperability amongst all groups.

InterDomain Specification Web Services

• https://wiki.internet2.edu/confluence/display/CPD/OSCARS+Web+Service+Definition

•  Specification is defined by a Web Service Desciption Language (WSDL) document and XML Schema files containing associated data types.

• OSCARS.wsdl - web service definition of OSCARS messages

• OSCARS.xsd - data types used by OSCARS.wsdl • nmtopo-ctrlp.xsd - NMWG control plane topology

schema used by OSCARS.xsd for topology-related data types

IDC - Web Service Based Definition

•  Four Primary Web Services Areas: •  Topology Exchange, Resource Scheduling, Signaling, User Request

I2 DCN Software Suite

•  OSCARS

–  Web service layer, InterDomain messaging, AAA, Scheduling

•  DRAGON

–  Control of domain network elements (Core Directors and/or Ethernet Switches) –  Intra and Inter Domain Path Computation

–  RSVP based signaling •  Version 0.3.1 of DCNSS released April 2008

–  https://wiki.internet2.edu/confluence/display/DCNSS

DRAGON

DOE OSCARS Office of Science

Internet2 DCN Deployment

Internet2 DCN Services

1-A-5-1-1 1-A-6-1-1

1-A-6-1-1

I2 DCN Data Plane • Ciena CoreDirectors • Ethernet Edge Ports mapped into variable

bandwidth SONET Circuits. Several Provisioning Modes –  tagged –  untagged –  tunnel (transparent passing of all Ethernet Frames)

• Provisioning granularity of STS-1 (~50 mbps) •  SONET Edge Ports, SONET Circuits

–  SONET Edge Ports and Interdomain Links not yet incorporated into the DCN control plane

DCN Provisioning Web Page or API

Web Page Based Provisioning

Internet2 IDC

USER API java createReservation https://dcn.internet2.edu:axis2/services/dcn

reservation.properties

Web Service

Circuit Status and History

Individual Circuit Details

DCN – Global Network Interoperation via IDCP

Connecting to and Using the DCN • DCN Workshops

–  http://events.internet2.edu/2008/DCN/

• User/API access mode • Dynamic Network Deployment for RONS

How do I connect? – Software Configuration • Option 1: No local IDC • Option 2: Install local IDC

How do I connect? – Software Configuration

• Option 1: No local IDC

How do I connect? – Software Configuration • Option 2: Install local IDC

DRAGON Control Plane - Key Elements

•  Virtual Label Switching Router – VLSR –  Open source protocols running on PC act as GMPLS network

element (OSPF-TE, RSVP-TE) –  Control PCs participate in protocol exchanges and provisions

covered switch according to protocol events (PATH setup, PATH tear down, state query, etc)

• Network Aware Resource Broker – NARB –  Intradomain listener, Path Computation, Interdomain Routing

and Path Computation

• More information: –  dragon.east.isi.edu –  dragon.maxgigapop.net

How do I write my own DCN application? •  Java library for making DCN calls • Can call simple command-line client

directly from application • Google Summer of Code students will be

developing PERL, C, and Python libraries

Additional Information

• DCN Software Suite – https://wiki.internet2.edu/

confluence/display/DCNSS/Home • Java Client API

– https://wiki.internet2.edu/confluence/display/CPD/OSCARS+Client+Java+API

Interdomain Dynamic Circuit Infrastructure

•  Structure of Multidomain Dynamic Circuit Network

•  Status – current Issues • Deployment DCN domains • Community Development Activities •  Interworking with other services

IDC and other dynamic circuit projects

IDC protocols developed in DICE (Dante, Internet2, Canarie and Esnet) control plane wg ) “IDC Protocol” is the name DICE gave to the protocol

Others groups working on dynamic circuits UCLP, G-Lambda, Enlighten, Optiputer Portal,

Phosphorous, Enlighten, Argia and others

Collaboration between groups to develop common interfaces and protocols in GLIF (Global Lambda Integrated Facility) and in a proposed OGF NSI (Network Services Interface) working group

Internet2 DCN Working Group

• DCN WG has been formed under NTAC – Chair: Linda Winkler (Argonne

National Laboratory) • DCN WG will drive directions and set

agenda in this area • Mailing list and Wiki available

–  dcn-wg@internet2.edu –  https://spaces.internet2.edu/display/DCN/Home

• DCN WG BOF on Monday, July 21, 12:30 PM 1:50 PM

InterDomain Control Protocol Standardization Activities

•  Standardization process and increasing community involvement continues

•  Optical Grid Forum (OGF) –  Network Services Interface (NSI-WG) –  Network Markup Language (NML) Working Group

• Standardizing topology schemas (perfsonar and control plane) –  Grid High Performance Networking (GHPN) Research Group –  Network Measurement (NM-WG) –  Network Measurement Control (NMC-WG) –  Information Services (IS-WG)

•  GLIF –  Control Plane Subgroup working on normalizing between various

interdomain protocols (IDCP, G-Lambda GNS-WSI, Phosphorus API) –  Also other GLIF subgroups in this and related space (global id

format, PerfSonar)

Global Dynamic Circuit Infrastructure

DCN Users 1.  Individual connections

–  talk directly to others already connected to DCN –  requires all participating users to connect with known

characteristics • [e.g. know each other’s IP address on DCN]

–  Examples are Connections between Syracuse and Wisconsin to support LIGO, DVTS over DCN

2.  Communities of Users –  talk to community members over DCN –  Encourages creation of application within community

3.  Communities may develop and are developing middleware to coordinate DCN with member applications

Middleware and Dynamic Circuits

Middleware and DCN • Example Middleware that works with DCN

–  LambdaStation and TeraPaths •  Interact with routers with DCN connections • Create temporary connection between routers • Modify routers to send specific flows new connection

–  Phoebus • File transfer support (works with or without DCN) • Phoebus Gateway DCN connection • Creates temporary connection to other Phoebus

Gateways • Uses new connection for one leg of file transfer

•  Issue – sharing Middleware among communities

Data Plane Issues

VLAN Mapping SONET / Ethernet Mapping

Interdomain Dynamic Circuit Infrastructure

•  Global infrastructure assumes multiple network providers interconnecting at both the data and control level to allow creation of circuits that cross multiple networks

•  Providers are autonomous – must authorize use of resources within their network.

•  Each network domain has a domain controller •  Each network domain controller communicates with other

domains using IDC (InterDomain Control) protocol •  Connections at Control and Data layer are independent •  Data and Control Interconnection may be hierarchical or mesh

or some combination and may be different from each other

Global Dynamic Circuit Infrastructure - 2

Adding arbitrary connections between domains

Data and Control Planes

Matching Data and Control Planes

Hierarchical Control Plane

Authorization Sequences

MetaScheduler Dynamic Circuit Interface

Internetwork Collaboration metascheduler selects networks

GRID computing and DCN •  GRID computing combines computing, storage and

communication resources •  Network Services are provided in a number of ways

–  IDC network can provide circuit communication resource –  Other groups have developed similar capabilities –  OGF starting a group to define Network Service Interface –  GLIF GNI-API group working to define best practices

interface between a number of groups working on Generic Network Interface • Expectation is that OGF will take GLIF work as input •  IDC protocol developers are participating in both

activities

•  Expectation is that applications or middleware will use interface from NSI or GNI to get network connections

Dynamic Exchange Points

DCN Interdomain Infrastructure Status

•  IDC / Domain controller •  Interdomain monitoring of Links between

domains • Monitoring of circuit segments over links* • Topology availability* • Lookup service* • AAA and Policy*

Questions Comments?