Supporting Optical Networking for Research and Education in the

United States

Christian Todorov

September 20, 2007

Christian Todorov2

Agenda

• Previous Generation Network

• New Requirements of R&E

• The Internet2 Network

• Looking Forward

Christian Todorov

Abilene Network

• Internet2’s previous network (Abilene) was IP only and based on unprotected 10G (OC-192) waves on the Qwest infrastructure

• Utilized IS-IS• Natively supported IPv4/v6 and multicast • IP is great… for most people… most of

the time…

3

Christian Todorov

Changing Needs

• As the capabilities available to the research community changed, the demands on the network also changed

• e-science applications and facilities grew and exerted greater performance pressures on the network where TCP and shared environments were no longer acceptable

• Dedicated infrastructure was becoming a requirement

4

Christian Todorov

The New Requirements

• High performance applications are dependent on high performance networks

• Networks must be fast, reliable, scalable, have flexible architectures, be cost effective, capable of delivering multiple services across multiple network layers, easy to operate and maintain, and have a view towards the future

• Enable the user – the network as a service

5

Christian Todorov

Demands on the Network

• Entering the age of large scientific facilities• Large Hadron Collider at CERN• Very Long Baseline Arrays (radio astronomy)• Large Synoptic Survey Telescope (2010-13) – 30TB/night

• An increasingly diverse set of demanding applications are utilizing network resources• Telemedicine: BIRN project, proteomics, tele-surgery, remote

ICU, radiology: high-resolution 3D color fMRI brain scan = 4.5PB • Telepresence: master classes, virtual classrooms, tele-psychiatry• High performance video delivery: Uncompressed HD, Cinegrid• Disaster Recovery and distributed storage

6

Christian Todorov

Christian Todorov

• People• 3000 CERN employees• 6500 visiting scientists from 500 Universities in 80 countries

• Physical Size• 27 Km circumference • 9300 magnets• 7 Tev nominal proton energy• 600 million collisions per second

• Experimental Facilities• ALICE (A Large Ion Collider Experiment) – Study quark-gluon plasma• ATLAS (A Toroidal LHC ApparatuS) – Search for Higgs boson• CMS (Compact Muon Solenoid) – Search for Higgs boson• LHCb (LHC-beauty) – Study the CP violation phenomenon• LHCf (LHC-forward) – Study astroparticle physics

CERN – Large Science Facility

CERNTier 0 Raw Data

FNAL BNLShared Data Storage and Reduction

Tier 1(12 orgs)

US Tier 2(15 orgs)

CMS Atlas

US Tier 3 (68 orgs)

US Tier 4 (1500 US scientists)

Scientists Request Data

LHCOPN

GEANT-ESNet-Internet2

Internet2/Connectors Internet2/Connectors

Local Infrastructure

Christian Todorov10

CERN

Tier 0 to Tier1: Requires 10-40 Gbps

Tier 1 to Tier 2: Requires 10-20 Gbps

LHCOPN

GEANT-ESNet-Internet2

Internet2/Connectors Internet2/Connectors

Tier 1 or 2 to Tier 3: Estimate: Requires 1.6 Gbps per transfer

Peak Flow Network Requirements

Local Infrastructure

Christian Todorov

Tier 1 to Tier 2 Traffic

Christian Todorov

The Internet2 Network Overview

• Layer 1: Managed wavelengths from Level(3) Communications • Level(3) owns and manages Infinera optical gear: responsible for

software upgrades, equipment maintenance, remote hands, sparing, NOC services

• Internet2 NOC has total provisioning control• Layer 2: Internet2 owned and managed Ciena CoreDirectors

• Using DRAGON GMPLS control plane • Layer 3: Internet2 owned and managed Juniper T640s • Expanded Observatory

• Platform for layer 1/3 network performance data collection, collocation, experimentation

• perfSONAR integration for intra- & inter-network performance analysis• International connectivity

• Layer 1 network extended to international exchange points in Seattle, Chicago and New York City

• Peering points in Seattle, PAIX, Equinix Chicago

12

Christian Todorov

Network Design

13

Christian Todorov

Network Numbers

• 13,500 long haul route miles, 64 metro fiber route miles

• Deployed and configured over 300 Infinera Network Elements

• 23 Ciena Core Directors

• 9 Juniper T640s

• Day 1 capacity of 100Gbps

• Built 27 custom collocation suites representing 3,365 sqft of space including:• 91 Racks - Internet2, ESnet, third-parties• 60 Individual bulk cables with 48 & 96 fiber count

• Internet2 and ESNet NOCs get same, real-time feeds as the Level(3) NOCs in Atlanta & Denver

• Developed the Virtual Network Operations Center – Provisioning and Troubleshooting Dashboard

14

Christian Todorov

Title of slide

• Level 1• Level 2

• Level 3• Level 4

• Level 5

Christian Todorov

• Best-Effort 10G IP Service• Enables delivery of advanced content, commodity services, etc.• Dual stack IPv4, IPv6; IPv4 & v6 multicast and jumbo frame enabled

• Point-to-Point Wavelength Services• Circuit Service for static or on-demand bandwidth

• Point-to-point Ethernet (VLAN) Framed SONET Circuit• Point-to-point SONET Circuit• Bandwidth provisioning available in 50 Mbps increments• Supports GFP, VCAT and LCAS• Various protection options for both waves and sub-rate circuits

• Physical Connection• 1 or 10 Gigabit Ethernet• OC-192 SONET

Flexible Infrastructure Supporting e-Science, Network Research & Education

Christian Todorov17

•Infinera DWDM Gear - Static at the start•Grooming capabilities in ADM to provide sub channels and HOPI types of activities at the start•Simplified and standardized interface to connectors, exchange points, and other global research and education networks - 2 x 10 Gbps interfaces•Measurement and control servers will support the node

Christian Todorov

Wavelength & Circuit Services

• Connection oriented services provide for:• Guaranteed bandwidth and predictable jitter and latency

(repeatable, dependable performance between collaborating sites)

• Traffic segregation (support specific policy or traffic engineering requirements)

• Router bypass: Express links created for high-bandwidth, limited duration long-haul traffic reducing the need for mid-path L3 interfaces

• Cost efficiency: L3 router blades cost > L2 ports > L1 or L0 interfaces

• Capability tradeoff but could possibly improve performance

18

Christian Todorov19

Lightpath Provisioning

Christian Todorov

Multi-Service/Domain/Layer/Vendor Provisioning

Regional Network Regional

Network

Internet2 Network

ESNet

Dynamic Ethernet Dynamic EthernetTDM

GEANT

IP Network (MPLS, L2VPN)

Ethernet

Router

SONET Switch

Ctrl Element

Domain Controller

LSP

Data PlaneControl Plane Adjacency

• Multi-Domain Provisioning• Interdomain ENNI (Web Service and OIF/GMPLS)• Multi-domain, multi-stage path computation process• AAA• Scheduling TDM

Slide from Tom Lehman, ISI-East

GUI

XML

AST

Christian Todorov

Wavelength & Circuit Services

• Automated circuit provisioning enable rapid deployment and efficient utilization of capital investment• Establishing end-to-end lightpaths is a non-trivial task: it is

resource intensive and error prone • Automated reservation, allocation, and provisioning enables co-

scheduling of network and non-network resources – i.e. radio telescopes

• Greater efficiency in the core network means these savings can be passed down to the members as lower cost wavelength and IP services.

Christian Todorov

Scalability and Operability

• The Internet2 Network is based on a unique arrangement with Level 3 that represents a hybrid approach to carrier provided resources.

• Internet2 has full control over the provisioning on the network but does not bear the responsibility of supporting and maintaining the physical infrastructure: fiber, amps, transport equipment, etc.

• Level 3’s support of the physical network frees Internet2 of having higher levels of specialized engineering resources dedicated to network support.

• The Internet2 NOC has a full view into the underlying transport equipment and works jointly with a dedicated NOC group within Level 3.

• The agreement with Level 3 is essentially one for capacity that has no upper limit. Internet2 can continue to add capacity to network even beyond the carrying capacity of the transport chassis and the fiber.

• The Internet2 network is constructed on a dedicated fiber pair and with dedicated transport equipment

• The Infinera, Ciena, and Juniper equipment used in the network are 40G capable and each has 100G on their roadmaps

22

Christian Todorov

• Dependence on the network is increasing• Distributed applications• Moving larger data sets

• Network is growing much more complex• Dynamic and static circuits• Network security issues

• Need to better understand the network• User must know what performance levels to expect• Network operators must be able to demonstrate that the

network meets or exceeds those expectations.• Application developers must have access to tools that

differentiate between network problems and application problems.

Expanded Network Measurement

Christian Todorov

• OWAMP (latency)• Regular tests between all routers, and on-demand

• BWCTL (throughput)• New version with more ‘testers’ available in August• Regular tests between all routers, and on-demand

• NDT (User Diagnostic)• 3.4.1 available now, with better logging and error handling

• NPToolKit (Knoppix system image that includes all tools)• Recent versions of Measurement Tools installed and pre-

configured• http://e2epi.internet2.edu/network-performance-toolkit.html

Internet2 Measurement Tools

Christian Todorov

• perfSONAR motivation• Most organizations monitor and diagnose their own network• Networking is becoming an increasingly cross-domain effort• Monitoring and diagnostics must also become a cross-domain

effort• perfSONAR

• A set of protocols and schemas for implementing a Service-Oriented Architecture for sharing/controlling network performance tools

• A global community of users and developers. Joint collaboration between GEANT2, ESnet, Internet2 and RNP (Brazil) as well as numerous connected participants

• Provides infrastructure for network performance monitoring on cross-domain links; contains a set of services delivering performance measurements in a federated environment

perfSONAR

Christian Todorov

Objectives

• The vision for the Internet2 Network is a seamless integrated network facility that allows for applications and users to transparently utilize the services and network layers that most appropriately serve their needs, when they need it, in a cost effective manner.

• This network facility will allow users to focus on their work and not on the network.

Christian Todorov

Questions?

Christian Todorov28