1

ESnet UpdateSummer 2007

Joint Techs Workshop

Joe BurresciaESnet General Manager

July 16,2007

Energy Sciences NetworkLawrence Berkeley National Laboratory

Networking for the Future of Science

2

TWC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

Lab DCOffices

MIT

ANL

BNL

FNALAMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTNNNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over

SONET Optical Ring and Hubs

ELP

DC

commercial peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P/

PA

cifi

cWav

e

ESnet 3 with Sites and Peers (Early 2007)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

Ab

ilen

e

CERN(USLHCnet

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight

SNV

Ab

ilene

JGI

LBNL

SLACNERSC

SNV SDN

SDSC

Equinix

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

AU

SEA

CH

I-SL M

AN

LA

NA

bile

ne

Specific R&E network peers

Other R&E peering points

UNM

MAXGPoP

AMPATH(S. America)

AMPATH(S. America)

ES

net

Scie

nce D

ata

Netw

ork

(S

DN

) core

R&Enetworks

Office Of Science Sponsored (22)

ATL

NSF/IRNCfunded

Equinix

3

ESnet 3 Backbone as of January 1, 2007

Sunnyvale

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)MAN rings (≥ 10 G/s)Lab supplied links

Seattle

San Diego Albuquerque

El Paso

Chicago New York

City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

4

ESnet 4 Backbone as of April 15, 2007

Clev.

Boston

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

5

ESnet 4 Backbone as of May 15, 2007

SNV

Clev.

Boston

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

6

ESnet 4 Backbone as of June 20, 2007

Clev.

Boston

Houston

Kansas City

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

7

ESnet 4 Backbone Target August 1, 2007

Clev.

Boston

Houston

Los Angeles

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

8

ESnet 4 Backbone Target August 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

9

ESnet 4 Backbone Target September 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Nashville

Future ESnet Hub

ESnet Hub

10

DC

ESnet 4 Backbone Target September 15, 2008

Houston 10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Los Angeles

Nashville

Future ESnet Hub

ESnet Hub

11ESnet4Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe(GEANT)

Asia-Pacific

New York

Chicago

Washington DC

Atl

anta

CERN (30+ Gbps)

Seattle

Albuquerque

Au

str

ali

a

San Diego

LA

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN (30+ Gbps)Canada(CANARIE)

Asi

a-Pac

ific

Asia Pacific

GLORIAD (Russia and

China)

Boise

HoustonJacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s

City

Au

str

ali

a

Core network fiber path is~ 14,000 miles / 24,000 km

162

5 m

iles

/ 2

545

km

2700 miles / 4300 km

Sunnyvale

Production IP core (10Gbps)

SDN core (20-30-40-50 Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

IP core hubs

Primary DOE LabsSDN hubs

High speed cross-connectswith Ineternet2/AbilenePossible hubs

USLHCNet

12Typical ESnet 4 Hub

M320 Router7609 Switch

10G Performance Tester

Peering Router

Secure Term ServerPower Controllers

OWAMP Measurement

Device

13ESnet 4 Factiods as of July 16, 2007

• Installation to date:o 10 new 10Gb/s circuits

o ~10,000 Route Miles

o 6 new hubs

o 5 new routers 4 new switches- Total of 70 individual pieces of equipment shipped

– Over two and a half tons of electronics

o 15 round trip airline tickets for our install team- About 60,000 miles traveled so far….

- 6 cities– 5 Brazilian Bar-B-Qs/Grills sampled

14ESnet Traffic Now Exceeding 2 PetaBytes/Month

TBytes Accepted

0.00

500.00

1000.00

1500.00

2000.00

2500.00

3000.00

Jan, 00 Jan, 01 Jan, 02 Jan, 03 Jan, 04 Jan, 05 Jan, 06 Jan, 07

2.4PBytes in May 2007

Te

raB

ytes

ESnet traffic historically has increased 10x every 47 months

1 PBytes in April 2006

15

ESnet Availability 8/2006 through 7/2007

0

500

1000

1500

2000

2500

OR

NL

100

.000

SL

AC

100

.000

NE

RS

C 9

9.99

8

SN

LL

99.

998

LB

L 9

9.99

7

PN

NL

99.

997

LL

NL

99.

996

FN

AL

99.

995

IAR

C 9

9.99

5

SR

S 9

9.99

5

Yu

cca

99.9

93

BJC

99.

991

AN

L 9

9.99

0

JGI

99.9

90

LA

NL

-DC

99.

986

LL

NL

-DC

99.

986

MIT

99.

985

JLab

99.

982

PP

PL

99.

982

GA

99.

973

BN

L 9

9.97

0

DO

E-A

LB

99.

955

LA

NL

99.

955

SN

LA

99.

955

All

ied

99.

951

Pan

tex

99.9

49

Bec

hte

l 99

.948

NR

EL

99.

941

DO

E-N

NS

A 9

9.91

8

DO

E-G

TN

99.

889

INL

99.

883

OR

AU

99.

878

OS

TI

99.8

68

Am

es-L

ab 9

9.85

1

NO

AA

99.

782

Lam

on

t 99

.624

Ou

tag

e M

inu

tes

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Jul

ESnet Availability

“5 nines” (>99.995%) “3 nines” (>99.5%)“4 nines” (>99.95%)

Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.

16OSCARS Overview

Path Computation• Topology

• Reachability• Contraints

Scheduling• AAA

• Availability

Provisioning• Signalling• Security

• Resiliency/Redundancy

OSCARSGuaranteedBandwidth

Virtual Circuit Services

On-demand Secure Circuits and Advance Reservation System

17OSCARS Status Update

• ESnet Centric Deploymento Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet

(1Q05)o Layer 2 (Ethernet VLAN) virtual circuit service under development

• Inter-Domain Collaborative Effortso Terapaths

- Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06)- Inter-domain interoperability for layer 2 virtual circuits under development

o HOPI/DRAGON- Inter-domain exchange of control messages demonstrated (1Q07)- Initial integration of OSCARS and DRAGON has been successful (1Q07)

o DICE- First draft of topology exchange schema has been formalized (in collaboration with NMWG)

(2Q07), interoperability test scheduled for 3Q07- Drafts on reservation and signaling messages under discussion

o UVA- Integration of Token based authorization in OSCARS under discussion

18ESnet Measurement Update

• perfSONAR Services Deployedo Measurement Archives serving Utilization information (Demo

later today)o E2EMP serving end-to-end circuit status informationo Visualization tools

• Bandwidth & Latency Measurement Pointso Deployed at ESnet4 hubs as they are configured

• Working with the LHC communityo To define a set of perfSONAR services that can meet the

network measurement & monitoring needs of large distributed science application communities.