1

ESnet Update

ESnet/Internet2 Joint Techs Albuquerque, New Mexico

February 6, 2005

R. Kevin Oberman

ESnet Network EngineerLawrence Berkeley National Laboratory

2

OutlinePhysical Connectivity

o What’s newo BAMAN in productiono LIMAN nearing productiono CHIMAN under constructiono New Circuits

Traffic Driving:o Interconnectivityo Architecture

IP core to Chicago (Qwest)

IP core to El Paso

NLR to Seattle

NLR to San Diego

SF Bay Area

λ4 future

λ3 future

λ2 SDN/circuits

λ1 production IP

SLACQwest /

ESnet hub

SNLL

Joint Genome Institute

LBNL

NERSC

LLNL

NASAAmes

Level 3hub

ESnet hubs and sites

DOE Ultra Science Net

(research net)

ESnet MAN ring (Qwest

circuits)

ESnet SF Bay AreaMAN Ring

10 Gb/soptical channels

• 2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits

- 10-50X current site bandwidth

• Dual site connection (independent “east” and “west” connections) to each site

• Used as a 10 Gb/s production IP ring and2 X 10 Gb/s paths (for circuit services) to each site

• Project completed 9/2005, on time and on budget

4

AbileneNYSERNetSINet (Japan)CANARIE (Canada)HEANet (Ireland)Qatar

Long Island MAN Initial Configuration

ESnet boarder

demarcationdevice

32 A

oA, N

YC

BrookhavenNational Lab, Upton, NY

10GEcircuit

BNLIP gateway

1 GE or 10 GE

MAN LAN

ESnetIP core

GEANT

Europe

Washington

(Qwest hub)

10 Gb/s circuitsInternationalLHCnet circuits (proposed)production IP coreSDN/provisioned virtual circuits

LI MAN DWDM

ESnet IP core(Qwest)

CERN Chicago(Qwest hub)S

tarli

ght

T320

US

LHC

net

Cisco 6509

5

Chicago MAN

Starlight 455 N. City Front

Fermilab Argonne

T320

T320

USLHCnet

ESnet IPSDN

3x10G

4x10G

6

New 10G Circuits• ESnet is nearing completion of a process to turn up 44 new

10G circuits in less than 12 months including:o MANs

- 16 circuits composing the San Francisco Bay Area MAN - 2 circuits comprising the Long Island MAN (New York to Brookhaven)- 15 circuits comprising the Chicago MAN (FNAL, ANL and StarLight)

o NLR- 3 circuits: San Diego to Sunnyvale, Sunnyvale to Seattle, hub to hub in

Sunnyvaleo Sites

- 1 circuit JLAB (via, MAX/NGIX-E, MATP, E-LITE) - 1 circuit between NYC and DC to reach GEANT - 1 circuit ORNL – Starlight - CHI-HUB(NBC)

o Peering- 1 circuit GEANT- 1 circuit SNV – NASA-AMES

7

Other Circuits

• Completed:o OC48 ALB-ELPo DS3 IARC-SNV (New Site) o OC3 ALB-KCP

• In the works: o OC3 ALB-HUB-PANTEX <-- In build-out

8

The Office of Science U.S. Community*

Institutions supported by SC Major User Facilities

DOE Multiprogram LaboratoriesDOE Program-Dedicated LaboratoriesDOE Specific-Mission Laboratories

Pacific NorthwestPacific NorthwestNational LaboratoryNational Laboratory Ames LaboratoryAmes Laboratory

Argonne National Argonne National LaboratoryLaboratory

BrookhavenBrookhavenNationalNational

LaboratoryLaboratory

Oak RidgeOak RidgeNational National

LaboratoryLaboratoryLos AlamosLos Alamos

National National LaboratoryLaboratory

Lawrence Lawrence LivermoreLivermore

&&SandiaSandia

National National LaboratoriesLaboratories

LawrenceLawrenceBerkeley Berkeley NationalNational

LaboratoryLaboratory

FermiFermiNationalNational

Accelerator Accelerator LaboratoryLaboratory

PrincetonPrincetonPlasmaPlasmaPhysicsPhysics

LaboratoryLaboratory

Thomas Jefferson Thomas Jefferson National National

Accelerator FacilityAccelerator Facility

NationalNationalRenewable Energy Renewable Energy

LaboratoryLaboratory

StanfordStanfordLinearLinear

Accelerator Accelerator CenterCenter

Idaho National Idaho National LaboratoryLaboratory

SC Program sites

General General AtomicsAtomics

SandiaSandiaNational National

LaboratoryLaboratory

9

NLR

Sup

plie

d Ci

rcui

ts

LVK

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

SNLAPANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

MIT

ANL

BNLFNAL

AMESLLNL

GA

OSC GTNNNSA

International (high speed)Lab Supplied10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (4)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

ESnet IP core

CA*net4 FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over SONET Optical Ring and

Hubs

ELP

ATL

DC

commercial and R&E peering points

MAE-E

PAIX-PAEquinix, etc.

PNW

GPo

P/PA

cific

Wav

e

ESnet’s Physical Connectivity (Spring 2006)*

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

StarlightChi NAP

CHI-SL

SNV

Abilene

SNV SDN

JGILBNL

SLACNERSC

SDSC

Equinix

MAXGPoP

SoXGPoP

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2)UltraLight

AU

AU

SEA

SINet (Japan)Russia (BINP)

LANL

NREL

AMPATH(S. America)

AMPATH

MA

N LA

NAb

ilene

MAN rings

Abile

ne

USN

USN

IARC

NA

SAA

mes

KCP-ALB UNM

LLNL/LANLDC Offices

LBNL DC

ORAU DC

DOE-ALB

10

Network UtilizationTB

ytes

/Mon

th

ESnet Accepted Traffic (Bytes)Jan, 1990 to Dec., 2005

ESnet is Currently Transporting About 750 Terabytes/mo.and this volume is increasing exponentially

Feb,

90

Jul,

90D

ec, 9

0M

ay,9

1O

ct, 9

1M

ar, 9

2A

ug, 9

2Ja

n, 9

3Ju

n, 9

3N

ov, 9

3A

pr, 9

4S

ept,

94Fe

b, 9

5Ju

l, 95

Dec

, 95

May

, 96

Oct

, 96

Mar

, 97

Aug

, 97

Jan,

98

Jun,

98

Nov

, 98

Apr

, 99

Sep

, 99

Feb,

00

Jul,

00D

ec, 0

0M

ay, 0

1O

ct, 0

1M

ar, 0

2A

ug, 0

2Ja

n, 0

3Ju

n, 0

3N

ov, 0

3A

pr, 0

4S

ep, 0

4Fe

b, 0

5Ju

l, 05

Dec

, 05

0

100

200

300

400

500

600

700

800

11

R2 = 0.991

0.0

0.1

1.0

10.0

100.0

1000.0

Feb, 90Aug, 90Feb, 91Aug, 91Feb, 92Aug, 92Feb, 93Aug, 93Feb, 94Aug, 94Feb, 95Aug, 95Feb, 96Aug, 96Feb, 97Aug, 97Feb, 98Aug, 98Feb, 99Aug, 99Feb, 00Aug, 00Feb, 01Aug,01Feb, 02Aug, 02Feb, 03Aug, 03Feb, 04Aug, 04Feb, 05Aug, 05

Network Utilization

Oct., 1993

Aug., 1990

Jul., 1998

39 months

57 months

42 months

ESnet traffic has increased by about 10X every 46 months, on average,since 1990

Dec., 2001

TByt

es/M

onth

12

OSCARS: Guaranteed Bandwidth VC Service For SC Science

• ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS)

• To ensure compatibility, the design and implementation is done in collaboration with the other major science R&E networks and end sites

o Internet2: Bandwidth Reservation for User Work (BRUW) - Development of common code base

o GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS)

- Extends to NRENso BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Peta-

scale Computing Research o GA: Network Quality of Service for Magnetic Fusion Research o SLAC: Internet End-to-end Performance Monitoring (IEPM) o USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science

• In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program

• A prototype service has been deployed as a proof of concepto To date more then 20 accounts have been created for beta users, collaborators, and

developerso More then 100 reservation requests have been processed

13

Observation

• In 4 years, we can expect a 10x increase in traffic over current levels without the addition of production LHC traffico Nominal average load on busiest backbone links is

~1.5Gbps todayo In 4 years that figure will be ~15Gbps if current trends

continue

• Measurements of this kind are science-agnostico It doesn’t matter who the users are, the traffic load on the

backbone is increasing

• Bandwidth trends drive requirement for a new network architecture

14

10-50 Gb/s circuitsProduction IP coreScience Data Network coreMetropolitan Area NetworksInternational connections

MetropolitanArea Rings

Primary DOE Labs

IP core hubs

ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings

possible hubs

SDN hubs

Europe(GEANT)

Asia-Pacific

New York

Chicag

o

Washington, DC

Atla

nta

CERN

Seattle

AlbuquerqueAus

.A

ustr

alia

San Diego

LA

Sunn

yval

e

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN

Loop off Backbone

Canada(CANARIE)

Europe(GEANT)

SDN Core

IP Core