Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy

Sciences • High Energy Physics • Nuclear Physics

ESnetUpdate

Steve Cotter, Dept Head Joint Techs Winter 2009

College Station, TX

•  Network Update •  OSCARS •  Futures

Agenda

ESnet4 – Jan 2009

StarLight

MAN LAN (32 A of A)

PNNL

FNL

ORNL

LLNL

GA

BNL

LANL

IP router

Lab

Optical node

SDN router Lab Link

MAN NLR 10G 20G SDN

SDN IP

Peering Link

SINet (Japan) Russia (BINP)

CERN/LHCOPN (USLHCnet:

DOE+CERN funded) GÉANT - France, Germany, Italy, UK, etc

CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2

MREN StarTap Taiwan (TANet2, ASCGNet)

AMPATH CLARA

(S. America)

CUDI (S. America)

Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren Transpac2 CUDI

KAREN/REANNZ ODN Japan Telecom America NLR-Packetnet Internet2 Korea (Kreonet2)

KAREN / REANNZ Transpac2 Internet2 Korea (kreonet2) SINGAREN Japan (SINet) ODN Japan Telecom America

CA*net4

GÉANT in Vienna (via USLHCNet circuit)

ESnet Confidential

2008 Hub & Wave Install Timeline

JUL AUG SEP OCT NOV DEC

CHIC STAR

NASH ATLA

DENV

PNWG

WASH

NEWY AOFA

CLEV BOST

HOUS KANS

ALBU ELPA

LOSA SDSC

DENV BOIS

Created by Mike O’Connor Mod by JimG

1st set of Juniper MX’s arrived at

LBNL in Mid June

2nd set of Juniper MX’s arrived at

LBNL in Mid Sept

MX480 IP MX960 SDN MX960 SDN

MX960 IP

MX960 SDN

MX960

MX960

MX960

MX960

MX960

MX480

MX480&MX960

MX960 x2

MX480 MX960

MX960 x2

MX960 x2

MX480

MX480

MX960

MX960

2008 HUB Installs 6 MX480’s

19 MX960’s SITE Installs 1 M120 PPPL

1 M10i LASV-HUB

PPPL M120

LASV M10i

NEW HUB INSTALL

EXISTING HUB UPGRADE

1 OC12 LANV-SUNN 1 10GE Internet2 STAR-CHICHIC

14 10GE Internet2 wave installed/split & accepted

19 10GE Internet2 waves installed/split & accepted

6 10GE Internet2 waves installed, split & accepted

1 10G Framenet XC in WASH

2 10GE Internet2 waves installed, split & accepted

1 10GE NLR AOFA-WASH 1 ORNL-NASH 10G IP

1 10GE MAN-LAN #2 1 10GE NRL Temp WASH-STAR 1 10GE CIC-OMNIPop at STAR

Current Hub Count: •  21 Completed: 32 AofA, NEWY*, WASH, ATLA, NASH, CLEV*, BOST*,

CHIC, STAR, KANS*, HOUS*, ELPA*, DENV, ALBU, BOIS*, PNWG, SUNN, SNV(Qwest), LOSA*, SDSC, LASV(SwitchNap)* *9 New Hubs since July 2008

Current Backbone Wave Count: •  Internet2 / Level3 Waves:

–  IP Waves: 17 new/split for a total of 25 –  SDN Waves: 25 new/split for a total of 30

•  NLR Waves: –  1 new wave for a total of 5 –  1 temp wave (STAR-WASH) for used during NLR northern path upgrade

Hub & Wave Count

MAN Upgrades Timeline

ESnet Confidential

JAN FEB MAR

MX960

SNV

Created by Mike O’Connor Mod by JimG

LBL-MR2, SNV-MR1, SNLL-MR2. LL NL-MR2 (complete)

MX960

LBNL

MX480

SNLL MX480

LLNL MX480

SLAC

MX960

NERSC

MX960

FNAL MX480

FNAL

MX960

LBNL

MX480

JGI SLAC-MR2 (complete)

FNAL’s MX Shipping Feb 3rd

MX960

BNL

MX960

ANL

MX480

BNL Final BAMAN 6509 replacements mid Feb.

1 10GE FRPG (upgrade from 1GE) DENV

2 new 10GE LIMAN (waves #3 & #4) & DF circuit between AofA-NEWY to BNL

ESnet4 Metro Area Rings

Newport News - Elite

JLab ELITE

ODU

MATP

Wash., DC

Atlanta MAN

180 Peachtree

56 Marietta (SOX)

ORNL (backup)

Wash., DC

Houston

Nashville

San Francisco Bay Area MAN

LBNL

SLAC

JGI

LLNL

SNLL

NERSC

SU

NN

USLHCNet

FNAL

600 W. Chicago

Starlight

ANL

West Chicago MAN Long Island MAN

BNL

111- 8th (NEWY)

32 AoA, NYC

USLHCNet

•  LI MAN expansion, BNL diverse entry •  FNAL and BNL dual ESnet connection •  Upgraded Bay Area MAN switches

•  1 OC-12 between SUNN-LASV (supporting Las Vegas sites and future OC-12 to GA)

•  1 10GE NLR AofA-WASH (backup for SDN & USLHCnet)

•  1 ORNL-NASH 10G IP •  1 10G FrameNet connection at WASH hub •  1 10GE MAN-LAN connection (second10G peering link) •  1 10GE NRL temp WASH-STAR •  1 10GE CIC-OMNIPop at STAR •  1 10GE FRPG (upgrade from 1GE) DENV

New ESnet4 Connections

•  New LIMAN 10G waves #3 & #4 to BNL (Early Feb) •  OC12 between LASV hub and General Atomic (Feb) •  10GE between BOST hub to MIT (Feb) •  OC12 between DENV hub and Pantex (TBD) •  1GE wave in BOIS to INL via IRON (TBD) •  10GE SDN wave between PNWG hub to PNNL (TBD) •  10GE SDN wave between NASH hun to ORNL (TBD)

Future Installs

0

500

1000

1500

2000

LIG

O

100.

000

MS

RI

100.

000

NG

A 10

0.00

0%

NS

TEC

10

0.00

0%

OR

NL

100.

000

GA

99.9

99

JGI

99.9

99

LBL

99.9

99

PN

NL

99.9

99

SLA

C

99.9

99

SN

LL

99.9

99

NE

RS

C

99.9

98

BN

L 99

.996

B

JC

99.9

95

FNA

L 99

.995

LL

NL

99.9

95

DO

E-

NN

SA

SR

S

99.9

94

Y12

99

.994

P

PP

L 99

.992

Yu

cca

99.9

92

MIT

99

.990

A

NL

99.9

89

IAR

C

99.9

89

DO

E-A

LB

99.9

78

LAN

L 99

.978

S

NLA

99

.978

P

ante

x 99

.977

LA

NL-

DC

99

.976

LL

NL-

DC

99

.976

K

CP

99.9

73

JLab

99

.964

N

RE

L 99

.961

IN

L 99

.892

D

OE

-GTN

99

.885

N

OA

A 99

.855

O

STI

99

.852

B

echt

el

99.8

08

Am

es-L

ab

99.7

67

Lam

ont

99.7

18

OR

AU

99

.660

Out

age

Min

utes

Site Availability 2/2006 to 1/2007 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

0

500

1000

1500

2000

AN

L 10

0.00

FN

AL

100.

00

Lam

ont

100.

00

LIG

O

100.

00

OR

NL

100.

00

PN

NL

100.

00

SN

LL

100.

00

DO

E-A

LB

99.9

99

JGI

99.9

99

LBL

99.9

99

MS

RI

99.9

99

NG

A 99

.999

S

LAC

99

.999

Yu

cca

99.9

99

NN

SA

99.9

98

SN

LA

99.9

98

LAN

L 99

.997

B

NL

99.9

94

MIT

99

.993

S

RS

99

.993

A

mes

-Lab

99

.991

P

PP

L 99

.991

IN

L 99

.989

N

ER

SC

99

.989

N

STE

C

99.9

89

NR

EL

99.9

85

LLN

L 99

.984

K

CP

99.9

81

IAR

C

99.9

73

GA

99.9

72

LAN

L-D

C

99.9

70

LLN

L-D

C

99.9

70

Pan

tex

99.9

55

DO

E-G

TN

99.9

20

JLab

99

.913

B

echt

el

99.8

96

Y12

99

.865

B

JC

99.8

64

NO

AA

99.8

64

OR

AU

99

.854

O

STI

99

.852

Out

age

Min

utes

Site Availability 2/2008 to 1/2009 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

5 9s 41%

4 9s 22%

3 9s 37%

Improved Site Availability

5 9s 39%

4 9s 19%

3 9s 32%

2 9s 10%

•  Network Update •  OSCARS •  Futures

Agenda

OSCARS Current (v0.5) Implementation

User User App

AAAS Authentication Authorization Auditing Subsystem

Notification Broker API Resv API

WBUI Web Based User Interface

OSCARS Core -  Reservation Management -  Path Computation -  Scheduling -  Inter-Domain Communications

PSS Path Setup Subsystem -  Network Element Interface

NS NotificationSubsystem

HTTPS HTTPS (SOAP) RMI SSHv2

ESnet Public WebServer (Proxy)

WS Interface

IDC InterDomain Controller

Source

Sink

SDN

SDN

IP

IP

IP

IP Link

ESnet WAN

SDN

Notification Call-back Event API

•  Well defined inter-module interfaces •  Exchange of static topology information •  PCE integrated into OSCARS Core

ESnet IDC (OSCARS)

OSCARS Future Implementation

User User App

AAAS Authentication Authorization Auditing Subsystem

Notification Broker API Resv API

WBUI Web Based User Interface

OSCARS Core -  Reservation Management -  Scheduling -  Inter-Domain Communications

PSS Path Setup Subsystem -  Network Element Interface

NS NotificationSubsystem

HTTPS HTTPS (SOAP) RMI SSHv2

ESnet Public WebServer (Proxy)

WS Interface

IDC InterDomain Controller

PCE Path Computation Engine

ESnet IDC (OSCARS)

Notification Call-back Event API

Source

Sink

SDN

SDN

IP

IP

IP

IP Link

ESnet WAN

SDN

•  Exchange of dynamic topology information •  includes time dimension

•  PCE separated from OSCARS Core •  PCEs can be daisy changed •  allows PCE to be pluggable •  facilitates a research framework for collaboration

•  Operational VC support –  As of 12/2/08, there are 16 long-term production VCs

instantiated, all of which support HEP •  4 VCs terminate at BNL •  2 VCs support LHC T0-T1 (primary and backup) •  12 VCs terminate at FNAL •  2 VCs support LHC T0-T1 (primary and backup) •  For BNL and FNAL LHC T0-T1 VCs, except for the ESnet PE router

at BNL (bnl-mr1.es.net) and FNAL (fnal-mr1-es.net), there are no other common nodes (router), ports (interfaces), or links between the primary and backup VC.

•  Short-term dynamic VCs –  Between 1/1/08 and 12/2/08, there were roughly 2650

successful HEP centric VCs reservations •  1950 reservations initiated by BNL using Terapaths •  1700 reservations initiated by FNAL using LambdaStation

OSCARS Use within ESnet

OSCARS is a Production Service

Site VLANS ESnet PE

ESnet Core

USLHCnet (LHC OPN)

VLAN

USLHCnet VLANS

USLHCnet VLANS

USLHCnet VLANS

USLHCnet VLANS

Tier2 LHC VLANS

T2 LHC VLAN

Tier2 LHC VLANS

OSCARS setup all VLANs

OSCARS generated and managed virtual circuits at FNAL – one of the US LHC Tier 1 data centers. This circuit map (minus the yellow callouts that explain the diagram) is automatically generated by an OSCARS tool and assists the connected sites with keeping track of what circuits exist and where they terminate.

Spectrum Now Monitors OSCARS Circuits

•  Network Update •  OSCARS •  Futures

Agenda

Historical ESnet Traffic Patterns

Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008

Tera

byte

s / m

onth

Oct 1993 1 TBy/mo.

Aug 1990 100 MBy/mo.

Jul 1998 10 TBy/mo.

38 months

57 months

40 months

Nov 2001 100 TBy/mo.

Apr 2006 1 PBy/mo.

53 months

Traffic Increases 10X Every 47 Months

July 2010 10 PBy/mo.

Network Traffic, Science Data, and Network Capacity

Historical Projection

All

Four

Dat

a S

erie

s ar

e N

orm

aliz

ed to

“1” a

t Jan

. 199

0

(HEP data courtesy of Harvey Newman, Caltech, and Richard Mount, SLAC. Climate data courtesy Dean Williams, LLNL, and the Earth Systems Grid Development Team.)

4 Pb/y 2010

40 Pb/y 2010

ESnet4 – 2010

StarLight

MAN LAN (32 A of A)

PNNL

FNL

ORNL

LLNL

GA

BNL

LANL

IP router

Lab

Optical node

SDN router Lab Link

MAN NLR 10G

30/40/50G SDN IP

50

50

30

30

50

40

40

40

40

40

40