Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy

Sciences • High Energy Physics • Nuclear Physics

Utilization of Transatlantic Circuits by European LHC Tier 2 Accesses of US LHC

Tier 1 Centers

February, 2010

William E JohnstonSenior Scientist, Energy Sciences Network

Lawrence Berkeley National Labwej@es.net, www.es.net

Managing Transatlantic Capacity Used by non-OPN LHC

• The Tier 2 accesses across the Atlantic (in both directions) utilize a mix of planned and ad hoc capacity as opposed to the carefully managed capacity of the OPN

• GÉANT provides transatlantic capacity for general IP traffic, as does NSF / IRNC (?), and several European countries

• ESnet partnered with USHLCnet some time ago to provide some capacity specifically to support European Tier 2 accesses of US Tier 1 centers

• What ESnet is seeing is LHC traffic coming from Europe on almost every available path, including once or twice over the commercial peerings

Managing Transatlantic Capacity Used by non-OPN LHC

• An analysis of the traffic from ESnet’s point of view seems essential in order to understand

- whether the current planned capacity for this traffic is adequate and - how it will need to evolve

• The current exercise (deliberately a bottom up view) is to- determine the traffic patterns,- see if they are optimal, and- establish a baseline for capacity prediction when the LHC starts producing

physics data

• Most of the European Tier 2 to US Tier 1 traffic is currently Layer 3 traffic, so focus on that

- What data is available to determine traffic patterns and how can it be used?

• What is presented here is definitely a work in progress

Layer 3 Paths Related to LHC traffic to ESnet

Clev.

Wash. DC

n

Atlanta

Nashville

USLHCUSLHC

ORNL

FNAL

BNLPhil

MAX

MANLAN

LHC OPNTo CERN

ESnet IP coreESnet Science Data Network core (N X 10G)ESnet SDN core, NLR links (backup paths)Other internationalLHC OPN (USLHCNet)Metro ring linkInternational GÉANT

5G

xGCH

FR

UK

AT

DE

FNAL

Notes:• All paths are 10Gb/s unless noted• The two ESnet transatlantic paths are carried on virtual circuits provided by USLHCnet to Amsterdam where

oGÉANT carries the 3G circuit to Vienna where it advertises ESnet routesoUSLHCNet carries the 2G circuit to CERN where CERN advertises ESnet routes internally

NL

Starlight

Logical Connectivity for ESnet, GÉANT, and USLHCnet

ESnet IP

32 AofA, New YorkMAN LAN

ESnet GÉANTVienna

Frankfurt

ESnetGÉANT

ESnet CERN

CERN (internal)

London

ESnetGÉANT

Amsterdam

ESnetGÉANT

ESnet IP

WashingtonMAX

Paris

ESnetGÉANT

Fermilab

5G

xG

What are Top Level Traffic Patterns?

ESnet IP

32 AofA, New YorkMAN LAN

ESnet GÉANTVienna

Frankfurt

ESnetGÉANT

ESnet CERN

CERN (internal)

London

ESnetGÉANT

Amsterdam

ESnetGÉANT

ESnet IP

WashingtonMAX

Paris

ESnetGÉANT

Fermilab

5G

xG

Who comes to me?

How do I get to

Fermilab?

Who Comes to Fermilab?

020406080

100120140160

CSTN

ET-A

S-AP

(749

7)VA

NDE

RBIL

T (7

212)

UN

L-AS

(789

6)JA

NET

(786

)RE

DIRI

S (7

66)

MIT

-GAT

EWAY

S (3

)DF

N-IP

(680

)Ru

ssia

n (6

801)

BELN

ET (2

611)

SON

NET

-AS

(159

)PR

INCE

TON

-AS

(88)

SWIT

CH (5

59)

NER

DCN

ET (6

356)

UCD

AVIS

-CO

RE (6

192)

DESY

-HAM

BURG

(175

4)IN

2P3

(789

)U

LTRA

LIGH

T (3

2361

)IT

EP (2

148)

UN

IV-O

F-O

KLAH

OM

A (2

5776

)RU

TGER

S (4

6)N

DGF

(395

90)

JHU

(572

3)TI

SK (2

501)

HEPN

ET-J

(250

5)CO

RNEL

L (2

6)RC

CN (1

930)

UVI

C-AS

(164

62)

NSC

KIPT

-AS

(352

96)

SIN

ET-A

S (2

907)

Fund

acao

(125

1)SI

NP-

MSU

(129

25)

KNU

-AS

(100

52)

IHEP

-SU

(264

3)ER

X-ER

NET

-AS

(269

7)BR

OW

N (1

1078

)GR

IDPN

PI (2

9493

)JIN

R/HE

PNET

(287

5)RA

DIO

-MSU

(268

3)YA

HOO

-GQ

1 (3

6647

)FR

-REN

ATER

(220

0)PI

ON

IER-

AS (8

501)

MBL

-AS-

AP (2

3674

)U

MN

-AGS

-NET

-AS

(217

)N

CREN

(81)

UCR

-EDU

(610

6)U

IUC

(38)

BNL-

AS (4

3)Re

de (1

916)

UM

DNET

(27)

EEN

et (3

221)

Mb/

s, a

vera

ge

FNAL as source, top 75 flows, past monthBase Date = 2009-12-28

Series1

020406080

100120140160

VAN

DERB

ILT

(721

2)AS

GARR

(137

)JA

NET

(786

)HE

CPER

N-A

S-PK

(457

73)

JHU

(572

3)LE

VEL3

(335

6)FU

NET

AS (1

741)

UN

L-AS

(789

6)U

CDAV

IS-C

ORE

(619

2)EE

Net

(322

1)U

NIV

-OF-

OKL

AHO

MA

(257

76)

REDI

RIS

(766

)CE

A-Sa

clay

(777

)BE

LNET

(261

1)IN

2P3

(789

)DF

N-IP

(680

)SU

RFN

ET-N

L (1

103)

UM

N-A

GS-N

ET-A

S (2

17)

ACO

NET

(185

3)RA

DIO

-MSU

(268

3)JIN

R/HE

PNET

(287

5)KF

KI-A

S (3

314)

PRIN

CETO

N-A

S (8

8)SO

NN

ET-A

S (1

59)

MIT

-GAT

EWAY

S (3

)SI

NET

-AS

(290

7)U

-CHI

CAGO

-AS

(160

)U

CSB-

NET

-AS

(131

)VI

RGIN

IA-A

S (2

25)

DESY

-HAM

BURG

(175

4)YA

HOO

-GQ

1 (3

6647

)N

ERDC

NET

(635

6)SW

ITCH

(559

)RC

CN (1

930)

KNU

-AS

(100

52)

WAS

HIN

GTO

N-A

S (7

3)TT

UN

ET (1

0421

)U

OI (

8581

)N

SCKI

PT-A

S (3

5296

)CS

TNET

-AS-

AP (7

497)

TISK

(250

1)Ru

ssia

n (6

801)

INDI

ANAG

IGAP

OP

(197

82)

BNL-

AS (4

3)N

CU-T

W (1

8420

)Re

de (1

916)

ERX-

CERN

ET-B

KB (4

538)

ULA

KNET

(851

7)HE

PNET

-J (2

505)

EEN

et (3

221)

Mb/

s, a

vera

ge

FNAL as source, top 75 flows, past monthBase Date = 2010-02-28

Series1

LHC Known European T2 sites per Ian Bird

LHC Unknown European sites

Nor

dune

t

Kha

rkov

, Ukr

aine

Ioan

nina

, Gre

ece

Kha

rkov

, Ukr

aine

How do they get to Fermilab on the US side?JANET (786)(UK)

aofa-sdn1 JINR/HEPNET (2875) star-cr1

REDIRIS (766)(Spain)

aofa-sdn1 RADIO-MSU (2683) star-cr1

DFN (680)(Germany)

aofa-sdn1 FR-RENATER (2200) aofa-sdn1

Russian (6801) star-cr1 PIONIER-AS (8501) aofa-sdn1

BELNET (2611)(Belgian)

aofa-sdn1 EENet (3221) aofa-sdn1

SWITCH (559)(Switzerland)

aofa-sdn1 ASGARR (137) aofa-sdn1

FUNETAS (1741) aofa-sdn1

DESY-HAMBURG (1754) (Germany)

aofa-sdn1 CEA-Saclay (777) aofa-sdn1

IN2P3 (789) (France)

aofa-sdn1 SURFNET-NL (1103) aofa-sdn1

ITEP (2148)(Russia)

star-cr1 ACONET (1853) aofa-sdn1

NDGF (Nordunet) 39590 (Nordic)

aofa-sdn1 KFKI-AS (3314) aofa-sdn1

RCCN (1930)(Portugal)

aofa-sdn1 NERDCNET (6356) star-cr1

NSCKIPT-AS (35296)(Ukraine)

eqx-chi-rt1(ouch)

UOI (8581) aofa-sdn1

IHEP-SU (2643) star-cr1 Rede (1916) aofa-sdn1

GRIDPNPI (29493) (Russia)

aofa-sdn1 ULAKNET (8517) aofa-sdn1

How do they get to Fermilab on the US side?

• The traffic through star-cr1 is on an IRNC (?) or country provided path or GLORIAD path

• Not all of FNAL’s traffic is LHC, though most of it is (e.g. FNAL is also the data center for the Sloan Digital Sky Survey)

Top Level Traffic Patterns

ESnet IP

32 AofA, New YorkMAN LAN

ESnet GÉANTVienna

Frankfurt

ESnetGÉANT

ESnet CERN

CERN (internal)

London

ESnetGÉANT

Amsterdam

ESnetGÉANT

ESnet IP

WashingtonMAX

Paris

ESnetGÉANT

Fermilab

5G

xG

Who comes to me?

How do I get to

Fermilab?

Austria

Belgian

UK

Switzerland

Estonia

Spain

Germany

Looking Glass to traceroute from both ends

ESnet – GEANT peering in Amsterdam

4.0G

Out In

ESnet – GEANT peering in Vienna

1.4G

ESnet – GEANT peering in Frankfurt

1.0G

By-Site Traffic: Europe -> FNAL (Mb/s)

By-Site Traffic: FNAL-> EuropeThe pattern of substantially more Tier 1 (FNAL) to Tier 2 (Europe) traffic compared to T2 to T1 is at least approximately consistent with theEuropean Tier 2s getting data from the US Tier 1s

Next Steps

• Sort by-site traffic by path• Disambiguate LHC site traffic in European ASNs• Decide what is useful to publish monthly

- Engage Internet2 to publish similar data for their network

• Would expect to see the complimentary pattern –US Tier 2s pulling data from European Tier 1s

- Probably need to also involve NLR and the new IRNC links as some of the US Tier2 connect directly to the New York exchange point (MAN LAN) and if GEANT peers with them, Internet2 would not see the traffic

• END of Slides

YUCCA MT

PNNL

SNLAAlliedSignal

ARM

KCP

NOAA

OSTI

ORAU

SRS

JLAB

PPPLLab DCOffices

MIT/PSFC

BNL

NREL

GA

DOE GTNNNSA

NNSA Sponsored (13+)Joint Sponsored (4)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

~45 end user sites

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD(Russia, China)

Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)SingarenTranspac2CUDI

commercial peering points

PAIX-PAEquinix, etc.

ESnet

ESnet core hubs

CERN/LHCOPN(USLHCnet:

DOE+CERN funded)

GÉANT- France, Germany,

Italy, UK, etc

JGILBNLSLAC

NERSC

Equinix

MRENStarTapTaiwan (TANet2,

ASCGNet)

NASA

Ames

AU

AU

Specific R&E network peers

AMPATHCLARA

(S. America)CUDI(S. America)

R&Enetworks

Office Of Science Sponsored (22)

NSF/IRNCfunded

KAREN/REANNZODN Japan Telecom AmericaNLR-PacketnetInternet2Korea (Kreonet2)

KAREN / REANNZ Transpac2Internet2 Korea (kreonet2)SINGAREN Japan (SINet)ODN Japan Telecom America

NETL

International peers (10 Gb/s)10-20-30 Gb/s

SDN core (I2, NLR)10Gb/s IP coreMAN rings (Nx10 Gb/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Salt Lake

DOE

Geography isonly representationalOther R&E peering points

USH

LCN

etto

GÉA

NT

CA*net4

GFDLPU Physics

UCSD Physics

NSTEC

An accurate map is located at http://www.es.net/pub/maps/current.pdf

ESnet Vienna peering with GÉANT (via USLHCNet circuit)

USLHCNet 1/2010

GEANT International