04/19/23 1

The Internet & SLACLes Cottrell1, SLAC

http://www.slac.stanford.edu

/grp/scs/net/talk/internet-connectivity-97/index.htm

Outline of Talk– I. SLAC’s connectivity – II. How is it Working?– III. Why is it like it is?– IV. What’s going on out there on the Internet?– V. What are we (DOE/Esnet, HEP, SLAC) doing?– VII. Summary & Future

Talk presented at SLAC, July 1997

04/19/23 2

Some Acronyms ARA - Appletalk Remote Access, protocol to connect up remote Macs ATM - Autonomous Transfer Mode, a high high speed network mechanism DSL - Digital Subscriber Loop, a proposed medium speed (100s kbps -

Mbps) leased line service (phone company answer to cable modems) ESnet - Energy Sciences network (DOE’s research network, SLAC’s main

connection to Internet ISDN - Integrated Switched Digital Network, new <= 128 kbps digital

switched phone service) POP - Point of Presence, a place where one or more networks have facilities SLIP/PPP protocols to provide Internet access over a serial line VPN - Virtual Private Network, a way of tunneling private data over the

public Internet WAN - Wide Area Network

04/19/23 3

Outline

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the

Internet? V. What are we (DOE/Esnet, HEP, SLAC)

doing? VII. Summary & Future

04/19/23 4

Dial in Accesshttp://www.slac.stanford.edu/comp/net/residential.html

Terminal/emulator dial in

– 7 ports, 14.4 kbps ARA

– 16 ports, <=33.6kbps, ~340 accounts (85 active/mo)

SLIP/PPP thru campus

– 14.4kbps, need campus account Netcom, $15/mo, nationwide

– 28.8 kbps Wireless via Ricochet ISDN Direct & via ISP

– 9 ports, <=128 kbps, in pilot mode ~25 users, production service late summer

Following VPN developments

04/19/23 5

SLAC’s WAN Connectivity 43Mbps to ESnet ATM

cloud (Sprint Oakland POP)

1.5Mps to Caltech/ESnet 1.5Mbps to LBNL/ESnet 10Mbps to Stanford

04/19/23 6

Outline

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the Internet? V. What are we (DOE/Esnet, HEP, SLAC)

doing? VII. Summary & Future

04/19/23 7

What is Important to User

We have to optimize the scarcest & therefore most valuable commodity - Time

How long does it take after I hit the button?

04/19/23 8

Value of Rapid Response Time Studies in late 70’s early 80s by Walt Doherty of

IBM & others showed the economic value of rapid response time:– 0-0.4s = High productivity interactive response

– 0.4-2s = Fully interactive regime

– 2-12s =Sporadically Interactive regime

– 12s-600s =Break in contact regime

– >600s =Batch regime

There is a threshold around 4-5s where complaints increase rapidly

04/19/23 9

Average Ping Response for Various Groups of Hosts Seen from SLAC Jan-95

thru Dec-96

0

100

200

300

400

500

600

700

Jan-

95

Mar

-95

May

-95

Jul-9

5

Sep-9

5

Nov-9

5

Jan-

96

Mar

-96

May

-96

Jul-9

6

Sep-9

6

Nov-9

6

Pin

g R

esp

on

se (

ms)

ESNET

Internatl

NAmericaE

NAmericaW

International little changeN. America E improving 210 ms -> 150ms N. America W improving 140 ms -> 80msESnet improving 100ms -> 50 ms

Ping Response for Groups of Hosts

04/19/23 10

European/Japan Packet Loss to SLAC

Packet Loss to Major HEP International Sites seen from SLAC Jan-95 thru Apr-97

0

5

10

15

20

25

30

35

40

45

Sep-94 Jan-95 Apr-95 Jul-95 Oct-95 Feb-96 May-96 Aug-96 Dec-96 Mar-97 Jun-97

% 1

00

By

te P

ing

Pa

ck

et

Lo

ss

CERN.CH

DESY.DE

IN2P3.FR

RL.AC.UK

KEK.JP

ROMA1.INFN.IT

Increase UK-US bandwidth

Improve Esnet - Internet connect

RAL: poor to unnacceptable, most others acceptable

Packet loss much more important– loss of packet typically causes 4-5s timeout

04/19/23 11

Quality by Host GroupPing Loss Quality

Distributions for Host Groups

0%

10%

20%

30%

40%

50%

60%

70%

Esnet

ISP L

ocal

Inte

rnat

ional

NAmer

icaE

NAmer

icaW

Pe

rce

nti

le

<= 1% Loss (==Good)>1% & <=5% Loss (==Acceptable)>5% & <=12% Loss (==Poor)> 12% & <=25% Loss (==Bad)>25% Loss (==Unusable)

(76, 5.46) (183, 7.18)(150, 0.79)

(199, 6.3) (188, 6.21)

(host-months, median loss)

0.0-1% Good, 1-5% Acceptable, 5-12% Poor

12-25% Bad, > 25% Unusable

Similar to Internet Weather Report (<6%, <12%, > 12%)

04/19/23 12

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the Internet? V. What are we (DOE/Esnet, HEP, SLAC)

doing? VII. Summary & Future

Outline

04/19/23 13

Driving Forces - Hosts

04/19/23 14

Driving Forces - New Apps

WWW, multimedia, Internet voice, video conferencing– > 60% graphics– <20% HTML

04/19/23 15

Driving Forces - Penetration

Countries with Internet access

US Domains

04/19/23 16

Current Internet Hosts

04/19/23 17

Challenge - Diversity of Traffic

it2%

gov 4%

com14%

net19%

edu43%

br3%

other domains13%

ch2%

com14%

Traffic out ofFNAL

04/19/23 18

Challenge - No single Mgmt for Links1 RTR-CGB4.SLAC.Stanford.EDU

2 RTR-DMZ.SLAC.Stanford.EDU

3 ESNET-A-GATEWAY.SLAC.Stanford.EDU

4 pppl-atms.es.net

5 nynap-pppl-atms.es.net

6 192.157.69.11 [Sprint NAP]

7 core3-hssi3-0.WestOrange.mci.net

8 core1.WestOrange.mci.net

9 border2-fddi-0.WestOrange.mci.net

10 border2-hssi1-0-gw.WestOrange.mci.net

11 192.204.183.3 [PREPnet]

12 DEFAULT1-GW.UPENN.EDU

13 NISC8.UPENN.EDU

04/19/23 19

Challenge

Commercial Internet focussed on staying alive as opposed to research or promoting advanced requirements

04/19/23 20

More Acronymns CalREN2 - a California initiative to provide better educational &

research networking CHEP97 - Computing in High Energy Physics meeting in Berlin,

April 1997 ESSC - ESnet’s Steering Committee ICFA - International Committee on Future Accelerators Internet 2 - University initiative to provide improved networking

between universities NGI - Next Generation Internet, Presidential initiative vBNS - very high-speed Backbone Network System, a high speed

NSF funded backbone network

04/19/23 21

Outline

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the

Internet? V. What are we (DOE/Esnet, HEP, SLAC)

doing? VII. Summary & Future

04/19/23 22

New Initiatives - California

CalREN2 – joint proposal NSF, UC, Stanford, Caltech …– includes Pac Bell & Cisco– Distributed GigaPOPs in SF & LA, also SD & Sac– Hi speed (622Mbps) ring around state envisioned

04/19/23 23

Bay Area

Bay Area CalREN2 GigaPOP nodes:– UCSF– UCB (links to Esnet, Sprint, MCI/vBNS, UC

Davis (state wide)– UCOP– Stanford (links to NASA/NSI, BBN, MCI &

statewide ring)

04/19/23 24

New U.S. Initiatives: vBNS

NSF initiative for interconnecting supercomputer centers for “meritorious applications”

Extended to promote University interconnectivity

622 Mbps backbone

04/19/23 25

New US Initiatives: Internet 2 Started out (Oct-96) as

consortium of ~ 34 major universities– Now there are over 100

covers 80% of US university sites we monitor

– ~$500K / university over several years, 25% seed

– Will use vBNS as backbone

– GigaPOPs in major areas

04/19/23 26

Next Generation Internet (NGI)

Presidential Initiative– $100M/yr for 3 years– 100 sites at 100 times bandwidth (1.5Mbps => 155Mbps

backbone)– 10 sites at 1000 times bandwidth– DARPA, DOE, NSF, NASA…

Internet2/NGI/ESnet relationship unclear– can Universities connect to Internet 2 & ESnet?

04/19/23 27

U.S. International Connections

Country Today (Jan-Mar '97) PlansBrazil 128kbps to FNALCanada 2*45Mbps Move to DC POPCERN 2Mbps Move to DC POP May-97France IN2P3 via CERN ?Germany (DFN) 1.54 Mbps to DC POP Add part of 2*45MbpsItaly 1.54Mbps to PPPL Look to move to DC POPJapan KEK to FixW 512kbps KEK 1.54Mbps to LBNLUK (9 + 8.5)Mbps (ANS+Sprint) 45Mbps via Teleglobe (DC POP?)

Only list those of interest to HEP Moving to colocate US end points at DC POP to

improve peering Discussing CERN<=>Esnet<=>KEK link STAR-TAP = proposed Int’l GigaPOP at Chicago

04/19/23 28

Europe: TEN-34 W. European and some E.

European countries interconnect at 4 - 34Mbps– de, it, ch, uk, gr, nl, pt, at, lu,

es, fr, be, hu, sw+dk+no+fi

– Several links in production, more by Jul-97

Intra country links generally good

Intra Europe links improving with TEN-34

Next step TEN-155

04/19/23 29

Asia & FSU

Most connections thru Japan, in general good to acceptable for KEK– US/Esnet/KEK 522kbps => 1.5Mbps– China 64kbps => 128kbps (via KEK)– => 128kbps BINP/Russia Jun-97

2Mbps satellite DESY <=> MSU (Moscow)

04/19/23 30

Outline

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the

Internet? V. What are we (DOE/ESnet, HEP, SLAC)

doing? VII. Summary & Future

04/19/23 31

ESSC End-user Connectivity WG

intra-ESnet connectivity good ESnet <=> University connectivity often bad ESSC set up WG to look at problem

– Ranked top 20 university sites by ER funding– Monitored from SLAC & FNAL– Identified worst (bad (6) to poor (8) performance)– Recommended ESnet look at 6 BAD sites to understand

costs of improving, DOE/ESnet will provide money– Typical Frame Relay 1.5Mbps connections

$1-3K/month

04/19/23 32

ESnet Peering Improved peering (63=>110

NSPs), examples:– MCI & Sprint to avoid public

interconnect swamps

– University of California (avoid Sprint)

– THEnet at UT Austin

– vBNS East Coast in place since Feb-97

(avoid W Orange MCI) West Coast May 1997 Chicago to come

– Hubs at DC, Oakland, San Diego, Chicago

– Now carry > 45K routes

04/19/23 33

Improved ESnet Internet connection

Weekday 7am - 7pm Packet Loss seen from SLAC '95 - '96

0

10

20

30

40

50

60

70

80

90

100

Cu

mu

lati

ve

% P

ac

ke

t L

os

s

Washington.eduUOregon.eduUCSC.eduUCDavis.eduColostate.eduColorado.edu

ESnet Peers with Sprint/MCI to avoid MAE-West

04/19/23 34

UC-Esnet Improved Peering & UCSC

25ms 16ms

04/19/23 35

Improved peeringbetween Esnet &vBNS

vBNS/Esnet Peering & U. Colorado

04/19/23 36

ICFA Internet Working Group

Mini-workshop CHEP97 Working groups on: monitoring, remote

regions, present status, requirements analysis, and the proposal

End 1998 come up with proposal on what to do & why

Next meeting Santa Fe, Sep-97

04/19/23 37

Monitoring - Why “You can’t manage what you can’t measure” Monitor to set “user” expectations, help with

problem detection, get long term trends End-to-end monitoring mainly using ping

Provides response time, packet loss, reachability, unpredictability

Short (trouble shooting) & long term (planning)

Most important metric is packet loss

04/19/23 38

Monitoring - Who Many major HEP sites are monitoring end-to-

end Internet performance to collaborators– several hundred remote sites monitored

Collaborative effort to provide HEP-wide and ESnet wide reports, requested by ICFA, ESnet– Partially funded by DOE FWP involving SLAC, LBL,

HEPNRC– Based on SLAC early work (ping based) will complement

LBL NIMI work– SLAC, HEPNRC/FNAL, LBL collaboration

04/19/23 39

Monitoring - How Plan to coordinate effort, centered on SLAC/HEPNRC code

– install common software– distributed architecture– SLAC, HEPNRC Analysis Sites– Umd, RAL, INFN, KEK, ARM, CMU, RMKI, IN2P3,

CERN, DESY, TRIUMF, MSU signed up to be Collection Sites

– 247 Remote Sites as of 7/7/97– Reduces network impact of full mesh monitoring

04/19/23 40

WWW

Analysis Analysis

Collecting

Collecting

CollectingCollecting

RemoteRemoteRemote

Remote

Remote

HTTP

Ping Data(via HTTP)

Pings

E.g. HEPNRC E.g. SLAC

E.g. RAL

Data Collection & Distribution Architecture

04/19/23 41

Results from ~70 Sites in 10 Countries Being Monitored from

SLAC

ESnet SiteN. American SiteInternational Site

SLAC

FNAL

ORNL

UMd

Monitoring Site

04/19/23 42

Putting it all Together

04/19/23 43

Outline

I. SLAC’s Connectivity II. How is it Working? III. Why is it like it is? IV. What’s going on out there on the Internet? V. What are we (DOE/Esnet, HEP, SLAC)

doing? VII. Summary & Future

04/19/23 44

Summary Driving forces:

– Internet user growth 8.4M => 28M US users (15 mos)– Computer power doubling every 12-18 months– new applications, WWW, Internet phone, VR, Video ...

Since Apr-95, no single management for planning, trouble reporting etc.

ESnet performance good to acceptable, N. America poor (~6% packet loss avg), International poor (~7% packet loss avg)

Bottlenecks at interchanges

04/19/23 45

Future Many separate initiatives:

– critical to make sure they interplay well– identify and avoid bottlenecks– understand and guide impact for HEP

Criticality of Internet to HEP collaborations means HEP should increase efforts in this area:– keep tuned in, understand issues– monitor end-to-end performance– work with other research and higher education users

04/19/23 46

SLAC Networking http://www.slac.stanford.edu/comp/net/net.html

SLAC WAN Monitoring Page, lots of pointers http://www.slac.stanford.edu/comp/net/wan-mon.html

ESnet: http://www.es.net/ vBNS: http://www.vbns.net/ Internet2: http://www.internet2.edu/ NGI: http://www.hpcc.gov/ngi

-concept-08Apr97/ TEN-34: http://www.scimitar.terena.nl

/projects/ten-34/ ICFA Workshop on HEP & the Internet:

http://www.slac.stanford.edu/xorg/icfa/chep97/wg.html