1

IEPM/PingERIEPM/PingER

Internet End-to-end Performance Monitoring and the PingER projectInternet End-to-end Performance Monitoring and the PingER project

IEPM/PingERIEPM/PingER

Internet End-to-end Performance Monitoring and the PingER projectInternet End-to-end Performance Monitoring and the PingER project

Warren Matthews and Les Cottrell (SLAC)

National Collaboratory Middleware and Network Research Project Review, ANL,

August 18-20, 2003.

04/19/23 20:43 2

OverviewOverviewOverviewOverview

• A little History• Evaluate the progress• Assess the Value• Interactions with other projects• Elements that should be added • Summary

3

HistoryHistoryHistoryHistory

• Ping End-to-end Reporting• Began early 1995

– Monitor network performance to sites collaborating with SLAC

• ESnet Network Monitoring Task Force (NMTF)– Extended to several DoE labs, strong support from

FNAL

• 1997 ICFA created Network Task Force– PingER spreads worldwide– Funded by DOE/MICS

4

RecentlyRecentlyRecentlyRecently

• In 2001, extended PingER to include bandwidth testing – IEPM-BW

• End-to-end user perception for high performance bulk-transfer

• Iperf, bbftp, GridFTP …• Heavy network impact compared to

lightweight PingER

5

Current StatusCurrent StatusCurrent StatusCurrent Status

• PingER funding is under Thomas Ndousse

• DoE/MICS funding runs out at end of year

• Continues to be extremely useful– Most recently began working with

ICTP/eJDS to quantify the Digital Divide• MAGGIE proposal to develop/extend

high performance monitoring (with PSC, ICIR, LBNL)

04/19/23 20:43 6

OverviewOverviewOverviewOverview

• A little History

•Evaluate the progress• Assess the Value• Interactions with other projects• Elements that should be added • Summary

7

PingERPingERPingERPingER

• Mature, Successful• Widely used in HENP• Utilization has been extended beyond

HENP– EDG, IAEA, XIWT– ICTP/eJDS– Many others

• Continues to be extended to meet new needs– Better visualization, web services access to

data …

8

PingER MethodologyPingER MethodologyPingER MethodologyPingER Methodology

• Simple ping monitoring• 1 ping to prime caches• Send, size

– Default is 10x100 Byte pkts, 10x1000 Byte pkts

• Record ping packet loss and RTT• Derive unreachability, quiescence,

unpredictability, jitter, TCP throughput• Also out-of-order packets, duplicate

packets

04/19/23 20:43 9

PingER SoftwarePingER SoftwarePingER SoftwarePingER Software

• Monitoring• Analysis• Visualization• Available from SLAC/FNAL websites• Package

04/19/23 20:43 10

Using PingERUsing PingERUsing PingERUsing PingER

• Since 1995• Trouble shooting• Identify Sites to Upgrade• Choosing a provider• Setting expectations for VoIP • Routing Choices for multihomed sites• Compare with http, ftp

– Strong correlation

11

PingER DeploymentPingER DeploymentPingER DeploymentPingER Deployment

• Currently 36 monitoring sites in 14 countries

• 473 target sites in 79 countries– 99% of the worlds on-line population

• Most extensive end-to-end active R&E network monitoring worldwide

• Special BaBar, PPDG, Digital Divide. etc groups and pages

04/19/23 20:43 12

End-to-end MonitoringEnd-to-end Monitoring

• In reality, most projects monitor end-to-end performance– End host effects are unavoidable

• Internet2 end-to-end Performance Initiative (e2epi) has recognized this– Most useful to users.

13

IEPM-BWIEPM-BW

• Throughput Monitoring– Traceroute– Iperf (+quick iperf), BBftp, BBCP (mem and

disk)– ABWE (available bandwidth)– GridFTP, UDPMON– Web100– Netflow

• Analysis

04/19/23 20:43 14

IEPM-BW DeploymentIEPM-BW Deployment

• Currently 10 monitoring sites– SLAC, FNAL, GATech (SOX)– INFN (Milan), NIKHEF, APAN (Japan)– UMich, Internet2 (Ann Arbor)– UManchester, UCL (UK)

• 50 unique target sites

15

Using IEPM-BWUsing IEPM-BWUsing IEPM-BWUsing IEPM-BW

• Usual– Baselines– Troubleshooting– Setting expectations

• Also on both testbeds and production nets– Compare measurement tools (ping vs ABwE

vs. iperf/quick iperf vs bbcp vs GridFTP vs tsunami)

– Compare advanced TCP stacks• Eliminate need for multiple streams

– Look at non TCP bulk transfer

04/19/23 20:43 16

OverviewOverviewOverviewOverview

• A little History• Evaluate the progress

•Assess the Value• Interactions with other projects• Elements that should be added • Summary

04/19/23 20:43 17

ExamplesExamplesExamplesExamples

• Long term trends • Short term glitches• Troubleshooting• Upgrades• Vacations• Peering

18Traffic on ESnet has doubled every year

2Mbps

Multiple OC12s

vacation

19

Packet Loss between DESY and FNAL in February and March 2000.

0

2

4

6

8

10

12

1 3 5 7 9 11 13 15 17 19 21 23 25 27 1 3 5 7 9 11 13 15 17 19

Day of the Month

Daily Packet Loss (%)

DFN closes Perryman POP and looses direct peering with ESnet

Peering re-established via Dante at 60 Hudson

February March

20

Median Packet Loss Seen From nbi.dk

0

5

10

15

20

25

30

35

40

45

50

11/1/9811/8/9811/15/9811/22/9811/29/98

12/6/9812/13/9812/20/9812/27/98

1/3/991/10/991/17/991/24/99

% 100 Byte Packet Loss During Day.

Ten-155 became Ten-155 became operational on operational on December 11.December 11.

Smurf Filtersmurf Filtersinstalled oninstalled onNORDUnet’sNORDUnet’sUS connection.US connection.

To North America

To Western Europe

21

Throughput from SLAC to RAL between May 2002 and February 2003

0

50000

100000

150000

200000

250000

5/13/20025/27/20026/10/20026/24/20027/8/2002

7/22/20028/5/2002

8/19/20029/2/2002

9/16/20029/30/200210/14/200210/28/200211/11/200211/25/2002

12/9/200212/23/2002

1/6/20031/20/2003

2/3/20032/17/2003

iperf

bbcpmem

bbcpdisk

bbftp

22

23

TrafficTrafficTrafficTraffic

Typically, Internet traffic is 70% http

24

ConclusionsConclusionsConclusionsConclusions

• Establish layer 3 connectivity exists

• Iperf vs Quick iperf

• BBftp vs BBCP => implementation

• IPERF vs BBftp => CPU, Disk

• Packet Loss < 0.1%

• TCP/IP must be tuned on high-speed long delay paths

• Web100/Net100

04/19/23 20:43 25

eJDSeJDSeJDSeJDS

• PingER continues to be useful

• Recently joined with electronic journal distribution service (eJDS)

• Distribute physics journals to member around the world

• Particularly concerned with quantifying the Digital Divide

26

LimitationsLimitationsLimitationsLimitations

• ICMP

• Do not monitor routers

• Rate limiting

• Blocking is common, especially in developing countries

• However, study indicates low impact from rate limiting

• Scheduling with cron

04/19/23 20:43 27

OverviewOverviewOverviewOverview

• A little History• Evaluate the progress• Assess the Value

• Interactions with other projects

• Elements that should be added • Summary

28

Comparison to Other Comparison to Other ProjectsProjects

Comparison to Other Comparison to Other ProjectsProjects

• Surveyor• RIPE• AMP• NIMI• SCNM• XIWT• NetPhysics

04/19/23 20:43 29

ComparisonsComparisonsComparisonsComparisons

• Typically results were closely correlated.

• Often tools complement each other and combined provide insight into network behaviour.

• Derived throughput from equation of Mathis et al (BW~MSS/(RTT*sqrt(loss)) ) shows good agreement

04/19/23 20:43 30

PublishingPublishingPublishingPublishing

• Network Performance information is critical to the Grid vision – Application steering

• Working with GGF/NMWG & PPDG• Monitoring data is available as

prototype Web Service • OGSI Grid service under

development

04/19/23 20:43 31

Internet2 PIPESInternet2 PIPESInternet2 PIPESInternet2 PIPES

• E2e pi• PIPES infrastructure• IEPM-BW Job manager• MAGGIE Analysis Engine

32

Available Bandwidth Estimator Available Bandwidth Estimator (ABwE)(ABwE)

Available Bandwidth Estimator Available Bandwidth Estimator (ABwE)(ABwE)

• Tool under development by SLAC/Rice– Part of the DoE/SCIDAC INCITE project

• Light weight– 60 packets in 1 second– Iperf 35,000 packets/s for 10-20 seconds

• No need to tune windows/streams• Replace iperf in test engine• FreeBSD version created for Abilene

Backbone Measurement Infrastructure

33

Quick IperfQuick IperfQuick IperfQuick Iperf

• Iperf is the tool of choice for many admins. – Considered accurate but intrusive.– Errors due to long slow start

• Use web100 to detect end of slow start. Modify iperf client. Web100 required on client only.

• Measurement within 10%• Save 94% time, 92% traffic

04/19/23 20:43 34

PingER-6PingER-6PingER-6PingER-6

• SLAC has native IPv6 service from ESnet

• PingER ported to IPv6• Monitoring started in November

1999 • 41 Sites in 10 countries • edu/ac.*, net/net.*, com/co.*

04/19/23 20:43 35

PingER -vs- PingER6

0

50

100

150

200

250

300

350

400

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 1 3 5 7 9 11 13 15 17 19 21 23 25 27 28 31

IPv6

IPv4

RTT between SLACand Purdue in Novand Dec 1999.

04/19/23 20:43 36

OverviewOverviewOverviewOverview

• A little History• Evaluate the progress• Asses the Value• Interactions with other projects

•Elements that should be added

• Summary

37

MAGGIEMAGGIEMAGGIEMAGGIE

• Need to further develop IEPM-BW– On-demand measurements,

visualization, automated trouble shooting

• Measurement and Analysis for the Global Grid and Internet End-to-end performance– A secure, scalable measurement

infrastructure providing measurement, analysis and access to data.

38

MAGGIEMAGGIEMAGGIEMAGGIE

MAGGIE

NIMISecurity and scheduling

IEPM-BWMeasurement Engine

Publishing

Fault FindingAnalysis Engine

Other tools

NMWG

AMP

RIPESLAC

SLAC

FNAL

PSCICIR

LBNL

SLAC

ANLSCIDAC

UCL

Akenti NWS

04/19/23 20:43 39

OverviewOverviewOverviewOverview

• A little History• Evaluate the progress• Assess the Value• Interactions with other projects• Elements that should be added

•Summary

40

Meeting the Objectives Meeting the Objectives (1/4)(1/4)

Evaluate the ProgressEvaluate the Progress

Meeting the Objectives Meeting the Objectives (1/4)(1/4)

Evaluate the ProgressEvaluate the Progress• The problem: The user cannot assume the network

will be there. – Even if it is, the user cannot assume it will perform to

their expectation. • The vision (realized): PingER has set expectation,

provides data for troubleshooting, provides data for research. Continues to be useful.

• A Unique contribution: Probably the largest monitoring project in the world. IEPM-BW comparing tools, leveraging other efforts.

04/19/23 20:43 41

Meeting the Objectives Meeting the Objectives (2/4)(2/4)

Assess the valueAssess the value

Meeting the Objectives Meeting the Objectives (2/4)(2/4)

Assess the valueAssess the value• PingER is widely used and

continues to be useful. • Goals get more ambitious• Challenges remain

42

Meeting the Objectives Meeting the Objectives (3/4)(3/4)

Interactions Across Interactions Across ProjectsProjects

Meeting the Objectives Meeting the Objectives (3/4)(3/4)

Interactions Across Interactions Across ProjectsProjects• Long history of involvement in

other projects– HENP, ESnet, Grid, High

Performance, ICFA-SCIC – Friends, colleagues and contacts

throughout the world (Other worlds coming soon)

• Bright future for MAGGIE.

04/19/23 20:43 43

Meeting the Objectives Meeting the Objectives (4/4)(4/4)

Assess the Integrated Assess the Integrated ImpactImpact

Meeting the Objectives Meeting the Objectives (4/4)(4/4)

Assess the Integrated Assess the Integrated ImpactImpact• The contribution to the big picture

by IEPM-PingER, IEPM-BW and especially the need for MAGGIE have been summarized by Mary-Anne and Thomas– But they may not have known it

04/19/23 20:43 44

The Big PictureThe Big PictureThe Big PictureThe Big Picture

All of the National Collaboratory and Network Research projects have specific goals and objectives, but all of you involved in those projects are also part of a much larger, longer term effort, namely creating an infrastructure that will enable geographically separated scientists to effectively work together as a team and that will facilitate remote access to both facilities and data.

-Mary-Anne and Thomas

04/19/23 20:43 45

Toward a Monitoring Toward a Monitoring InfrastructureInfrastructure

Toward a Monitoring Toward a Monitoring InfrastructureInfrastructure

• Certainly the need– DOE Science Community, SCIDAC Testbed– Grid, Large Scale Networking– Troubleshooting / E2Epi

• Many of the ingredients– Many monitoring projects– Many tools– PIPES, MAGGIE (Cross domain)

46

SummarySummary

“Unfortunately, network management research has historically been very under-funded, because it is difficult to get funding bodies to recognize this as legitimate networking research.”

Sally FloydIAB Concerns & Recommendations Regarding

Internet Research & Evolution.http://www.ietf.org/internet-drafts/draft-iab-research-funding-00.txt

04/19/23 20:43 47

LinksLinksLinksLinks

• Accompanying paper

• IEPM-BW Home

• 7 papers and 35 talks in the last 12 months

• ABwE

• RIPE-TT

• E2E PI

• GGF NMWG

• AMP TroubleShooting

• Quick Iperf

48

CreditsCreditsCreditsCredits

• Connie Logg, Jerrod Williams (SLAC), Jiri Navratil (CESnet/SLAC), David Martin, Frank Nagy, Al Thomas, Maxim Grigoriev (FNAL), Fabrizio Coccetti (INFN/SLAC).

• Brian Tierney, Eric Boyd, Jeff Boote, Matt Zekauskas, Matt Mathis, Russ Hobby, Vern Paxson, Andy Adams, kc Claffy, Iosif Legrand, Ajay Tirumala, Tom Dunigan.

• Local admins and other volunteers• DoE/MICS