HIDDEN SLIDE –Summary · 2017-08-28 · HIDDEN SLIDE –About •Title:...

HIDDENSLIDE– Summary• Theseslidesaremeanttobeused‘asis’togiveanupperlevelviewofperfSONAR foranaudiencethatisnotfamiliarwiththeconcept.• You*ARE*allowedtodeletethingsyoudon’tunderstand,ormodifythemessagetofitaspecificgroup• You*ARE*allowedtoaddmoreiftheaudienceismoretechnical• Presentationstylesvary– pleasestudyupbeforepresenting

HIDDENSLIDE– About• Title:perfSONAR-9-Hardware-Therory• ExpectedPresentationTime:Theseslidesshouldtake30minutestopresent.• ExecSummary:Thisisaverytheoreticaldiscussiononnetworkperformance– startingwithnetworkdevices,hosts,operatingsystems,andapplications.• ExpectedAudience:Engineers(midtohighlevel),notmanagement.• AudienceSkillLevel:Intermediatetoexpertknowledgeofhownetworkswork.• LastReviewed:SSConJuly13th 2017

HIDDENSLIDE– ContentOutline• Outline:

• Overviewofthelayersofnetworking• Discussionaboutwhatisbeingmeasured

• WhatCanbeCut:• Nothingreallyshouldbecut– ifyoudon’tunderstandthematerialthisisnotaslidedeckyouwanttoaimtopresent.

NetworkPerformance-Theory

ThisdocumentisaresultofworkbytheperfSONAR Project(http://www.perfsonar.net)andislicensedunderCCBY-SA4.0(https://creativecommons.org/licenses/by-sa/4.0/).

Event

Presenter,Organization,EmailDate

HIDDENSLIDE– Slide6• Thissetofmaterialsisverytheoretical– andismeanttocomebeforethehosthardwaresection• Thepurposeistomotivatewhynetworkperformancemeasurementis‘hard’andwhyweneedhardwaretobeperformingatpeakefficiency.• Itconcludeswithatiebacktowhatmeasurementswilltellyou.

Introduction

• perfSONAR isatooltomeasure“end-to-end”networkperformance.Whatdoesthisimply:• End-to-end:theentirenetworkpathbetweenperfSONAR Hosts

• ApplicationsSoftware• OperatingSystem• HostHardware• @EachHop:transitionbetweenOSILayersinrouting/switchingdevices(e.g.TransporttoNetworkLayer,etc.),buffering,processingspeed

• Flowthroughsecuritydevices• Noeasywaytoseparateouttheindividualcomponentsbydefault– thenumberthetoolgiveshasthemallcombined

August28,2017©2016,http://www.perfsonar.net 6

HIDDENSLIDE– Slides8- 12• Thenextcoupleofslidesare‘networking101’.Ifyouaren’tcomfortablewiththematerial,schoolup• BasicallyyouneedtounderstandthelayersofanOS,anetwork,andhowtheyallrelateviatheconceptoftheOSIstack(e.g.transitionsandreasonsthereissuchamodel)• Hostsaretalkedaboutfirst.

InitialEnd-to-EndNetwork

August28,2017©2016,http://www.perfsonar.net 8

InitialEnd-to-EndNetwork

August28,2017©2016,http://www.perfsonar.net 9

Src HostDelay:• ApplicationwritingtoOS

(kernel)• Kernelwritingviamemory

tohardware• NICwritingtonetwork

Dst HostDelay:• NICreceivingdata• Kernelallocatingspace,

sendingtoapplication• Applicationreading/acting

onreceiveddata

Src LAN:• Bufferingoningress

interfacequeues• Processingdatafor

destinationinterface• Egressinterfacequeuing• Transmission/Serialization

towire

WAN:• Propagationdelayforlongspans• Ingressqueuing/processing/egressqueuing/serializationforeachhop

Dst LAN:• Bufferingoningress

interfacequeues• Processingdatafor

destinationinterface• Egressinterfacequeuing• Transmission/Serialization

towire

OSIStackReminder

August28,2017©2016,http://www.perfsonar.net 10

• ThedemarcationbetweeneachlayerhasanAPI(e.g.the‘narrowwaist’ofanhourglass)

• Somelayersaremorewelldefinedthanothers:• Withinanapplicationthejobof

presentationandsessionmaybehandled

• TheoperatingsystemhandlesTCPandIP,althoughtheseareseparatelibraries

• Network/DataLinkoccurwithinnetworkdevices(Routers,Switches)

HostBreakout

August28,2017©2016,http://www.perfsonar.net 11

• Mostapplicationsarewrittenin‘userspace’,e.g.specialsectionoftheOSthatisjailedfrom‘kernelspace’.

• RequeststousefunctionslikethenetworkaredonebyusingsystemcallsthroughanAPI(e.g.openasocket soIcancommunicatewitharemotehost)

• TheTCP/IPlibrariesarewithinthekernel,theyreceivetherequestandtakecareoftheheavyliftingofconvertingthedatafromtheapplication(e.g.alargechunkofmemory)intoindividualpacketsforthenetwork

• TheNICwillthenencapsulateintoLinklayerprotocol(e.g.ethernet frames)andsendontothewireforthenexthoptodealwith

HostBreakout

August28,2017©2016,http://www.perfsonar.net 12

• Thereceivesideworkssimilar– justinreverse

• FramescomeoffofthenetworkandintotheNIC.Theonboardprocessorwillextractpackets,andpassthemtothekernel

• Thekernelwillmapthepacketstotheapplicationthatshouldbedealingwiththem

• TheapplicationwillreceivethedataviatheAPI

• Note– theTCP/IPlibrariesmanagethingslikedatacontrol.Theapplicationonlyseesasocket,andknowsthatitwillsendindata,anditwillmakeittotheotherside.Itisthejobofthelibrarytoensurereliabledelivery

HIDDENSLIDE– Slides14-16• Thenextcoupleofslidestalkabouttheinternalsofanetworkdevice– assumeswitchorrouter•Wethenstarttorollthisbackandexplainhowthisrelatedtonetworkperformanceonanendtoendbasis.

NetworkDeviceBreakout

August28,2017©2016,http://www.perfsonar.net 14

NetworkDeviceBreakout

August28,2017©2016,http://www.perfsonar.net 15

• Dataarrivesfrommultiplesources

• Buffershaveafiniteamountofmemory• Somehavethisperinterface• Othersmayhaveaccesstoashared

memoryregionwithotherinterfaces• Theprocessingenginewill:

• Extracteachpacket/framefromthequeues

• Pulloffheaderinformationtoseewherethedestinationshouldbe

• Movethepacket/frametothecorrectoutputqueue

• Additionaldelayispossibleasthequeuesphysicallywritethepackettothetransportmedium(e.g.opticalinterface,copperinterface)

NetworkDeviceBreakout– Delays

August28,2017©2016,http://www.perfsonar.net 16

HIDDENSLIDE– Slides18-22• Nowweputittogether– traversinganetworkmeanstraversingtheOSUstack(partsofitatleast)ineverysingledeviceonthepath.• Thistakestime,anditcausesproblemsifthereareslownessesinpiecesofthenetwork

NetworkDevices&OSI

August28,2017©2016,http://www.perfsonar.net 18

• Noteverydevicewillcareabouteverylayer

• Hostsunderstandthemallviavariouslibraries

• Networkdevicesonlyknowuptoapoint:• ‘Routers’knowuptothe‘NetworkLayer’.

TheywillmakethechoiceofsendingtothenexthopbasedonNetworkLayerheaders.E.g.TCPinformation– IPaddresses

• ‘Switches’knowuptothe‘LinkLayer’.TheywillmakethechoiceofsendingtothenexthopbasedonLinkLayerheaders.E.g.MACaddressingfromtheIP

• Eachhophasthehardware/softwaretopulloftheencapsulateddatatofindwhatitneeds

“End-to-End”

• Anetworkuserinterfaceswiththenetworkviaatool(datamovementapplication,portal).Theyonlygetasinglepieceoffeedback– howlongtheinteractiontakes• Inreality– it’sacomplexseriesofmovestogetthedataendtoend,withlimitedvisibilitybydefault• Delaysonthesourcehost• DelaysinthesourceLAN• DelaysintheWAN• DelaysinthedestinationLAN• Delaysonthedestinationhost

August28,2017©2016,http://www.perfsonar.net 19

“End-to-End”• Theonlywaytogetvisibilityistorelyoninstrumentationatthevariouslayers:• Hostlevelmonitoringofkeycomponents(CPU,memory,network)• LAN/WANlevelmonitoringofindividualnetworkdevices(utilization,drops/discards,errors)• End-to-endsimulations

• Thelateroneistricky– wewantto‘simulate’whatauserwouldseebyhavingourown(welltuned)applicationtellushowitcandoacrossthecommonnetworksubstrate

August28,2017©2016,http://www.perfsonar.net 20

DereferencingIndividualComponents• HostPerformance• SoftwareTools– Ganglia,HostSNMP+Cacti/MRTG

• NetworkDevicePerformance• SNMP/TL1PassivePolling(e.g.interfacecounters,internalbehavior)

• SoftwarePerformance• ???Thisdependsheavilyonhowwellthesoftware(e.g.operatingsystem,application)isinstrumented.

• End-to-end• perfSONAR activetools(iperf3,owamp,etc.)

August28,2017©2016,http://www.perfsonar.net 21

Takeaways• Sincewewant“network”performance– wewanttoremovethehosthardware/operatingsystem/applicationsfromtheequationasmuchaspossible• Thingsthatwecandoonourown,orthatwegetforfreebyusingperfSONAR:• HostHardware:Choosinghardwarematters.Thereneedstobepredictableinteractionsbetweensystemcomponents(NIC,motherboard,memory,processors)

• OperatingSystem:perfSONAR featuresatunedversionofCentOS.Thisversioneliminatesextrasoftwareandhasbeenmodifiedtoallowforhighperformancenetworking

• Applications:perfSONAR applicationsaredesignedtomakeminimalsystemcalls,anddonotinvolvethedisksubsystem.Theperformancetheyreportisdesignedtobeaslow-impactonthehosttoachieverealisticnetworkperformance

August28,2017©2016,http://www.perfsonar.net 22

HIDDENSLIDE– Slides24-27• Nowwegetdownintowhatthisallmeansfornetworkperformance• Toolslikeiperf (andothers)aredesignedtotellus‘whatisgoingonrightnow’givensomeconfig• perfSONAR istunedtoperformwell,soitshouldgiveyouapurenetworkperformanceindication(aslongasthehostsareadequate).

LetsTalkaboutIPERF• Startwithadefinition:

• networkthroughputistherateofsuccessfulmessagedeliveryoveracommunicationchannel

• Easierterms:howmuchdatacanIshovelintothenetworkforsomegivenamountoftime

• Thingsitincludes:theoperatingsystem,thehosthardware,andtheentirenetoworkpath

• Whatdoesthistellus?• Oppositeofutilization(e.g.itshowmuchwecangetatagivenpointintime,minuswhatisutilized)

• Utilizationandthroughputaddedtogetherare“capacity”• Toolsthatmeasurethroughputareasimulationofarealworkusecase(e.g.howwellcouldbulkdatamovementperform)

August28,2017©2016,http://www.perfsonar.net 24

WhatIPERFTellsUs• Letsstartbydescribingthroughput,whichisvague.

• Capacity:linkspeed• NarrowLink:linkwiththelowestcapacityalongapath• Capacityoftheend-to-endpath=capacityofthenarrowlink

• Utilizedbandwidth:currenttrafficload• Availablebandwidth:capacity– utilizedbandwidth

• TightLink:linkwiththeleastavailablebandwidthinapath• Achievablebandwidth:includesprotocolandhostissues(e.g.BDP!)

• Allofthisis“memorytomemory”,e.g.wearenotinvolvingaspinningdisk(morelater)

August28,2017©2016,http://www.perfsonar.net 25

45 Mbps 10 Mbps 100 Mbps 45 Mbps

Narrow Link Tight Link

source sink

(Shaded portion shows background traffic)

BWCTLExample(iperf3)[zurawski@wash-pt1 ~]$ bwctl -T iperf3 -f m -t 10 -i 2 -c sunn-pt1.es.netbwctl: run_tool: tester: iperf3bwctl: run_tool: receiver: 198.129.254.58bwctl: run_tool: sender: 198.124.238.34bwctl: start_tool: 3598657653.219168Test initializedRunning clientConnecting to host 198.129.254.58, port 5001[ 17] local 198.124.238.34 port 34277 connected to 198.129.254.58 port 5001[ ID] Interval Transfer Bandwidth Retransmits[ 17] 0.00-2.00 sec 430 MBytes 1.80 Gbits/sec 2[ 17] 2.00-4.00 sec 680 MBytes 2.85 Gbits/sec 0[ 17] 4.00-6.00 sec 669 MBytes 2.80 Gbits/sec 0[ 17] 6.00-8.00 sec 670 MBytes 2.81 Gbits/sec 0[ 17] 8.00-10.00 sec 680 MBytes 2.85 Gbits/sec 0[ ID] Interval Transfer Bandwidth Retransmits

Sent[ 17] 0.00-10.00 sec 3.06 GBytes 2.62 Gbits/sec 2

Received[ 17] 0.00-10.00 sec 3.06 GBytes 2.63 Gbits/sec

iperf Done.bwctl: stop_tool: 3598657664.995604

SENDER END

August28,2017©2016,http://www.perfsonar.net 26

N.B.ThisiswhatperfSONARGraphs– theaverageofthecompletetest

Summary

•Wehaveestablishedthatourtoolsaredesignedtomeasurethenetwork• Forbetterorforworsethe‘network’isalsoourhosthardware,operatingsystem,andapplication• Togetthemostaccuratemeasurement,weneed:• Hardwarethatperformswell• Operatingsystemsthatperformwell• Applicationsthatperformwell

August28,2017©2016,http://www.perfsonar.net 27

NetworkPerformance-Theory

ThisdocumentisaresultofworkbytheperfSONAR Project(http://www.perfsonar.net)andislicensedunderCCBY-SA4.0(https://creativecommons.org/licenses/by-sa/4.0/).

Event

Presenter,Organization,EmailDate