Upload
haliem
View
224
Download
2
Embed Size (px)
Citation preview
perfSONAR: The Road to 100k Nodes
Eric Boyd, Internet2, [email protected] Luke Fowler, Indiana University, [email protected]
Brian Tierney, ESnet, [email protected]
What is perfSONAR? • perfSONAR is a tool to:
– Set (hopefully raise) network performance expectaIons – Find network problems (“soQ failures”) – Help fix these problems
• All in mulI-‐domain environments • These problems are all harder when mulIple networks are involved
• perfSONAR is provides a standard way to publish acIve and passive monitoring data – This data is interesIng to network researchers as well as network operators
2
Current perfSONAR components • Measurement tools
– iperf3, bwctl, owamp, traceroute, etc. • Measurement archive • Central test mesh management tools • Host management tools
– Configure tests, configure NTP, etc. • Data analysis tools
– Plot data from the archive – Dashboard tools
• Lookup Service
October 15, 2015 3
perfSONAR History • perfSONAR can trace its origin to the Internet2 “End 2 End performance IniIaIve”
from the year 2000. • What has changed since 2000?
– The Good News: • TCP is much less fragile; Cubic is the default CC alg, autotuning is and larger TCP buffers are
everywhere • Reliable parallel transfers via tools like Globus Online • High-‐performance UDP-‐based commercial tools like Aspera • more good news in latest Linux kernel, but it will take 3-‐4 years before this is widely deployed
– The Bad News: • The wizard gap is sIll large • Under-‐buffered and switches and routers are sIll common • Under-‐powered/misconfigured firewalls are common • SoQ failures sIll go undetected for months • User performance expectaIons are sIll too low
4
The perfSONAR collaboraIon • The perfSONAR collaboraIon is a Open Source project lead by ESnet, Internet2,
Indiana University, and GEANT. – Each organizaIon has commiced 1.5 FTE effort to the project – Plus addiIonal help from many others in the community (OSG, RNP, SLAC, and more)
• The perfSONAR Roadmap is influence by – requests on the project issue tracker – annual user surveys sent to everyone on the user list – regular meeIngs with VO using perfSONAR such as the WLCG and OSG – discussions at various perfSONAR related workshops
• Based on the above, every 6-‐12 months the perfSONAR governance group meets to prioriIze features based on: – impact to the community – level of effort required to implement and support – availability of someone with the right skill set for the task
5
perfSONAR Vision • Grow perfSONAR to be the ubiquitous, indispensible performance monitoring framework for R&E network customers that meets their needs in the long-‐term, supported in a self-‐sustaining way.
October 15, 2015 6
perfSONAR Challenge • Get to 100k nodes deployed worldwide – An increase of 2 orders of magnitude
• Get to 100+ nodes per campus
October 15, 2015 7
perfSONAR Challenge ImplicaIons • Cut the price per node by an order of magnitude – $4k (perfSONAR 3.3) – $2k (perfSONAR 3.4) – $100 (perfSONAR 3.5)
October 15, 2015 8
perfSONAR Hardware • These days you can get a good 1U host capable of
pushing 10Gbps TCP for around $500 (+10G NIC cost). – See perfSONAR user list
• And you can get a host capable of 1G for around $100! – Intel Celeron-‐based (ARM is not fast enough) – e.g.: hcp://www.newegg.com/Product/Product.aspx?Item=N82E16856501007
• VMs are not recommended – Tools work becer if can guarantee NIC isolaIon
6/2/15 9
perfSONAR Challenge ImplicaIons • Scale the infrastructure to support 100k nodes – Does the Lookup Service scale? – Do the various UIs scale? – Does the support model scale? – Grow the number of operaIng systems supported? – Re-‐evaluate accessibility of nodes?
October 15, 2015 10
perfSONAR Challenge ImplicaIons • What is the total cost of ownership per node? – O(n2)? O(n)? O(k)?
• How auto-‐configurable are the nodes? – If you drop a node in a wiring closet, how much work is required to integrate it into a campus mesh?
October 15, 2015 11
perfSONAR 3.5: What’s new?
perfSONAR 3.5 Themes • Impact: Increase the number of nodes, reduce total cost of ownership, improve security
• Usability: Transformed user experience, reduced cost for NOC integraIon
• Efficiency: Reduced development costs
October 15, 2015 13
perfSONAR 3.5 ImplementaIon • Modernize the GUIs • Support for central host management and node auto-‐configuraIon
• Support for Debian, VMs, and other installaIon opIons
• Support for low cost, 1Gbps nodes
October 15, 2015 14
Expanded perfSONAR Use Cases • Current Use Case – perfSONAR Toolkit
• Includes CentOS 6 and all perfSONAR components
• New Use Cases – perfSONAR tools only
• Support for both RHEL-‐based and Debian-‐based hosts – perfSONAR hosts that are centrally managed – perfSONAR hosts that ‘self-‐configure’
October 15, 2015 15
Improved Support for Central Management
• Goals: – Make it easy to incorporate perfSONAR hosts into exisIng host management systems (puppet, chef, SaltStack, cfengine, etc.) • Include sample puppet config files
– Make it easy to manage many perfSONAR hosts at a single insItuIon
– New rpm and debian bundles to support this
October 15, 2015 16
perfSONAR and Puppet
• Puppet is a configuraIon management system that allows you to configure and maintain many nodes easily
• IniIal perfSONAR puppet modules have been created – Alpha/beta level – Only CentOS6 supported – We need your help to test/contribute improvements! – Hoping to expand in perfSONAR 3.6
October 15, 2015 17
Puppet Support Features • Node classes can be defined, i.e. endpoint or measurement archive • Puppet will install soQware/dependencies, configure it, keep
packages up-‐to-‐date • Includes support for Internet2 repos, web100 kernel, Esmond,
bwctl, owamp, rsyslog client or server, apache, meshconfig – Most services have configurable port ranges – Puppet also maintains iptables setngs for the configured services
October 15, 2015 18
New perfSONAR InstallaIon opIons • In addiIon to the tradiIonal “Toolkit” install, you now have the these addiIonal opIons:
– perfSONAR-‐Tools: • iperf3, bwctl, owamp, nuccp, etc • Install this on DTNs, etc to help with troubleshooIng • Does not support scheduled tesIng • CentOS and Debian support
– perfSONAR-‐TestPoint: • tools plus Lookup Service registraIon and ‘mesh agent’ • For use in environments with a central measurement archive • For use on low end/older hardware (e.g.: $100 nodes) • Supports scheduled tesIng • CentOS and Debian support
• See: hcp://docs.perfsonar.net/install_opIons.html
October 15, 2015 19
New perfSONAR InstallaIon opIons (cont.) • perfSONAR-‐Core:
– Includes everything except the web interface – Use this in environments where your site sysadmins want to fully manage the
host configuraIon, but don’t want to setup a central measurement archive – CentOS only
• perfSONAR-‐CentralManagement: – Includes measurement archive, test mesh manager, dashboard – Use this to manage a collecIon of perfSONAR hosts at your site/campus – CentOS only
October 15, 2015 20
New perfSONAR InstallaIon opIons (cont.) • perfSONAR-‐Complete
– All perfSONAR packages – Use this environments where your sysadmins want to manage the install, but sIll use
the toolkit web interface, system setng, etc • the toolkit install will override certain changes every update.
– CentOS only
• Other packages to note: – Separate rpms/debs for iptables config, sysctl config, and ntp packages so you can add
them on top of perfSONAR-‐Core as desired.
October 15, 2015 21
perfSONAR for Network Researchers • AcIve measurement interesIng for network researchers
– Traceroute data automaIcally collected along with bwctl/owamp results
• Data easy to download for analysis – esmond-‐ps-‐get-‐bulk
• Output CSV or JSON • See: hcps://pypi.python.org/pypi/esmond_client
• AddiIonal InformaIon at: – hcp://docs.perfsonar.net/client_apis.html
October 15, 2015 22
User Interface Refresh • Easy to use web interface • Targeted at users with variety of levels of technical experIse.
October 15, 2015 23
Our goals for perSONAR 3.5: • improve easy of use and
visual appeal • integraIon of state of the
art UI components • to extend UI lifespan.
Current Interface
New Host Management Interface
October 15, 2015 © 2014, hcp://www.perfsonar.net 24
perfSONAR on Low Cost Hardware • MoIvaIon: make perfSONAR affordable enough to deploy on all subnets
• AssumpIons: – 1Gbps test nodes – Centralized measurement archive – Centralized configuraIon management – Debian Linux
October 15, 2015 25
Auto-‐Updates Rock!
October 15, 2015 26
psUI: the perfSONAR web UI
TechEX 2015
What is psUI? • Perfect tool for
– On demand measurements – Accessing mulIple archived measurements
• Access MP and MA from – Global Lookup Service (~ 9000 services) – Locally maintained list
• Calls bwctl through OPPD (calls 2 remote bwctl instances) • Java applicaIon running in tomcat
– Java 7 and Tomcat 7 – CentOS and Debian packages
• To be deployed on separate host (VM is ok) • Federated login with eduGAIN credenIals
October 15, 2015 28
October 15, 2015 29
1
2
3
October 15, 2015 30
1 2 3
Other resources • Install your own psUI for your NOC – InstallaIon instrucIons: hcp://docs.perfsonar.net/install_psui
– Usage instrucIons: hcp://docs.perfsonar.net/using_psui
• Public demo psUI instance (eduGAIN enabled): hcps://psui.geant.net
October 15, 2015 31
Low Cost Node Project
perfSONAR on Low Cost Hardware • MoIvaIon: make perfSONAR affordable enough to deploy on all subnets
• AssumpIons: – 1Gbps test nodes – Centralized measurement archive – Centralized configuraIon management – Debian Linux?
October 15, 2015 33
Many Possible SoluIons
August 10, 2015 34
• Small, Low-‐cost Nodes Examples include the above; NUC, Gigabyte, Pi, cubox, …
Started with ARM-‐based soluIon • Cheap ARM-‐based
perfSONAR nodes • e.g.: Cubox
(“perfCUBE”) $120
• TCP max only 350Mbps
• owamp works well
35
Evaluated these Celeron-‐based devices
10/15/15 36
ASUS CHROMEBOX-‐M004U ($160) GIGABYTE GB-‐BXBT-‐2807 ($115) Intel NUC Kit DCCP847DYE ($170)
LIVA by ECS • $100-‐125 USD • Home:
hcp://www.ecs.com.tw/ECSWebSite/Product/Product_LIVA.aspx?DetailID=1560&LanID=0#Overview
• hcp://www.newegg.com/Product/Product.aspx?Item=N82E16856501007 – $105 in Sept, 2015
© 2015, hcp://www.perfsonar.net 37
© 2015, hcp://www.perfsonar.net 38
SpecificaIons
Assembly Required • Contents of box:
– Case top and bocom – Motherboard with onboard memory,
RAM, and 1GE network adapter – Wireless adapter card – Antennae for wireless card (1 long,
black; 1 short, white) – Power Supply (US) with adapter plate
(EU included)
© 2015, hcp://www.perfsonar.net 39
Assembly Completed
© 2015, hcp://www.perfsonar.net 40
Strengths • Complete package with limited assembly required
– 32Gig of flash onboard – 2Gig of RAM – True 1GE (tesIng over clean paths 920-‐980M – Only 15W power required
• POE PossibiliIes
• Small and lightweight • ComparaIvely Low cost to other more robust
hardware/server opIons
© 2015, hcp://www.perfsonar.net 41
eMMC Flash (Onboard) • perfSONAR’s recommended CentOS 6 does not include driver
support for the onboard storage • Using instead as a recommended build Ubuntu 12.04.5
Desktop – Server doesn’t have proper EMMC driver
• Debian (Ubuntu) perfSONAR packages: hcp://docs.perfsonar.net/install_debian#configuraIon
© 2015, hcp://www.perfsonar.net 42
Power Bucon / USB Issues (minor) • Power bucon instability has been found on about
1 in 10 boxes. • Easily overcome by
– mulIple boot acempts or; – Long-‐pressing the power bucon for a firm connecIon or;
• Assume this is either a quirk of the manufacturing process and hope it is worked out on future models
• USB instability has been found on about 3 in 20 boxes; independent of the power bucon issue
© 2015, hcp://www.perfsonar.net 43
Passive Cooling • No fan in the box for cooling • No issues with this were found during
tesIng, but worth considering when placing in warmer spaces.
© 2015, hcp://www.perfsonar.net 44
TesIng Next GeneraIon Liva…
October 15, 2015 © 2014, hcp://www.perfsonar.net 45
• “Liva X” – new run of hardware, expected to resolve minor issues.
• Nearly idenIcal hardware, very similar cost profile • Arrives pre-‐assembled • perfSONAR project has acquired (arrived on Monday!),
beginning tesIng/validaIon now. Stay tuned! • hcp://www.newegg.com/Product/Product.aspx?
Item=N82E16856501011
More InformaIon… hcps://www.perfsonar.net/deploy/hardware-‐selecIon/low-‐cost-‐hardware/ hcp://docs.perfsonar.net/low_cost_nodes.html hcp://github.com/perfsonar/project/wiki/perfSONAR-‐Endpoint-‐Node-‐Project
October 15, 2015 © 2014, hcp://www.perfsonar.net 46
plans for perfSONAR 3.5.1 and 3.6
perfSONAR Plans • We are in the early stages of refining the perfSONAR roadmap
• ExpecIng a minor release in the February Imeframe and a major release in late 2016 (Imeframe TBD)
October 15, 2015 48
perfSONAR Community • Planning a survey of the perfSONAR user community in October, 2016
• Planning to publish roadmap in November, 2016
October 15, 2015 49
DRAFT: v3.5.1 (February 2016) • New Test configuraIon GUI • New plots based on ESnet Tools team react-‐based plotng tools
• Node ‘autoconfig’ • minimize configuraIon required to bring up a new centrally managed
perfSONAR node • More automated tesIng • Bug fixes
October 15, 2015 50
DRAFT: v3.6 (Summer 2016) • Theme for v3.6 will be “Control and Scalability” • perfSONAR is successful because of the ‘default open’ model.
• BUT, as the number of perfSONAR hosts worldwide grows, we need a way to control – Who is running tests – How oQen are they allowed to run tests
October 15, 2015 51
Roadmap for v3.6 (cont.) • CentOS 7 / Debian 8 support • Pre-‐packaged perfSONAR VM images • GUI to configure test meshes • Maddash 2.0 – See next slide
October 15, 2015 52
MaDDash 2.0 • AlerIng
– Working in conjuncIon with MaDAlert project to analyze dashboard pacerns and create useful alerts
• Scalability – Dashboards keep growing and want to increase frequency and number of checks that can be
run. – Exploring alternaIve back-‐ends and new modes for distribuIng check execuIon
• Usability – Redesign of the web interface. – Major goals include:
• Making dashboards easier to understand • Making it easier to search for and find checks you care about
Test Node Auto-‐ConfiguraIon • Goal: minimize configuraIon required to bring up a new centrally managed perfSONAR node
• Site will register in the lookup service: – What hosts to run tests to – Where to send the test results
• Eventual Goal is a ‘zero config’ mode
October 15, 2015 54
Email Lists and Reference Materials
AcIve and Growing perfSONAR Community
• AcIve email lists and forums provide: – Instant access to advice and experIse
from the community. – Ability to share metrics, experience
and findings with others to help debug issues on a global scale.
• Joining the community automaIcally increases the reach and power of perfSONAR – The more endpoints means
exponenIally more ways to test and discover issues, compare metrics
56
• The perfSONAR collaboraIon is working to build a strong user community to support the use and development of the soQware.
• perfSONAR Mailing Lists – Announcement Lists:
• hcps://mail.internet2.edu/wws/subrequest/perfsonar-‐announce
– Users List: • hcps://mail.internet2.edu/wws/subrequest/perfsonar-‐users
perfSONAR Community
57
Useful URLs • hcp://docs.perfsonar.net/ • hcps://www.perfsonar.net/ • hcp://fasterdata.es.net/ – hcp://fasterdata.es.net/performance-‐tesIng/network-‐troubleshooIng-‐tools/
• hcps://github.com/perfsonar – hcps://github.com/perfsonar/project/wiki
58