Monitoring at Cloud Scale

;

Monitoring at Cloud ScaleMonitoring at Cloud Scale

Julien PivottoJulien Pivotto

Build a cloud Day AmsterdamBuild a cloud Day AmsterdamJune 13th, 2013June 13th, 2013

;

IntroductionAround monitoring

ToolsConclusion

Table of contents

1 IntroductionDevOpsmonitoringsucksmonitoringsucks

2 Around monitoringThe cloudThe pastEnvironmentChallengesInfrastructure as code

3 ToolsCollectdLogstashStatsdGraphiteIcinga

4 ConclusionThey work togetherSharingThere are solutions

Julien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion

Julien PivottoJulien Pivotto• sysadmin @ inuitssysadmin @ inuits• open-source defender for 7+ yearsopen-source defender for 7+ years• devops believerdevops believer• @roidelapluie on twitter/github@roidelapluie on twitter/github


;


ToolsConclusion

DevOpsmonitoringsucksmonitoringsucks

What is that DevOps stuff again?

• CultureCulture• (Lean)(Lean)• AutomationAutomation• MeasurementMeasurement• SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis


;


ToolsConclusion


#monitoringsucks

• https://github.com/monitoringsuckshttps://github.com/monitoringsucks• a movement to find a solution to monitoringa movement to find a solution to monitoring• the feeling that monitoring is stucked in the pastthe feeling that monitoring is stucked in the past


;


ToolsConclusion


#monitoringlove

• then it turned into #monitoringlovethen it turned into #monitoringlove• relevant tools existrelevant tools exist• they just need to be usedthey just need to be used• following the unix philosophyfollowing the unix philosophy

we are going to explore some of themwe are going to explore some of them


;


ToolsConclusion

The cloudThe pastEnvironmentChallengesInfrastructure as code

What is different in the cloud?

• ScaleScale• VelocityVelocity• More changes, more oftenMore changes, more often


;


ToolsConclusion


What do you need?

• scalabilityscalability• automationautomation


;


ToolsConclusion


time for retirement

• forget all-in-one toolsforget all-in-one tools• forget auto-discovery toolsforget auto-discovery tools• forget non-scalable toolsforget non-scalable tools• forget tools you can not automateforget tools you can not automate


;


ToolsConclusion


forget about. . .

http://www.flickr.com/photos/mourner/150844753/

ZabbixZabbixJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


CentreonCentreonJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


GroundWorkGroundWorkJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


CactiCactiJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


HypericHypericJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


BigBrotherBigBrotherJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


MuninMuninJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


forget about. . .


ZenossZenossJulien Pivotto Monitoring at Cloud Scale

;


ToolsConclusion


Your infrastructure today

http://www.flickr.com/photos/bjbrake/235217140/


;


ToolsConclusion


Your infrastructure tomorrow



;


ToolsConclusion


Your infrastructure in 6 months



;


ToolsConclusion


Challenges

• ReproducabilityReproducability• SpeedSpeed• MetricsMetrics• OrchestrationOrchestration


;


ToolsConclusion


WANTED

http://www.flickr.com/photos/pagedooley/3124443099/

• Small toolsSmall tools• Collect / MangleCollect / Mangle• Analyse / ActAnalyse / Act• VisualizeVisualize


;


ToolsConclusion


WANTED

The UNIX philosophyThe UNIX philosophy


;


ToolsConclusion


Automation

• One source of trust: puppet, chef, . . .One source of trust: puppet, chef, . . .• Exported resourceExported resource• Monitor in the same location you deployMonitor in the same location you deploy• Infrastructure-as-CodeInfrastructure-as-Code• no autodiscovery toolsno autodiscovery tools


;


ToolsConclusion


Automation

If it is not automated || not monitoredIf it is not automated || not monitoredthen it does not exist!then it does not exist!


;


ToolsConclusion


Example in puppet

• Create a definition for your applicationCreate a definition for your application• In that definition, add the configuration, the vhosts. . .In that definition, add the configuration, the vhosts. . .• Then export the monitoring (@@icinga_service)Then export the monitoring (@@icinga_service)• In bonus you can export DB configuration, etc. . .In bonus you can export DB configuration, etc. . .• Use only the "meta" definitionUse only the "meta" definition• Collect the exported ressources (Nagios_service <||>)Collect the exported ressources (Nagios_service <||>)


;


ToolsConclusion

CollectdLogstashStatsdGraphiteIcinga

Collectd

• Statistics collection daemonStatistics collection daemon• A lot of plugins available. . .A lot of plugins available. . .• Can send data to graphiteCan send data to graphite• Simple configurationSimple configuration


;


ToolsConclusion


Collectd plugins

http://www.flickr.com/photos/juhansonin/3141561416/


;


ToolsConclusion


Collectd plugins

AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt LoadLogFile LPAR MadWifi MBMon memcachec memcached Memory ModbusLogFile LPAR MadWifi MBMon memcachec memcached Memory Modbus

Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginxNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock

Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMSWrite_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB

Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC


;


ToolsConclusion


Logstash

• Ship logs from any sourceShip logs from any source• Filter themFilter them• Index themIndex them• Search themSearch them• Backed with elasticsearchBacked with elasticsearch


;


ToolsConclusion


Kibana

http://kibana.org/images/screenshots/searchss.png


;


ToolsConclusion


Statsd

• Stats aggregationStats aggregation• Simple countersSimple counters• Flushes every XX seconds to graphiteFlushes every XX seconds to graphite• Text over UDPText over UDP


;


ToolsConclusion


Statsd

echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125


;


ToolsConclusion


Graphite

• Graphing made simpleGraphing made simple• A lot of helpers functionsA lot of helpers functions• Listening on UDP and TCPListening on UDP and TCP• Text over UDP/TCPText over UDP/TCP


;


ToolsConclusion


Send data to graphite

echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003


;


ToolsConclusion


Graphite API


;


ToolsConclusion


gdash

https://github.com/ripienaar/gdash


;


ToolsConclusion


Icinga

• Fork of nagiosFork of nagios• Large and vibrant communityLarge and vibrant community• Configuration compatible with nagiosConfiguration compatible with nagios• User-friendly interfaceUser-friendly interface• Use Icinga Classic!Use Icinga Classic!


;


ToolsConclusion


Icinga

https://icinga.org


;


ToolsConclusion

They work togetherSharingThere are solutions

Toolchain from apache to nagios

• Apache ships logs to rsyslogApache ships logs to rsyslog• Rsyslog ships logs to logstashRsyslog ships logs to logstash• Logstash ships metrics to statsdLogstash ships metrics to statsd• Statsd ships metrics to GraphiteStatsd ships metrics to Graphite• Icinga query metric from graphiteIcinga query metric from graphite• https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools


;


ToolsConclusion


Reusing Icinga/Nagios perfdata

• Icinga performs various checksIcinga performs various checks• Icinga sends perfdata to graphiteIcinga sends perfdata to graphite• Graphite stores the dataGraphite stores the data• Gdash serves them inside dashboardsGdash serves them inside dashboards• https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite


;


ToolsConclusion


The metrics

• Everything can become a metricEverything can become a metric• Performance metricsPerformance metrics• Usage metricsUsage metrics• Business-valuable metricsBusiness-valuable metrics• People metricsPeople metrics• Metrics are knowledgeMetrics are knowledge


;


ToolsConclusion


Metrics that matter

http://codeascraft.com/2011/02/15/measure-anything-measure-everything/


;


ToolsConclusion


What have we seen?

• We have seen only open-source softwareWe have seen only open-source software• Small, pluggable daemonsSmall, pluggable daemons• Robust solutionsRobust solutions• Nice & user-friendly outputNice & user-friendly output• They play togetherThey play together


;


ToolsConclusion


Homework

• SensuSensu• RiemannRiemann• dashingdashing• ExtreMonExtreMon• EsperEsper• SkylineSkyline• OculusOculus


;


ToolsConclusion


Try them yourself

https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash


;


ToolsConclusion


Contact

Julien PivottoJulien [email protected]@inuits.eu@roidelapluie@roidelapluie

INUITS bvbaINUITS bvbaDuboisstraat 50Duboisstraat 502060 Antwerp2060 AntwerpBelgiumBelgium+32 473 441 636+32 473 441 636https://inuits.euhttps://inuits.eu


Technology

Monitoring at Cloud Scale