Monitoring in the CloudMonitoring in the Cloudwith Puppet

PuppetCamp Ghent 2013

Kris Buytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops

Do you know what your children do at 5 am Do you know what your children do at 5 am in the morning ?in the morning ?

● Sleeping ?Sleeping ?

● Crashing a Party ?Crashing a Party ?

● Why are the cops at your front door ?Why are the cops at your front door ?

● What happened ?What happened ?

● How long have they been gone ?How long have they been gone ?

devopsdevops● CultureCulture

● (Lean)(Lean)

● AutomationAutomation

● MeasurementMeasurement

● SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Gene KimGene Kim

Monitoring is usually an Monitoring is usually an aftertoughtaftertought

ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME

A sub movementA sub movement● #monitoringsucks trending#monitoringsucks trending

● https://github.com/monitoringsucks/https://github.com/monitoringsucks/

● 2008 Study :Nagios + Friends2008 Study :Nagios + Friends

● 2011 Conclusion : Nagios/Icinga are 2011 Conclusion : Nagios/Icinga are the only automatable alternatives the only automatable alternatives

#monitoringsucks#monitoringsucksMonitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:

● Having my hands tied with the model of host and service bindings. Having my hands tied with the model of host and service bindings.

● Having to set up "fake" hosts just to group arbitrary metrics together Having to set up "fake" hosts just to group arbitrary metrics together

● Having to either collect metrics twice - once for alerting and another for trending Having to either collect metrics twice - once for alerting and another for trending

● Only being able to see my metrics in 5 minute intervals Only being able to see my metrics in 5 minute intervals

● Having to chose between shitty interface but great monitoring or shitty monitoring Having to chose between shitty interface but great monitoring or shitty monitoring but great interface but great interface

● Dealing with a monitoring system that thinks IT is the system of truth for my Dealing with a monitoring system that thinks IT is the system of truth for my environment environment

● Not actually having any real choicesNot actually having any real choices

John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.htmlmonitoring-sucks.html

Let's forget aboutLet's forget about● GroundWorkGroundWork

● ZenossZenoss

● ZabbixZabbix

● HypericHyperic

● CactiCacti

● MuninMunin

● BigBrotherBigBrother

What's different in the cloud ?What's different in the cloud ?

● ScaleScale

● VelocityVelocity

● ChangeChange

ChallengesChallenges● ReproducabilityReproducability

● SpeedSpeed

● AuditingAuditing

● Keeping stuff in sync Keeping stuff in sync

Thank Thank godgod Luke for Puppet Luke for Puppet

And Mark for CfengineAnd Mark for Cfengine

}}#monitoringsucks #monitoringsucks 20132013

What we wantWhat we want

● Small , wel suited componentsSmall , wel suited components

• CollectCollect

• Transport / MangleTransport / Mangle

• Analyse / ActAnalyse / Act

• VisualizeVisualize

CollectdCollectd● CollectsCollects

● Zillion PluginsZillion Plugins

• Nginx,apache, mysql, diskNginx,apache, mysql, disk

● Graphite Carbon PluginGraphite Carbon Plugin

● Send metrics to graphiteSend metrics to graphite

● https://github.com/KrisBuytaert/pupphttps://github.com/KrisBuytaert/puppet-collectdet-collectd

● Collect from Collect from anywhereanywhere

● FilterFilter

● Send anywhereSend anywhere

● Queuing supportQueuing support

Puppet and LogstashPuppet and Logstash● https://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash

● Includes Logstash / Kibana / Grok / ES modules as Includes Logstash / Kibana / Grok / ES modules as submodules ! submodules !

git clonegit clone

● git submodule init git submodule init

● git submodule update git submodule update

● vagrant up vagrant up

Graphite Graphite ● Graphing at ScaleGraphing at Scale

● Graphing at EaseGraphing at Ease

● Any metric is a graphAny metric is a graph

● echo "somestring $somevalue $timestamp" | echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003

Graphite ComposerGraphite Composer

x

Graphite APIGraphite API

Gdash In actionGdash In action

Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/

● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !

● git clonegit clone

● git submodule init git submodule init

● git submodule update git submodule update

● vagrant up vagrant up

Collectd & GraphiteCollectd & Graphite

Configuring a ServiceConfiguring a Service● PackagePackage

● ConfigConfig

● Service Service

<- we pray status is correct<- we pray status is correct

Apache Example:Apache Example:

Definition of Done:Definition of Done:

Done means : being monitoredDone means : being monitored

Icinga ?Icinga ?• Isn't nagios dead ?Isn't nagios dead ?

• Vibrant CommunityVibrant Community

• Throw great parties in NurnbergThrow great parties in Nurnberg

• Nobody can pronounce it anyhowNobody can pronounce it anyhow

• https://github.com/Inuits/puppet-icinga/https://github.com/Inuits/puppet-icinga/

Monitoring BaselineMonitoring Baseline● Deploy a host,Deploy a host,

● Add it to the monitoringAdd it to the monitoring

● Add collection toolsAdd collection tools

● Add check definitionsAdd check definitions

● Update the monitoring tool configUpdate the monitoring tool config

Stored ConfigsStored Configs

Collection and ExportCollection and Export

Export :Export :

@@resource { @@resource {

... }... }

Collect:Collect:

Resource <<| Resource <<| query |>>query |>>

Clean out nodes that dissapearClean out nodes that dissapear

puppet node clean puppet node clean

Exporting and Collecting Exporting and Collecting

Monitoring a VhostMonitoring a Vhost

It is not in production if it is not It is not in production if it is not monitoredmonitored

Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics

● JMXTransJMXTrans

● Export JMXConfigsExport JMXConfigs

● Configure NRPE CheckConfigure NRPE Check

● Export NagiosCheckExport NagiosCheck

● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode

● Graph EmGraph Em

Collect Nagios Configs Collect Nagios Configs on Nagios Serveron Nagios Server

Triggers on GraphsTriggers on Graphs

Triggers on GraphsTriggers on Graphs

Conclusion:Conclusion:● Reproducable monitoring setupReproducable monitoring setup

● Dynamically generated monitoring Dynamically generated monitoring configconfig

● Code is available at Code is available at github.com/KrisBuytaert/snippetsgithub.com/KrisBuytaert/snippets

But,But,(pronounced in Ozzy dialect)(pronounced in Ozzy dialect)

htt

p:/

/ww

w.k

art

ar.

net/

201

3/0

1/m

onit

ori

ng

-suck

s---

a-r

ant/

#monitoringlove#monitoringlove

HomeworkHomeworkSensuSensu

ExtrememonExtrememon

......

RiemannRiemann

EsperEsper

ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221

http://loadays.orghttp://loadays.org