Monitoring (with) PuppetMonitoring (with) Puppet

PuppetCamp Dusseldorf

@KrisBuytaert

KrisKris BuytaertBuytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Offcer and Open Source Consultant Chief Trolling Offcer and Open Source Consultant

@inuits.eu@inuits.eu● Everything is an effng DNS ProblemEverything is an effng DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Organising too many confs , #devopsdays, Organising too many confs , #devopsdays,

#loadays, ... #loadays, ... ● Evangelizing devopsEvangelizing devops

Do you know what your children do at 5 am in the Do you know what your children do at 5 am in the morning ?morning ?

● Sleeping ?Sleeping ?

● Crashing a Party ?Crashing a Party ?

● Why are the cops at your front door ?Why are the cops at your front door ?

● What happened ?What happened ?

● How long have they been gone ?How long have they been gone ?

devops = clamsdevops = clams● CultureCulture

● (Lean)(Lean)

● Automate all the things ... Automate all the things ...

• Build Automation Build Automation

• Test Automation Test Automation

• IACIAC

● Monitoring , Metrics ... Monitoring , Metrics ...

● SharingSharing

Monitoring is usually an aftertoughtMonitoring is usually an aftertoughtENOBUDGET, ENOTIMEENOBUDGET, ENOTIME

#monitoringsucks#monitoringsucks● John Vincent (@lusis)John Vincent (@lusis)

● A sub movement A sub movement

● https://github.com/monitoringsucks/https://github.com/monitoringsucks/

#monitoringlove#monitoringlove• #monitoringlove hacksessions#monitoringlove hacksessions

• #monitorama#monitorama

For years we've tolerated humans to to For years we've tolerated humans to to make structural manual changes to the make structural manual changes to the infrastructure our critical applications are infrastructure our critical applications are running on.running on.

Whilst at the same time demanding those Whilst at the same time demanding those critical applications to go trough rigid test critical applications to go trough rigid test

scenarios.scenarios.

Who let this happen ?Who let this happen ?

Infrastructure as CodeInfrastructure as Code● Model our infrastructure Model our infrastructure

● A fast reproducable platformA fast reproducable platform

● Disaster discovery for “free”Disaster discovery for “free”

Infrastructure as CodeInfrastructure as Code● Code = Code Code = Code

● Version Control Version Control

● Quality ChecksQuality Checks

● TestingTesting

● Continuous IntegrationContinuous Integration

● Continous DeliveryContinous Delivery

Infrastructure as CodeInfrastructure as Code● Core Infrastructure Core Infrastructure

● Middleware deployment and integrationMiddleware deployment and integration

● Automated continuous application Automated continuous application deploymentdeployment

● Integrated Security enforcementIntegrated Security enforcement

● Host, Service and Application Monitoring Host, Service and Application Monitoring confguredconfgured

Why #monitoringsucksWhy #monitoringsucks● Manual confg (gui)Manual confg (gui)

● Not in sync with realityNot in sync with reality

● Hosts onlyHosts only

● Services sometimesServices sometimes

● Appliccation neverAppliccation never

● ChaosChaos

• Really ?Really ?

• Service,Service,

• FunctionalitiesFunctionalities

• eg. vhosts etceg. vhosts etc

• Single Source of TruthSingle Source of Truth

But tools do AutodetectionBut tools do Autodetection

Monitoring 101Monitoring 101● Deploy a host,Deploy a host,

● Add it to the monitoringAdd it to the monitoring

● Add collection toolsAdd collection tools

● Add check defnitionsAdd check defnitions

● Update the monitoring tool confgUpdate the monitoring tool confg

● RestartRestart

Collect Metrics 101Collect Metrics 101

CollectdCollectd● CollectsCollects

● Zillion PluginsZillion Plugins

• Nginx,apache, mysql, diskNginx,apache, mysql, disk

● Graphite Carbon PluginGraphite Carbon Plugin

● Send metrics to graphiteSend metrics to graphite

● https://github.com/KrisBuytaert/puppet-https://github.com/KrisBuytaert/puppet-collectdcollectd

Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/

● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Tattle and more modules as submodules !

● git clonegit clone

● git submodule init git submodule init

● git submodule update git submodule update

● vagrant up vagrant up

2014 style dashboards2014 style dashboards

Alerting 101Alerting 101

Alert all the thingsAlert all the thingsAnd get alertfatigueAnd get alertfatigue

● We are in Germany, EuropeWe are in Germany, Europe

● It needs to be confgurable It needs to be confgurable

=> Icinga=> Icinga

But Monitoring confg is Data ?But Monitoring confg is Data ?● If it can be generated it's not user If it can be generated it's not user

generated content anymoregenerated content anymore

● Your computer can generate your confg !Your computer can generate your confg !

Stored ConfgsStored Confgs

Collection and ExportCollection and Export

Export :Export :

@@resource { @@resource {

... }... }

Collect:Collect:

Resource <<| query Resource <<| query |>>|>>

Clean out nodes that dissapearClean out nodes that dissapear

puppet node clean puppet node clean

Exporting and Collecting Exporting and Collecting

Default Puppet TypesDefault Puppet Types

Puppet-icinga modulePuppet-icinga module● https://github.com/inuits/puppet-icingahttps://github.com/inuits/puppet-icinga

Monitoring a VhostMonitoring a Vhost

Alternative ApproachesAlternative Approaches

● https://gist.github.com/jfryman/5808537https://gist.github.com/jfryman/5808537

● https://github.com/favoretti/puppetdb-https://github.com/favoretti/puppetdb-external-naginatorexternal-naginator

● Deploy a new appDeploy a new app

● Add monitoringAdd monitoring

● Add Real application monitoringAdd Real application monitoring

● Both on infra and on app levelBoth on infra and on app level

Monitoring Puppet & FriendsMonitoring Puppet & Friends

Puppet RunsPuppet Runs

PuppetMasterPuppetMaster

PuppetMasterPuppetMaster @@nagios_service{"check_socket_8140_puppet_${::fqdn}":@@nagios_service{"check_socket_8140_puppet_${::fqdn}":

check_command => 'check_tcp!8140',check_command => 'check_tcp!8140',

service_description => 'TCP puppet on port 8140',service_description => 'TCP puppet on port 8140',

host_name => $::fqdn,host_name => $::fqdn,

use => 'generic-service',use => 'generic-service',

contact_groups => $::environment,contact_groups => $::environment,

notifcation_period => $::icinga::notifcation_period,notifcation_period => $::icinga::notifcation_period,

notifcations_enabled => $::icinga::notifcations_enabled,notifcations_enabled => $::icinga::notifcations_enabled,

target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",

}}

PuppetDBPuppetDB

PuppetDB(2)PuppetDB(2)● check_puppetdb_memorycheck_puppetdb_memory

• Java heap memoryJava heap memory

● check_puppetdb_processedcheck_puppetdb_processed

• Nr of reports processedNr of reports processed

● check_puppetdb_populationscheck_puppetdb_populations

• Resources, nodes, resources per nodeResources, nodes, resources per node

● check_puppetdb_queuecheck_puppetdb_queue

• Is pgsql down ? :)Is pgsql down ? :)

Puppet DashboardPuppet Dashboard @@nagios_service{"check_http_puppet_dashboard_${::fqdn}":@@nagios_service{"check_http_puppet_dashboard_${::fqdn}":

check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",

service_description => 'HTTP PuppetDashboard on port 3000',service_description => 'HTTP PuppetDashboard on port 3000',

host_name => $::fqdn,host_name => $::fqdn,

use => 'generic-service',use => 'generic-service',

contact_groups => $::environment,contact_groups => $::environment,

notifcation_period => $::icinga::notifcation_period,notifcation_period => $::icinga::notifcation_period,

notifcations_enabled => $::icinga::notifcations_enabled,notifcations_enabled => $::icinga::notifcations_enabled,

target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",

}}

Puppet Dashboard(2)Puppet Dashboard(2)

/usr/share/puppet-dashboard/spool/usr/share/puppet-dashboard/spool

#MonitoringSucks#MonitoringSucks● Puppetruns break our Icinga boxenPuppetruns break our Icinga boxen

● BadlyBadly

● FrequentlyFrequently

It ain't borkenIt ain't borken● Successful puppet runSuccessful puppet run

● Successful Icinga reconfgureSuccessful Icinga reconfgure

● Disk usage growsDisk usage grows

● FastFast

Be aware of bucketsBe aware of buckets

A Puppet BugA Puppet Bug

Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics

● JMXTransJMXTrans

● Export JMXConfgsExport JMXConfgs

● Confgure NRPE CheckConfgure NRPE Check

● Export NagiosCheckExport NagiosCheck

● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode

● Graph EmGraph Em

Collect Nagios Confgs on Collect Nagios Confgs on Nagios ServerNagios Server

Triggers on GraphsTriggers on Graphs

Triggers on GraphsTriggers on Graphs

SummarySummary● Honour your parentsHonour your parents

● Don't manually do what machines can do Don't manually do what machines can do for youfor you

● Monitor your puppet infrastructure too !Monitor your puppet infrastructure too !

● Send Pull RequestsSend Pull Requests

● Icinga2 module in the worksIcinga2 module in the works

ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221