Leveling up monitoring: A decade of automating and scaling Nagios

Leveling Up Monitoring:

A Decade of Automating and Scaling Nagios

Katherine Daniels and Laurie Denness

@beerops - @lozzd Velocity 2016


Katherine Daniels@beerops

Senior Operations Engineer, Etsy Co-Author of Effective DevOps

Laurie Denness @lozzd

Staff Operations Engineer, Etsy Official Graph Enthusiast

3

Agenda


Au to mat i o n

2

D e p loy i nato r

3

S c a l i ng + To o l i ng

4

I n T h e B e g i n n i ng . . .

1

25MActive Buyers

About Etsy

1.6MActive Sellers

$2.39B2015 Annual GMS

(As of March 31, 2016)

Monitoring!



bit.ly/yaynagios

http://bit.ly/yaynagios

https://kartar.net/2015/08/monitoring-survey-2015---tools/




In The Beginning




Sometimes your statement needs emphasis with a black background.



L E S S O N S L E A R N E D :

Templates are awesome.






define service { use generic-service hostgroups Linux_hosts,!email-only-servers service_description SSH check_command check_ssh }


define service { use disk-space-service hostgroup_name email-only-servers contact_groups ops_nonurgent }




Start small.


Nagios and Chef



24



Automation is awesome!



Automation is awesome!

HA HA JUST KIDDING





Trust but verify.


How Many Repos?





?!?!?!?!??!?!



Try, fail, learn, and try again.

Problems

Problems

• Four git repos, inconsistent mess, duplication

Problems


• Broken semi-useful automation - need to regain trust

Problems



• Some shared config, some unique

Problems




• Gain confidence in changes

Problems




• Gain confidence in changes

• Stop editing on the production box


Nagios and Chef


Nagios and Chefand Deployinator!


Solution 1: Merge everything: find and remove duplication,

shared configs


Thanks Murphy!


Super Secret Option!!!





Solution 2:

Using Jenkins CI to test changes before production


Solution 3:

Use Deployinator to run Chef recipe to generate automated configs

Chart Tit le

Chart Tit le


Solution 4:

Use Deployinator to rsync config to all boxes

• git pull repo on deploy host


• Run Chef recipe to add automated pieces



• Re-run the try-nagios script against that




• rsync copy from deploy box to Nagios hosts





• Create symlink for nagios.cfg





• Create symlink for nagios.cfg

• Restart Nagios



Use the tools you have.


Scaling things up!






Core Workers


Core Workers




If at first you don’t succeed, rub some webscale on it.


Iterating and Iterating



Iterate

Iterate

Iterate


To Infinity and Beyond


http://github.com/etsy/opsweekly

http://github.com/etsy/opsweekly

Chart Tit le

Chart Tit le

Final Lessons Learned

• Templates are awesome

• Start small

• Automation is awesome

• Trust but verify

• Learn from (y)our mistakes

• Iterate on the tools you have

Open Source Summary

Open Source Summary

• http://github.com/etsy/deployinator

• http://github.com/etsy/pushbot

• http://github.com/etsy/trylib

• http://github.com/etsy/opsweekly

• http://github.com/etsy/nagios-herald

• http://github.com/RJ/irccat

THANK YOU!


Software

Leveling up monitoring: A decade of automating and scaling Nagios