Upload
puppet
View
41
Download
0
Embed Size (px)
Citation preview
Joey [email protected] Shepherd
Automating Datastore Fleets with Puppet
Logic
Web Search Ads API Logging Svc 5 Platform
Data
MySQL Lucene Cass ... Kafka DB 5 Zookeeper
...
...
Logic
Web Search Ads API Logging Svc 5 Platform
Data
MySQL Lucene Cass ... Kafka DB 5 Zookeeper
...
...
It’s Complicated
MUST support● Different datastores● Different configurations● Cloud and datacenter ready● Ready for production● Cope with Data Gravity
Yelp’s Datastore Problem
Using Puppet to Manage Datastores
● About 100 clusters● Over 500 datastore nodes● Puppet runs take ~5 minutes● Run Puppet every hour, CI/CD
○ Automation runs via separate scheduler
Monitor Clusters
https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html
Totally fine maybeBAD
DatabaseClient
configure_nerveconfigure_synapse
JSON files hacheck
ZookeeperSynapse Nerve
Healthchecks
ConfigurationConfiguration
Configuration
Healthchecks
Registration
Service Discovery
Service Discovery
DatabaseClient
configure_nerveconfigure_synapse
JSON files hacheck
ZookeeperSynapse Nerve
Healthchecks
ConfigurationConfiguration
Configuration
Healthchecks
Registration
● Server rack, unknown authorhttp://www.clipartkid.com/server-rack-cabinet-clip-art-at-clker-com-vector-clip-art-online-IPTtWc-clipart/
● Devops venn diagram, Kharnagyhttps://commons.wikimedia.org/wiki/File:Devops.svg
● Steve Jobs, waldrynohttps://pixabay.com/en/steve-jobs-technology-illustration-1249665/
Image Citations