April 11, 2023

Erik Dalén

Puppet at Spotify

2Section name

● Over 24 million monthly active users● Launched in 28 countries● Over 20 million songs● More than 1 billion playlists

Growing quickly

Spotify

System Engineer in Site Reliability Engineering at Spotify

Operational system owner for Puppet, playlist system and Cassandra

Community contributor to Puppet

whoami

● More than 450 changes per month● 220 committers to our Puppet git repository● 325 puppet modules

Code review by SRE team using Gerrit

Puppet users since 3 years

Puppet Infrastructure

● Roughly 5500 nodes

● 3 different Puppet installations

● Each with their own CA and PuppetDB

● One or more puppetmasters per data centre

● Run using cron

Git branch = = Puppet environment

Everyone can push to private branches and run puppet against those

Code review mandatory to push to “production” branch

Puppet Infrastructure

Built as Debian Packages

Deployed using Puppet

Backend services

●Client connections are proxied through the accesspoints●Most other backend services are stateless●Storage in Cassandra, PostgreSQL or Tokyo Cabinet

Architecture overview

Accessp

oint

Service 1 DB

Service 2 DB

Service 3

●Puppet module for the service●Deployed and tested in test environment●Hardware requested from SRE team and service

deployed in production

Backend service deployment

Using SRV records to discover services

Puppet module dalen-dnsquery can be used to lookup them from inside Puppet manifests.

Service Discovery

At the moment using different ENCs in different Puppet installations

Will be switching to using Hiera for node classification

Node Classification

Hostname standard:

sto1-puppet-a1

<site>-<role>-<pool><poolid>

Mainly role fact used to determine the classes a machine should have

Node Classification

25 pull requests from Spotify merged to core Puppet and Facter in last 12 months.

Many improvements to puppetlabs modules sent upstream.

Puppet contributions

Tired of writing queries like this?

["and", ["in", "name", ["extract", "certname", ["select-resources", ["and", ["=", "exported", false], ["=", "type", "Class"], ["=", "title", "Cassandra"], ["=", ["parameter", "version"], "1.1"]]]]], ["in", "name", ["extract", "certname", ["select-facts", ["and", ["=", "name", "site"], ["=", "value", "lon"]]]]]]

Querying PuppetDB

Finding nodes using dalen-puppetdbquery:

$ puppet query nodes ‘Class[Cassandra]{version=“1.1”} and site=lon’

$ puppet query nodes ‘processorcount > 16 and manufacturer ~”Dell.*”’

github.com/dalen/puppet-puppetdbquery

Querying PuppetDB

Use PuppetDB as a backend to the datamapper ORM

Node.get(’foo.example.com’).facts.each do |fact|

puts “#{fact.name}: #{fact.value}”

end

dm-puppetdb-adapter

A Puppet face to list files managed by puppet

# puppet ls /etc/systemd/system

nagios-nrpe-server.service

declared in /etc/puppet/environments/production/modules/systemd/manifests/unit.pp:15

content from a "content" parameter

puppet ls

● Splitting the repo out● Remove SRE review requirement on large parts● Support testing using vagrant● Building images using a masterless puppet apply

The future

Consists of a node terminus and a forge implementation

Builds a per node environment dynamically on demand

Will be open sourced Real Soon Now™

Spikor

Questions?

April 11, 2023

Check out spotify.com/jobs or @Spotifyjobs for more information.

Want to join the band?