Baking in the Cloudwith Packer and

Puppet

Alan Parkinson

@alan_parkinson

Large jumps in demand, we have to scale fast

Load Balancer Behave Pro App

Our problems with provisioning on start-up

Reliability

Many single points of failure

Require Puppet Master redundancy

and datacentre replication

Mirror dependant software repositories

Provisioning Latency…9 minutes for puppet to prepare a app server

Total 15 minute response time to a scaling

request

Baking images like

Produce a preconfigured

machine image that is rolled out

to the autoscaling groups

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration

PACKER.IO A HASHICORP PROJECT

PACKER.IO A HASHICORP PROJECT

packer.json

{ "variables": { "aws_access_key": "", "aws_secret_key": "" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-9eaa1cf6", "instance_type": "t2.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }]}

$ packer build -var 'aws_access_key=YOUR ACCESS KEY' \ -var 'aws_secret_key=YOUR SECRET KEY' \ example.json

==> amazon-ebs: amazon-ebs output will be in this color. ==> amazon-ebs: Creating temporary keypair for this instance... ==> amazon-ebs: Creating temporary security group for this instance... ==> amazon-ebs: Authorizing SSH access on the temporary security group... ==> amazon-ebs: Launching a source AWS instance... ==> amazon-ebs: Waiting for instance to become ready... ==> amazon-ebs: Connecting to the instance via SSH... ==> amazon-ebs: Stopping the source instance... ==> amazon-ebs: Waiting for the instance to stop... ==> amazon-ebs: Creating the AMI: packer-example 1371856345 ==> amazon-ebs: AMI: ami-19601070 ==> amazon-ebs: Waiting for AMI to become ready... ==> amazon-ebs: Terminating the source AWS instance... ==> amazon-ebs: Deleting temporary security group... ==> amazon-ebs: Deleting temporary keypair... ==> amazon-ebs: Build finished.

==> Builds finished. The artifacts of successful builds are: --> amazon-ebs: AMIs were created:

us-east-1: ami-19601070

Add some provisioners{ "variables": {…}, "builders": […], "provisioners": [ { "type": "shell", "script": "../common/install-puppet.sh" }, … { "inline": [ "sudo apt-get purge --yes puppet", "sudo apt-get autoremove --yes" ], "type": "shell" } ]}

No need for a Puppet Master

"provisioners": [ … { "type": "puppet-masterless", "hiera_config_path": "../puppet/hiera.yaml", "manifest_file": "../puppet/manifests/default.pp", "module_paths": [ "../puppet/modules" ] }, …]

Manifests, modules and hiera data can all be stored in git and git submodules

How do we protect sensitive configuration data?

hiera-eyamlbackend for Hiera that provides per-value

encryption of sensitive data within yaml files

---

duo-security-skey: ENC[PKCS7,MIIBmQYJKoZIh……Anc=]

behave_pro: logentries_api_key: ENC[PKCS7,MIIBmQYJKoZIh……uW8=] application_secret: ENC[PKCS7, MIIBmQYJKoZIh……FRg==]

common.eyaml

hiera.yaml

---:backends: - eyaml - yaml

:hierarchy: - "%{environment}" - common

:yaml: :datadir: '/tmp/hieradata':eyaml: :datadir: '/tmp/hieradata'

:pkcs7_private_key: /tmp/hierakeys/private_key.pkcs7.pem :pkcs7_public_key: /tmp/hierakeys/public_key.pkcs7.pem

Note: Use a temporary folders or data will be baked into the final image

Basic asymmetric encryption (PKCS#7)

Private key decrypts data

Puppet Master or agent only needs this at runtime

$ eyaml encrypt -s 'hello there'

Public key encrypts data

Safe to distribute to developers and ops engineers

Git diff allows peer review without decrypting values

The eyaml keys are stored in a private S3 bucket with access controlled by a IAM

Policy

Distributing the keysto the Bakery

Use a IAM Role in Packer to access the S3 bucket

"builders": [{ "type": "amazon-ebs", … "iam_instance_profile" : "puppet-provisioner", …}]

IAM Policy

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1425244502000", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::puppet.behave.pro/*" ] } ]}

install-heira-key.sh

Download the keys to the EC2 Instance

sudo apt-get install --yes python-pipsudo pip install s3cmd

s3cmd get s3://puppet.behave.pro/private_key.pkcs7.pem /tmp/hierakeys/private_key.pkcs7.pem

s3cmd get s3://puppet.behave.pro/public_key.pkcs7.pem/tmp/hierakeys/public_key.pkcs7.pem

"provisioners": [ { "type": "shell", "script": "../common/install-hiera-key.sh" }, ]

packer.json

Summary

If scaling fast or reliably are important, bake images

Git makes a great alternative to Puppet Master when baking

Secure data with hiera-eyamlhttps://github.com/TomPoulton/hiera-eyaml

Questions