Getting Started with PoolParty and EC2

Getting Started with PoolParty & EC2

PoolParty

PoolParty

• PoolParty makes it easy and simple to configure any cloud of computers

PoolParty

• PoolParty makes it easy and simple to configure any cloud of computers

• Written in Ruby and Erlang

PoolParty

PoolParty

• Not Ruby or Rails specific

PoolParty

• Not Ruby or Rails specific

• Not even tied to Amazon EC2

sample config

Amazon

EC2

EC2 S3

EC2 S3

EBS

EC2 S3

EBS AMI

EC2 S3

EBS AMI

Elastic IP

EC2 S3

EBS AMI

Elastic IP Keypairs

EC2 S3

EBS AMI

Elastic IP Keypairs

Security Groups

EC2 S3

EBS AMI

Elastic IP Keypairs

Security GroupsCloudFront

EC2 S3

EBS AMI

Elastic IP Keypairs

Security GroupsCloudFront???

Amazon: EC2

Amazon: EC2

• Platform to launch machine instances

Amazon: EC2

• Platform to launch machine instances

• ec2-* (git-style) commands such as:

Amazon: EC2

• Platform to launch machine instances

• ec2-* (git-style) commands such as:

• ec2-run-instances

Amazon: EC2

• Platform to launch machine instances

• ec2-* (git-style) commands such as:

• ec2-run-instances

• ec2-terminate-instances

Amazon: EC2

• Platform to launch machine instances

• ec2-* (git-style) commands such as:

• ec2-run-instances

• ec2-terminate-instances

• ec2-allocate-address

Amazon: EC2

• Platform to launch machine instances

• ec2-* (git-style) commands such as:

• ec2-run-instances

• ec2-terminate-instances

• ec2-allocate-address

• etc.

Amazon: EC2 (AMIs)

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

• Conceptually similar to a Xen or VMware image

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

• Conceptually similar to a Xen or VMware image

• Base to build on

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

• Conceptually similar to a Xen or VMware image

• Base to build on

• Hundreds pre-built

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

• Conceptually similar to a Xen or VMware image

• Base to build on

• Hundreds pre-built

• Snapshots

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

• Conceptually similar to a Xen or VMware image

• Base to build on

• Hundreds pre-built

• Snapshots

• PoolParty default is the widely used ami-1cd73375 - ec2ubuntu

Amazon: EC2

Amazon: EC2

• Primarily uses ephemeral storage

Amazon: EC2

• Primarily uses ephemeral storage

• e.g. node crashes = data gone (sortof)

Amazon: EC2

• Primarily uses ephemeral storage

• e.g. node crashes = data gone (sortof)

• requires new way to think about servers

Amazon: EC2

• Primarily uses ephemeral storage

• e.g. node crashes = data gone (sortof)

• requires new way to think about servers

• forces redundancy and clustering issues to the front

Amazon: S3

Amazon: S3

• Simple Storage Service (S3)

Amazon: S3

• Simple Storage Service (S3)

• Persistent

Amazon: S3

• Simple Storage Service (S3)

• Persistent

• Data stored in “buckets”

Amazon: S3

• Simple Storage Service (S3)

• Persistent

• Data stored in “buckets”

• EC2 does not use S3 *directly*

Amazon: EBS

Amazon: EBS

• Elastic Block Store

Amazon: EBS

• Elastic Block Store

• Persistent disks for EC2 instances

Amazon: EBS

• Elastic Block Store

• Persistent disks for EC2 instances

• Can store snapshots on S3

Amazon: EBS

• Elastic Block Store

• Persistent disks for EC2 instances

• Can store snapshots on S3

• Fast

Amazon: EBS

• Elastic Block Store

• Persistent disks for EC2 instances

• Can store snapshots on S3

• Fast

• Mounts to only one instance at a time (not a SAN)

Amazon: Elastic IPs

Amazon: Elastic IPs

• Instance IP addresses are dynamic

Amazon: Elastic IPs

• Instance IP addresses are dynamic

• Elastic IPs are static

Amazon: Elastic IPs

• Instance IP addresses are dynamic

• Elastic IPs are static

• Free to use, but $0.01/hr if unassociated with an instance

EC2 Security Groups

EC2 Security Groups

• Amazon has a strict firewall

EC2 Security Groups

• Amazon has a strict firewall

•ec2-authorize

EC2 Security Groups

• Amazon has a strict firewall

•ec2-authorize

• default security group nodes have unlimited (network) access to each other

EC2 Security Groups

• Amazon has a strict firewall

•ec2-authorize

• default security group nodes have unlimited (network) access to each other

• everything else is turned off

EC2 Security Groups

ec2-add-group web -d "Web server group" ec2-authorize -p 80 webec2-authorize -p 22 web

EC2 Security Groups

ec2-add-group web -d "Web server group" ec2-authorize -p 80 webec2-authorize -p 22 webec2-authorize -P icmp -t -1:-1 web

Amazon: CloudFront

Amazon: CloudFront

• Amazon’s “self service” CDN

Amazon: CloudFront

• Amazon’s “self service” CDN

• Pay-as-you-go

Amazon: CloudFront

• Amazon’s “self service” CDN

• Pay-as-you-go

• Upload to S3

Amazon: CloudFront

• Amazon’s “self service” CDN

• Pay-as-you-go

• Upload to S3

• Content is distributed & cached on Amazon’s machines

PoolParty

PoolParty overview

PoolParty overview

• binary tools

PoolParty overview

• binary tools

• configuration files

PoolParty overview

• binary tools

• configuration files

• erlang-driven messenger

PoolParty: Binary Tools

PoolParty: Binary Tools

• pool-* binaries

PoolParty: Binary Tools

• pool-* binaries

• server-* binaries

PoolParty: Binary Tools

Shell Example

PoolParty Configuration

PoolParty Configuration

PoolParty Configuration

PoolParty Configuration

PoolParty Configuration

Shell Examplea look at clouds/plugins/poolparty-mysql-plugin/mysql.rb

how does this actually work?

Puppet

Puppet

Puppet

• Puppet is a system for automating system administration tasks.

Puppet

• Puppet is a system for automating system administration tasks.

• Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

Puppet

• Puppet is a system for automating system administration tasks.

• Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

• Red Hat - In the process of moving legacy systems onto Puppet.

Puppet

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

• Mini-language

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

• Mini-language

• Dependencies

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

• Mini-language

• Dependencies

• If/unless conditions

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

• Mini-language

• Dependencies

• If/unless conditions

• Variables

Puppet

Puppet

• Client/Server architecture

Puppet

• Client/Server architecture

• puppetmasterd

Puppet

• Client/Server architecture

• puppetmasterd

•puppetd

Puppet: Language Example

Puppet: Language Example

Puppet: Language Example

Puppet: Language Example

Puppet: Language Example

Puppet

Puppet

• Puppet base config:

/etc/puppet/puppet.conf

Puppet

• Puppet base config:

/etc/puppet/puppet.conf

• PoolParty’s Puppet config:

/etc/puppet/manifests/classes/poolparty.pp

Puppet Configuration

Shell Examplea look at /etc/puppet/manifests/classes/poolparty.pp

Puppet

• Cron jobs:* 1 * * * . /etc/profile && puppetmasterd --verbose

*/15 * * * * /usr/bin/puppetrunner

Puppet

Puppet

• Builds dependency graph

Puppet

• Builds dependency graph

• Provisions the machines (e.g. runs the commands)

Puppet Security

Puppet Security

• Uses SSL certificates as the required and only form of authentication

Puppet Security

• Uses SSL certificates as the required and only form of authentication

• Master authenticates the certificates

Puppet Security

• Uses SSL certificates as the required and only form of authentication

• Master authenticates the certificates

• Test certificate from node:

/usr/sbin/puppetd --waitforcert 60 \ --server master --test --verbose

Puppet Security

• Uses SSL certificates as the required and only form of authentication

• Master authenticates the certificates

• Test certificate from node:

/usr/sbin/puppetd --waitforcert 60 \ --server master --test --verbose

• If certificate doesn’t match, probably cached on server. Try on master:

puppetca --clean node1.compute-1.internal

Puppet Cache

Puppet Cache

• Keeps a cache of the parsed configuration in:

/var/lib/puppet/localconfig.yaml

Troubleshooting Puppet

Troubleshooting Puppet• tail -f /var/log/syslog

Troubleshooting Puppet• tail -f /var/log/syslog

• Can the nodes contact master?

Troubleshooting Puppet• tail -f /var/log/syslog

• Can the nodes contact master?

• Do the certificates match?

Troubleshooting Puppet• tail -f /var/log/syslog

• Can the nodes contact master?

• Do the certificates match?

• Does /var/lib/puppet/localconfig.yaml contain the right information?

Troubleshooting Puppet• tail -f /var/log/syslog

• Can the nodes contact master?

• Do the certificates match?

• Does /var/lib/puppet/localconfig.yaml contain the right information?

• Does /etc/puppet/manifests/classes/poolparty.pp contain the right information?

Troubleshooting Puppet• tail -f /var/log/syslog

• Can the nodes contact master?

• Do the certificates match?

• Does /var/lib/puppet/localconfig.yaml contain the right information?

• Does /etc/puppet/manifests/classes/poolparty.pp contain the right information?

• Does /var/poolparty/poolparty.pp contain the right information? (more on that later)

Troubleshooting Puppet

Troubleshooting Puppet

•server-provision -n app -i 1 \

--slave --verbose --debug

master

PoolParty DNS

root@master:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 master75.101.128.147 master puppet localhost10.252.166.18 node1.app node110.252.167.47 master.app master

on master

PoolParty DNS

root@master:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 master75.101.128.147 master puppet localhost10.252.166.18 node1.app node110.252.167.47 master.app master

on master

root@node1:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 node175.101.128.147 puppet master10.252.167.47 master.app master10.252.166.18 node1.app node1

on node1

PoolParty DNS

• /etc/hosts modified by PoolParty when you cloud provision

HAProxy

HAProxy

HAProxy

• Extremely fast

HAProxy

• Extremely fast

• Reports of HAProxy saturating gigabit fiber (http://is.gd/4myI)

HAProxy

• Extremely fast

• Reports of HAProxy saturating gigabit fiber (http://is.gd/4myI)

• Reputation of being reliable and secure

HAProxy

• Extremely fast

• Reports of HAProxy saturating gigabit fiber (http://is.gd/4myI)

• Reputation of being reliable and secure

• Installed and setup by PoolParty by default

HAProxy

HAProxy

• Config kept in:

/etc/haproxy.cfg

HAProxy

• Config kept in:

/etc/haproxy.cfg

•Reload config by:

/etc/init.d/haproxy reload

HAProxy• View stats by:

http://ec2-your-ip/poolparty

HAProxy• View stats by:

http://ec2-your-ip/poolparty

PoolParty Messenger

PoolParty Messenger

PoolParty Messenger

• Three parts:

PoolParty Messenger

• Three parts:

• Master

PoolParty Messenger

• Three parts:

• Master

• Node

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

• “Instance Glue”

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

• “Instance Glue”

• Master provisions nodes (will change)

PoolParty Messenger

PoolParty Messenger

• Erlang - excellent node-to-node monitoring

PoolParty Messenger

• Erlang - excellent node-to-node monitoring

• Gathers load data

PoolParty Messenger

• Erlang - excellent node-to-node monitoring

• Gathers load data

• Decides when to launch new nodes

PoolParty Messenger

PoolParty Messenger

• Changes happening in architecture

PoolParty Messenger

• Changes happening in architecture

• Will become neighborhood-based instead of master/node-based

PoolParty Messenger

• Changes happening in architecture

• Will become neighborhood-based instead of master/node-based

• The brains of the operation when humans aren’t around

Using PoolParty

Using PoolParty

Using PoolParty

• cloud start

Using PoolParty

• cloud start

• cloud configure --verbose --debug

Using PoolParty

• cloud start

• cloud configure --verbose --debug

• cloud provision --verbose --debug

Using PoolParty: What Happens

Using PoolParty: What Happens

• PoolParty reads clouds.pool

Using PoolParty: What Happens

• PoolParty reads clouds.pool

• Generates templates, puppet configs, etc in a local storage directory (/tmp/poolparty)

Using PoolParty: What Happens

• PoolParty reads clouds.pool

• Generates templates, puppet configs, etc in a local storage directory (/tmp/poolparty)

• Files are rsync’d to /var/poolparty on master

Using PoolParty: What Happens

• PoolParty reads clouds.pool

• Generates templates, puppet configs, etc in a local storage directory (/tmp/poolparty)

• Files are rsync’d to /var/poolparty on master

• A gang of bootstrapping tasks are run (via Capistrano)

Using PoolParty: What Happens

• PoolParty reads clouds.pool

• Generates templates, puppet configs, etc in a local storage directory (/tmp/poolparty)

• Files are rsync’d to /var/poolparty on master

• A gang of bootstrapping tasks are run (via Capistrano)

• Puppet config is cp’d from /var/poolparty to /etc/puppet/manifests/classes/poolparty.pp

Using PoolParty: What Happens

Using PoolParty: What Happens

• Puppet is started

Using PoolParty: What Happens

• Puppet is started

• Puppet tries to provision everything

Using PoolParty: What Happens

• Puppet is started

• Puppet tries to provision everything

• Puppet starts PP Messenger master

Using PoolParty: What Happens

Using PoolParty: What Happens

• PP Messenger master contacts Amazon and boots more instances if needed.

Using PoolParty: What Happens

• PP Messenger master contacts Amazon and boots more instances if needed.

• PP Messenger bootstraps the new node

Using PoolParty: What Happens

• PP Messenger master contacts Amazon and boots more instances if needed.

• PP Messenger bootstraps the new node

• The bootstrap process starts PP Messenger Node

Using PoolParty: What Happens

• PP Messenger master contacts Amazon and boots more instances if needed.

• PP Messenger bootstraps the new node

• The bootstrap process starts PP Messenger Node

• PP Messenger initiates Puppet provisioning on the new node

Using PoolParty: What Happens

• PP Messenger master contacts Amazon and boots more instances if needed.

• PP Messenger bootstraps the new node

• The bootstrap process starts PP Messenger Node

• PP Messenger initiates Puppet provisioning on the new node

• time passes

Using PoolParty: What Happens

Using PoolParty: What Happens

• Puppet runs on master and master discovers a new node has been started.

Using PoolParty: What Happens

• Puppet runs on master and master discovers a new node has been started.

• In the case of HAProxy/Apache, Puppet re-generates the HAProxy config to include the new node.

Using PoolParty: What Happens

• Puppet runs on master and master discovers a new node has been started.

• In the case of HAProxy/Apache, Puppet re-generates the HAProxy config to include the new node.

• As specified in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.

Using PoolParty: What Happens

• Puppet runs on master and master discovers a new node has been started.

• In the case of HAProxy/Apache, Puppet re-generates the HAProxy config to include the new node.

• As specified in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.

• HAProxy now starts distributing the load to the new node

Using PoolParty: What Happens

Using PoolParty: What Happens

• Similar config changes happen throughout the system

Real World Misc.

Real World Misc.

Real World Misc.

• tail -f /var/log/syslog is your friend

Real World Misc.

• tail -f /var/log/syslog is your friend

• ssh into master:

cloud ssh

Real World Misc.

• tail -f /var/log/syslog is your friend

• ssh into master:

cloud ssh

• ssh into i’th node:

cloud ssh -i 1

Real World Misc.

Real World Misc.

•scp scp -i /Users/me/.ec2/id_rsa-clouds_app \

root@ec2-75-101-191-10.compute-1.amazonaws.com:/ \

etc/snmp/snmpd.conf .

Real World Misc.

•scp scp -i /Users/me/.ec2/id_rsa-clouds_app \

root@ec2-75-101-191-10.compute-1.amazonaws.com:/ \

etc/snmp/snmpd.conf .

•rsyncrsync -av -e "ssh -i /var/poolparty/id_rsa-clouds_app" \

/var/www/cpohunterfan.com/documents node1:/var/www/cpohunterfan.com/

Workshop

Agenda:

Agenda:

• Launch, login, terminate an EC2 instance

Agenda:

• Launch, login, terminate an EC2 instance

• cloud ssh into our instances and look around

Agenda:

• Launch, login, terminate an EC2 instance

• cloud ssh into our instances and look around

• Take a closer look at clouds.pool

Agenda:

• Launch, login, terminate an EC2 instance

• cloud ssh into our instances and look around

• Take a closer look at clouds.pool

• Examine a basic PoolParty plugin