Getting Started with PoolParty and EC2

Getting Started with PoolParty & EC2

PoolParty

PoolParty

• PoolParty makes it easy and simple to configure any cloud of computers

PoolParty

• PoolParty makes it easy and simple to configure any cloud of computers

• Written in Ruby and Erlang

PoolParty

PoolParty

• Not Ruby or Rails specific

PoolParty

• Not Ruby or Rails specific

• Not even tied to Amazon EC2

sample config

Amazon

EC2

EC2 S3

EC2 S3

EBS

EC2 S3

EBS AMI

EC2 S3

EBS AMI

Elastic IP

EC2 S3

EBS AMI

Elastic IP Keypairs

EC2 S3

EBS AMI

Elastic IP Keypairs

Security Groups

EC2 S3

EBS AMI

Elastic IP Keypairs

Security GroupsCloudFront

EC2 S3

EBS AMI

Elastic IP Keypairs

Security GroupsCloudFront???

Amazon: EC2

Amazon: EC2

• Platform to launch machine instances

Amazon: EC2


• ec2-* (git-style) commands such as:

Amazon: EC2



• ec2-run-instances

Amazon: EC2




• ec2-terminate-instances

Amazon: EC2





• ec2-allocate-address

Amazon: EC2





• ec2-allocate-address

• etc.

Amazon: EC2 (AMIs)

Amazon: EC2 (AMIs)

• AMI - Amazon Machine Image

Amazon: EC2 (AMIs)


• Conceptually similar to a Xen or VMware image

Amazon: EC2 (AMIs)



• Base to build on

Amazon: EC2 (AMIs)




• Hundreds pre-built

Amazon: EC2 (AMIs)





• Snapshots

Amazon: EC2 (AMIs)





• Snapshots

• PoolParty default is the widely used ami-1cd73375 - ec2ubuntu

Amazon: EC2

Amazon: EC2

• Primarily uses ephemeral storage

Amazon: EC2


• e.g. node crashes = data gone (sortof)

Amazon: EC2



• requires new way to think about servers

Amazon: EC2



• requires new way to think about servers

• forces redundancy and clustering issues to the front

Amazon: S3

Amazon: S3

• Simple Storage Service (S3)

Amazon: S3


• Persistent

Amazon: S3


• Persistent

• Data stored in “buckets”

Amazon: S3


• Persistent

• Data stored in “buckets”

• EC2 does not use S3 *directly*

Amazon: EBS

Amazon: EBS

• Elastic Block Store

Amazon: EBS


• Persistent disks for EC2 instances

Amazon: EBS



• Can store snapshots on S3

Amazon: EBS




• Fast

Amazon: EBS




• Fast

• Mounts to only one instance at a time (not a SAN)

Amazon: Elastic IPs

Amazon: Elastic IPs

• Instance IP addresses are dynamic

Amazon: Elastic IPs


• Elastic IPs are static

Amazon: Elastic IPs


• Elastic IPs are static

• Free to use, but $0.01/hr if unassociated with an instance

EC2 Security Groups

EC2 Security Groups

• Amazon has a strict firewall

EC2 Security Groups


•ec2-authorize

EC2 Security Groups


•ec2-authorize

• default security group nodes have unlimited (network) access to each other

EC2 Security Groups


•ec2-authorize

• default security group nodes have unlimited (network) access to each other

• everything else is turned off

EC2 Security Groups

ec2-add-group web -d "Web server group" ec2-authorize -p 80 webec2-authorize -p 22 web

EC2 Security Groups

ec2-add-group web -d "Web server group" ec2-authorize -p 80 webec2-authorize -p 22 webec2-authorize -P icmp -t -1:-1 web

Amazon: CloudFront

Amazon: CloudFront

• Amazon’s “self service” CDN

Amazon: CloudFront


• Pay-as-you-go

Amazon: CloudFront


• Pay-as-you-go

• Upload to S3

Amazon: CloudFront


• Pay-as-you-go

• Upload to S3

• Content is distributed & cached on Amazon’s machines

PoolParty

PoolParty overview

PoolParty overview

• binary tools

PoolParty overview

• binary tools

• configuration files

PoolParty overview

• binary tools

• configuration files

• erlang-driven messenger

PoolParty: Binary Tools


• pool-* binaries


• pool-* binaries

• server-* binaries


Shell Example

PoolParty Configuration





Shell Examplea look at clouds/plugins/poolparty-mysql-plugin/mysql.rb

how does this actually work?

Puppet

Puppet

Puppet

• Puppet is a system for automating system administration tasks.

Puppet


• Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

Puppet


• Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

• Red Hat - In the process of moving legacy systems onto Puppet.

Puppet

Puppet

• Resources: files, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

Puppet


• Mini-language

Puppet


• Mini-language

• Dependencies

Puppet


• Mini-language

• Dependencies

• If/unless conditions

Puppet


• Mini-language

• Dependencies

• If/unless conditions

• Variables

Puppet

Puppet

• Client/Server architecture

Puppet


• puppetmasterd

Puppet


• puppetmasterd

•puppetd

Puppet: Language Example





Puppet

Puppet

• Puppet base config:

/etc/puppet/puppet.conf

Puppet

• Puppet base config:

/etc/puppet/puppet.conf

• PoolParty’s Puppet config:

/etc/puppet/manifests/classes/poolparty.pp

Puppet Configuration

Shell Examplea look at /etc/puppet/manifests/classes/poolparty.pp

Puppet

• Cron jobs:* 1 * * * . /etc/profile && puppetmasterd --verbose

*/15 * * * * /usr/bin/puppetrunner

Puppet

Puppet

• Builds dependency graph

Puppet

• Builds dependency graph

• Provisions the machines (e.g. runs the commands)

Puppet Security

Puppet Security

• Uses SSL certificates as the required and only form of authentication

Puppet Security


• Master authenticates the certificates

Puppet Security



• Test certificate from node:

/usr/sbin/puppetd --waitforcert 60 \ --server master --test --verbose

Puppet Security



• Test certificate from node:

/usr/sbin/puppetd --waitforcert 60 \ --server master --test --verbose

• If certificate doesn’t match, probably cached on server. Try on master:

puppetca --clean node1.compute-1.internal

Puppet Cache

Puppet Cache

• Keeps a cache of the parsed configuration in:

/var/lib/puppet/localconfig.yaml

Troubleshooting Puppet

Troubleshooting Puppet• tail -f /var/log/syslog


• Can the nodes contact master?



• Do the certificates match?




• Does /var/lib/puppet/localconfig.yaml contain the right information?





• Does /etc/puppet/manifests/classes/poolparty.pp contain the right information?





• Does /etc/puppet/manifests/classes/poolparty.pp contain the right information?

• Does /var/poolparty/poolparty.pp contain the right information? (more on that later)



•server-provision -n app -i 1 \

--slave --verbose --debug

master

PoolParty DNS

root@master:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 master75.101.128.147 master puppet localhost10.252.166.18 node1.app node110.252.167.47 master.app master

on master

PoolParty DNS

root@master:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 master75.101.128.147 master puppet localhost10.252.166.18 node1.app node110.252.167.47 master.app master

on master

root@node1:~# cat /etc/hosts127.0.0.1 localhost.localdomain localhost127.0.0.1 node175.101.128.147 puppet master10.252.167.47 master.app master10.252.166.18 node1.app node1

on node1

PoolParty DNS

• /etc/hosts modified by PoolParty when you cloud provision

HAProxy

HAProxy

HAProxy

• Extremely fast

HAProxy

• Extremely fast

• Reports of HAProxy saturating gigabit fiber (http://is.gd/4myI)

http://is.gd/4myI

http://is.gd/4myI

HAProxy

• Extremely fast


• Reputation of being reliable and secure

http://is.gd/4myI

http://is.gd/4myI

HAProxy

• Extremely fast


• Reputation of being reliable and secure

• Installed and setup by PoolParty by default

http://is.gd/4myI

http://is.gd/4myI

HAProxy

HAProxy

• Config kept in:

/etc/haproxy.cfg

HAProxy

• Config kept in:

/etc/haproxy.cfg

•Reload config by:

/etc/init.d/haproxy reload

HAProxy• View stats by:

http://ec2-your-ip/poolparty

HAProxy• View stats by:

http://ec2-your-ip/poolparty

PoolParty Messenger

PoolParty Messenger

PoolParty Messenger

• Three parts:

PoolParty Messenger

• Three parts:

• Master

PoolParty Messenger

• Three parts:

• Master

• Node

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

• “Instance Glue”

PoolParty Messenger

• Three parts:

• Master

• Node

• Client

• “Instance Glue”

• Master provisions nodes (will change)

PoolParty Messenger

PoolParty Messenger

• Erlang - excellent node-to-node monitoring

PoolParty Messenger


• Gathers load data

PoolParty Messenger


• Gathers load data

• Decides when to launch new nodes

PoolParty Messenger

PoolParty Messenger

• Changes happening in architecture

PoolParty Messenger


• Will become neighborhood-based instead of master/node-based

PoolParty Messenger


• Will become neighborhood-based instead of master/node-based

• The brains of the operation when humans aren’t around

Using PoolParty

Using PoolParty

Using PoolParty

• cloud start

Using PoolParty

• cloud start

• cloud configure --verbose --debug

Using PoolParty

• cloud start

• cloud configure --verbose --debug

• cloud provision --verbose --debug

Using PoolParty: What Happens


• PoolParty reads clouds.pool



• Generates templates, puppet configs, etc in a local storage directory (/tmp/poolparty)




• Files are rsync’d to /var/poolparty on master





• A gang of bootstrapping tasks are run (via Capistrano)





• A gang of bootstrapping tasks are run (via Capistrano)

• Puppet config is cp’d from /var/poolparty to /etc/puppet/manifests/classes/poolparty.pp



• Puppet is started



• Puppet tries to provision everything



• Puppet tries to provision everything

• Puppet starts PP Messenger master



• PP Messenger master contacts Amazon and boots more instances if needed.



• PP Messenger bootstraps the new node




• The bootstrap process starts PP Messenger Node





• PP Messenger initiates Puppet provisioning on the new node





• PP Messenger initiates Puppet provisioning on the new node

• time passes



• Puppet runs on master and master discovers a new node has been started.



• In the case of HAProxy/Apache, Puppet re-generates the HAProxy config to include the new node.




• As specified in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.




• As specified in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.

• HAProxy now starts distributing the load to the new node



• Similar config changes happen throughout the system

Real World Misc.

Real World Misc.

Real World Misc.

• tail -f /var/log/syslog is your friend

Real World Misc.


• ssh into master:

cloud ssh

Real World Misc.


• ssh into master:

cloud ssh

• ssh into i’th node:

cloud ssh -i 1

Real World Misc.

Real World Misc.

•scp scp -i /Users/me/.ec2/id_rsa-clouds_app \

[email protected]:/ \

etc/snmp/snmpd.conf .

Real World Misc.

•scp scp -i /Users/me/.ec2/id_rsa-clouds_app \

[email protected]:/ \

etc/snmp/snmpd.conf .

•rsyncrsync -av -e "ssh -i /var/poolparty/id_rsa-clouds_app" \

/var/www/cpohunterfan.com/documents node1:/var/www/cpohunterfan.com/

Workshop

Agenda:

Agenda:

• Launch, login, terminate an EC2 instance

Agenda:


• cloud ssh into our instances and look around

Agenda:



• Take a closer look at clouds.pool

Agenda:



• Take a closer look at clouds.pool

• Examine a basic PoolParty plugin

Technology

Getting Started with PoolParty and EC2