Deploying Rails Applications on EC2

With help from Capistrano/Rubber

About Me

• Matthew Conway

• Director of Engineering at SnapMyLife

• email: matt@snapmylife.com

• twitter: mattconway

• github: wr0ngway

About SnapMyLife

• Mobile Photo Sharing Site

• http://snapmylife.com

• Deploys to EC2 with rubber/capistrano

• Approximately 10m pageviews/month

Those that went to RailsConf probably recognize my ploy to garner your affection.For the rest, this was a tip from Joel Spolsky’s keynote at RailsConfClaims that showing a pic of Angelina Jolie improves his presentation approval rating.

Rubyists turn me on!

Those that went to RailsConf probably recognize my ploy to garner your affection.For the rest, this was a tip from Joel Spolsky’s keynote at RailsConfClaims that showing a pic of Angelina Jolie improves his presentation approval rating.

Amazon Elastic Compute Cloud (EC2)• Hardware (virtual) as a service

• Scriptable cloud computing

• Have to assume all instances are transient (this has benefits!)

• Easy ami bundling for custom servers, support for static IPs, security groups

EC2 Problems

• Transient

• Redundancy

• Backups!

• Scripted instance construction for short downtimes while rebuilding

An instance could go away at any point in timeSolve this with redundant instancesTransience is a bug not a feature forces one to treat failures as imminent, which is a good assumption to make for a deployment scenarioWe’ve only had a single instance soft fail since Dec we were notified we probably should kill it as the H/W it was running on was badMention Availability zones lets one create a new instance with some guarantee that its not going to fail at the same time as some other group of instances.

EC2 Problems (cont.)

• DNS Issue

• Use elastic (static) ips for web/mail roles

• DNS round robin with a short TTL

• Monitor web instances from each other, remove from round robin if a failure

Even with redundancy, backups, we have some issuesWe haven’t done this yet, I’ll add it to rubber when I do Basically a monit script that firs off a request to nettica when the server is down

EC2 Problems (cont.)

• Not good for sending email

• SPAM central

• IPs are blacklisted

• Use something like authsmtp.com or smtp.com, or host your mail server elsewhere

Ok, for utility emails (e.g. monit/munin), since recipient list small (i.e. the recipients can whitelist)Ok for emails you don’t mind getting shoved into a spam folder.Mention google for your domain for small email quantities.

Our Deployment Needs

• Configure multiple instances as easily as one

• Easy bootstrapping and extension - didn't want to force need for a custom image at start

Rubber

• Free and open sourced

• http://github.com/wr0ngway/rubber/wikis

• Makes it easy to get up and running on ec2 with a single instance

• Allows one to add in instances to scale as needed on a role by role basis

Start Demo

• Create a simple rails app

• Setup a single instance to host it

• Lifted from quickstart on rubber wiki: http://github.com/wr0ngway/rubber/wikis

Longer when building instance from scratch, but I built from scratch last night and used rubber to bundle so that the builds today would be quicker.

Create your rails app

rm -rf rubbertest; rails rubbertest; cd rubbertest./script/generate scaffold post title:string body:text

vi app/views/posts/index.html.erb:<%= `hostname` %> <br /><%= ObjectSpace.each_object(Mongrel::HttpServer) { |i| @port = i.port }; @port %> <br /><%= `hostname -i` %> <br /><%= require 'open-uri'; open("http://169.254.169.254/latest/meta-data/public-ipv4").read() %> <br />

Install rubber

./script/plugin install git://github.com/wr0ngway/rubber.git

./script/generate vulcanize complete_mysql# slow network so manual copy# cp -R ../rubber vendor/plugins

Configure rubber

cp ~/.ec2-matt/rubber-secret.yml config/rubber/

Create a single instance with all roles

cap rubber:create# use alias: production# use roles: app,memcached,mysql_master,web,web_munin

Demo - create rails app

rails rubbertest; cd rubbertest./script/generate scaffold post title:string body:text

Add to app/views/posts/index.html.erb :<%= `hostname` %> <br /><%= ObjectSpace.each_object(Mongrel::HttpServer) {

|i| @port = i.port};@port %> <br />

<%= `hostname -i` %> <br /><%= require 'open-uri' open("http://169.254.169.254/latest/meta-data/public-ipv4").read() %> <br />

Demo - Install rubber./script/plugin install git://github.com/wr0ngway/rubber.git./script/generate vulcanize complete_mysql

edit config/rubber/rubber.yml:

aws_access_key: ****aws_secret_access_key: ****aws_account: ****ec2_key_name: ec2_key_file:staging_roles: app,memcached,mysql_master,web,web_munin

Demo - create instancecap rubber:create# use alias: production# use roles: app,memcached,mysql_master,web,web_munincap rubber:bootstrapcap deploy:setupcap rubber:monit:stopcap deploy:cold

Or use convenience task to do all the above:cap rubber:create_staging

Demo - view results

• Uses /etc/hosts for demo

• site: http://production.foo.com

• munin: http://production.foo.com:8080

Mention nettica/dyndns - if you set it up, the act of creating an instance also sets up the CNAME in nettica. Nettica preferred due to full SOAP api.LAN screwy? /etc/hosts normally works fine, but something about the LAN setup here causes OS X to ignore my /etc/hosts so using IPs (anyone know why?)

Rubber (cont.)

• My attempt at getting people to share deployment recipes

• Uses rails generators, so you can customize your config as you grow

• ec2 conveniences such as easy ami bundling, support for static IPs, security groups

Talk about templates rubber just copies file out of generator template tree, so adding a new template is easyBy making the templates generators, you can tweak them in place in your project re-running the generator with newer versions of rubber, you can see which files need updating, and show the diffs, etc.EC2 has a lot of features I’ve tried to wrap some of these features to make them easier to use - e.g. static_ips.

Rubber Features

• Capistrano rocks! rubber is a capistrano plugin

• DRY configuration

• Role and/or host based configuration of instances

All cap features available, some cap annoyances fixed e.g. ROLES/HOST runs all tasks for those hosts Vs FILTER runs just the tasks for that serverrubber*.yml files I tried to split out into different files per module. You should look in these files to figure out what variables you can tweak. For more substantial tweaks, you can edit the config files yourself Send me any settings you make declarative that others might find useful (code in conf, var in yml)deploy*.rb - custom tasks per moduleInstances are commodities whats important is how you define what runs in those instances. By setting up roles to contain chunks of your deployment scenario, you have a lot of flexibility when it comes time to run those roles on concrete instances. That is, you can define your role ONCE and use it for many instances.

Rubber Configuration System

• All configuration for your instances lives in your project's source tree and gets applied on deploy

• Configuration system uses ERB templates for config files

• Configuration system handles dynamically adding instances

Configuration is the heart of rubberBased on current host config is getting generated for (Global Settings -> Roles Settings -> Host settings) * Config file -> specific config file

Config templatemunin host config

<% rubber_instances.each do |i| %>[<%= i.full_name %>] address <%= i.internal_ip %> use_node_name yes<% end %>

Part of the munin config file Adds an entry responsible for polling each server in our deployment config. Dynamically expands when we add new servers to the mix.

Configuration (cont.)

• Configuration templates for sharing deployment scenarios (vulcanize generator)

• Start with a single instance and add more up as needed - no configuration edits needed

• Use whatever AMIs you want (so long as they are debian based :)

Rubber Features (cont.)

• Use /etc/hosts or updates dynamic dns entries automatically

• Easily/quickly create standalone staging/dev/loadtest instances that mirror production setup

Anti-Features

• Initial experience is fairly turnkey, but you will need some expertise as you grow

• Tied to Rails, shouldn’t be hard to divorce

• Not (yet) a gem

• Many other TODOs, feel free to contribute

• No dynamic instances (like poolparty) - you create them all statically. Works for us.

Mention TODO file in project

Demo +1 app serverALIAS=app01 ROLES=app,memcached cap rubber:createcap rubber:bootstrapcap deploy:setupcap deploy... check load balancing at http://production.foo.com/postsALIAS=app01 cap rubber:destroy... check load balancing at http://production.foo.com/posts

Add in another app server

ALIAS=app01 ROLES=app,memcached cap rubber:createcap rubber:bootstrap # if paranoid, FILTER=app01cap deploy:setupcap deploy

Hit url, show load balancingremove app01, show load balancing

ALIAS=app01 cap rubber:destroy

Individual Staging Servers

• Incredibly valuable, you know it works before you deploy for real

• Quick and easy

• cp config/environments/production.rb config/environments/matt.rb

• RAILS_ENV=matt cap rubber:create_staging

• Mimics production

Every engineer deploys their branch to their own on demand staging server

cp config/environments/production.rb config/environments/matt.rbRAILS_ENV=matt cap rubber:create_staging

Complex setup

ALIAS=web01 ROLES=web,internal_web cap rubber:createALIAS=app01 ROLES=app cap rubber:createALIAS=app02 ROLES=app cap rubber:createALIAS=db01 ROLES=mysql_master cap rubber:createALIAS=db02 ROLES=mysql_slave cap rubber:createcap rubber:bootstrapcap deploy:setupcap deploy:cold

Note the mysql_slave bootstrap sets up master -> slave replication for youThere is also a mysql_cluster generator template

Mysql cluster setup./script generate mysql_clusterALIAS=data01 ROLES=mysql_data cap rubber:createALIAS=data02 ROLES=mysql_data cap rubber:createALIAS=data03 ROLES=mysql_data cap rubber:createALIAS=sql01 ROLES=mysql_sql cap rubber:createALIAS=sql02 ROLES=mysql_sql cap rubber:createALIAS=mgm01 ROLES=mysql_mgm cap rubber:createcap rubber:bootstrapcap deploy:setupcap deploy:cold

Still to do - sqlrelay on localhost to load balance between sql nodes

Bring it all Down

• cap rubber:destroy_all

• Elapsed cost, a few cents

• No idle hardware

• No obsolete hardware

• We can do it all again without any additional effort

Questions?

• Alternatives listed in FAQ in rubber github wiki