Agile Development at GoGrid with Pallet and JClouds

02/01/11

Presented to JClouds Meetup @ Twitter’s HQToni Batchelli, Consultant and Entrepreneur wannabe

[email protected], @tbatchelli, @disclojureFebruary 2, 2011

Complex Infrastructure Made Easy™

Agile Development at GoGrid with Pallet and JClouds

mailto:[email protected]


GoGrid’s Challenges

GoGrid is a cloud provider, building their own software

Developing and testing their cloud management software is complex. They need to create and maintain realistic environments in order to load them with their latest software builds

~10 different servers types~20 different services

The problem?

Current time to setup a development/QA environment is 2 days• Mostly manual work

Building the environments manually is very error prone

A high level of interconnectivity between services

Goal

To be able to create development and testing environments dynamically, in a

customizable way, and on cloud infrastructure

A Cloud in a Cloud

Additional ChallengesEnvironments need to be verified as fully functional before deploying software on them

Must support multiple locations

Must be able to scale dynamically

Must support concurrent development• Different feature branches

Must manage/version non-software components in a lockstep with the software:• Network configuration• System-level configurations• Application-level configurations

02/01/11

tools

Abstraction layer on resource management

If you code your infrastructure management code to the jclouds API, you can port your code to all cloud providers

JVM-based

Provides access to GoGrid’s infrastructure by abstracting the communication GoGrid’s Cloud API

Uses JClouds to create and manage nodes

Provisions nodes with software and configuration

Verifies node and service correct operation

Operates the services in the nodes

Nodes are grouped by groups (formerly known as ‘tags’). All nodes in a group are equivalent

Clojure

Pallet is built on ClojureClojure is a JVM-based lisp with superb integration with Java and concurrency

Can use any existing Java library (Tomcat, jdbc, etc...)

Provides a high level of abstraction, which is what is needed in this case

Growing community: there are 7 books on Clojure either published or in preparation

02/01/11

how?

GoGrid Cloud Infrastructure

GoG

rid

Clo

ud A

PI

ServiceServiceService

ServiceService

Service

Service Service

Service ServiceService


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


jclouds


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


gimmenodes!

jclouds


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


gimmenodes!

jclouds


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service



GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


pallet

pallet


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


do stuff!

pallet

pallet


GoG

rid

Clo

ud A

PI


ServiceService

Service

Service Service


do stuff!

pallet

pallet


GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

jclouds


GoG

rid

Clo

ud A

PI

make mea dev cloud!

jclouds


GoG

rid

Clo

ud A

PI

make mea dev cloud!

jclouds

ServiceService

Service



Service Service


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

install cloudinfrastructure!

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

install cloudinfrastructure!

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

is all workingfine?

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

is all workingfine?

ServiceService

Service



Service Service

pallet

pallet

OK!


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

deploynew build!

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service Service

pallet

pallet

deploynew build!


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service Service

pallet

pallet


GoG

rid

Clo

ud A

PI

deploynew build!


GoG

rid

Clo

ud A

PI

ServiceService

Service



Service ServiceGoGrid Cloud Infrastructure

GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

awesome!

ServiceService

Service




GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

ServiceService

Service




GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

ServiceService

Service




GoG

rid

Clo

ud A

PIjclouds


GoG

rid

Clo

ud A

PI

let’s test:gimme nodes!

ServiceService

Service




GoG

rid

Clo

ud A

PIjclouds


GoG

rid

Clo

ud A

PI

let’s test:gimme nodes!

ServiceService

Service




GoG

rid

Clo

ud A

PIjclouds


GoG

rid

Clo

ud A

PI

ServiceService

Service




GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

jclouds

ServiceService

Service




GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

I’m done,thanks!

jclouds

ServiceService

Service




GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

I’m done,thanks!

jclouds


GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

environmentfor branch A?


GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

branch A



GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

branch A


environmentfor branch B?


GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

branch A



GoG

rid

Clo

ud A

PI

branch B



GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

branch A



GoG

rid

Clo

ud A

PI

branch B


environmentfor QA please?


GoG

rid

Clo

ud A

PI


GoG

rid

Clo

ud A

PI

branch A



GoG

rid

Clo

ud A

PI

branch B



GoG

rid

Clo

ud A

PI

QA

environmentfor QA please?

02/01/11

development

GoGrid’s Development Process

Create a feature branch

Create an environment (cloud infrastructure) for this branch

Verify new environment (end-to-end tests)

Automatically build and deploy the new software every time new code is committed to the branch.

Destroy environment and free resources when done

An Environment

Usually one environment for active development branch

Each environment is specified by:• The development branch it tracks• Service types and node count for each service type• Network configuration• External services/DB provisioning

Development Environments

Each development environment tracks a development feature branch

Each environment has a node with Hudson CI, configured to track the environment’s branch

When new code is pushed to the development branch, the environment’s Hudson server builds the software and updates the environment with it.

02/01/11

more on pallet

Why Pallet?

There are well known solutions for node provisioning in the cloud, why pallet?• It integrates with infrastructure management• Service interconnections are first class entities in

the framework• Scalable programming model• Integrates well with SCM and with other OPS tools• Comprehensive support for all development cycles

✓No central server

✓No agents on nodes

✓Works with standard and custom images

✓First-class support for complex multi-service configurations

✓All node configuration done remotely via generated scripts over SSH

✓DSL for generating scripts, resources

✓Testable

crates

Pallet’s basic configuration unit

A crate is a grouping of functions around a service

Functions usually:• manipulate/create internal representations• result in resources deployed or scripts being run in

the target nodes

proxied

bootstrap

automatedadmin user

tomcat

java tomcat

tomcatdeploy

reverse proxy

haproxy

Custom Crates

Standard Crates

A Node(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase

(crates/tomcat-‐deploy "webapp.war")) :restart-‐tomcat (resource/phase (service/service

"tomcat6":action :restart)))

jclouds section of a node(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase


"tomcat6":action :restart)))

Execution Phases(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase


"tomcat6" :action :restart)))

Cloud Infrastructure

Pallet

Crates

Node definitions

Clo

ud A

PI

webapp webapp webapp

proxy proxy

DB DB


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

jclouds


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy

jclouds


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy

webapp webapp webappjclouds


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

jclouds


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

squid squid


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

squid squid


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

squid squid

tomcat tomcattomcat


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

squid squid

tomcat tomcattomcat


Pallet

Crates

Node definitions

Clo

ud A

PI


proxy proxy

DB DB

proxy proxy


DB DB

squid squid

tomcat tomcattomcat

mysql mysql

How are systems interconnected?

Phase 1• Retrieve and/or create needed nodes• For each node in each group: run phases for crates

anda) Collect configuration infob) Schedule configuration actions to be performed

on node

Phase 2• Generate and execute scripts and resources

02/01/11

how does this look today?

(def branch-1-environment {:branch "http://svn....../branches/branch-1" :compute-service {:provider "gogrid" :identity "<username>" :credential "<password>" :endpoint "http://10...:8080/api/"} :db-name "branch-1-db" :db-properties env-network-properties :pre-bootstrap-fn #'setup-environment-db :bootstrap-tags {com.gogrid.nodes.services/LDAP 1 com.gogrid.nodes.services/Cache 1} :bootstrap-phases [:restart-cache :restart-LDAP] :verify-phases [:verify]})

Environment Definition

http://svn.servepath.com/repos/gogrid/branches/kauai

http://svn.servepath.com/repos/gogrid/branches/kauai

http://10.128.36.56:8080/api/

http://10.128.36.56:8080/api/

user => (start branch-1-environment)

user => (verify branch-1-environment)

user => (destroy branch-1-environment)

Operation

It’s a newcomer in the field.• Documentation needs work• Untested support for some combinations of flavors

and versions of Linux

Steep learning curve, a new language

OMG! It’s A Lisp!

Challenge with Pallet

Some pointers...

http://jclouds.org Adrian Cole et al.

http://palletops.com Hugo Duncan et al.

http://gogrid.com For more info on GoGrid services please contact:

Paul LappasVP Engineering & Co-Founder of [email protected]

GoGrid is hiring devops and SW engineers for their SF office (contact Paul too)

http://jclouds.org

http://jclouds.org

http://palletops.com

http://palletops.com

http://gogrid.com

http://gogrid.com



02/01/11

Toni [email protected]

@tbatchelli@disclojure

Questions?



Technology

Agile Development at GoGrid with Pallet and JClouds