Deploying software at Scale

Deploying Software at Deploying Software at ScaleScale

Kris Buytaert

@krisbuytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops

Todays GoalsTodays Goals● A reproducable way to deploy and upgrade A reproducable way to deploy and upgrade

softwaresoftware

● AutomaticallyAutomatically

● FastFast

● ConsistentConsistent

What's this devops thing What's this devops thing anyhow ? anyhow ?

The community of developers whose work you The community of developers whose work you see on the Web, who probably don’t know what see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy ADO or UML or JPA even stand for, deploy betterbetter systems at systems at less costless cost in in less timeless time at at lower risklower risk than we see in the Enterprise. This is true even than we see in the Enterprise. This is true even when you factor in the greater flexibility and when you factor in the greater flexibility and velocity of startups.velocity of startups.

Tim Bray , on his blog January 2010Tim Bray , on his blog January 2010

What's the problem ?What's the problem ?

The Old DaysThe Old Days● ““Put this Code Live, here's a tarball” NOW! Put this Code Live, here's a tarball” NOW!

● What dependencies ?What dependencies ?

● No machines available ?No machines available ?

● What database ?What database ?

● Security ?Security ?

● High Availability ? High Availability ?

● Scalability ?Scalability ?

● My computer can't install this ?My computer can't install this ?

devopsdevops● CultureCulture

● (Lean)(Lean)

● AutomationAutomation

● MeasurementMeasurement

● SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Gene KimGene Kim

devops (<)> continuous delilvery devops (<)> continuous delilvery

NirvanaNirvanaAn “ecosystem” that supports continuous delivery, from An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to infrastructure, data and configuration management to business.business.

Through automation of the build, deployment, and testing Through automation of the build, deployment, and testing process, and improved collaboration between developers, process, and improved collaboration between developers, testers, and operations, delivery teams can get changes testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code matter what the size of a project or the complexity of its code base.base.

Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble

How many times a day ?How many times a day ?● 10 @ Flickr10 @ Flickr

● Deployments used to be painDeployments used to be pain

● Nobody dared to deploy a siteNobody dared to deploy a site

● Practice makes perfectPractice makes perfect

● Knowing you can vs constantly doing it Knowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, QA, " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to support, everyone in the company actually) is to enable the business goals. We strongly feel that enable the business goals. We strongly feel that in order to do that you must have in order to do that you must have the ability to the ability to deploy code quickly and safelydeploy code quickly and safely. Even if the . Even if the business goals are to deploy strongly QA’d code business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all once a month at 3am (it’s not for us, we push all the time), having a reliable and easy the time), having a reliable and easy deployment should be deployment should be non-negotiablenon-negotiable." ."

Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator

http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

How do we get there ?How do we get there ?

OS BaselineOS Baseline● Automated DeploymentsAutomated Deployments

● Reproducable Reproducable

● Kickstart, FAI, Preseeding, Kickstart, FAI, Preseeding,

● JeOSJeOS

Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code

● Development best practicesDevelopment best practices

• Model your infrastructureModel your infrastructure

• Version your cookbooks / manifestsVersion your cookbooks / manifests

• Test your cookbooks/ manifestsTest your cookbooks/ manifests

• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra

● Model your infrastructureModel your infrastructure

● A working service = automated ( Application Code + A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring )

● Think Puppet, Chef, Cfengine, Ansible , ....Think Puppet, Chef, Cfengine, Ansible , ....

Version ControlVersion Control● Git !Git !

● Version Version ALLALL the things: the things:

• Source code Application Source code Application

• Source code Infrastructure Source code Infrastructure

• BuildsBuilds

• TestsTests

• PipelinesPipelines

• ScriptsScripts

• Documentation Documentation

• Monitoring scriptsMonitoring scripts

•

•

•

Continuous IntegrationContinuous Integration

● Builds Builds

● Nightly Builds Nightly Builds

● Builds with tests Builds with tests

● Nightly Builds with tests Nightly Builds with tests

● Frequent integration Frequent integration

● Continuous Integration Continuous Integration

JenkinsJenkins● Open Source Continuous Integration ServerOpen Source Continuous Integration Server

● A zillion plugins (400)A zillion plugins (400)

● Have developers build stable and deployable Have developers build stable and deployable codecode

● Test Infra code Test Infra code

Jenkins Pipeline Jenkins Pipeline

What's in your Pipeline ?What's in your Pipeline ?

A pipelineA pipeline● Checkout codeCheckout code

● SyntaxSyntax

● StyleStyle

● Code CoverageCode Coverage

● TestsTests

● BuildBuild

● More TestsMore Tests

● Package Package

App RequirementsApp Requirements● TestableTestable

● Configuration isolatedConfiguration isolated

● Automated DeploymentsAutomated Deployments

● ““If my computer can't install it , the installer is If my computer can't install it , the installer is borken” borken” Luke Kanies at Fosdem (2007)Luke Kanies at Fosdem (2007)

● Bulk provisioning of data Bulk provisioning of data

● http://www.krisbuytaert.be/blog/how-i-my-javahttp://www.krisbuytaert.be/blog/how-i-my-java

Why ops like to package Why ops like to package ● Packages give you featuresPackages give you features

•Consistency, security, dependenciesConsistency, security, dependencies

● Uniquely identify where files come fromUniquely identify where files come from

•Package or cfg-mgmt Package or cfg-mgmt

● Source repo not always availableSource repo not always available

•Firewall / Cloud etc .. Firewall / Cloud etc ..

● Weird deployment locations , no easy accessWeird deployment locations , no easy access

● Little overhead when you automateLittle overhead when you automate

● CONFIG does not belong in a packageCONFIG does not belong in a package

#packagingsucks#packagingsucks• Really ..Really ..

● Missing upstreamMissing upstream

● Ancient upstreamAncient upstream

● Unneeded dependenciesUnneeded dependencies

● Broken upstreamBroken upstream

● Distro PoliciesDistro Policies

● ““Maximum RPM” ?Maximum RPM” ?

● Packaging is needed Packaging is needed

● Dependencies, tooling, repositoriesDependencies, tooling, repositories

● Anger driven developmentAnger driven development

NotNot all all packagespackages are are equalequal

#packaginlove#packaginlove

fpmfpmfpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit 0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ /usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+ /usr/lib/rpm/brp-strip-static-archive+ /usr/lib/rpm/brp-strip-comment-noteProcessing files: hornetq-2.2.5-1.x86_64Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created /usr/local/hornetq-2.2.5.x86_64.rpm

fpm in actionfpm in action● https://github.com/Inuits/build-gemshttps://github.com/Inuits/build-gems

● Fork, pull Fork, pull

● Jenkins pulls , builds , pushes to repoJenkins pulls , builds , pushes to repo

● (variants for Nagios Plugins / Jenkins Plugins (variants for Nagios Plugins / Jenkins Plugins available) available)


● SyntaxSyntax

● StyleStyle


● TestsTests

● BuildBuild


● Package Package

● Upload to RepoUpload to Repo

Pulp Pulp Pulp is a Python application for managing Pulp is a Python application for managing software repositories and their associated software repositories and their associated content, such as packages, errata, and content, such as packages, errata, and distributions. It can replicate software distributions. It can replicate software repositories from a variety of supported sources, repositories from a variety of supported sources, such as http/https, file system, ISO, and RHN, to such as http/https, file system, ISO, and RHN, to a local on-site repository. It provides mechanisms a local on-site repository. It provides mechanisms for systems to gain access to these repositories, for systems to gain access to these repositories, providing centralized software installation.providing centralized software installation.

Pulp Pulp ● Redhat Community Redhat Community

● Redhat Emerging TechnologyRedhat Emerging Technology

● Part of Katello Part of Katello

Pulp Pulp ● ““manages” its own apache instancemanages” its own apache instance

● Symlinks , no copiesSymlinks , no copies

● QueuesQueues

•Syncing in the background Syncing in the background

•No more screens ;)No more screens ;)

● Actions are not instantlyActions are not instantly

•e.g. Add / sync / deletee.g. Add / sync / delete

● Hello mongodb :(Hello mongodb :(

● v1 vs v2v1 vs v2

● Only use repo functionality , cfgmgmt is in charge of packagesOnly use repo functionality , cfgmgmt is in charge of packages

Repository ManagementRepository Management

Version vs LatestVersion vs Latest● Version your repos ?Version your repos ?

ensure => latestsensure => latests

● Latest your environments ?Latest your environments ?

● Strict versioning in config ?Strict versioning in config ?

ensure => '0.98.4'ensure => '0.98.4'


● SyntaxSyntax

● StyleStyle


● TestsTests

● BuildBuild


● Package Package


● Deploy on TestDeploy on Test

The Marionette CollectiveThe Marionette Collective● Distributed ssh ++Distributed ssh ++

● What version of ssh do I have installed on my What version of ssh do I have installed on my servers ? servers ?

● On what servers is XYZ running ? On what servers is XYZ running ?

● Clean all my ssl certs ? Clean all my ssl certs ?

● Restart apache on all servers with fact X Restart apache on all servers with fact X

mc-packagemc-packagemc-package -W /dev/ status jdk * [ ============================================================> ] 33 / 33

servicesdb01.dev.com version = -absent services.dev.google.com version = jdk-1.6.0_13-fcs drbdtest02.dev.google.com version = -absent services3.dev.google.com version = jdk-1.6.0_20-fcs um.dev.google.com version = jdk-1.5.0_19-fcs devtools03.uat.com version = jdk-1.6.0_29-fcs alexandria02.dev.google.com version = -absent weblink01.dev.com version = -absent wikitest.dev.google.com version = jdk-1.6.0_24-fcs payment.dev.google.com version = jdk-1.5.0_17-fcs tiff2pdf01.dev.com version = -absent devdoos.dev.com version = jdk-1.6.0_30-fcs wiki.dev.google.com version = jdk-1.6.0_24-fcs reporting01.dev.com version = -absent devtools01-dev.uat.com version = jdk-1.6.0_23-fcs devtools02.uat.com version = jdk-1.6.0_29-fcs drbdtest01.dev.google.com version = -absent

---- package agent summary ---- Nodes: 33/33 Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0_29-fcs, 1 * 1.6.0_30-fcs, 1 * 1.6.0_31-fcs, 22 * absent Elapsed Time: 1.73 s

What to Trigger ?What to Trigger ?● Update PackageUpdate Package

• Only updates packageOnly updates package

● Trigger Puppet Run Trigger Puppet Run

• Updates config + packageUpdates config + package


● SyntaxSyntax

● StyleStyle


● TestsTests

● BuildBuild


● Package Package


● Deploy on TestDeploy on Test


● PromotePromote

● Deploy on UATDeploy on UAT


● PromotePromote

● Deploy on ProdDeploy on Prod

Done ?Done ?● Close the feedback loop,Close the feedback loop,

● Send metric on deployment Send metric on deployment

echo "deployed.$package_name 1 `date +echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003 %s`" > /dev/tcp/<%= graphite_host %>/2003

Done ?Done ?

A Software project is not done until your last A Software project is not done until your last enduser is in his grave ! enduser is in his grave !

But remember But remember Everything is a Fscking DNS Problem

No really, Everything is a Fscking DNS ProblemIf it's not a fucking DNS Problem ..

It's an arp problemIf it's not an arp problem...

It's a Full Filesystem ProblemIf your filesystem isn't full

It's a Spanning Tree problemIf it's not a spanning Tree problem...

It's a USB problemIf it's not a USB Problem

It might be an ntp problem If it's not an ntp problemIt's a sharing IRQ Problem

If it's not a sharing IRQ ProblemBut most often .. its a Freaking Dns Problem !

Or someone playing tricks on you

Jan 2006

ContactContactKris Buytaert Kris Buytaert [email protected]@inuits.be

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/bhttp://www.krisbuytaert.be/blog/log/http://www.inuits.be/http://www.inuits.be/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221

mailto:[email protected]

http://www.krisbuytaert.be/blog/

http://www.krisbuytaert.be/blog/

Documents

Deploying software at Scale