Upload
rantav
View
2.150
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Continuous Deployment
@rantav@outbrain
How much time does it take your code to meet the users?
How much time does it take your code to meet the users?• More than a year?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
• 1 - 24 Hours?
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
• 1 - 24 Hours?
• < 12 Minutes?
~ $ svn ci -m "Implement the super-sharp image scaler. #deloy:ImageServer #to:ny"
Outbrain enables readers to discover the most interesting, relevant and timely links to stories (paid and organic)
Multi Billion Page Views per month
Note: Outbrain is typically installed on *every* article/blog post on each of these sites, immediately under the content.
Who's in?
WHY
WHY
For the Business
For Fun
WHYWhat was so bad before ?
WHY
What is a startup?
• A startup is always on a quest to find Product-Market fit
• => You have to iterate Fast• ... before running out of money
WHY
• Feedback loop speed is important (see REPL in programming)
WHY
What's used to be the case before:
• Inefficient Waits (wait for QA, wait for other features etc)
• Inefficient Context Switch• Feature Delayed - b/c of other features• Big Changes – Big Problems.
HOW
HOW
HOWContinuous Deployment Themes
• Release is a marketing decision.
• Deployment is an engineering decision!
HOWContinuous Deployment Themes
• Small Changes – Reduce Risks
• Kent Beck: You can spill a bucket but you can't spill a Hose.
HOWContinuous Deployment Themes
• Deploy fast, release often
• Fast turnarounds lead to happy customers
• ... and happy developers
HOWCulture
• Everyone have to care about everything!o buildo testso qualityo productiono monitoringo business
HOWCulture
• No Broken Windows!
Broken Windows Theory
HOWCulture
“What's the worse that could happen?”
WORKFLOW
WORKFLOWDev
WORKFLOWDev
The
INGREDIENTS
INGREDIENTSTrunk Stable
•Everyone run tests before commit•No branches, really•Trunk may get released any
moment. If you commit now, users will see it really soon.
=> Test your code, really good•Use feature flags (but only if you
must)• Forward and backward compatibility
INGREDIENTSAutomated Tests
•We have about 2000 test cases•They run in < 4 minutes•We use TeamCity•We regularly test production
Desired state: No manual QA whatsoever.
INGREDIENTSInfrastructure Automation
"Infrastructure as code"
•Using kickstart script to install OS and Chef agent.
•All infrastructure is deployed by Chef.
•All apps are deployed by Glu•All scripts, and chef cookbooks, glu
configs are SVNed.•Very easy to deploy large number of
machines.
INGREDIENTSDeployer
INGREDIENTSServicization
• At outbrain we have ~ 25 services• Each service is deployed at > 1 server• If there's damage, it's contained• It's easy to make small changes• If there's an error it's easy to find it• If rollback is needed it's easier• Everything is either proxies (HAProxy) or queued (AMQ)
• Challenge: It's sometimes hard to find the right balance:o number of services that need maintananceo complexityo performanceo api conformance
INGREDIENTSThe Immune System
• Codingo Reviewo Testing
• Monitoringo We use Nagios for the vitalso Regularly check servers via instrumentationo KeyNoteo Test Production (selenium,APT)
• Instrumentationo Self-Testo Performance
• Monitoring KPIso PVs, o 3 different CTRs,o clicks, revenue, RPMo etc...
INGREDIENTSThe Immune System
• The line of defense will always be broken
=> Multiple lines of defense
INGREDIENTSVisibilty
• Everyone monitor the services • When there's a deployment, you see it on the graph
INGREDIENTSVisibilty
• Yammer
INGREDIENTSVisibilty
• svn changelog• glu changelog• glu audit log
INGREDIENTSLearn Fast, Adapt Fast
• 5 Ys
ARSENALOutbrain's arsenal
Architecture:
• Java in most parts, Spring for the most and Struts2• SOA (dah...) with REST• Message queues (ActiveMQ)• Service multiplicity with HAProxy• Data stored in:
o MySQL, Cassandra, HDFS, NFS• Caching: Memcached
ARSENALOutbrain's arsenal
Testing:
• JUnit• TeamCity• Selenium• Staging environment (for high risk deployments)
ARSENALOutbrain's arsenal
Deployment:
• Kickstart• Chef• Glu• TeamCity• In-house Deployment Manager• RPM + YUM
ARSENALOutbrain's arsenal
Monitoring:
• Nagios• "Testing production" every 10 minutes. • Keynote
ARSENALOutbrain's arsenal
Communications:
• Yammer
Fun Numbers
• 5-50 production changes a day!!!• More then 2000 code tests running in
less then 4 minutes.• More then 600 production services tests
runs every 10 minutes.• It takes ~30 minutes from code
complete to ~100 machines deployed.
References
• Why Continuous Deployment / Eric Ries• Continuous Deployment at outbrain / Ran Tavory • Deployment Infrastructure for Continuous Deployment /
WealthFront• Continuous Deployment presentation / Eishay Smith• Quantum of Deployment / Etsy• Chrome Release Cycle / Anthony Laforge