Upload
gene-kim
View
3.401
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Presentation with Jez Humble, James Turnbull, John Willis
Citation preview
DevOps: It’s Not Just For WebOps And We Have The Metrics To Prove It
Gene Kim (IT Revolu0on Press) Jez Humble (ThoughtWorks) James Turnbull (Puppet Labs) John Willis (Enstra0us / Dell)
Presenters James Turnbull, VP of Community, Evangelism, and Business
Development • Prominent voice in DevOps community • Author of several technical books including The LogStash
Book
Gene Kim, IT researcher and author • Founder and former CTO of Tripwire for 13 years • Author "The Phoenix Project: A Novel About IT, DevOps, and
Helping Your Business Win” and “Visible Ops Handbook”
Jez Humble, Principal, ThoughtWorks • Co-author of Continuous Delivery • Loud mouth on effective practices for software delivery
The Big Why?
• Desire to understand the health and habits of DevOps community
• Understand the link between behaviors and performance
• Mi0gate the “echo chamber” effect: what is the evidence of improved performance
• Prove that DevOps isn’t just for unicorns… it’s for the horses, too.
Goals For This Presentation
• Show you how to do cross-‐sec0on popula0on studies
• Survey Design • Survey Execu0on • Survey Analysis
• Share our top findings • Get your help for our 2013-‐2014 study
Where Did The High Performers Come From?
Over Ten Years, We Benchmarked 1500+ IT Orgs
Intuitive Analytical
Academic Study
Population Sample Population Sample
Academic Study
Population Sample
What Is Cross-Section Population Study
Population Sample
Academic Study
Population Study
Benchmark
ROI Story Best
Practice
Peer Recommendation
Anecdote
Prior Experience
We Are Here
Vendors Consultants Industry
Analyst
Tradeshows
Industry Press
Blogshere
High Performing IT Organizations
§ High performers maintain a posture of compliance § Fewest number of repeat audit findings § One-‐third amount of audit prepara0on effort
§ High performers find and fix security breaches faster § 5 /mes more likely to detect breaches by automated control § 5 /mes less likely to have breaches result in a loss event
§ When high performers implement changes… § 14 /mes more changes § One-‐half the change failure rate § One-‐quarter the first fix failure rate § 10x faster MTTR for Sev 1 outages
§ When high performers manage IT resources… § One-‐third the amount of unplanned work § 8 /mes more projects and IT services § 6 /mes more applica0ons
Source: IT Process Institute, 2008
Visible Ops: Playbook of High Performers
• The IT Process Institute has been studying high-performing organizations since 1999
• What is common to all the high performers?
• What is different between them and average and low performers?
• How did they become great?
www.ITPI.org
The Story Begins…
Picture credit: Flickr: Floralgal
I Know!
12
Lesson:
You are only as smart as the average
of the top 5 people you hang out with
13
2012 DevOps Survey
• Written by Puppet Labs, Gene Kim, and Jez Humble
• Ran through December of 2012
• Goal: Get a snapshot of the current state of the health
and habits of the DevOps community
How To Do A Cross Section Population Study
• Survey design • Survey execu0on • Survey analysis
The Vision: DevOps Survey Of Practice II
• Performance • Behaviors • Demographics • Psychographics
• Planned target was 30 ques0ons • We asked for a budget of 8 ques0ons
16
Survey Design
In the ideal, we want to be able to say, “the more you do [A, B, C], the more performance improves”
X axis: independent variables DevOps behaviors hypothesized to improve performance
Y axis: dependent variables DevOps performance measures
Y-Axis: What Performance Looks Like
• How ofen do you deploy code? (e.g., daily, weekly, monthly, yearly)
• What is the average lead 0me of a change? (i.e., how long does it take to go from “code commihed” to “code successfully running in produc0on?”)
• What percentage of your changes require rollbacks or hoiixes?
• On average, how long does it take to restore service when something goes wrong? (e.g., a day, a week, a month, a year)
18
X-Axis: What Behaviors Result In Performance
• Are environment and infrastructure changes (i.e., everything except for code) checked into revision control?
• Is there an automated process to deploy environment and infrastructure changes?
• Who performs code deployments? (e.g., Dev, Ops, both)
• Who is on the hook for produc0on support (e.g., Dev, Ops, both)
19
Next Step: Survey Execution
• Once the survey instrument is ready, then you need people to take the survey
• Michelle Carroll and the Puppet Labs marke0ng machine went to work
• Mailing lists • Twiher • Reddit
21
The Results
• 4039 completed survey responses in 30 days!!!
• This is amazing. With a primary research firm, this type of popula0on size usually would cost $100-‐200K!
22
Lesson: Do Not Change Answers Mid-Survey
Uh, Oh… Now what?
Lesson:
Get data analyst on the team BEFORE you do survey design!
25
Where Do You Find Your Own Wally?
• Steal a consultant ;) • R User Groups • University students in search of a thesis
• Economics students a plus (sta0s0cs, applied) • Ex-‐Wall Street investors (not recommended)
Wally Zabaglio, Puppet Labs Data Analyst – Former specialized analytics consultant at The Nielsen
Company; analyst for the US Department of Energy's Office of Environmental Management
– Primary analyst on DevOps survey results
27
Data Cleanup
• Used Google Refine to clean up data and compensate for “Gene screwup”
• Used R to do analysis • Look for signal
• Explore data • Run correla0ons
Key Findings
High Performing DevOps Teams
• They’re more agile • 30x more frequent deployments • 8,000x shorter lead 0me (minutes/hours vs. months/quarters)
• They’re more reliable • 2x the change success rate • 12x faster MTTR
31
Technical Practices Matter
• 89% are using infrastructure version control • 82% are using automated code deployments
Deliberate Practice Matters
Organizations that implemented DevOps practices over 12
months ago were 5x more likely to be high performing than
organizations that weren’t implementing DevOps at all.
The Lost Hypotheses *
o Who performs code deployments?
o Who is on the hook for production support?
Measuring Culture
“I’ll tell you EXACTLY what devops means.
Devops means giving a shit about your job enough to not pass
the buck. Devops means giving a shit about your job enough to
want to learn all the parts and not just your little world.
Developers need to understand infrastructure. Operations
people need to understand code. People need to fucking work
with each other and not just occupy space next to each other.”
John Vincent | @lusis | http://bit.ly/12DkRhf
Measuring Culture
Trust (and Verify)
Continuous Improvement vs Risk Management Theater
“Human Error”
Heroes / "high performers"
The tribe is operationally strong *
70% Opera0ons people
The tribe is operationally strong
This is not a Dev takeover
The tribe is enterprise too!
26% -‐ 500 to 9999 16% -‐ 10K plus
Overcoming barriers to DevOps adoption
Biggest barriers were cultural
• Lack of manager buy-in
• Lack of team buy-in
• Value of DevOps not understood outside my
group
What are organizations looking for in DevOps skills?
• Coding/scripting (84%)
• People skills (60%)
• Process re-engineering skills (56%)
• Experience with specific tools (19%)
Challenges to culture
It's easier to teach tools than teach people skills
Preparing For DevOps Survey 2013-2014
• How do we measure culture? • High management vs. low trust management styles (and link to team size)
• Industry code • Beher define performance variables to enable regression analysis (Likert-‐type scale)
• Business cri0cality: system of engagement vs. system of record
• What else? We want to hear your thoughts!
If you … Then … … want to learn more about the findings • Read the full-length report:
http://puppetlabs.com/devops • Share the infographic:
http://puppetlabs.com/2013-state-of-devops-infographic
• Get these slides: http://slideshare.net/realgenekim/DevOpsSurveyOfPractice
… have ideas on the DevOps Survey Of Practice 2013-2014
• Come talk with us!
• Join our Google+ Community: https://plus.google.com/communities/104363243715545285555
… share your stories of transformation • Go to FlowCon! http://flowcon.org
… get a free copy of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win
• Go to the AppFirst booth at 2:40pm. Gene will be signing books!