DevOps IRL
Sarah Zelechoski & Jonathan Frappier
Sarah ZelechoskiLead Site Reliability Engineer
@ReactiveOps
@szelechoski
/in/szelechoski
reactiveops.com
github.com/sairezgithub.com/reactiveops
Jonathan Frappier
Senior Technical Consultant, Education Services @EMCCorp
@jfrappier
/in/jonathanfrappier
#vBrownBag professionalvmware.combit.ly/vbrownbagitunes
#Commitmasgithub.com/jfrappiergithub.com/commitmasbit.ly/Commitmas2
Agenda
● Trust Exercise● This is a story all about how
we...● Discovered DevOps● Enabled business through
automation● Supported new business
requirements in weeks not months (or years)
*Disclaimer
As always…
● Assume when we say how awesome something is, we really mean:
“In most cases…” everything is awesome “if it meets your business requirements”
● If you think we are repeating things, it’s because we are, but we did that on purpose
Trust Exercise...
via 9gag.com
It’s coming...
Actually it’s here!
Actually it’s here!
Actually it’s here!
Actually it’s here!
What is DevOps?
Cultural● what your company look like today?
○ interactions between groups○ silos of expertise○ responsibility/blame○ bottlenecks/waiting○ methods of communications○ different toolsets#notDevOps
developer <> build <> QA <> ops <> support <> marketing <> financeWho are your customers?
developer <> build <> QA <> ops <> support <> marketing <> finance
“DevOps is a cultural and operational model that fosters collaboration to enable high-performance IT to achieve business goals”
Brian Gracely, Wikibon
Cultural● management supported
○ no more silos○ cross-pollination between groups○ understanding motivations○ increase communication○ shared responsibility○ learn to trust#DevOps
Operational
● from feature code to deploy
○ all groups have to work in concert
to get product out the door● pipeline has to function
○ consistent effort■ no bottlenecks■ waiting derails productivity
○ be continuous■ fits and starts■ feedback cycle
#DevOps
Collaboration
● everyone on the same page ○ what you’re working on
■ new feature■ new tests■ new requirements■ new tools■ new campaign
○ how it affects me● focus effort
○ not on task○ on supporting the project
#DevOps
Enable#DevOps
● to achieve business goals○ make your customers happy○ everyone is your customer
■ marketing■ support■ sales
○ help me help you
Big DevOps
ZOMG!
DevOps IRL
“There’s no new problem that someone hasn’t already had and written about it in a book”
Will Smith
● you are not a beautiful and unique snowflake ○ business needs in common with others○ working with the same groups
● you don’t have the resources or time to develop new tools new and techniques○ focus on your own product dev!
● techniques and tools already exist to help○ open source software○ pipeline/workflow examples○ collaboration/communication
methodologies● community
○ real stories of success○ help when you need it
● if it’s sink or swim○ don’t jump into the deep end first○ nothing wrong with swimming
lessons○ use existing tools to help buoy you○ good technique will move you faster
and more efficiently○ use the buddy system○ practice makes perfect
Storytime
Cultural
StorySJ -2
● Devs dev’d● QA QA’d● IT IT’d
StorySJ
● Management support● Team collaboration● Empathy● Blameless post-mortems● Continuous improvement● Go play darts...or pool...or
cards...or make a run
Techniques● Join team meetings/standups
(hopefully standups)● Retrospectives● Shared task boards● Go sit somewhere else● Go play darts...or pool...or
cards...or make a run
Tools● Slack, HipChat, giant group
Lync chat● Social media● Conferences, meetups● Human-ie stuff like talking to
other “humans”
Community
● The Phoenix Project● Mentorship Programs
○ bit.ly/mentorex1○ bit.ly/mentorex2○ vsensei.expert *
● Etsy Blameless Post-Mortems○ bit.ly/ETSYbp-m
● 8 DevOps Secrets○ bit.ly/8devopsec
● Breaking Down Silos Doesn’t Happen Overnight○ bit.ly/breakingsilos
● Unicorns and the language of otherness○ bit.ly/igniteunicorns
Operational
StorySJ -2
● hand-crafted artisan infrastructure○ installs done by who? when? ○ vmware fusion, samba, postgres
● devs/qa○ had local VMs or
workstation/system setup
● “staging” ○ rolled by hand / modified by all
parties
● production was under CM○ layered bash scripts
○ long running instances / never
updated
● deployment
StorySJ
● codify infrastructure○ make repeatable from dev to qa
to staging to production
○ try for no hand-rolled or modified
anything
● config management ○ idempotent ansible
● strict procedure ○ hand rolled gold images○ eventually moved to packer
● build products for everyone○ vagrant boxes○ vm templates
● convincing everyone to participate
Techniques
● source control○ quality control○ infrastructure reviewed/audited○ input from all groups
● leverage automation ○ avoid repeating effort○ eliminate hand rolled resources
● configuration management● immutable infrastructure
○ for everything (even vcenter)○ dev has same box as production
Tools
● source control○ github
● automation○ PowerShell/PowerCLI (vcenter)○ ansible○ packer
● configuration management○ puppet/ansible/chef
● immutable infrastructure○ vagrant, VMware, terraform
vSphere provider○ packer
Community
● About Immutable Infrastructure via Codeship○ https://blog.codeship.
com/immutable-deployments/
● devopsdays○ http://www.devopsdays.org/
● The Ship Show○ http://theshipshow.com
● Hacking Ansible○ http://t.co/IFYdUBImqn
Collaboration
StorySJ -2
● Switch replacement take 1…○ Teams were siloed○ Dev was... something… ¯\_(ツ)_/¯
○ No knowledge of work process
between groups
StorySJ
● Switch replacement take 2 except…○ Management support on culture
○ Dev was Agile(ish) and Ops was
Kanban(ish) (different but
compatible)
○ More knowledge of work process
between groups
● Inserted into the Dev process
“The kind of trust that makes a team great is vulnerability trust...the ability to say I don’t know the answer” - Patrick Lencioni
Techniques● Lots of methodologies
○ Agile, Kanban○ Okay to morph what fits your org
● Human communication…○ Yup...I’m gonna talk about pool
and again
Tools● Trello, Asana● Jira (all things for small
teams), Zendesk● Git, GitHub, any SCM● Slack, HipChat● Whiteboards & Post-Its
Community● bit.ly/usingtrello● bit.ly/brenderasana● https://geekspeak.slack.com● http://community.emccode.
com/ ● #vBrownBag Slack coming
soon
Enable
StorySJ
● everyone a customer● enabled dev with repeatable
dev boxes ○ moved to vagrant○ provided upgrade paths
● enabled build & qa with standardized resources○ eliminated IT waste
○ long standing boxes and archived
versions were unnecessary○ immutable vm guests○ shorter lead time
HELP ME HELP YOU
HELP ME HELP YOU
StorySJ
● enabled external customers○ coherent monitoring and alerting○ less downtime○ quicker releases (hotfixes, etc)○ heartbleed, critical security fixes
● enabled ourselves○ made everything repeatable and
reliable○ free DR○ self documenting
Techniques
● Shared responsibility ○ dev helping with alerts/support
○ ops helping with feature
requirements
● Feedback loop ○ including sales/marketing
● Immutable infrastructure● Continuous Build/Deployment● Performance for Business
Tools
● Ticketing & Tasking○ JIRA, Zendesk, Asana, Trello
● Logging○ loggly, logentries, ELK, papertrail
● Metrics○ statsd, Graphite, Jenkins
● Infrastructure Monitoring○ Splunk, Nagios, Veeam One,
Sensu
● Application Monitoring○ New Relic, App Dynamics
● Alerting & SLA Tracking○ Pager Duty, Pingdom
Community● Foundations of a Successful
DevOps Team○ https://www.pagerduty.
com/blog/devops-foundations/
● The Fallacy of Fast○ https://www.youtube.com/watch?
v=iIT18m6bHmw
Where do you Start?
Culture● Own it● Make a new friend● Lunch ‘n Learn / Beer ‘n Learn● Desk swap / desk move● Get in on meetings● Goal board
Operational
● Infrastructure as code○ CM and SCM of all things○ collaborate in Github (or other)
■ everyone has access + input■ @ people for conversations■ enforce code review
● Stop using the GUI○ start writing your VM infra as
code○ PowerCLI or ansible 2○ orchestrate stand-up/tear down
○ try orchestrating the GUI with
Onyx
● Start small○ pick a task you have to do
manually and automate it
○ if you find yourself hardcoding
something, extract it to a variable
Collaboration
● Pick a starting point○ Agile? Kanban? Whatever…
● Pick a tool, try it for a small project○ Asana, Trello, Jira…○ If it doesn’t work, punt
● Find the process and tools that work for you○ Talk to the community
○ Remember someone has tackled
this problem○ Different != wrong, ask why
● Talk more○ Slack, HipChat, actual talking
Enable
● Pick a project with organization wide visibility
● Give everyone a part to play○ how do they contribute?○ how do they give feedback?○ how do they get help?○ how do they get work done?
● Everyone knows the goal○ agree on well defined success
criteria
○ turns in to SLA or KPI. business
enabled.
● Over-communicate ● Trust, empower, encourage,
reward○
Learning
● #vBrownBag DevOps Series○ http://professionalvmware.
com/vbrownbag-devops-series/
● #Commitmas○ http://bit.ly/Commitmas2
○ https://github.
com/commitmas/30-days-of-
commitmas-2015
● Pluralsight.com● CodeSchool.com● FreeCodeSchool.com● LinuxAcademy.com/devops
Thank You