Upload
rainya-mosher
View
301
Download
2
Tags:
Embed Size (px)
Citation preview
Release Engineering Downstream of an OpenStack
ProjectBest Practices for the Path from Idea to Production with
an Open Source Code Base
Rainya Mosher
@rainyamosher | www.rainyamosher.com
People don’t want…a quarter inch drill. They want a quarter inch hole.
Clayton M. ChristensenThe Innovator’s Solution
About Rainya• 10+ Years Web App Devoplment
(ASP & .NET C#)
• 3+ Years OpenStack Operations
• Master of Science in Technology
Commercialization
• Musical Theatre Nerd, Singer, and
Actor
• Twitter: @rainyamosher
• Blog: rainyamosher.com That about sums it up.
ENFP
Release Engineering (RelEg / RE) is the sub-discipline of software engineering concerned
with the compilation, assembly, and delivery of source code into finished products or other
software components.
https://en.wikipedia.org/wiki/Release_engineering
http://docs.openstack.org/infra/manual/developers.html
• Clone the code from github• Branch your change• Make fixes• Submit review• Submit patch revisions until you
get passing tests and a +2• Change is merged
Average Review Wait Times as of 7/20/2015
• Nova – 22 days• Neutron – 9 days• Glance – 26 days
http://stackalytics.com
Pull OpenStack
Master
Consume Third-Party
Distro
Internal Integrations & Configs
More ownership over release
cadence
More in-house resources for packaging
More flexiblity with to
integration of internal
changes into python source
code
Third party guides release
cadence
Fewer in-house packaging
needs
Less flexibility in integrating
internal changes into
python source code
No matter the upstream relationship, you’ll need to account for your
internal configurations and system integrations!
Even understanding the upstream
landscape of the project, there will be
times when someone – usually a product
or engineering manager – will say “I don’t care about the community, just ship
the feature/bug fix/widget.” What can
you do?
PDLC
SDLC
AGILE
Scrum, Kanban, XP, etc.
PDLC is the management of a good or service from inception,
design, engineering, build, sale, maintenance, and disposal
through the integration of people, data, process, and
business systems to make an optimal experience for a
customer (internal or external).
SDLC is a framework of steps and phases providing a model for development teams to follow when creating a piece of software to improve the quality of the software and the overall process.
AGILE SDLCs are a collection of methodologies valuing
flexibility, collaboration, communication, and rapid
iterations in the development process.
Simple Easy
One role / task / concept / dimension
vs Complex
Familiar, part of our skillset, near at handvs Hard
Source Image: Rich Hickey, “Simple Made Easy”, 11/2012
Early on, the easy path gives you speed and a sense of doing.
Over time, easy ignores complexity, slows you down, and leads to painful technical debt.
Being intentional with your PDLC & SDLC approach can help you get from mad baby to successful sand eating baby.
BUILD RELEASE DEPLOY
SYSTEMS
Pipeline
• Invest in your non-production environments
• Engineers cannot deliver stable software to production if the places they have to develop and validate are unstable
Dev
• Prod-like functionality (as much as possible)
• Unit tests at a minimum
Integration
• First place for Dev to integrate code with upstream packages
• Unit tests • Basic smoke tests
Test
• Full smoke tests
• High level integration testing with tightly coupled services
Pre-Prod
• Mirrors Prod• Full integration testing with all services
• Load and performance testing when major changes
Production
• Run baseline tests prior to an upgrade
• Automate the deploy to keep human error out of the way
• Don’t touch it!
Speed: Find failures quickly Accuracy: Find all failures Stability!
BUILD RELEASE DEPLOY
SYSTEMS
Config Management
• Make all system and software changes through a config management system
• Start with immutable servers and services (it’s harder to give up root later)
• Ansible, Puppet, Chef, Salt all have strengths & weaknesses – take the time to determine which is best for your situation!
BUILD RELEASE DEPLOY
SYSTEMS
Packaging
• Successful packaging has been done with RPMs and DEBs on physical hosts and VMs
• Containers are rapidly maturing into a viable approach for the control plane
• Finding an approach everyone can agree to is perhaps the biggest challenge!
• Configs can be bundled with the source code or distributed separately based on frequency of updates and scale of system
BUILD RELEASE DEPLOY
SYSTEMS
Orchestration
• Thoughtful orchestration for deploy-time activities can overcome many shortfalls of software
• Dedicate time and energy to find the best order of operations for introducing changes to services
• Deploy patterns - blue-green, rolling, shotgun – are limited by the architecture of the software being deployed, and that’s ok!
We need to create a culture that reinforces the value of taking risks and learning from
failure and the need for repetition and practice to create mastery.
Gene KimThe Phoenix Project
CI & DevOpsRelease
Management
Engineering Practices
Always code as if the person who ends up maintaining your code will be a violent psychopath who knows where you live. ~ John F. Woods, c. 1991
Until zero-downtime deploys are a reality, CI/CD may as well stand for continual impact and constant downtime. Adopt a release management process to stay sane!
Automation first! This is difficult for many organizations to do. It often means slowing down, taking the long way rather than the easy way, and telling people “not right now.”
References Images
- Gripp & Drouhard. Space Unicorn. (2012). http://youtu.be/17o1OlroNSE
- Miltonious Blog Unicorn of Technical Difficulties, Source Unknown
- Random Internet Meme’s found via Google Search
- Hickey, R. 2012. Simple Made Easy. http://www.slideshare.net/evandrix/simple-made-easy
Quotes
- Christensen, C. M. (2003). The Innovator’s Solution: Creating and Sustaining Successful Growth
- Kim, G., Behre, K., Spafford, G. (2013). The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
References Useful Links
- OpenStack Dev: http://docs.openstack.org/infra/manual/developers.html
- Review Wait Times: http://stackalytics.com/
- OpenStack Releases: https://wiki.openstack.org/wiki/Releases
- Rel Eng Overview: https://en.wikipedia.org/wiki/Release_engineering