Embed Size (px)
DESCRIPTION
10 practical field-proven tips for building a continuously delivered service, based on Kenshoo's experience with its RTB service - a critical, high throughput, highly available component, serving millions of requests per minute in under 50 milliseconds. From coding practices to test automation, from monitoring tools to feature A/B testing - the entire development chain should be focused around removing blockers and manual steps between your code and your clients, without ever settling for quality. Join to see what makes our clients and developers happy and effective.
Citation preview
Continuous Delivery In Practice
Lessons from Kenshoo’s RTB project
Who, What, Where
Tzach Zohar:● System Architect● [email protected]
Kenshoo: ● Founded 2006● Online Marketing Technology● ~450 employees● 12 World Wide locations
Continuous Delivery: Why bother?
Faster development, delivery and feedback makes our clients and us happier.
The “us” alone is worth it.
Continuous Delivery: Why bother?
Better suited productResponsiveness
Less wasteHigher quality
Background: RTB Project
● ~1.5 years ● ~4 developers● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability● ~5-10 deployments / week
How?
1.The Obvious
● Single branch● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously
Deploying
● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade
2. Four-Layer Test Suite
2. Four-Layer Test Suite
Staging: verify compatibility of new build with other components’ production builds
2. Four-Layer Test Suite
3. Keep Builds Stable
Do not overlook a test that “sometimes fails”, trusting build status is crucial
3. Keep Builds Stable
Be Suspicious of:● Random data tests● Asynchronous tests● Integration tests
4. Master Is Always
Shippable
On every commit? Not Quite:Use GitHub Pull Requests
“Merge” == Build and Deploy
credit: [email protected]
5. Rigorous Code Reviews
● Remember “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!
5. Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter
6. Real-Time Feedback
Detect issues immediately and visually
7. Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when changing cross-node APIs / Data
1.Write: oldRead: both
2.Write: new Read: both
3.Write: new Read: new
deploy deploy
8. Keep Upgrade in Mind (2)
Verify backward compatibility in tests
9. A/B Testing
Apply new features to a limited user-group Measure business results per-group
(Not by branching)
10. Own It
Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am
10. Own It
That’s It.
Appendix A. Partial Tool List
Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHubProvisioning: Puppet
Deployment: Fabric, botoMonitoring: Metrics, Graphite
Appendix B. Are You Ready?
Unit Coverage > 90%?
Good Staging Tests?
Informative Monitors?
Builds Are Kept Green?
No API Breaking Changes?
Rigorous Code Reviews?
Support Has Your Phone Number?
Do You Own it?
Not Ready
No Yes
credit: [email protected]
Thanks. Questions?
