Upload
isotopp
View
339
Download
1
Tags:
Embed Size (px)
Citation preview
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
8 Rollouts a dayKristian Köhntopp
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Most interesting!
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
This is what we do
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
This is what we do
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
This is what we do
• Most hotels worldwide are not in a chain.
• Many hotels communicate by fax.
• Guests need help.
• Self-Service okay, Personal if needed.
• 110+ countries, 40+ languages.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Problem statement
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Most likely your problem is different
Problem statement
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
• Rapid growth - constantly:
• Last years maximumreached early in the new year
• Year after year
Problem statement
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
• The growth pattern is predictible and stable.
• Due to growth, good practice is constantly obsolete.
• New bottlenecks appear all the time, the larger part of the year we operate in 'blue water'.
Problem statement
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Problem statement
The Mission Statement:
Create the required
organisational
personal
technical
structure to match the expected size in time.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Problem statement
There are no constants. Embrace Change.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
About booking
• 2006:
• Under 40 persons in IT.
• They know the code, because they wrote it. And the CPAN modules it uses.
• Conditional code in production. Collection of statistics.
• No Ops team.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
About booking
• Long startup time for new devs.
• Unstable, slow rollouts.
• Code hard to test, too many variants.
• Code hard to test, changing too fast.
• Drowning in Ops work, too many variants.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
• Validate requirements: Do they actually help improving the biz?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
HIghest Paid Persons OpinionExperiments kill HIPPOs
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
• Data proves:
• HIPPO ideas are 0% better than improvements by random strangers.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
• Data proves:
• HIPPO ideas are 0% better than improvements by random strangers.
• Conclusion:
• The only reliable way to improve is to run many experiments.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
• Data proves:
• HIPPO ideas are 0% better than improvements by random strangers.
• Conclusion:
• The only reliable way to improve is to run many experiments.
• Velocity: Roll out code as fast as possible,
• and prove it with real users.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/
Experiments
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/
Experiments
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/
Experiments
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Experiments
• »Experiment« Framework
• Conditional Code,
• measure impact on conversion, performance, etc.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Velocity
• Focus is on capturing quantitative (business) performance data.
• and a low Time To Rollout.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Proven requirements
ExperimentalImplementation
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Proven requirements
• Successful Experiment:
• Full On, even if it is painful! It is earning money!
• Optimize that code.
• Trailing performance team sees only business-positive code.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Proven requirements
• Better insight than tests: The best simulation of a real user is…
• There is a term for this: Testing In Production
http://www.thetestingplanet.com/2011/11/the-future-of-software-testing-part-one-testing-in-production/http://www.setheliot.com/blog/wp-content/uploads/2012/03/STP2012_503_A_to_Z_Testing_in_Production-v2.0.pdf
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Testing in production
• Experiment framework
• decoupling deployment/activation
• ramped activation
• exposure control
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Testing in production
• Load/Capacity test in production
• determine true capacity, true failure mode
• use a probe to determine latency
• concentrate real user load on few frontends, observe probe latency
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Testing in production
• Telemetry
• modern browsers can report page load times back
• add additional system data
• Data mining
• real time monitors
• offline data mining
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Decision
• Velocity. Scalability. Performance.
• Prove business impact.
• Scale the method. Create more great old ones.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Scale the method
Wait.What method?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Risk compensation is an effect whereby individual people may tend to adjust their behaviour in response to perceived changes in risk. …
Another way of stating this is that individuals will behave less cautiously in situations where they feel "safer" or more protected.
http://en.wikipedia.org/wiki/Risk_compensation
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Education
Visibility
Empowerment
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Education: You should know your trade.
We're painting you blue.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Visibility: Effects of changes should be obvious.
Linear, continuous systems.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Shared Spaces
Empowerment:
Take away all mechanism that makes decisions = Give back responsibility
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Booking Rule #1
If you break it,
will you even notice?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Booking Rule #2
If you break it,
can you fix it?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Craftsmanship
The ability to debug a user story front to back,
from Biz to deployment.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Culture of Failure
A good fail is fast and early.
There is a lesson to take away from a good fail.
Bad fails are extended suffering and pain.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
So what exactly?
• Hire experienced people
• Real-world goal vs. SE academicscode earning money vs. beautiful code
• with a proven track record in OSS
• working in a cooperative environment, self-organizing, self-driven personality
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
So what exactly?
• We are not building on a green lawn in the middle of nowhere
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
So what exactly?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
So what exactly?
• We are not building on a green lawn in the middle of nowhere,
• but we are also very aware of the fact that code has a finite lifetime.
• In fact, more than 90% of our code changes are thrown away.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Components
• Experimentation
• Testing in Production
• Shared Spaces
• Education, Visibility, Empowerment
• A culture of failure
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
So how does this relate to DevOps?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Kill all sysadmins!
• BOFH is a poisonous term
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Kill all sysadmins!
• BOFH is a poisonous term
• Infrastructure Developer
• automate infrastructure tasks
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Kill all sysadmins!
• BOFH is a poisonous term
• Infrastructure Developer
• automate infrastructure tasks
• Inhouse Operations Expert
• support developers before and after failure
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Infrastructure Developer:
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Infrastructure Developer:
• Make sure the same features work at a 10x scale.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Infrastructure Developer:
• Make sure the same features work at a 10x scale.
• Make sure things work with zero manual interactions.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Infrastructure Developer:
• Make sure the same features work at a 10x scale.
• Make sure things work with zero manual interactions.
• Mindset differs from feature dev a lot.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Methods differ
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Methods differ
• Scrum-ish vs. Kanban-ish vs. Firefighting
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Methods differ
• Scrum-ish vs. Kanban-ish vs. Firefighting
• All have prio lists
• Feature dev: release oriented, timeboxed
• Infrastructure dev: mission oriented, # of things in flight
• Translator, Agent: look at top10 things in flames right now, pick any
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Inhouse Operations Expert
• Feature dev works on feature front to back, biz to deploy
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Inhouse Operations Expert
• Feature dev works on feature front to back, biz to deploy
• Infrastructure expert offer local special knowledge, possible solutions and their 'prices'
• Enable dev to make informed decision
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Inhouse Operations Expert
• Feature dev works on feature front to back, biz to deploy
• Infrastructure expert offer local special knowledge, possible solutions and their 'prices'
• Enable dev to make informed decision
• Dev then still may choose to break things
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Example:
• DBAs are not 'owning' their databases
• Dev can get advice on schema design, query help, etc at any timπe
• Dev can also elect to go forward against this advice
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations POV
• Example:
• DBAs are not 'owning' their databases
• Dev can get advice on schema design, query help, etc at any timπe
• Dev can also elect to go forward against this advice
• The experts do not have control
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Lock in
• We are in fact locked in…
• Code changes must be small to debug, so we MUST roll out often.
• "Book": No rollout for x months, took weeks to reintegrate.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Lock in
• Dependence on functional and fast monitoring:
• Eventlogging breaks: We cannot roll out.
• Monitors all over the place.
• Obsessed with data-based reasoning.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Lock in
• Individual changes must be small and rolled out fast:
• We cannot use GIT branches, they are tools to accumulate large diffs.
• Unified codebase everywhere.
• We need to roll out systems even if they do not change.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Lock in
• We are load testing weekly:
• Codebase changes quickly.
• A lot of really bad code is put life in experiments, has capacity impact.
• Load testing?
• In production, playing w/ LB weights, of course.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
It works for us
"Have you broken the site already?""No.""What am I paying you for?" -- CEO to new dev over lunch
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Craftsmanship
• Hire experts: "Fachwissen" They know all possible design choices and the price tags for each of them.
• Teach history: "Sachwissen"The choices we made.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Craftsmanship
• C-Level endorsement.
• Teach business to devs, dev to business people.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Craftsmanship
• No cop out: You can get counselling, but the choice to roll out is yours.
• You wrote it. You are the only expert on that code we have. Your call.
• Avoiding risk compensation.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
End-to-end responsibility
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
End-to-end responsibility
• DevOps is magic:
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
End-to-end responsibility
• DevOps is magic:
If you treat adult expertsas responsible adult experts, they will handle themselves as responsible adult experts.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
No biz vs. dev
• We are doing many small projects.
• We are creating tight feedback loops.
• We are working interdisciplinary. Business people sit with devs, as needed. Operations people sit with devs, as needed.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations as a service
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations as a service
• Primary goal: Velocity.
• Everything else we do as needed.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations as a service
• Primary goal: Velocity.
• Everything else we do as needed.
• We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules. Tough luck. Deal.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Operations as a service
• Primary goal: Velocity.
• Everything else we do as needed.
• We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules. Tough luck. Deal.
• Some people can't. They quit.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Should you do it?
• Should you do it?
• Website. We control the rollout.
• High growth environment.
• 8 months/year in blue water.
• »Slack is good.« Baby fat eaten by growth rapidly anyway.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Hell yes
• Treat your responsible adults as responsible adults, and they will handle themselves as responsible adults.
• Kill HIPPOs with data.
• Measure. Real world data. For that, you need to be live.
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
Live by the rules
If you break it, will you even notice?
If you break it, can you fix it?
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://booking.com/jobs
Join us!
Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices
http://booking.com/jobs
Join us!