Embed Size (px)
DESCRIPTION
Continuous Delivery from Skunkworks to Continuous Delivery as a Service.
Citation preview
Continuous Delivery
From “Skunkworkz” to “as a Service”
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvementHorrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: [email protected]: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
Agenda
Why wanted to do Continuous Delivery
How Continuous Delivery affected us our Organization and Individuals
How Continuous Delivery has matured within the organization
We had to change!
Web/Mobile
Integrations & Business
Logic
Vehicle Connectivity
Web/Mobile
Integrations & Business
Logic
Integrations & Business
Logic
Vehicle Connectivity
Web/Mobile
Integrations & Business
Logic
Vehicle Connectivity
Customer A Customer B Customer C Customer D
We had to change!
Web/Mobile
Integrations & Business
Logic
Vehicle Connectivity
Customer A Each new delivery started the same way.
1. Create and staff a new team PO, PM, Arch, Dev, Test, ect2. Branch the most similar previous delivery3. Rewrite to fit4. Maintain
We had to change!
Web/Mobile
Integrations & Business
Logic
Customer B We couldn't scale our business
1. Competence was stuck within a delivery2. New teams didn’t benefit from the
competence gained in previous delivery3. Huge adaption costs4. Huge maintenance costs5. All work payed by one customer
Lets do it!
Web/Mobile
Integrations
Web/Mobile
Integrations Integrations
Vehicle Connectivity
Web/Mobile
Integrations
Customer A Customer B Customer C Customer D
Services
Reality check!
Sales & Solution Management OperationsDevelopment
Organization not ready!
One managed service for multiple customers was made impossible by...
... internal accounting required a customer to pay for a runtime environment
... operational fear of change driven by one customer impacting other customers
... conceptual fear on management level that customer data can be kept separated in one service
... development fear that conflicting requirements will break common and other customers functionality
... general fear that dependencies will slow down development
Lets do what we can!
Web/Mobile
Integrations
Customer E Customer D
Services
Vehicle Connectivity
Integrations
Integrations
Web/Mobile Web/Mobile
Integrations
Services
Web/Mobile
Vehicle Connectivity
Integrations
Integrations
We do Scrum!! We are Agile !!???
Pre Planning
Dev Sys Test Reg Test
Pre Planning
Dev Sys Test Reg Test
Scrummerfall with multiple teams and dependencies between teams to delivery to multiple customers is a potential nightmare.
We are not Agile enough to do this.
Sprint 2-4 weeks
Regression testing is the root of all evil !!???
Pre Planning
Dev Sys Test Reg Test
Pre Planning
Dev Sys Test Reg Test
Sprints are long because its too expensive to regression test.
We can only release as often as we can regression test.
Sprint 2-4 weeks
We need Continuous Regression Testing
Pre Planning
Dev
Test Automation
Reg Test
If we regression test each and every little change to our components and services we can ensure that we never break commitments.
Pre Planning
Reg Test Reg Test
Verification
Verification
We need Acceptance Test Driven Development
Pre Planning
Dev
Test Automation
Reg Test
First verification of a feature is the first regression test of that feature and it needs to happen as soon as feature is complete.
Pre Planning
Reg Test Reg Test
Verification
Verification
Organization not ready!
One managed service for multiple customers was made impossible by...
... internal accounting required a customer to pay for a runtime environment
... operational fear of change driven by one customer impacting other customers
... conceptual fear on management level that customer data can be kept separated in one service
... development fear that conflicting requirements will break common and other customers functionality
... general fear that dependencies will slow down development
Lets do it!
Build new delivery platform
Continuous Regression Testing
Deliver to new Customer
How hard can it be?
Super Easy!!!
Acceptance Test Driven Development worked like a charm
Continuous Delivery automation up and running in days
Very high payback on investment
Very easy to evolve features overtime
Super Easy!!!
Why??
First customer only needed subset of the delivery platform
Small team of 5 experienced T shaped individuals with complementing personalities
100% shared vision
Constant focus and priority on Customer Delivery
High confidence from Product Owner
Then we tried to scale it some more
We got more customers and started expanding our delivery platform.
Suddenly we started to fail more and more with our Continuous Delivery.
What went wrong??
We had no clue that we did was cultural and behaviour change
In 2011 no one talked about Continuous Delivery as Cultural and Organizational change.
What went wrong??
We knew that the value of our Test Automation was questionable and coverage was insufficient.
We needed testers. We asked for testers. We got testers.
We lost our Acceptance Test Driven Development.
We became one big automated waterfall.
What went wrong??
We didn't understand that a automated Acceptance Test Driven Development model changes the Test Profession.
When we asked for a Tester we got the same testers as we had recruited for years...
...GUI Clickers
What went wrong??
Testers didn't want to touch code.
Testers didn't want developers to touch tests.
“Developers shouldn't test”“Testers shouldn't have to code”“Automation doesn’t find bugs”
What went wrong??
When we asked for Developers we got responsible Developers but soon they were revolting
“We spend too much time writing test code”“We spend too much time fixing regression”
“We spend too little time coding”
What went wrong??
Our Continuous Delivery infrastructure didn’t scale
Test Servers where a bottleneck
Deploy scripts weren't generic enough
We needed to invest more than the spare time of two developers into our CD Implementation
What went wrong??
We asked for Roles.
We didn't ask for competences.
We didn't ask for vision.
We got tons of “opinion management”.
We also learnt and evolved..
We got support!
We had done enough to prove Continuous Delivery to management.
We had done enough to prove the concept of Shared Services.
We got full management support!
What did we do about it??
Domain Teams with more end to end responsibility
Coaching & Cross Functional Communities
Recruit more on competence less on roles
Domain Team to deliver Continuous Delivery as a Service
What did we do about it??
Without consensus any CD Implementation becomes an overpriced build server.
CD as a ServiceJoint responsibility
Pipe Status
Code Config FunctionTests
LoadTests
Infra Deploy Process Visualize
Domain Team
Delivery Engine Team
So how did it go??
30+ Components/Services delivered to 6 Customer
100 individuals involved
No code freezes, No branch/merge activities
Sustainable development pace
From change to QA fully tested in less than an hour
Still not deploying often enough to production
So how did it go??
I will never again work within an organization that employs manual procedures to ship software.
What about Operations and Runtime??
Out biggest failure is our constant inability to include Operations and Infrastructure in our efforts.
We develop Continuous Delivery without being able to and allowed to adopt our production infrastructure to new
patterns and technologies.
Continuous Delivery cannot be done well without DevOps
Continuous Delivery is Individual and Organizational Change
If you want to change the behaviour of your organization you need to be prepared to change your organization.
Continuous Delivery is Individual and Organizational Change
Start small!
Empower Teams and Individuals
Understand the change to the traditional Roles of your Organization.
IT doesn't solve politics
Is Automation the Holy Grale?
Automated Regression Testing helps to ensure we Continuously Deliver the specified application.
Which only enables us to Continuously Improve our application....
