Reduce Time to ValueFocus First on Configuration Management Debt

Chris Sterling, Product Owner, CenturyLink Cloud

Bio - Chris Sterling

2

Author of “Managing Software Debt: Building for Inevitable Change” Supported change efforts to adopt Lean, Agile & Continuous Delivery behaviors for organizations of 10 up to 800+ people Product Owner at CenturyLink Cloud Entrepreneur & Lean Startup Practitioner Blog: managingsoftwaredebt.com

Social

3

@csterwa #swdebt

Types of Software Debt Configuration Management Debt: Integration and release management become more risky, complex, and error-prone.

Platform Experience Debt: The availability and alignment of people to business objectives that involve software changes is becoming more limited or cost-prohibitive.

Design Debt: The cost of adding features is increasing toward the point where it is more than the cost of writing from scratch.

Quality Debt: There is a diminishing ability to verify the functional and technical quality of software: the “Break/Fix” mentality.

Technical Debt: These are the activities that a team or team members take shortcuts on now that will impede future development if left as is.

4

Reducing Time To Value

5

Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt Reduce hand-offs and dependencies in Org to reduce Platform Experience Debt Malleable architectures with Pluggable UI, Microservices and APIs to reduce Design Debt Increase efficiency of validation processes to reduce Quality Debt Make software more testable to reduce Technical Debt

Configuration Management

Debt

“If releases are like giving birth, then you must be doing something wrong.” — Robert Benefield, CTO - Evolve Beyond

6

Case Study: Web Property

7

180+ person “Web 2.0” product organization Waterfall SDLC that development uses to deliver in 6 month release cycles Want to use Agile methods to be more responsive to users and keep up with other “Web 2.0” companies Transitioned to Agile methods on 15 teams in 3 months Changed release management strategy, added XP technical practices, and implemented Scrum product development framework for scaled coordination Able to release every week to users within 4 months Used streamlined deployment environment process to validate product changes daily using Continuous Integration and automated promotions

Case Study: Ad Platform

8

700+ person Ad Platform organization

Extended dependencies out to 2000+ people throughout company

Millions of lines of code + Hundreds of unique apps/services

Continuous Integration server involved thousands of jobs & 1M+ builds per year

9-18 month overlapping release cycles

Found opportunity to not branch for each overlapping release

After 2 years can release to 20+ data centers globally every day

4 years later they release to production with no humans for nearly 100% of all their company apps/services

Principle:

Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt

9

Where Does CM Debt Source control practices that impact delivery velocity

Too many variations/versions of the software supported in production

Over-burdened release engineering and operations teams

High coupling with centrally managed architecture element/component

Too many hand-offs between teams in order to release software to users

Poor integration processes across architecture components and scaled team delivery

Code changes feel too risky and takes too long to validate before releasing into production

Poor documentation practices10

Traditional Source Control

11

Traditional Source Control

11

Main Branch

Traditional Source Control

11

Main Branch

Version 1 Branch

Integrate for Version 2

Code Complete

Traditional Source Control

11

Main BranchDebt

Death March

Version 1 Branch

Integrate for Version 2

Code Complete

Traditional Source Control

11

Main BranchDebt

Death March {Debt accrues quickly within stabilization periods

Version 1 Branch

Integrate for Version 2

Code Complete

Worse Source Control

12

Main Branch

Version 1

Version 2 Version 3 Version 4

Worse Source Control

12

Main Branch

Version 1

Version 2 Version 3 Version 4

{We wish branch points were this clean

Flexible Source Control

13

Flexible Source Control

13

Main Branch

Flexible Source Control

13

Main Branch

Version 1

Flexible Source Control

13

Main Branch

Version 1 Version 2

Flexible Source Control

13

Main Branch

Version 1 Version 2{Not Easy! Must have proper infrastructure to do this.

Scaled Continuous Delivery

14

Component Validation

Integrated Component Validation

End-to-End & Load/Stress

Canary Deployments

15

Embrace DevOps Culture

Cultures, for better or worse, are very stable. — Jeff Bezos, CEO Amazon

16

Principle: Teams own software delivery from solution to deployment to operations.

17

Monitor App Performance

18

Assess Anomalies

19

Monitor Valuable Trends

20

Platform Experience Debt

“As in Nature, if an organization is too inflexible or stands still too long it will get eaten.”

- James Burke

Principle: Rather than creating teams to work on projects, let’s find ways to give work to cross-functional teams.

22

Feature Team “Feature Team” structure Uses common Product Backlog Integration is done in parallel Requires high levels of communication across teams to resolve integration issues Forces Product Owners to be more coordinated Sprints should be synchronized Cross team fertilization is a requirement to successfully deliver in parallel

23

Design DebtTechnical features that involve improving

software quality attributes can be prioritized based on the cost of not addressing them.

25

Malleable architectures and design enable nimble business capabilities.

Feature Toggles

Problem: Need to deliver changes to production on master with features that are not sufficient or validated with customers

Introduce feature behind toggle ▶ ON: Show and allow access to feature ▶ OFF: Don’t show or allow access to feature

Should be secondary option to introducing smaller capabilities that lead to sufficient feature

26

Microservices

27

Pluggable UI

API Routing

Service 1 Service 2 Service 3 Service 4

Identity Management

Authorization Authorization Authorization Authorization

Data Data Data Data

Messaging Platform

Application Programming

28

API

Deploy to a Platform

29

Quality Debt“Promises make debt, and debt makes

promises.” — Dutch proverb

The Three Amigos

Quickly get testers, coders, and business on the same page before building based on a requirement

31

* The Three Amigos pattern originally coined by George Dinwiddie http://www.stickyminds.com/s.asp?F=S17232_COL_2

The Three Amigos Pattern

32

As a Shopper I want to receive updates on incredible deals that are located near my home so that I can save money on my purchases

Acceptance Criteria: Save Shopper’s location Ask Shopper if they want to receive localized deals daily Send notification of incredible deals to Shoppers located within 10 miles each morning Allow Shopper to stop receiving localized deals

The Three Amigos Pattern

33

As a Shopper I want to receive updates on incredible deals that are located near my home so that I can save money on my purchases

What areas of the application will this affect? What is the overall design? (UI, API, UX, etc…) What are the details test cases for this user story and it’s acceptance criteria? What about negative test conditions? What about boundary conditions? How might we put existing functionality at risk?

The Three Amigos At minimum include tester, coder & business rep in discussion Should only take 30 minutes to 1 hour for user stories Focus on clarification and design through testable inputs/outputs Extend to Operations to reduce CM debt

34

Acceptance Test-Driven

35

ATDD Case StudyTest Automation Reduces Cost of Change

Manual Regression Testing

Testing was taking 75 person hours during 2 full test runs consisting of: ▶ Comprehensive manual regression testing ▶ Data conversion and validation Cost for testing was $17,000 each iteration

37

Introducing Fit into Testing After 8 iterations team had introduced healthy amount of Fit fixtures and automated tests Reduced 70+ hour test runtime down to 6 hours which now included: ▶ Fit automated regression testing ▶ Data conversion and validation automated with Fit

fixtures Reduced cost of testing each iteration from $17,000 to $7,000

38

The Agile Regression Testing

39

* The Agile Triangle has been modified from Mike Cohn’s original version

The Agile Regression Testing

39

* The Agile Triangle has been modified from Mike Cohn’s original version

Automated Unit Tests Make up largest portion of regression tests and are developed by programmers

The Agile Regression Testing

39

* The Agile Triangle has been modified from Mike Cohn’s original version

Automated Unit Tests Make up largest portion of regression tests and are developed by programmers

Integration TestsAutomated &Exploratory

The Agile Regression Testing

39

* The Agile Triangle has been modified from Mike Cohn’s original version

Automated Unit Tests Make up largest portion of regression tests and are developed by programmers

Integration TestsAutomated &Exploratory

Smoke++ Tests Risk-based UI & API Automated Tests

Technical Debt

“For every [dollar] of competitive advantage gained by cutting quality, it costs $4 to restore it; and software is an organizational asset and decisions to cut quality must be made by executive

management and reflected in the financial statements.” — Ken Schwaber, co-creator of Scrum

Source: http://www.infoq.com/presentations/agile-quality-canary-coalmine

Continuous Integration

41

Early Warning Signs

42

Early Warnings: • Broken Builds • Broken Automated Tests • Broken Custom Thresholds

Quality Dashboard - Sonar

43

Early Warning Dashboard

44

Measuring Early Warning Signs: • Design Debt in Duplication (DRY) • Technical Debt in Code Complexity • Quality Debt in Bug DB (Break/Fix) • Other Custom Thresholds

Reducing Time To Value

45

Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt Reduce hand-offs and dependencies in Org to reduce Platform Experience Debt Malleable architectures with Pluggable UI, Microservices and APIs to reduce Design Debt Increase efficiency of validation processes to reduce Quality Debt Make software more testable to reduce Technical Debt

Q&A