How to Break the Software Rewrite Cycle

Software Rewrite Cycle How to Break

the

CTO, New Iron Janelle Klein

©2015 New Iron Group

My Career

Maintenance Team My First Project

Rewrite Team The Next Generation

My Career

Maintenance Team My First Project

Rewrite Team The Next Generation

My Career

My 2nd Project Third Generation Rewrite

My Career

My 3rd Project I Kicked Off the Rewrite!!

My Career

My 4th Project 3rd Generation Rewrite

Consulting...

My Career

Does anyone not have this challenge?

We Start with the Best of Intentions

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

Stages of Escalating Project Risk

Product Owner: “We’ve got more important things to do.”

Manager: “Good job everyone! Keep up that great work ethic!”


Manager: “We need to go faster! Let’s hire more developers.”


Developer: “I give up. I don’t care anymore if the project fails.”


The further down the path, the more tempting it is...

With invisible problems and business pressure that doesn’t let up, we just keep repeating the same cycle.

RESET

Software Rewrite Cycle

RESET

“A description of the goal is not a strategy.” -- Richard P. Rumelt

What’s wrong with our current strategy?

Our “Strategy” for Success

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

RESET “A good strategy is a specific and coherent response to—and approach for overcoming—the obstacles to progress.”

-- Richard P. Rumelt

The problem is we don’t have a strategy...

RESET

So what are the biggest obstacles that cause the software rewrite cycle?

What specific approach are we going to take to overcome the obstacles?

RESET

How I turned my project around...

We did all the “right” things.

RESET

How I turned my project around...

We did all the “right” things.

...and brought down production 3x in a row.

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

Mistakes ?

Most of our mistakes were in the most well-written parts of the code.

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes =

There had to be more to the story...

...pain...

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The Amount of PAIN was Driven By...

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

What causes PAIN?

Familiarity Mistakes

Stale Memory Mistakes

Semantic Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Generation

Using Debugger



What causes PAIN?


Stale Memory Mistakes Semantic Mistakes



Copy-Edit Mistakes



False Alarm


Ambiguous Clues

Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup


Using Debugger



What causes PAIN?


Stale Memory Mistakes

Semantic Mistakes



Copy-Edit Mistakes



False Alarm

Once we understood causes, most problems were avoidable.


Ambiguous Clues Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup


Using Debugger

An Avoidable Problem

7:01

Iterative Validation with Unit Tests 7:01 0:00

14:23 0:00

Skipping Tests and Validating at the End

7:01

Iterative Validation with Unit Tests 7:01 0:00

14:23 0:00

Skipping Tests and Validating at the End

Urgency Leads to High-Risk Decisions

If I make no mistakes I save ~2 hours.

If I make several mistakes I lose ~8 hours.

An Avoidable Problem

Software is a Reflection of our Decisions

RESET


Technical Debt is just a Symptom of the Problem


How much work does it take to mop the floor?



How much work does it take to complete a software task?

= Side-Effects from Ignoring the Risk

Trade-off Decisions

Writing Unit Tests Troubleshooting Mistakes Direct Cost Indirect Costs

or

We make high-risk decisions because the indirect costs are hard to quantify.

Likeliness of Event

Potential Impact x =

Risk

Troubleshooting Learning Rework

Side-Effects in Software

Troubleshooting Risk

Learning Risk

Rework Risk


Ignore Tango (risk not obvious)

Cycle of Escalating

Risk

Ignore Risk

Tangos Multiply

Cycle of Escalating

Risk

Ignore Risk

More Mopping Work

Cycle of Escalating

Risk

Ignore Risk

Further Behind on Deliverables

Cycle of Escalating

Risk

Productivity plummets because of the overwhelming side-effects

Ignore Risk

I can’t go any faster!

Case Study 1: Healthy project about 10 months old

Troubleshooting

Progress

Learning

Rework 10-20% friction

Effects of Escalating Risk

Case Study 2: Thrashing project about 18 months old

Troubleshooting

Progress

Learning


0:00 28:15

12:23 0:00


Case Study 3: Post-meltdown project about 12 years old

Troubleshooting

Progress

Learning


7:07 0:00

0:00 19:52


Case Study 1

Case Study 2

Case Study 3

1 day

2 days

1 day

3 days

1 day

3 days

We can’t see these effects by measuring velocity or task lead-time.


What specific approach are we going to take to overcome the obstacles?

Obstacle: Business Pressure “We aren’t given time to work on improvements!”

(nobody understands the problems)

Developers are stuck because they don’t have control.

What are we supposed to do?

Management is stuck because they don’t have visibility.

Organizational Deadlock

1. Don’t ask for Permission 2. State your Goal "I want to make the business case to management for fixing things around here. No more chaos and working on weekends, this needs to stop. But I need data to make the case so I need everyone's help." 3. State the Plan "Here's what I'm thinking. I want to run an experiment to record data for one month on all the time we spend troubleshooting. We can look at the data together and identify our biggest problems, then I’ll write it up and present the case to management to get things fixed.” 4. Enlist the Team “Will you guys help me make this happen?”

Make the Decision to Lead

1. Don’t ask for Permission 2. Make the Goal Clear to Your Team "I want to make the business case to management for fixing things around here. No more chaos and working on weekends, this needs to stop. But I need data to make the case so I need everyone's help." 3. State the Plan "Here's what I'm thinking. I want to run an experiment to record data for one month on all the time we spend troubleshooting. We can look at the data together and identify our biggest problems, then I’ll write it up and present the case to management to get things fixed.” 4. Enlist the Team “Will you guys help me make this happen?”






Share the PAIN!

Over the last month, we’ve spent ~50% of our time troubleshooting

Troubleshooting JavaScript Errors

Share the PAIN!

With 25 developers on the team, rushing to save a couple hours caused over 1000 hours of developer downtime

Blame the System...

The likelihood of mistakes and the potential impact

is difficult to quantify and explain.

The product owner can’t help making

ill-informed decisions.

Blame the System...

“...we keep deferring the problems and taking risks because we get stuck in a cycle of urgency.”

Constant Urgency

Blame the System...

“...now the problems are out of control.”

Take Responsibility

Dedicated resources (1 or 2 developers)

Control of all decisions for improvement work

Make a commitment to show results in 3 months

Ask for a 3-Month Trial

Now the challenge is getting results.

Obstacle: Knowing What to Fix

Idea Flow Learning Framework “Idea Flow” is a metaphor for the

human interaction in software development

Idea Flow Map

Idea Flow Learning Framework

Troubleshooting Learning Rework

Strategy for improving software predictability by optimizing developer experience.

Need a Feedback Loop...

Mentorship Process


1.  Make the Problems Visible •  Understand the Problems •  Improve Decision Habits



Idea Flow Learning Framework Strategy for improving software predictability

by optimizing developer experience.

Visibility Framework Visibility helps us Identify and the Problems

2. Record an Idea Flow Map

Idea Flow Mapping Commands

Troubleshooting

Progress

Learning

Rework


Troubleshooting

Progress

Learning

Rework


Troubleshooting

Progress

Learning

Rework


Troubleshooting

Progress

Learning

Rework


3. Reflect on Decisions Learn to Read the Visual Indicators in Idea Flow Maps

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

Learn to Read the Visual Indicators in Idea Flow Maps

3. Reflect on Decisions

= Solution Strategy

The “Heart” of Software Development (the problem-solving machine)

= Solution Strategy


= Solution Strategy


= Solution Strategy


= Solution Strategy


Depending on where the disruptions are in the process, we see a different effect in Idea Flow.

"How did you evaluate the possible options and choose a strategy?”

"What was wrong with the different strategies you tried?”

"What was the discovery that made you choose a different direction?"

Problems with Evaluating Alternatives

"Were you working with something that you were unfamiliar with?”

"Did you run into code that was noisy, ambiguous, or misleading?”

"What do you think made it difficult to learn?"

Problems with Modeling

"Did your task involve changes to complex code or business rules?”

"Were there a lot of details that you had to keep in your head?”

"What was causing the complexity in the validation cycles?"

Problems with Refining

"What experiments did you run to troubleshoot the problem?”

"How many times did you run the experiment?"

"How long did it take to get through each experiment cycle?"

Problems with the Validation Cycle

"Was there something in the code that made these changes especially mistake-prone?”

"How familiar were you with the language and tools you were working with?”

"Were you tired or distracted when you did the work?"

Problems with Execution

1. Make the Problems Visible

The goal of visibility is to ask the right questions so we can identify problems with our strategy.

Visibility Framework Visibility helps us Identify and the Problems




Brainstorming a List

“Technical Debt” Backlog

Our improvements don’t make much difference

Clarity Framework Understand what’s Causing your Biggest Problems

Understand what’s Causing your Biggest Problems Clarity Framework

1. Choose a Focus

Categorize the problem by type to see the impact

Focus on the problem with the highest friction

1. Choose a Focus

Tags: #problemtag

Categorize the problem by type

1. Choose a Focus

“But what problem categories do we use?”

The Ten Pains of Software Development


Design-Fit Pain - When the new feature doesn’t fit well into the existing design.


Requirements Pain - Bad assumptions about what functionality to build


Collaboration Pain - Challenges collaborating with other developers on the team.


Modeling Pain - When it’s difficult to build a conceptual model of the existing software.


Cognitive Pain - Challenges with complexity and intense thinking


Alarm Pain - Challenges with false alarms and test maintenance


Experiment Pain - Challenges with getting feedback by running experiments


Execution Pain - When changing the code is highly mistake-prone


Have an amplifying effect on other problems

Add up Friction by Pain Type

100 hours

50 hours

Troubleshooting

Learning

Rework

Focused Improvements Visible Feedback



Create a Vocabulary of Patterns

Pain Types

Problem Types

Strategy Types

Symptoms Causes Fixes

2. Breakdown the Patterns

2. Breakdown the Patterns

What Causes Experiment Pain?

100 hours

50 hours

Create a Vocabulary of Patterns

Why did it take an hour to find a typo?

Slow Feedback Loop - Ran experiments by firing up application and sending email

Numerous Feedback Loops - Problem was tricky to trackdown, sent 23 emails

Tags: #ExperimentPain



What Strategies Do We Use?

Iterative Unit Testing

Vertical Slices

Talk to the Familiar

Incremental Integration Test

Intelligent Logging

Isolate Hard-to-Test Code

Tight Control

State Controller

Front Load the Risk



How do we know if we’re making things better?

What does “better” really mean?

“Better” following best practices

“Better” solving the problems

Best Practices (solution-focused)

Decision Principles (problem-focused)

What’s a “Decision Principle”?

Answers two questions:

How do we evaluate our situation?

What are we trying to optimize for?

Decision Principles

“You know that thing that happens when you make too many changes at once and it's really hard to troubleshoot problems? We want to avoid that.”

“If I decide to skip the unit tests, how will that affect my haystack size?”

Haystack Effect

The number of unvalidated changes has a huge impact on the

difficulty of tracking down a problem.

“You know when you run an experiment and it’s really hard to figure out what’s going on? We want to avoid that.”

“How can I design my code sandwich to make this easier to troubleshoot?”

Setup the Inputs

Interpret the Outputs

Code Sandwich

By reducing the thickness of my code sandwich I can reduce diagnostic difficulty.

Decision Principles

We can learn what works by running “Strategy Experiments”

3. Run Strategy Experiments

1.

2. 3.

1. Decide on a Strategy 2. Predict the effects in Idea Flow 3. Explain the results


“I’m going to use Iterative Unit Testing to mitigate the risk of a big haystack."

Prediction: My strategy of #IterativeUnitTesting seemed successful, troubleshooting should take <15m


Prediction: My strategy of #IterativeUnitTesting seemed successful, troubleshooting should take <15m

Explain Why? Situation Pattern: #HeavyIntegrationLogic

“I’m going to use Iterative Unit Testing to mitigate the risk of a big haystack."


2. Understand the Problems

The goal of understanding the problems is to identify the “critical few”

that will make the biggest difference.

Pareto’s Law - (80/20 Rule)




Clarity without Awareness...

“Well that was a stupid thing to do. Clearly, I shouldn’t have created a big haystack because troubleshooting would obviously be difficult.”

We have to be aware in the moment

we make the decision.

Stop and Think Practice asking the right question at the right time

“What does your code sandwich look like for this experiment?”

“What are you planning to observe in order to understand the behavior?”

“What parts of the system are you trying to manipulate in order to see variations in behavior?”

Situation

Developer struggling with an experiment (poke the developer)

Ask Questions

Awareness Framework

Think: What questions could I ask my future self to recognize this problem in the future?

Ask the right question at the right time

3. Improve Decision Habits

The goal of improving decision habits is to make the improvements permanent.

Idea Flow Learning Framework (Software Control = Predictability)

So how do we break the software rewrite cycle?

We Learn.

Thank you!

If you liked the talk, please tweet about #ideaflow!

Free e-book if you sign up by July 15th!

@janellekz [email protected]

Twitter: Email: