Learn Your Way to AWESOME

Janelle Kleinopenmastery.org @janellekz

Point A Point AWESOME.

Learn Your Way to AWESOME.

Focus of this Talk:

Using Science to Fuel Software Mastery“Data-Driven Software Mastery”

Overview

How can we measure development pain?

How do we create a data-driven feedback loop?

What does this look like in practice?

Overview

How can we measure development pain?

How do we create a data-driven feedback loop?

What does this look like in practice?

Great TeamDisciplined with Best PracticesConstantly Working on Improvements+

Project FAILURE

About 8 years ago…

Technical Debt Mistakes

I thought the problem wasTechnical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakesin code that we didn’t write ourselves.

LowerFamiliarity

MoreMistakes=

There had to be more to the story...

PAIN

This is what I knew...

What made development feel painful?

UnexpectedBehavior

ProblemResolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

What Causes Unexpected

Behavior (likeliness)?

What Makes Troubleshooting

Time-Consuming (impact)?Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm

Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger

Most of the pain was caused by human factors.

What causes PAIN?

What Causes Unexpected

Behavior (likeliness)?

What Makes Troubleshooting

Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger

What causes PAIN?

Most of the pain was caused by human factors.

Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm

What Causes Unexpected

Behavior (likeliness)?

What Makes Troubleshooting

Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger

What causes PAIN?

Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm

Most of the pain was caused by human factors.

What Causes Unexpected

Behavior (likeliness)?

What Makes Troubleshooting

Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger

What causes PAIN?

PAIN is a consequence of how we interact with the code.

Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm

PAIN occurs during the process of understanding and extending the software

PAIN

Not the Code.

Optimize “Idea Flow”

My team spent tons of time working on improvements that didn’t make much

difference.

We had tons of automation, but the automation didn’t catch our bugs.

My team spent tons of time working on improvements that didn’t make much

difference.

We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problemsit’s identifying the right problems to solve.

“What are the specific problemsthat are causing the team’s pain?”

PAIN occurs during the process of understanding and extending the software

PAIN

Not the Code.

Optimize “Idea Flow”

The Rhythm of “Idea Flow”

Write a little code.

Work out the kinks.

Write a little code.

Work out the kinks.

Write a little code.

Work out the kinks.

Write a little code. Work out the kinks.

The Rhythm of “Idea Flow”

Does the behavior match our expectations?

The Rhythm of “Idea Flow”

Write a little code. Work out the kinks.

Adjustment LoopProgress Loop

“Friction” in Idea Flow

Write a little code. Work out the kinks.

Adjustment LoopProgress Loop

Idea Flow Mapping Tools

(Open Source, Supported GA ~August 2016)github.com/ideaflow/tools

“Idea Flow Map”

“Friction”

“What caused the painin this case?”

Categorize the Problems with #HashTags

#ReportingEngine

#Hibernate

#MergeHell

1. Problem A

2. Problem B

3. Problem C

Add up the Pain by Category

What’s the biggest problem to solve?

Overview

How can we measure development pain?

How do we create a data-driven feedback loop?

What does continuous improvement look like?

What should we be trying to optimize for?

The Lean StartupEric Reis

Pioneers in the Art of “Everyday Science”

“The Art of the Scientist” Workshop, Ash Maurya

Ash: “What’s the most important thing that scientists do?”

“Run Experiments!”“Analyze Data!”

Getting Involved with Lean Startup

“The Art of the Scientist” Workshop, Ash Maurya

NO!

??

?

?

?

??

Getting Involved with Lean Startup

“The Art of the Scientist” Workshop, Ash Maurya

Ash: “The most important things scientists do is build models.”

Getting Involved with Lean Startup

Ceiling of Achievement, Ash Maurya

Lots of learning, but not learning the “right things.”

Floor of Improvement?

Lots of improving, but not improving the “right things.”

Modeling(Inductive)

Experimentation(Deductive)

Empirical Data

We can’t ignorethis part.

The Scientific Method

Experimentation = Deductive Reasoning

Modeling = Inductive Reasoning

Modeling(Inductive)

Experimentation(Deductive)

Empirical Data

Creating an explicit model makesour beliefs testable and our discoveries additive.

Science is a Feedback Loop

We can’t ignorethis part.

“The Customer Factory”, Ash Maurya

Optimize the Rate of Happy Customer Flow

Unaware Visitors

Users

RetentionUser Repeats Experience

Currency(not necessarily money)

Customers

Referrals

Customer Value > Customer Cost > Your Costs

Amusement Park

Interested Prospects

“The Art of the Scientist” Lean Startup Workshop, Ash Maurya

Ash: “The goal isn’t learning, the goal is traction.”

We can measure flow throughthe Customer Factory.

How do we know if we’re getting traction?

The GoalEliyahu Goldratt

Focus onLimiting

Constraint

Optimize the Rate of Flow in a Supply Chain

“The Customer Factory”, Ash Maurya

Optimize the Rate of Happy Customer Flow

Identify the Limiting Constraint

The Three Steps of Science

1. Measure the “What”2. Explain the “Why”3. Experiment to Learn “How”

Lean Startup: Science-Driven Refactoring

Target: Optimize Rate of Factory Flow

Pain Signal: “Friction” in Flow

This is “Out of Control”

Lower Variability = Better Control

Process Control in Lean Manufacturing

Quality Target

Upper Control Limit

Lower Control Limit

Symptom

Process Control in Lean Manufacturing

Symptom

Symptom

Cause

Cause

Fix

Fix

Fix

FixCause

Cause

Symptom

Symptom

Symptom

Symptom

Cause

Cause

Fix

Fix

Fix

FixCause

Cause

Symptom

What Why How

Process Control in Lean Manufacturing

Average Friction per Customer

This is “Traction”

Optimize for Customer Happiness

HighFriction

MinFriction

“No”Threshold

CustomerHappiness

Categorize the “No” Type

How do we use science to create a feedback loop in

software development?

Problems Look Avoidable in Hindsight

z

14:230:00

Big haystack of changes will make troubleshooting difficult

“That was a dumb thing to do,I’ll avoid that next time…

Imagine your brain is adecision-making engine

written in code.

Breakpoint

Human Brain = Decision-Making Machine

The ChallengeWe tend to make auto-pilot decisions and do what’s familiar

How do we modify decision habits?

BreakpointStop and Think!

Turn Auto-Pilot Decisions into Conscious Decisions

The Awareness Challenge:

We have to recognize the situationand predict the pain

in order to modify the decision.

Now we can modify the decision code!

Improve Quality of Decisions

Data-Driven Software Mastery

Target - The direction of “better”

Data-Driven Software Mastery

Input - The constraints that limit our short-term choices…

Data-Driven Software Mastery

Output - The pain signal we’re trying to improve

Data-Driven Software Mastery

Focus on the biggest pain…

Data-Driven Software Mastery

1. Visibility - Identify the specific patterns

1.Visibility

Data-Driven Software Mastery

2. Clarity - Understand cause and effect

2.Clarity

Data-Driven Software Mastery

3.Awareness

3. Awareness - Stop and think to adjust habits

Data-Driven Software Mastery

4. Run Experiments to Learn What Works

Data-Driven Software Mastery

Average Pain per Incident

This is Traction

Target Flow

Control Limit

Categorize the Pain Type

“Pain Control” in Software Development

Symptom

Symptom

Symptom

Cause

Cause

Fix

Fix

Fix

FixCause

Cause

Symptom

What Why How

Refactor BrainPain Type

BreakpointStop and Think!

We have to recognize the situationand predict the pain

in order to modify the decision.

The Awareness Challenge:

What —> Why —> How

George LakoffMetaphors We Live By

Jeff HawkinsOn Intelligence

+

The Foundational Theory

Theory of Understanding Theory of Prediction

What —> Why

Observe Pattern

“Do I know this pattern?”

Metaphors are patterns of conceptual “Shapes”

“What is Clojure?”

Lisp

property: Functional Language

Groovy

property: JVM Language

property: Object-Oriented Language

Clojure!

property: JVM Language

property: Functional Language

How Our Brain Creates a New Shape Recipe

Three Different Types of Metaphors

Object Patterns(Thing)

Context/Relationship Patterns(Space)

Process Patterns (Time)

Technical Debt

Debt is a “Thing”

“This looks likea tech debt thing.”

1. Categorize patterns by concept type

“This looks likea painful tech debt

thing.”

2. Categorize variations by intensity.

Pain level = 8

= Little Debt

= Medium Debt

= Big Debt

Beliefs About Developer Pain

PAIN occurs during the process of understanding and extending the software

PAIN

Pain occurs over TIME.

What do you think happens when we make this transition?

Process Patterns (Time)

Object Patterns(Thing)

What Causes Unexpected

Behavior (likeliness)?

What Makes Troubleshooting

Time-Consuming (impact)?

Take a Closer Look at the Patterns

Familiarity MistakesStale Memory MistakesSemantic MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm

Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger

Most of the pain was caused by human factors.

JourneyBeginning End

Think about the Developer’s Experience as a

Problem-Solving Journey

Journey

Constraints

What were the biggest challenges on the journey?

Friction

Journey

What were the biggest challenges on the journey?

Journey Strategy

What were the biggest challenges on the journey?

Journey Uncertainty

What were the biggest challenges on the journey?

???

8:150:00

George’s Painful Experience

Look for Patterns in Experience Reviews

What pain did you experience?What situation caused the pain?

What strategies did you try?

What strategies would you try if you could do it again?

Situation Strategy Consequence

Developer Journeys

Clarity = Understand Why Breakdown the Journeys into Patterns

PastStrategiesSituation Consequence

Prediction Machine

“Gut Feel” arises through pattern reinforcement

Clarity = Understand Why Breakdown the Journeys into Patterns

Symptom

Symptom

Symptom

Cause

Cause

Fix

Fix

Fix

FixCause

Cause

Symptom

What Why How

Refactor BrainPain Type

Why —> How

BreakpointStop and Think!

Turn Auto-Pilot Decisions into Conscious Decisions

The Awareness Challenge:

We have to recognize the situationand predict the pain

in order to modify the decision.

Now we can modify the decision code!

The Checklist ManifestoAtul Gawande

Scaling LeanAsh Maurya

+

Change Habits with Checklist Experiments

The Haystack Principle

Lots of unvalidated changes

Easier to find the needle.

The Haystack Principle

Optimize for small manageable haystacks.

14:230:00

I want to avoid this…

Thinking ChecklistIs my current approach likely to cause a big haystack?

Situation: start of subtask

Let’s Make a Checklist!

“What question could I ask my future self to recognize similar risks in the future?”

“In what situation would I ask the question?”

0:00

Stop and Think:

Is my current approach likely tocause a big haystack?

Predict: Small haystackLow Friction

Checklist Experiments

18:120:00

Stop and Think:

Is my current approach likely tocause a big haystack?

False Prediction

Predict: Small haystackLow Friction

Checklist Experiments

18:120:00

Stop and Think:

Is my current approach likely tocause a big haystack?

False Prediction

High-Risk Situations

1. Unraveling sweater2. Integration-heavy change3. High state variation4. Minimum scope is big

Q: Is my current approach likely tocause a big haystack?

Start of Subtask

Checklist Experiments

Strategy Types (“do”)

1. DependencyAnalysis2. IncrementalIntegrationTest3. DataDrivenTest4. IsolateHardToTestCode

Situation Types (“see”)

1. UnravelingSweater2. HeavyIntegrationLogic3. HighStateVariation4. CoupledExternalDependencies

Haystack Decisions

Adhoc Predictions While You Work

I predict this <strategy> will avoid this <pain>

General rule of thumb: Predict the pain and throw as many patterns as you can in a

sentence

I see a risk of this <situation>, and I predict this <strategy> will help me

avoid it

Symptom

Symptom

Symptom

Cause

Cause

Fix

Fix

Fix

FixCause

Cause

Symptom

What Why How

Refactor BrainPain Type

Improve Quality of Decisions

Data-Driven Software Mastery

If any part of the feedback loop is broken:

Our improvements don’t make much difference.

Average Pain per Incident

This is Traction

Target Flow

Control Limit

Categorize the Pain Type

“Pain Control” in Software Development

Overview

How can we measure development pain?

How do we create a data-driven feedback loop?

What does this look like in practice?

Continuous Improvement Framework

1. Clarify the Strategy

1. Clarify the Strategy

1. Clarify the Strategy

Look for #Patterns.

1. Clarify the Strategy

Janelle’s Painful Experience

2. Record an Idea Flow Map

Make adhoc predictions as you work

3. Reflect on Decisions (Experience Review)

3. Reflect on Decisions (Experience Review)

3. Reflect on Decisions (Experience Review)

Look for #Patterns.

3. Reflect on Decisions (Experience Review)

Continuous Improvement Framework

Focus Meeting: Mastery Circle

Circle Leader

Circle Member

Focus: What pain did we experience?Observe: Ask questions about the factsConclude: What major factors caused pain?Optimize: What strategies to reduce the pain?Learn: What experiments could we run?

ObservationQuestions

Make a F.O.C.O.L Point!

Summarize lessons learned in a #HashTagged Journey.

(focused problem-solving discussion)

Dedicated Capacity for Improvement Work

Long-term feedback loop

LEARN YOUR WAY TO AWESOME.

Free to Join Industry Peer Mentorship Network

openmastery.org

Developer Insight Tools Platform

Idea Flow Mapping Tools

Team Insight Tools

TeamJoeSallyMarkEric

Community Insight Tools

AnonymizedData

(REST)

Shared Taxonomyof Patterns & Principles

(with example data)

Janelle Kleinopenmastery.org @janellekz

Check out openmastery.org for details.

Read my Book.

Think About It.

FREE withMembershipBuy It

How to Measure the PAINin Software Development

Janelle Klein