Janelle Kleinopenmastery.org @janellekz
Point A Point AWESOME.
Learn Your Way to AWESOME.
Focus of this Talk:
Using Science to Fuel Software Mastery“Data-Driven Software Mastery”
Overview
How can we measure development pain?
How do we create a data-driven feedback loop?
What does this look like in practice?
Overview
How can we measure development pain?
How do we create a data-driven feedback loop?
What does this look like in practice?
Great TeamDisciplined with Best PracticesConstantly Working on Improvements+
Project FAILURE
About 8 years ago…
Technical Debt Mistakes
I thought the problem wasTechnical Debt
?Most of our mistakes were in the
most well-written parts of the code.
Mistakes
We made significantly more mistakesin code that we didn’t write ourselves.
LowerFamiliarity
MoreMistakes=
There had to be more to the story...
PAIN
This is what I knew...
What made development feel painful?
UnexpectedBehavior
ProblemResolved
Tracking Painful Interaction with the Code (Friction)
Troubleshooting
Progress
5 hours and 18 minutes of troubleshooting...
PAINFUL
What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm
Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger
Most of the pain was caused by human factors.
What causes PAIN?
What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger
What causes PAIN?
Most of the pain was caused by human factors.
Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm
What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger
What causes PAIN?
Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm
Most of the pain was caused by human factors.
What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger
What causes PAIN?
PAIN is a consequence of how we interact with the code.
Semantic MistakesStale Memory MistakesAssociation MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm
PAIN occurs during the process of understanding and extending the software
PAIN
Not the Code.
Optimize “Idea Flow”
My team spent tons of time working on improvements that didn’t make much
difference.
We had tons of automation, but the automation didn’t catch our bugs.
My team spent tons of time working on improvements that didn’t make much
difference.
We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.
The hard part isn’t solving the problemsit’s identifying the right problems to solve.
“What are the specific problemsthat are causing the team’s pain?”
PAIN occurs during the process of understanding and extending the software
PAIN
Not the Code.
Optimize “Idea Flow”
The Rhythm of “Idea Flow”
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.
Write a little code. Work out the kinks.
The Rhythm of “Idea Flow”
Does the behavior match our expectations?
The Rhythm of “Idea Flow”
Write a little code. Work out the kinks.
Adjustment LoopProgress Loop
“Friction” in Idea Flow
Write a little code. Work out the kinks.
Adjustment LoopProgress Loop
Idea Flow Mapping Tools
(Open Source, Supported GA ~August 2016)github.com/ideaflow/tools
“Idea Flow Map”
“Friction”
“What caused the painin this case?”
Categorize the Problems with #HashTags
#ReportingEngine
#Hibernate
#MergeHell
1. Problem A
2. Problem B
3. Problem C
Add up the Pain by Category
What’s the biggest problem to solve?
Overview
How can we measure development pain?
How do we create a data-driven feedback loop?
What does continuous improvement look like?
What should we be trying to optimize for?
The Lean StartupEric Reis
Pioneers in the Art of “Everyday Science”
“The Art of the Scientist” Workshop, Ash Maurya
Ash: “What’s the most important thing that scientists do?”
“Run Experiments!”“Analyze Data!”
Getting Involved with Lean Startup
“The Art of the Scientist” Workshop, Ash Maurya
NO!
??
?
?
?
??
Getting Involved with Lean Startup
“The Art of the Scientist” Workshop, Ash Maurya
Ash: “The most important things scientists do is build models.”
Getting Involved with Lean Startup
Ceiling of Achievement, Ash Maurya
Lots of learning, but not learning the “right things.”
Floor of Improvement?
Lots of improving, but not improving the “right things.”
Modeling(Inductive)
Experimentation(Deductive)
Empirical Data
We can’t ignorethis part.
The Scientific Method
Experimentation = Deductive Reasoning
Modeling = Inductive Reasoning
Modeling(Inductive)
Experimentation(Deductive)
Empirical Data
Creating an explicit model makesour beliefs testable and our discoveries additive.
Science is a Feedback Loop
We can’t ignorethis part.
“The Customer Factory”, Ash Maurya
Optimize the Rate of Happy Customer Flow
Unaware Visitors
Users
RetentionUser Repeats Experience
Currency(not necessarily money)
Customers
Referrals
Customer Value > Customer Cost > Your Costs
Amusement Park
Interested Prospects
“The Art of the Scientist” Lean Startup Workshop, Ash Maurya
Ash: “The goal isn’t learning, the goal is traction.”
We can measure flow throughthe Customer Factory.
How do we know if we’re getting traction?
The GoalEliyahu Goldratt
Focus onLimiting
Constraint
Optimize the Rate of Flow in a Supply Chain
“The Customer Factory”, Ash Maurya
Optimize the Rate of Happy Customer Flow
Identify the Limiting Constraint
The Three Steps of Science
1. Measure the “What”2. Explain the “Why”3. Experiment to Learn “How”
Lean Startup: Science-Driven Refactoring
Target: Optimize Rate of Factory Flow
Pain Signal: “Friction” in Flow
This is “Out of Control”
Lower Variability = Better Control
Process Control in Lean Manufacturing
Quality Target
Upper Control Limit
Lower Control Limit
Symptom
Process Control in Lean Manufacturing
Symptom
Symptom
Cause
Cause
Fix
Fix
Fix
FixCause
Cause
Symptom
Symptom
Symptom
Symptom
Cause
Cause
Fix
Fix
Fix
FixCause
Cause
Symptom
What Why How
Process Control in Lean Manufacturing
Average Friction per Customer
This is “Traction”
Optimize for Customer Happiness
HighFriction
MinFriction
“No”Threshold
CustomerHappiness
Categorize the “No” Type
How do we use science to create a feedback loop in
software development?
Problems Look Avoidable in Hindsight
z
14:230:00
Big haystack of changes will make troubleshooting difficult
“That was a dumb thing to do,I’ll avoid that next time…
Imagine your brain is adecision-making engine
written in code.
Breakpoint
Human Brain = Decision-Making Machine
The ChallengeWe tend to make auto-pilot decisions and do what’s familiar
How do we modify decision habits?
BreakpointStop and Think!
Turn Auto-Pilot Decisions into Conscious Decisions
The Awareness Challenge:
We have to recognize the situationand predict the pain
in order to modify the decision.
Now we can modify the decision code!
Improve Quality of Decisions
Data-Driven Software Mastery
Target - The direction of “better”
Data-Driven Software Mastery
Input - The constraints that limit our short-term choices…
Data-Driven Software Mastery
Output - The pain signal we’re trying to improve
Data-Driven Software Mastery
Focus on the biggest pain…
Data-Driven Software Mastery
1. Visibility - Identify the specific patterns
1.Visibility
Data-Driven Software Mastery
2. Clarity - Understand cause and effect
2.Clarity
Data-Driven Software Mastery
3.Awareness
3. Awareness - Stop and think to adjust habits
Data-Driven Software Mastery
4. Run Experiments to Learn What Works
Data-Driven Software Mastery
Average Pain per Incident
This is Traction
Target Flow
Control Limit
Categorize the Pain Type
“Pain Control” in Software Development
Symptom
Symptom
Symptom
Cause
Cause
Fix
Fix
Fix
FixCause
Cause
Symptom
What Why How
Refactor BrainPain Type
BreakpointStop and Think!
We have to recognize the situationand predict the pain
in order to modify the decision.
The Awareness Challenge:
What —> Why —> How
George LakoffMetaphors We Live By
Jeff HawkinsOn Intelligence
+
The Foundational Theory
Theory of Understanding Theory of Prediction
What —> Why
Observe Pattern
“Do I know this pattern?”
Metaphors are patterns of conceptual “Shapes”
“What is Clojure?”
Lisp
property: Functional Language
Groovy
property: JVM Language
property: Object-Oriented Language
Clojure!
property: JVM Language
property: Functional Language
How Our Brain Creates a New Shape Recipe
Three Different Types of Metaphors
Object Patterns(Thing)
Context/Relationship Patterns(Space)
Process Patterns (Time)
Technical Debt
Debt is a “Thing”
“This looks likea tech debt thing.”
1. Categorize patterns by concept type
“This looks likea painful tech debt
thing.”
2. Categorize variations by intensity.
Pain level = 8
= Little Debt
= Medium Debt
= Big Debt
Beliefs About Developer Pain
PAIN occurs during the process of understanding and extending the software
PAIN
Pain occurs over TIME.
What do you think happens when we make this transition?
Process Patterns (Time)
Object Patterns(Thing)
What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Take a Closer Look at the Patterns
Familiarity MistakesStale Memory MistakesSemantic MistakesBad Input AssumptionTedious Change MistakesCopy-Edit MistakesTransposition MistakesFailed Refactor MistakesFalse Alarm
Non-Deterministic BehaviorAmbiguous CluesLots of Code ChangesNoisy OutputCryptic OutputLong Execution TimeEnvironment CleanupTest Data CreationUsing Debugger
Most of the pain was caused by human factors.
JourneyBeginning End
Think about the Developer’s Experience as a
Problem-Solving Journey
Journey
Constraints
What were the biggest challenges on the journey?
Friction
Journey
What were the biggest challenges on the journey?
Journey Strategy
What were the biggest challenges on the journey?
Journey Uncertainty
What were the biggest challenges on the journey?
???
8:150:00
George’s Painful Experience
Look for Patterns in Experience Reviews
What pain did you experience?What situation caused the pain?
What strategies did you try?
What strategies would you try if you could do it again?
Situation Strategy Consequence
Developer Journeys
Clarity = Understand Why Breakdown the Journeys into Patterns
PastStrategiesSituation Consequence
Prediction Machine
“Gut Feel” arises through pattern reinforcement
Clarity = Understand Why Breakdown the Journeys into Patterns
Symptom
Symptom
Symptom
Cause
Cause
Fix
Fix
Fix
FixCause
Cause
Symptom
What Why How
Refactor BrainPain Type
Why —> How
BreakpointStop and Think!
Turn Auto-Pilot Decisions into Conscious Decisions
The Awareness Challenge:
We have to recognize the situationand predict the pain
in order to modify the decision.
Now we can modify the decision code!
The Checklist ManifestoAtul Gawande
Scaling LeanAsh Maurya
+
Change Habits with Checklist Experiments
The Haystack Principle
Lots of unvalidated changes
Easier to find the needle.
The Haystack Principle
Optimize for small manageable haystacks.
14:230:00
I want to avoid this…
Thinking ChecklistIs my current approach likely to cause a big haystack?
Situation: start of subtask
Let’s Make a Checklist!
“What question could I ask my future self to recognize similar risks in the future?”
“In what situation would I ask the question?”
0:00
Stop and Think:
Is my current approach likely tocause a big haystack?
Predict: Small haystackLow Friction
Checklist Experiments
18:120:00
Stop and Think:
Is my current approach likely tocause a big haystack?
False Prediction
Predict: Small haystackLow Friction
Checklist Experiments
18:120:00
Stop and Think:
Is my current approach likely tocause a big haystack?
False Prediction
High-Risk Situations
1. Unraveling sweater2. Integration-heavy change3. High state variation4. Minimum scope is big
Q: Is my current approach likely tocause a big haystack?
Start of Subtask
Checklist Experiments
Strategy Types (“do”)
1. DependencyAnalysis2. IncrementalIntegrationTest3. DataDrivenTest4. IsolateHardToTestCode
Situation Types (“see”)
1. UnravelingSweater2. HeavyIntegrationLogic3. HighStateVariation4. CoupledExternalDependencies
Haystack Decisions
Adhoc Predictions While You Work
I predict this <strategy> will avoid this <pain>
General rule of thumb: Predict the pain and throw as many patterns as you can in a
sentence
I see a risk of this <situation>, and I predict this <strategy> will help me
avoid it
Symptom
Symptom
Symptom
Cause
Cause
Fix
Fix
Fix
FixCause
Cause
Symptom
What Why How
Refactor BrainPain Type
Improve Quality of Decisions
Data-Driven Software Mastery
If any part of the feedback loop is broken:
Our improvements don’t make much difference.
Average Pain per Incident
This is Traction
Target Flow
Control Limit
Categorize the Pain Type
“Pain Control” in Software Development
Overview
How can we measure development pain?
How do we create a data-driven feedback loop?
What does this look like in practice?
Continuous Improvement Framework
1. Clarify the Strategy
1. Clarify the Strategy
1. Clarify the Strategy
Look for #Patterns.
1. Clarify the Strategy
Janelle’s Painful Experience
2. Record an Idea Flow Map
Make adhoc predictions as you work
3. Reflect on Decisions (Experience Review)
3. Reflect on Decisions (Experience Review)
3. Reflect on Decisions (Experience Review)
Look for #Patterns.
3. Reflect on Decisions (Experience Review)
Continuous Improvement Framework
Focus Meeting: Mastery Circle
Circle Leader
Circle Member
Focus: What pain did we experience?Observe: Ask questions about the factsConclude: What major factors caused pain?Optimize: What strategies to reduce the pain?Learn: What experiments could we run?
ObservationQuestions
Make a F.O.C.O.L Point!
Summarize lessons learned in a #HashTagged Journey.
(focused problem-solving discussion)
Dedicated Capacity for Improvement Work
Long-term feedback loop
LEARN YOUR WAY TO AWESOME.
Free to Join Industry Peer Mentorship Network
openmastery.org
Developer Insight Tools Platform
Idea Flow Mapping Tools
Team Insight Tools
TeamJoeSallyMarkEric
Community Insight Tools
AnonymizedData
(REST)
Shared Taxonomyof Patterns & Principles
(with example data)
Janelle Kleinopenmastery.org @janellekz
Check out openmastery.org for details.
Read my Book.
Think About It.
FREE withMembershipBuy It
How to Measure the PAINin Software Development
Janelle Klein