View
324
Download
0
Category
Preview:
Citation preview
Janelle Kleinopenmastery.org @janellekz
Tour Speaker
Bringing Science to Software Development
How to Measure the PAIN in Software Development
Janelle Klein
How to Measure the PAIN in Software Development
Janelle Klein
After five years…
Data-Driven Software Mastery
leanpub.com/ideaflow
How to Measure the PAIN in Software Development
Janelle Klein
==
2.0
Visibility Changes Everything
Visibility
+
Lean Manufacturing
“The Art of the Scientist” Lean Startup Workshop, Ash Maurya
Ash: “The goal isn’t learning, the goal is traction.”
==
2.0
+
Traction (Lean Startup)
We get a design template for how to build a learning organization.
Visibility Changes Everything
Data-Driven Software Mastery
Five Talks
Stop Getting Crushed By Business Pressure
How to Build a Learning Organization
Top 5 Reasons Why Improvement Efforts Fail
Make a F.O.C.O.L. Point!
Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Shared Vision
Team Learning
Systems Thinking
These disciplines are emergent with science.
, Developer & Consultant for 15+ yearsSpecialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I <3 cool data)
Continuous Delivery infrastructure, design & automation strategies
Janelle Klein
Who Am I?
How to Measure the PAIN in Software Development
Janelle Klein
leanpub.com/ideaflow
Data-Driven Software Mastery
I’m also a hobbyist Cognitive Scientist
Great Team Disciplined with Best Practices Constantly Working on Improvements+
Project FAILURE
About 8 years ago…
We made significantly more mistakes in code that we didn’t write ourselves.
Lower Familiarity
More Mistakes=
There had to be more to the story...
Unexpected Behavior
Problem Resolved
Measure Painful Interaction with the Code (Friction)
Troubleshooting
Progress
5 hours and 18 minutes of troubleshooting...
PAINFUL
The amount of PAIN was caused by…
Likeliness(of((Unexpected(Behavior(
Cost(to(Troubleshoot(and(Repair(
High(Frequency(Low(Impact(
Low(Frequency(Low(Impact(
Low(Frequency(High(Impact(
PAIN(
What Causes Unexpected Behavior (likeliness)?
What Makes Troubleshooting Time-Consuming (impact)?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
Most of the pain was caused by human factors.
What causes PAIN?
What Causes Unexpected Behavior (likeliness)?
What Makes Troubleshooting Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Most of the pain was caused by human factors.
PAIN occurs during the process of understanding and extending the software
Complex(So*ware(
PAIN
Not the Code.
Optimize “Idea Flow”
My team spent tons of time working on improvements that didn’t make much difference.
We had tons of automation, but the automation didn’t catch our bugs.
My team spent tons of time working on improvements that didn’t make much difference.
We had well-modularized code,
but it was still extremely time-consuming to troubleshoot defects.
The hard part isn’t solving the problems it’s identifying the right problems to solve.
“What are the specific problems that are causing the team’s pain?”
measures the time spent on:
Idea Flow
x
Troubleshooting
x
Learning
x
Rework
Quality Risk Familiarity Risk Assumption Risk
Idea Flow Mapping Tools
(Open Source, Supported GA ~June 2016)github.com/ideaflow/tools
“What caused the pain in this case?”
Categorize the Problems with #HashTags
#ReportingEngine
#Hibernate
#MergeHell
1. Test Data Generation
2. Merging Problems
3. Repairing False Alarms
1000 hours/month
Add up the Pain by Category
0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Chaos Reigns Unpredictable work stops
fitting in the timebox
Troubleshooting
Progress
Learning
Analyzing Long-Term Trends
Time%Pressure%
Compromise%Safety%for%
Speed%
Increase%Number%&%Severity%of%Hazards%
%
More%Pain%and%Higher%Task%Effort%
Constant'Urgency'
Cycle of Chaos High-Risk Decision Habits
0%
100%
Release 1 Release 2 Release 3
Troubleshooting
Progress
Learning
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
(extrapolated from samples)
Gambling Habit —> Gradual Loss of CONTROL
Fewer%Problems%to%
Fix%
Stop%%and%Think%
Mi8gate%the%Risk%
Increased%Produc8vity%
and%Innova8on%
Safety'
Cycle of Safety Low-Risk Decision Habits
Optimize for Safety
We don’t improve productivity by trying to go faster, we improve productivity by improving control.
The Safety Principle
Product(Requirements( So1ware(Task(
Possible(Decision(Paths(
So1ware(
Software is a Reflection of our Decisions
Product(Requirements( So1ware(Task(
High%Risk%Decision%Habits%
So1ware((and(all(our(problems)(
(
RESET
Software is a Reflection of our Decisions
Product(Requirements( So1ware(Task(
High%Risk%Decision%Habits%
So1ware((and(all(our(problems)(
(
Technical Debt is just a Symptom of the Problem
Software is a Reflection of our Decisions
Quality Target
Lower Control Limit
Upper Control Limit
X"
This is “Out of Control”
Lower Variability = Better Control
Steering Wheel = Process Control
“Pain Control” in Software Development
Optimal Friction
Upper Control Limit X"
“Out of Control”
20min
0m
Average Pain per Incident
This is Better
Target
Control Limit
Categorize the Pain Type
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Improve Quality of Decisions
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Target - The direction of “better”
Target: Optimize the Rate of Idea Flow
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Input - The constraints that limit our short-term choices…
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Output - The pain signal we’re trying to improve
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Focus on the biggest pain…
F ocus!
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
1. Visibility - Identify the specific patterns
1. Visibility
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
2. Clarity - Understand cause and effect
2. Clarity
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
3. Awareness
3. Awareness - Stop and think to adjust habits
Idea Flow Learning Framework
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
4. Run Experiments to Learn What Works
Idea Flow Learning Framework
Optimal Friction
Upper Control Limit X"
“Out of Control”
20min
0m
Average Pain per Incident
This is Mastery
Target
Control Limit
Categorize the Pain Type
This is the Process of Mastery
“The Art of the Scientist” Workshop, Ash Maurya
Ash: “What’s the most important thing that scientists do?”
“Run Experiments!”
“Analyze Data!”
Getting Involved with Lean Startup
“The Art of the Scientist” Workshop, Ash Maurya
Ash: “The most important things scientists do is build models.”
Getting Involved with Lean Startup
Test the Model (Experiments)
Refine the Model (Patterns)
False&Predic,on&
Scien,fic&Model&
Modeling (Inductive)
Experimentation (Deductive)
Empirical Data
The Scientific Method
We can’t ignore this part.
Test the Model (Experiments)
Refine the Model (Patterns)
False&Predic,on&
Scien,fic&Model&
Modeling (Inductive)
Experimentation (Deductive)
Empirical Data
Science is a Feedback Loop that Fuels Discovery
Creating an explicit model makes our beliefs testable and our discoveries additive.
We can’t ignore this part.
“The Art of the Scientist” Lean Startup Workshop, Ash Maurya
Ash: “The goal isn’t learning, the goal is traction.”
Optimize the Rate of Flow in a Supply Chain
Focus on Limiting
Constraint
(Goldratt’s Theory of Constraints, “The Goal”)
Optimal Friction
Upper Control Limit X"
“Out of Control”
20min
0m
Average Friction per Customer
This is Traction
This is the Process of Lean Startup
High Friction
Min Friction
“No” Threshold
Customer Happiness
Categorize the “No” Type
“The Idea Flow Factory” (supply chain model)
Optimize the Rate of Idea Flow
Across the Software Supply Chain
100 hours
50 hours
Troubleshooting
Learning
Rework
Focus on the Biggest Pain Type
Focused Effort = Visible Improvement
Optimal Friction
Upper Control Limit X"
“Out of Control”
20min
0m
Average Pain per Incident
This is Traction
Target Rate
Control Limit
Categorize the Pain Type
This is the Process of Idea Flow Mastery
We can apply the entire suite of cost & risk reduction tools from Lean Manufacturing!!
Supply Chain Optimization
Statistical Process Control (SPC)
Optimal Friction
Upper Control Limit X"
“Out of Control”
20min
0m
Average Pain per Incident
Target
Control Limit
Throughput Accounting
Test the Model (Experiments)
Refine the Model (Patterns)
False&Predic,on&
Scien,fic&Model&
Clarity (Why)
Awareness (How)
Visibility (What)
Use Science to Fuel “Explicit Mastery”
Model “Decision Principles”
What’s a Decision Principle?
1. How do I evaluate my situation?
2. What should I optimize for?
Answers Two Questions
The Haystack Principle
Lots of unvalidated changes
Easier to find the needle.
Decision Principle: The Haystack Effect
Optimize for small manageable haystacks.
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Use Science to Fuel “Explicit Mastery”
Systematically codify Decision Principles from lessons learned
The Visibility Principle
Our mind filters what we see and experience
in terms of the “shapes” we know.
Three Different Types of Metaphors
Object Patterns (Thing)
Context/Relationship Patterns (Space)
Process Patterns (Time)
We “see” the world through a fabric of metaphor.
PAIN occurs during the process of understanding and extending the software
Complex(So*ware(
PAIN
Pain occurs over TIME.
What do you think happens when we make this transition?
Process Patterns (Time)
Object Patterns (Thing)
What Causes Unexpected Behavior (likeliness)?
What Makes Troubleshooting Time-Consuming (impact)?
Take a Closer Look at the Patterns
Familiarity Mistakes
Stale Memory Mistakes
Semantic Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
Most of the pain was caused by human factors.
“What caused the pain in this case?”
We Start to Experience the World Differently
#ReportingEngine
#Hibernate
#MergeHell
2.#Record#an#Idea#Flow#Map#
So#ware(Task(Strategy((Design(
Experience(Review(
Iden:fy(the(Pa>erns(
1.#Clarify#the#Strategy#
So#ware(Task(
3.#Reflect#on#Decisions#
Focus(Mee:ng(
4.#Share#Discoveries#
Visibility Framework
The Clarity Principle
Our understanding deepens when we explain
the similarities and differences of experiences
and the relationships of cause and effect
Circle Leader
Circle Member
Focus: What’s the problem to solve?
Observe: Ask questions about the facts
Conclude: What’s causing the pain?
Optimize: How do we reduce pain?
Learn: What questions does this raise?
Observation Questions
Make a F.O.C.O.L Point!
Summarize lessons learned in a #HashTagged Blog.
Deep Reflection in Open Mastery Circles
Clarity Framework (Why)Repeat&Weekly&
Focus&Mee1ng&
1.#Explain#Decisions# 3.#Search#for#Archetypes#
4.#Model#Decision#Principles#
Zoom In Zoom Out
2.#Model#the#Journey#
Focus#
Reset Password Login
Username:
Password:
To continue to our site, please login:
Task: Implement THIS
Me: "So first, George, can you describe what you were trying to do before you encountered the conflict?”
George: "This was back when I was getting the token to work in the email. I wrote the code to generate the token, edited the email template to include the token in the email link, but the token wasn't showing up in the actual email."
Me: "So what did you do next?"
George: At first, I thought there was something wrong with the token variable code, but the problem turned out to be a misspelled variable name in the email template. Doh!"
Focus: What Major Factors Caused the Pain?
Observe: Ask Questions About the Facts.
Painful Consequences: Consequence
Conclusions: What Caused the Pain?
Ran experiments by sending emails over and over again (23 emails with a missing tokenId)
Redeployed application when uncertain about the cause
Ambiguous Clue
Manual Execution (slow)
Optimize: How could we have avoided the pain?
What’s an alternative strategy that could have mitigated these risks?
George: My main strategy was to do a small #VerticalSlice by getting the “email send” part working, before adding the token logic for the security part. I understand how the #ManualExecution and an #AmbiguousClue caused the friction, but I'm not really sure what I would do differently if I could do it again.
Me: "George, can you describe your original strategy to mitigate risk?”
Optimize: How could we have avoided the pain?
What’s the strategy pattern we’re trying to implement?
Alternative Strategy:
APIs for each email workflow
Wire up email last
Strategy
Optimize: How could we have avoided the pain?
What’s the strategy pattern we’re trying to implement?
Alternative Strategy:
APIs for each email workflow
Wire up email last
Strategy
Optimize: How could we have avoided the pain?
Alternative Strategy:
APIs for each email workflow
Wire up email last
Situation
What’s the situation we need to recognize in order to decide to use the strategy?
Situation Strategy Consequence
Developer Journeys
#HiddenOutputs Create #ObservableInterfaces
Avoid #AmbiguousClues & #ManualExecution
Optimize: How could we have avoided the pain?
Learn: What questions does this raise?
What questions should we ask ourselves in the future?
In what situations should we ask the question?
Experiment Time
Our Perception of Time is WAY OFF
Setup Experiment Analyze Results & Decide Next Experiment
Execute
waiting - time goes slow
doing - time zooms by
The Code Sandwich Principle
The thickness of the sandwich increases
troubleshooting difficulty
Behavior Complexity
Observability
Ease of Manipulation
Optimize for experimentation.
The Clarity Principle
Our understanding deepens when we explain
the similarities and differences of experiences
and the relationships of cause and effect
The Awareness Principle
We can only avoid a mistake
if we predict the consequences
of the decision in the moment.
18:120:00
George’s Painful Experience
I know I made a big haystack, I’ll do better next time.
This is hindsight bias, it’s not that easy!
The Experience Review
The ChallengeWe tend to make auto-pilot decisions and do what’s familiar
How do we break decision habits?
Imagine your brain is a decision-making engine
written in code.
BreakpointStop and Think!
Changing Decision Habits
BreakpointStop and Think!
Changing Decision Habits
We have to predict the big haystack during the decision.
The Checklist Manifesto
Atul Gawande
Scaling Lean (Lean Startup)
Ash Maurya
+
Change Decision Habits with Strategy Experiments
14:230:00
I want to avoid this…
Thinking Checklist
Is my current approach likely to cause a big haystack?
Situation: start of subtask
Let’s Make a Checklist!
“What question could I ask my future self to recognize similar risks in the future?”
“In what situation would I ask the question?”
0:00
Stop and Think:
Is my current approach likely to cause a big haystack?
Predict: Small haystack
Strategy Experiments
18:120:00
Stop and Think:
Is my current approach likely to cause a big haystack?
Predict: Small haystack
False Prediction
Strategy Experiments
18:120:00
Stop and Think:
Is my current approach likely to cause a big haystack?
False Prediction
Strategy Experiments
High-Risk Situations
1. Unraveling sweater 2. Integration-heavy change 3. High state variation 4. Minimum scope is big
Q: Is my current approach likely to cause a big haystack?
Start of Subtask
Strategy Types (“do”)
1. DependencyAnalysis 2. IncrementalIntegrationTest 3. DataDrivenTest 4. IsolateHardToTestCode
High-Risk Situation Types (“see”)
1. UnravelingSweater 2. HeavyIntegrationLogic 3. HighStateVariation 4. CoupledExternalDependencies
Haystack Decisions
Codify What Works
DoSee
So#ware(Task(
Software Task Strategy((Design(
Thinking(checklists(in((key(situa6onal(contexts(
!(!(!( !(!(
!( !(!(!(
Experience(Review(
!(!(3.#Refine#the#
Checklists#
!(!(!(
Failed(Experiments(
Successful(Experiments(
2.#Run#Checklist#Experiments#
1.#Design#Thinking#Checklists#
+(
Awareness Framework (How)
The Awareness Principle
We can only avoid a mistake
if we predict the consequences
of the decision in the moment.
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
Use Science to Fuel “Explicit Mastery”
Systematically codify Decision Principles from lessons learned
Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Shared Vision
Team Learning
Systems Thinking
These disciplines are emergent with science.
Input: Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term loop long-term loop
1. Visibility
2. Clarity
3. Awareness
F ocus!
Output: “Friction” in Idea Flow
How do my decisions affect your experience?
Use Science to Fuel “Explicit Mastery”
Management
Dev Team
Fire x10
What if our organization was a robot?
If the feedback loop is broken, we burn.
Pain Sensor
Bigger Social Network of BrainsCollaboration to Perform a Bigger Function (Organization)
Component
ComponentHub
Hub
Hub
The Challenge: Knowledge & Decision-Making are Distributed
Learn & Adapt
Role
Decision(Type(
Required(Knowledge(
Visibility and Decisions Coupled
Visibility and Decisions De-coupled
Role A
Required(Knowledge(
Role B
Decision(Type(
ROOT CAUSE of the Broken Feedback Loop
Manager
Alloca&on(Decisions(
Knowledge(of(Risks(
Risk(Mgmt(Decisions(
Developer
Communication Breakdown
Broken Feedback Loop (Manager Role)
So#ware(Task(
Mi.gate(the(Risks(
Product(Development(
Product(Work(Queue(
Risk(Mgmt(Work(Queue(
Product(Owner(
Product(Decisions(
Knowledge(of(Customers(
Technical(Risk(Manager(
Risk(Mgmt(Decisions(
Knowledge(of(Risks(
Dev$Team$Capacity$
Manager
Alloca.on(Decisions(
Turn Risk Management into an Investment Decision
Now we can steer!!
Three Dimensions of Organizational Knowledge
Discovery (Expansion)
Peer Mentorship (Diffusion)
Optimize the Whole (Acceleration)
“Decision Principles” (models)
Discovery (Expansion)
Peer Mentorship (Diffusion)
Optimize the Whole (Acceleration)
Priority 1
Priority 2
Priority 3
Priority 1
Priority 2
Priority 3
Priority 1
Priority 2
Priority 3
Biggest Pains
$$Biggest Skill Gaps
$Limiting Constraints
$$$$
Turn Risk Management into an Investment Decision
4. Optimize the Whole
2. Design the Strategy
1. Identify Obstacles (Focus)
3. Learn What Works (Traction)
Steering Wheel
Create a Steering Wheel to Drive the Organization!
$$ $$
Focus: Quality of Improvement Decisions
Focus Traction
$$$ $$
Focus: Quality of Investment Decisions
Risk Translator
(interface)
Management(Coordination)
Engineering(Execution)
Open Mastery Learning Framework
Idea Flow Pain Sensor
Open Mastery Learning Platform
Idea Flow Mapping Tools
Team Mastery Platform
TeamJoe Sally Mark Eric
Community Collaboration Platform
Anonymized Data
ProjectTiger
ProjectBear
(REST)
Shared #HashTag Glossary of Patterns & Principles
with Examples
The Conversation is on Slack
We’re debating the patterns and principles of Idea Flow in Wikipedia
It’s a Developer Party!
We’re sharing our ideas over Blogs
(and managers are invited)
Industry Peer Mentorship Network
Companies
Community Groups
HQ in Austin
Open Mastery Austin
meetup.com/Open-Mastery-Austin
Janelle Kleinopenmastery.org @janellekz
Check out openmastery.org for details.
Read my Book.
Think About It.
FREE with Reading GroupBuy It
How to Measure the PAIN in Software Development
Janelle Klein
Recommended