Embed Size (px)
DESCRIPTION
Testers are often evaluated by metrics that don’t really quantify the value of their work. Metrics such as tests planned, tests executed, coverage achieved, and defects reported all measure effort rather than results. Since people generally want to meet metrics goals, measurements that focus on activity rather than effectiveness often encourage unintended behaviors. Since the true value of testers lies in their ability to analyze and communicate risks and impacts, we must change the focus of metrics from numbers to insights. Scott Barber shares what stakeholders are really looking for when they request specific metrics, how the metrics they request frequently fail, and how to help your organization create metrics that do provide real insight. Discover the tools you need to explain what can be measured, what those measurements mean, and how to combine measurements into metrics that tell insightful stories about your testing.
Citation preview
nt Session
Presented by:
Scott
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
W8 Concurre4/9/2014 12:45 PM
“How Metrics Programs Can Destroy Your Soul”
Barber
SmartBear
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Scott Barber SmartBear
Chief performance evangelist for SmartBear Scott Barber is a respected leader in the advancement of software testing practices, an industry activist, and load testing celebrity of sorts. Scott authored several books―Performance Testing Guidance for Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web Load Testing for Dummies―and more than 100 articles and blog posts. Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of the AST and CMG, and is a founding member of ISST. His industry writing, speaking, and activism focus on improving the effectiveness and business alignment of software development practices. Learn more about Scott Barber.
2/25/2014
1
How Metrics ProgramsCan Destroy Your Soul
Created for:
How Metrics Programs Can Destroy Your Soul
By:
Scott BarberProduct Owner – Load Testing Tools
SmartBear Software, Inc.
2/25/2014
2
Product Owner – Load Testing ToolsSmartBear Software, Inc. scott barber@smartbear [email protected]@sbarber
Co‐Founder: Workshop On Performance and Reliabilitywww.performance‐workshop.org
Co‐Author:Author: Contributing Author:
Books: www.perftestplus.com/pubsAbout me: about.me/scott.barber
Have valid needs to: • Assess & compare performance & quality of:
di id l
Managers & Execs
• Individuals• Teams• Projects• Products
• Compute costs• Identify risks & trends
2/25/2014
3
Commonly satisfy those needs via: • Instinct
• Not Scalable
Managers & Execs
• Not Scalable• Non‐transferable
• Qualitative Metrics• “Group Gut”• Inconsistent
i i i• Quantitative Metrics• Scalable• Transferable
Commonly satisfy those needs via: • Instinct
• Not Scalable
Managers & Execs
• Tied to individualsC ’ b i d• Not Scalable
• Non‐transferable• Qualitative Metrics
• “Group Gut”• Inconsistent
i i i
• Can’t be trained• Predicated on trust
(not data)• Virtually impossible to
defend• Generally reserved for
“hands on” managers• Quantitative Metrics• Scalable• Transferable
hands‐on managers with one project or team
2/25/2014
4
Commonly satisfy those needs via: • Instinct
• Not Scalable
Managers & Execs
• Very difficult to do ll• Not Scalable
• Non‐transferable• Qualitative Metrics
• “Group Gut”• Inconsistent
i i i
well• Expensive• Time consuming• Not good for detailed
comparisons• Not good for
estimation• Quantitative Metrics• Scalable• Transferable
estimation• Few managers/execs
trained in using them
Commonly satisfy those needs via: • Instinct
• Not Scalable
Managers & Execs
• Scale wellP id i li i &• Not Scalable
• Non‐transferable• Qualitative Metrics
• “Group Gut”• Inconsistent
i i i
• Provide simplicity & consistency
• Easy to use• Good for detailed
comparisons• Good for estimation• Good for trends• Quantitative Metrics
• Scalable• Transferable
• Good for trends• Most managers/Execs
trained • Validated in many
areas over long periods of time
2/25/2014
5
Are based on measurements that are: • Verifiable
St d di d
Effective Metrics
• StandardizedAre explicitly linked to goals
• Correlated/causal• Compliance
F t d b l tFavor trends over absolutesEvolveAre not used in isolation
“Most metrics are created, collected, and reported to satisfy a leader’s request. The leader’s role is to supply clarity and direction by providing the proper questions. Middle management’s role is to
th ti M t i ff f idi th
What is a Metric?
answer the questions. Metrics offer a means of providing the answers so that all involved can have faith in them.
Unfortunately, leaders often don’t know exactly what they want. Chances are you have played the Guessing Game with a leader, where the data you provided wasn’t what he needed, so he asked for different data, figuring he would know the right data when he saw it Despite repeated failures you continued to chase data as ifsaw it. Despite repeated failures, you continued to chase data as if all the effort invested in collecting the wrong data would eventually prove worth your perseverance.”
From: “Do-It-Yourself Metrics”by Martin Klubeck, Michael Langthorne, and Don Padgetthttp://www.educause.edu/ero/article/do-it-yourself-metrics
2/25/2014
6
Provide insight into the health of: • Individuals• Teams
Metrics Should
• Teams• Projects• Products
Help to:• Assess efficiency & progress• Inform business decisions • Enhance communication• Maintain focus
However;
Even in well studied & historically validated areas, quantitative metrics are…quantitative metrics are…
2/25/2014
7
Quantitative Metrics: • Obscure Details
ll & i
Far From Perfect
• Roll‐ups & summaries• Focus on “big picture”
• Risk Measurement Dysfunction• People tend to optimize metrics• Metrics optimization invites “bad stuff”
• Are “invitations to conversations”• Don’t tell the whole story in isolation• Can be wildly misleading
… and…
Common quantitativeCommon quantitative management metrics
models are not a good fit for software development
2/25/2014
8
… and…
Within software development, testing is the *worst* fit for common quantitative managementquantitative management
metrics models
Inconsistent Units:• Test/Test Cases
Why?
• Size/Importance of defectsUnknown Unknowns:
• How many defects are there?• How many tests would it take to…?
Quality is Subjective:• How good is “good enough”?• What do Pass/Fail *mean*?
2/25/2014
9
Test/Test Cases Planned?Test/Test Cases Executed?
These metrics indicate… ?
Defects Found?Defect Removal Rate?Coverage?
Test/Test Cases Planned?Test/Test Cases Executed?
These metrics encourage… ?
Defects Found?Defect Removal Rate?Coverage?
2/25/2014
10
Communicate impacts to revenue:• Help the CEO get a bigger yacht, sooner.
What do we *want* to do?
Identify risks to the business:• Generally revenue centric.
Identify risks to the buyers/users:• If they don’t want to buy, we lose.
Streamline Development:• Get to “good enough” faster.
Do your test metrics help with…• Risks to revenue?
Test Metrics Effectiveness
• Risks to the business?• Risks to the buyers/users?• Increasing delivery velocity?
2/25/2014
11
Change Your PerspectiveRisk as a common language
Risk
Security
FunctionalCompliance
Whether explicitly or implicitly, all forms of testing revolve around the reduction and
PerformanceUsability
reduction and management of risk.
© 2011 PerfTestPlus, Inc. All rights reserved.
The Secret to Managing Risk
To effectively manage risk, you must effectively manage knowledge.
© 2011 PerfTestPlus, Inc. All rights reserved.
2/25/2014
12
Control Model Testing
Control Model Testing is a business-a businessaligned approach to software testing that derives “test cases” from knowledge models of the system
© 2011 PerfTestPlus, Inc. All rights reserved.
based on a risk-based taxonomy.
Basic Entities
© 2011 PerfTestPlus, Inc. All rights reserved.
2/25/2014
13
Risk Assessment Framework
The Open Group (http://www3.opengroup.org/):Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156
Testing For Threats & Risk
Controls prevent or mitigate risk whichmitigate risk which may impact business objectives.
Control Model Testing helps identify
© 2011 PerfTestPlus, Inc. All rights reserved.
Testing helps identify and assess these controls.
2/25/2014
14
Possibly Better Test MetricsRisks/Threats Identified?Risk/Threat Tests Planned/Executed?Risks/Threats Dismissed?Risks/Threats Uncontrolled/Unmitigated?Risks/Threats Controlled/Mitigated?
Questions?
2/25/2014
15
Contact Info
about me/scott barberabout.me/scott.barber
Product Owner – Load Testing ToolsSmartBear Softwear, Inc.
E-mail:Scott barber@smartbear com
Web Site:[email protected]
Blog:scott-barber.blogspot.com
SmartBear.com
Twitter:@sbarber
