Embed Size (px)
Citation preview
My background
• 1991: MSc in Computer Science and Engineering – KTH (Royal Institute of Technology), Stockholm
− Test process audits− Test process improvements − Test mentor− Training provider (ISEB/ISTQB)
• 15 years consultant with Enea– Tester– Test project manager– Test configuration manager– Test analyst
• 2007: PhD in Software Testing– University of Linköping– Thesis title: Handling Combinatorial Explosion in Software Testing
Agenda
• Quick Introduction to Risk-Based Testing
• Our Implementation
• The Case Study
• Tailor RBT to your Needs
Risk-based testing
• Primarily consider product risks
• Used to decide where to start testing and where to focus the testing
• Use risks to – Determine the test techniques used– Determine the extent of testing to be carried out– Prioritize testing and test cases– Determine whether any other non-testing activity
could be employed to reduce the project risks
Our Approach
• Risk Identification– Identify potential risks - often in a brainstorming session
• Risk Analysis– Estimate consequences and likelihood
• Risk Evaluation– Classify risks in two – four groups to determine which risks are
important
• Risk Mitigation– Devise test cases for the most important risks or– Use risk levels to determine test depth
Our Approach – Risk Identification
• Start with existing high-level requirements– Treat each requirement as a separate risk:
• Existing risks documented but not analyzed in advanced
• During risk analysis workshop add new risks as they appear
What if this requirement does not work?
Our Approach – Risk Analysis
• Conducted as a workshop– Project Leader, testers, one designer, one SW architect
• One “risk” at a time is graded with respect to several properties– Each property is judged on a scale 1(low) – 4(high)
• Risk analysis is based on guesses and assumptions
• Property assignments are relative not absolute
Our Approach – Dimensions and Properties
Probability
Requirements Quality
Implementation Complexity
Functional Dependency
Usage Complexity
Usage Frequency
Consequence
Usage Consequence
Test Consequence
Business Consequence
Test Uniqueness
Our Approach – Risk Evaluation
• Evaluation completely automated– Evaluation logic coded in Excel spreadsheet
• Based on values of properties, weighted averages forProbability and Consequence are calculated
• Risks are classified as High, Medium or Low based on– Probability alone
– Consequence alone
– Risk (= P * C)
See upcomingslides for details
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ...
7 ...
1. Estimate probability properties
1 1 434
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ...
7 ...
2. Calculate weighted average
1 1 434 3.31
Values 1 and 2 are counted onceValue 3 is counted three timesValue 4 is counted four times
Ex: ( 1 + 4*4 + 3*3 + 1 + 4*4 ) / 13 = 3.31
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ...
7 ...
3. Filter the weighted average
1 1 434 3.31
1 ≤ AVG ≤ 1.8 ⇒ LOW1.8 < AVG ≤ 3.4 ⇒ MEDIUM3.4 < AVG ≤ 4.0 ⇒ HIGH
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... M
7 ...
3. Filter the weighted average
1 1 434 3.31
1 ≤ AVG ≤ 1.8 ⇒ LOW1.8 < AVG ≤ 3.4 ⇒ MEDIUM3.4 < AVG ≤ 4.0 ⇒ HIGH
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... M 1 2 2 4 3.00 M
7 ...
4. Repeat for Consequence Properties
1 1 434 3.31
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... M 1 2 2 4 3.00 M
7 ...
4. Calculate and Filter Risk
1 1 434 3.31
1 ≤ AVG ≤ 4.0 ⇒ LOW4.0 < AVG ≤ 9.0 ⇒ MEDIUM9.0 < AVG ≤ 16.0 ⇒ HIGH
9.93
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... M 1 2 2 4 3.00 M H
7 ...
4. Calculate and Filter Risk
1 1 434 3.31
1 ≤ AVG ≤ 4.0 ⇒ LOW4.0 < AVG ≤ 9.0 ⇒ MEDIUM9.0 < AVG ≤ 16.0 ⇒ HIGH
9.93
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... M 1 2 2 4 3.00 M H
7 ...
5. Calculate the final risk level
1 1 434 3.31
Final Risk Level = Max(Prob level, Cons level, Risk level)
9.93
Our Approach – Example illustration
Probability Consequence RISK
Id Slogan Final
Risk Level
Req.Quality
Impl.Complex.
Func.Dep.
UsageComplex.
Usage Freq.
AVG ProbLevel
Usage.Cons.
TestCons.
Busin.Cons.
TestUniq.
AVG Cons.Level
P*C Level
1 ...
4 ... H M 1 2 2 4 3.00 M H
7 ...
5. Calculate the final risk level
1 1 434 3.31
Final Risk Level = Max(Prob level, Cons level, Risk level)
9.93
Our Approach – Risk Mitigation
• Focus on test depth (rather than order of risks)• Example:
– High level risks• Cover with several positive and negative test cases• At least 75% of positive test cases should be prio 1, rest prio 2• At least 50 % of negative test cases should be prio 1
– Medium level risks• Cover with several positive and some negative test cases• Between 50% and 75% of positive test cases should be prio 1, rest prio 2
– Low level risks• Cover with only positive test cases• One test case per risk should be prio 1
Case Study
• Main problem– “Testing takes too long time, where can we cut”
• Large complex system– Mix of commercial and proprietary hardware– Distributed and embedded
• Focus of RBT study– System test (functionality) of one node– Participants: testers, designers, SW architects
Case Study – Results and Observations
• Method is fast– 10 risks took less than 30 minutes to analyze– Risk values easy to update
• Results seem viable– Final result corresponds to ”gut feeling”
• Backwards traceability– Track defect slip-through back to decisions
• Common goal– The whole team agrees on what is important
Tailor RBT to Your Needs 1(2)
• Risk Identification– How do you find risks?– What is a suitable granularity?
• Risk Analysis– Which properties should be considered?– How many levels?– Each level of each property, what do they mean?
Tailor RBT to Your Needs 2(2)
• Risk Evaluation – How to transform property values into a risk level?
• Risk Mitigation– What are suitable directions for each risk level?
• Handle changes– Risks may change– RBT process may need alterations
Tips
• RBT is not an absolute science – Don’t waste time by going into too much detail
• New information will arrive in the project – Repeat analysis with some frequency
• It is hard to make RBT perfect the first time – Be prepared to revise the method, including thresholds
Thank you for listening!
Questions?
