Embed Size (px)
DESCRIPTION
Details approaches to software testing, as presented at a Twilight session at Intergen.
Citation preview
Producing Testable Requirements
Piers Chamberlain [email protected] &
Robert Nugent [email protected]
Producing Testable Requirements
Requirements Inspection
THE PROBLEM
What the project team built wasn’t what I asked for…and I can’t believe that no-one
picked up these issues before I started looking at the system myself!
…but was what you asked for what you meant!
1. Natural language is very poor for communicating precise detail.
2. Human beings are so good at “interpreting” language that we make assumptions
without thinking.
Producing Testable Requirements
Quality Requirements
The idea is to get your specification as ship-shape as possible.
Feasible Scope
Necessary
Correct
Complete Function
Consistent
Verifiable Testability
Modifiable
Traceable Maintenance
Prioritized
Producing Testable Requirements
What is a Testable Requirement
Testable (in principal) means
Precise
Unambiguous
Absolute state – The requirement is either met or not met
…but it needs not be tested – can be either untested or indirectly tested
Indirectly testable in practice
Cost or risk reasons may prevent “real” testing
Simulations, other tests that can provide some confidence
Producing Testable Requirements
Validation through Test Design
Define a set of test cases to prove a requirement in principal
If you have a large number of tests (> 5 or 6) the requirement is probably too vague
Especially if the tests are of different types
Test Design RulesYou can assume any other requirements in the spec are met
You cannot assume anything not in the spec
Producing Testable Requirements
An Example
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.
Producing Testable Requirements
Ambiguous…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.
Producing Testable Requirements
Inconsistent…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.
Producing Testable Requirements
Imprecise…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.
Producing Testable Requirements
Incomplete…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.
Producing Testable Requirements
Let‟s assume some validation rules…
The system will validate a user's password
The password must be at least 6 and no more than 40 characters
The password can consist only of standard alphanumeric characters (i.e. a-z, A-Z and 0-9)
Until the password is validated, it cannot be saved or used to authenticate a user.
The system will validate a user's username
The system will validate that the username conforms to the RFC 2822 standard as an email address
The system will verify that the username is valid by sending an activation code as a hyperlink to that destination
The username is accepted as valid if the user clicks the activation code sent
Until the username is activated, it cannot be used to authenticate a user.
Producing Testable Requirements
Version 2
1. Change Password1.1 A user must be currently authenticated to change their password
1.1.1 The new password must meet the standard password validation rules*
1.2. A user must be currently authenticated to change their username
1.2.1 The new username is subject to the standard username validation rules*
2. Authentication2.1 To authenticate a user must supply a valid username and password credentials
at the login screen
2.1.2 A user‟s authentication is expired after 30 minutes
2.1.3 A user‟s authentication is expired immediately if they have elected to logout
Producing Testable Requirements
Why it works
It forces a user-centric view of how the system should
hang together
Introduces workflow, temporal elements
Checks for consistency and completeness
Can highlight design considerations where they are either
required (and missing) or unnecessary
Can provide behavioural and psychological insights
Producing Testable Requirements
Guidelines for Quality Requirements
Keep sentences short and break up paragraphs
Use the active voice
Supply all required detail
Granularity – The “smaller number of tests” principle
Avoid aggregation (conjunctions like “AND” and “OR” can
suggest combined requirements)
Write at a consistent level of detail throughout the
document
Producing Testable Requirements
Requirement in good shape … now
what am I prioritisingRequirements (source of the tests)
At the same level of extraction or type
Need to determine a measure to use (I am using
Risk)
Producing Testable Requirements
How can I conduct an assessmentNeed to determine:
The source of requirements
What method
What criteria to use
Who should be involved
Why Assess the RisksNot all requirements are the same
I can choice what to test first
Testing duration is NOT guaranteed
Producing Testable Requirements
Example – Methods*RISK
Usage frequency
Damage
(cost of failure)
Probability of failure
Damage / Loss of Use Functional Volume
*Risk Based Testing, Strategies for Prioritising Tests against Deadlines Hans Schaefer.
Quality
Example – CriteriaCritical area
Visible area
Usage frequency
Complex areas
Changed areas
New component
Impacted user base
Deadlines
Defect history
Producing Testable Requirements
Example – risk calc spreadsheet
Risk calculation schema
Impact factors Probability factors Prob. of defect detection
visibility user impact frequency deadline pressure new people complexity
weights weights Risk
Risks addressed 1 3 10 1 3 3
function A 1 1 2 1 5 5 1 744
performance 0 1 1 1 1 4 1 208
usability of x 5 1 2 3 1 5 1 588
… 2 2 2 2 2 5 1 644
… 1 1 1 1 1 2 1 140
1 2 1 2 1 1 1 136
Producing Testable Requirements
Prioritising the RequirementsBy Requirement Type
Must Have, Should Have, Nice to Have (I Wish)
Core \ Discretionary
1 to 100
From two factor risk assessment
Clustering First HelpsBy Function
By Functional Area
Core vs. Auxiliary
By sprint, cycle, release
By Architectural layer
Producing Testable Requirements
Example –using risk spreadsheet
Risk calculation schema
Impact factors Probability factors Prob. of defect detection
visibility user impact frequency deadline pressure new people complexity
weights weights Risk
Risks addressed 1 3 10 1 3 3
function A 1 1 2 1 5 5 1 744
Performance profile of … 0 1 1 1 1 4 1 208
usability of x 5 1 2 3 1 5 1 588
UC213.1 2 2 2 2 2 5 1 644
Password must have … 1 1 1 1 1 2 1 140
… 1 2 1 2 1 1 1 1361
2
3
4
Producing Testable Requirements
Prioritising Wrap up
To successfully prioritise requirements you need to:
Determine Risk assessment criteria
Use one or two factor method
Weight the results
Rank by chosen method
And you will have:
Objective view of project deliverables (the Risk if not delivered)
Assistance in test scope analysis
Prioritising of test effort
Producing Testable Requirements
Other Benefits
From Requirements Inspection
Higher First Release quality
Lower Cost of software maintenance
Improved team dynamics / education
From Risk\Prioritisation
Objective estimates
Defect clustering analysis
Defect resolution order
Business perspective progress reporting
Producing Testable Requirements
Some Simple First Steps
Evaluate Requirements
Review for Testability
Conduct a risk assessment
Must have, Should have or Could have
Prioritise the Results
Group the requirements
Test this one first … to test this one last
Producing Testable Requirements
Questions & Discussion
Producing Testable Requirements
References & Further Reading
Generating Testable Requirements
Methodology for Writing High Quality Requirement Specifications Linda Rosenberg
September 24 1998
Writing Quality Requirements by Karl E. Wiegers
Writing Testable and Code-able Requirements Murat Guvenc / Borland
On-Track Requirements by Rodger Drabick Better Software Magazine May/Jun 1999
(Vol. 1, Issue 3)
Producing Testable Requirements
References & Further Reading
Risk-Based Prioritisation
Requirement Ranking – NO, No, No to High, Medium or Low Posted April 25th, 2008 by Steven Davis
Risk-Based Quality Management Posted April 24th, 2008 by Tracy Lynne Dedore
Prioritizing Requirements Posted April 9th, 2008 by Patrick Walsh
Theory and Practice of Risk-based Testing by Felix Redmill
Published in Software Testing, Verification and Reliability, Vol. 15, No. 1, March 2005
*Risk Based Testing, Strategies for Prioritizing Tests against Deadlines by Hans Schaefer,
Published in Methods & Tools Volume 13 number 4, Winter 2005.
Exploring Risk-based Testing and Its Implications by Felix Redmill
Published in Software Testing, Verification and Reliability, Vol. 14, No. 1, March 2004
Risk-based Test Planning During System Development by Felix Redmill
Invited paper at KKIO 2004, Gdansk, Poland, 5-8 October 2004
*Risk Based Testing and Metrics by Stale Amland, presented at EuroSTAR „99, November 8 – 12,
1999 Barcelona, Spain
