© LogicaCMG 2006. All rights reserved
LogicaCMG & Testing: TestFrame & RRBT
Mike Peters
Senior Test Consultant/Delivery Manager
Risk & Requirement Based Testing & TestFrame 2
About LogicaCMG
• > 40,000 employees worldwide
• 41 countries
• > 40 years experience in IT services
• Offers a wide range of IT-solutions
• Specialised in industry-specific solutions
• Aquisition of WM-data in 2006
Risk & Requirement Based Testing & TestFrame 3
LogicaCMG & Software Testing
• 2500 Test Professionals world-wide
– Test consultancy
– Managed testing
– Test outsourcing
• Test Research Centre in the Netherlands
– TestFrame method & tooling
– Test management: Risk & Requirement Based Testing
– Business Acceptance Management & - Testing
– ERP & Testing
– Embedded Software Testing
– STAPPIN (Standard infrastructure testing, e.g. Vista, Exchange)
Risk & Requirement Based Testing & TestFrame 4
… LogicaCMG’s knowledge & experience in testing is based on more than 1,000 test projects over the world
Testing credentials
Risk & Requirement Based Testing & TestFrame 5
Agenda
• Everyday testing challenges
• Risk & Requirement Based Testing (RRBT)
• TestFrame
• Discussion
Risk & Requirement Based Testing & TestFrame 6
The later you find faults, the more it costs to solve them
Testing of systems
Source: Boehm
Definition Design Build Use
Costs
Time
Risk & Requirement Based Testing & TestFrame 7
Testing is often under pressure
Planning andspecification
Development Test
Risk & Requirement Based Testing & TestFrame 8
Testing in parallel to the development activities
Development activities
Testing activities
Risk & Requirement Based Testing & TestFrame 9
Agenda
• Everyday testing challenges
• Risk & Requirement Based Testing (RRBT)
• TestFrame
• Discussion
Risk & Requirement Based Testing & TestFrame 10
Position various solutions
Test analysis
and execution
Test Management
Business
objectives
Structure/Fitting/ToolingStandard
RepeatableReliable
Maintainable
InsightControlFocus
ReportingPredictable
Commitment
Risk ControlQuality to MarketTime to Market
Business AlignmentBudget Control
Reputation Management
Risk & Requirement Based Testing & TestFrame 11
Risk & Requirement Based Testing
Risk & Requirement Based Testing & TestFrame 12
Test Management Model
EVALUATIONEVALUATIONEVALUATIONEVALUATION
ANDANDANDAND
TRANSFERTRANSFERTRANSFERTRANSFER
REPORTINGREPORTINGREPORTINGREPORTING
ANDANDANDAND
IMPLEMENTATIONIMPLEMENTATIONIMPLEMENTATIONIMPLEMENTATION
ADVICEADVICEADVICEADVICE
ISSUEISSUEISSUEISSUE
MANAGEMENTMANAGEMENTMANAGEMENTMANAGEMENT
PROGRESSPROGRESSPROGRESSPROGRESS
MANAGEMENTMANAGEMENTMANAGEMENTMANAGEMENT
TESTTESTTESTTEST
ORGANIZATIONORGANIZATIONORGANIZATIONORGANIZATION
PLANNINGPLANNINGPLANNINGPLANNING
ESTIMATIONESTIMATIONESTIMATIONESTIMATION
RISK ANALYSISRISK ANALYSISRISK ANALYSISRISK ANALYSIS
ANDANDANDAND
TEST STRATEGYTEST STRATEGYTEST STRATEGYTEST STRATEGY
Risk & Requirement Based Testing & TestFrame
How to reduce risk . . .
13
And still test effectively ?
Risk & Requirement Based Testing & TestFrame 14
Product risks versus requirements
Productrisks
Require-ments
Matching risks and requirements
Matching requirements and risks
A risk, no requirement:
• Add requirement (find faults earlier)
• Remove risk (don’t test more than necessary)
A requirement, no risk:
• Adjust risk list (better coverage of testing)
• Remove requirement (don’t develop more than necessary, no “frills”)
Risk & Requirement Based Testing & TestFrame
RRBT & Quality Attributes (ISO 9126)
FunctionalitySuitability
Accuracy
Interoperability
Security
Functionality compliance
15
ReliabilityMaturity
Fault tolerance
Recoverability
Reliability Compliance
PortabilityAdaptability
Installability
Co-existence
Replaceability
Portability compliance
UsabilityUnderstandability
Learnability
Operability
Attractiveness
Usability compliance
EfficiencyTime behaviour
Resource utilisation
Efficiency compliance
MaintainabilityAnalysability
Changeability
Stability
Testability
Maintainability compliance
40%40%40%40% 15%15%15%15%
10%10%10%10%
15%15%15%15%
20%20%20%20%
0%0%0%0%
Risk & Requirement Based Testing & TestFrame 16
Should test:• …• …• …• …• …• ...
Won’t test:• …• …
Could test:• …• …• …• …
Testing under pressure
Must test:• …• …• …• …
OUCH!!OUCH!!OUCH!!OUCH!!
Risk & Requirement Based Testing & TestFrame 17
Risk & Requirement Based Testing
Added value:
• Product risks: use the same language as stakeholders
• Unambiguous method for prioritizing product risks
• Detect gaps in requirements
• Good base for managing progress of test project
Risk & Requirement Based Testing & TestFrame 18
Agenda
• Everyday testing challenges
• Risk & Requirement Based Testing (RRBT)
• TestFrame
• Discussion
Risk & Requirement Based Testing & TestFrame 19
TestFrame
Risk & Requirement Based Testing & TestFrame 20
TestFrame
• LogicaCMG’s method for structured testing– Testing plays a role in quality
• Verifying test process and products
• Working in a structured way
– Testing takes effort
• Re-use defined products
• Optimal use of tools
– Testing is not a goal, it is used to reach a goal
• Use consciously and goal-oriented
• Risk analysis
– Testing has to fit with development method and organization
• Flexible and fitting
Risk & Requirement Based Testing & TestFrame 21
TestFrame model
Reusable Test ProductsReusable Test ProductsReusable Test ProductsReusable Test ProductsReusable Test ProductsReusable Test ProductsReusable Test ProductsReusable Test Products
QualityQualityQualityQualityQualityQualityQualityQuality--------totototototototo--------marketmarketmarketmarketmarketmarketmarketmarket
& Time& Time& Time& Time& Time& Time& Time& Time--------totototototototo--------marketmarketmarketmarketmarketmarketmarketmarket
SSSSTTTTRRRRUUUUCCCCTTTTUUUURRRRIIIINNNNGG
TTTTOOOOOOOOLLLLIIIINNNNGGGG
FFFFIIIITTTTTTTTIIIINNNNGGGG
Adjust testing to organization
(not vice versa)
Structured processStructured products
Use of tools for test analysis, test execution and test management
Risk & Requirement Based Testing & TestFrame 22
TestFrame phases
Preparation
Analysis Navigation
Execution
Preparation
Analysis Navigation
Execution
Risk & Requirement Based Testing & TestFrame 23
Assessment
Report
Risk
analysis
Plan of
ApproachTest plan
Technical
test
Engine
Navigation
scripts
Navigation
structure
Test
report
Evaluation
report
Handover
set
Findings
Scoping
Test
Conditions
Test
cases
Clusters
TestFrame products
Analysis
Preparation
Navigation
Execution
Risk & Requirement Based Testing & TestFrame
Test specification – Breakdown & automation
2424
Test reportsTest tool& Engine
Actionwords
Automated test scripts
System Under Test
Test cases
Test conditions
Test clusters
Risk & Requirement Based Testing & TestFrame 25
Èxample setup using Excel
Risk & Requirement Based Testing & TestFrame 26
Testing with action words (1)
• Identify the smallest functional test step
• Assign an action word to every step
• Add parameters
• Use the action words when defining test cases and test scenario’s
• When necessary: automate the action words
Risk & Requirement Based Testing & TestFrame 27
Test preparation and - specification
Test execution
Test cluster
A B C D…check balance Taylor 0 transfer Houston Taylor 210check balance Taylor 210transfer Smith Taylor 150check balance Taylor 360. . .
A B C D…check balance Taylor 0 transfer Houston Taylor 210check balance Taylor 210transfer Smith Taylor 150check balance Taylor 360. . .
Testing with action words (2)
Automated test execution
Navigation script
…check balancetransfer…
Risk & Requirement Based Testing & TestFrame 28
TestFrame: facts & advantages
• Easy accessible and well maintainable test products
• Early start possible and therefore time saving at the end of the project
• Effort “scarce” experts is minimized
• Better steering and control test execution process
• Test automation possible without adjustments to test documentation
• Test automation also possible when application is not 100% correct
• Easy reporting on test execution status and test results
Risk & Requirement Based Testing & TestFrame 29
Conclusion
Risk & Requirement Based Testing:Risk & Requirement Based Testing:Risk & Requirement Based Testing:Risk & Requirement Based Testing:
Make the right choicesMake the right choicesMake the right choicesMake the right choices
TestFrameTestFrameTestFrameTestFrame::::
Do it structuredDo it structuredDo it structuredDo it structured
“… for a list of all the ways technology has failed to improve the quality of life, please press 3.”
(Alice Kahn)
Risk & Requirement Based Testing & TestFrame 30
Agenda
• Everyday testing challenges
• Risk & Requirement Based Testing (RRBT)
• TestFrame
• Discussion
Risk & Requirement Based Testing & TestFrame 31
ne global team