Upload
doandang
View
215
Download
0
Embed Size (px)
Citation preview
1
www.nandbtesting.com © Newell & Budge, 2004
Better by Design
www.newellandbudge.com
Graham FreeburnNewell & Budge Testing Solutions
Tel: +44(0)1698 464 239Fax: +44(0)1698 464 250
www.nandbtesting.com © Newell & Budge, 2004
Main Course
• Learning from Engineering• Working with developers
Desert
The Menu
• Summary• Some caveats
Starter
• Bad Design, No Design…• A light bulb..
Aperitif • Introduction
2
www.nandbtesting.com © Newell & Budge, 2004
A Light Bulb..
• Testers only find failure..• Testing is (still) often too little, too late• How real are requirements?• Tester’s aren’t the only ones with
problems• A tale of two alarm clocks….
– Is good design (with tester involvement) a way forward?
www.nandbtesting.com © Newell & Budge, 2004
Bad design.. No Design..
• Why people don’t design• Some examples
– Software– Tale of Two Alarm Clocks
“You can use an eraser on the drafting table or a sledge hammer on the construction site.”
Frank Lloyd Wright
3
www.nandbtesting.com © Newell & Budge, 2004
Why people don’t design
• Old Days…– Get the &*!%*? Coded!– Fixed Architectures– Try it and see
• New World– Time Pressure– Development Tools– Requirements Code– Re-use– Packages / COTS
www.nandbtesting.com © Newell & Budge, 2004
Software
So many… so little time….
4
www.nandbtesting.com © Newell & Budge, 2004
• August 2004– Generic command and control
system– Originally for Fire Brigade– Modified for use by Coastguard– Crashed and reverted to orignal
design– “Turn left into High Street..”
• In a helicopter!
www.nandbtesting.com © Newell & Budge, 2004
Software
5
www.nandbtesting.com © Newell & Budge, 2004
Last month!
• UK Government Department
• Desktop upgrade• Caused 80,000 machines
(80% of total) to fail• No e-mail so officials had
to communicate by handwritten faxes
www.nandbtesting.com © Newell & Budge, 2004
Tale of Two Alarm Clocks – the sublime
AlarmOn/Off
Hour
Time set
Alarm set
Drowse
Minute
Source: The Power of Modern Testing,Les Hatton, Scottish Testing Group (STG), Oct 2003
6
www.nandbtesting.com © Newell & Budge, 2004
Tale of Two Alarm Clocks – the ridiculous
RandomNumbers
MoreButtons
Mode- ARGH!
Desk
Source: The Power of Modern Testing,Les Hatton, Scottish Testing Group (STG), Oct 2003
www.nandbtesting.com © Newell & Budge, 2004
Alarm Clock 2 – some examples
• Time setting– Press MODE button THREE times. Then press SELECT and SET
repeatedly• Alarm Setting
– Press MODE button TWO times. Then follow time setting.• Hourly chime on/off
– Press SELECT and MODE button simultaneously• Alarm on
– Press DATE and SELECT button simultaneously• Stop watch/lap counter
– Press MODE button ONCE and then beg for mercy as clock gibbers with entertaining series of random beeps
• Turning alarm off– Hurl out of window after standing at clock screaming.
Source: The Power of Modern Testing,Les Hatton, Scottish Testing Group (STG), Oct 2003
7
www.nandbtesting.com © Newell & Budge, 2004
Working with Developers
• Make the case for design• Get involved
– Designing for Testability– Testing the Design
• UML • Design Mapping• Quality Planning
– Early Test Design• Exploiting Bad Design
www.nandbtesting.com © Newell & Budge, 2004
Working with developers - designing for Testability
As applications become more complex, spreading over multi-tiers, across varying networks, and including different client configurations, application architecture testing will become increasingly mandatory
» GIGA GROUP
Transistors (1960s)
Design for Testabilitystandards (late 80s)
IntegratedCircuits (late 60s)
Large scaleICs (early 70s)
Assembly language (1960s)
UML (late 90s)
3GLs, 4GLs ( 70s & 80s )
Object Languages (early 90s)
Software
Design for Testabilitystandards (00s)
Source: Achieving eBusiness Quality by Design,Sam Guckenhiemer, Rational, Software Test Automation, 2001
8
www.nandbtesting.com © Newell & Budge, 2004
Working with developers - testing the design
UML
Use Case Diagrams
Class Diagrams
Packages & Object Diagrams
Component & Deployment Diagrams
Collaboration Diagrams
Statechart Diagrams Activity Diagrams
Sequence Diagrams
www.nandbtesting.com © Newell & Budge, 2004
Working with developers - testing the design
Design Mapping - “4C” compliance
Actor HierarchyDiagram
Sequence Diagram
Class/Type Diagram
Activity Diagram State Diagram
Use Case TextualDescription
Use Case Diagram Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
States should correspond State events should correspond with Sequence messages
1. Use Case steps should correspond with Sequence Diagram message2. Sequence begins in a state that satisfies the Use Case pre conditions3. Sequence ends in a state that satisfies the Use Case post conditions
Use Case Name is identicalDiagram Associations are described in DescriptionExtension PointscorrespondIncludes correspond
Use Cases correspond with Activities
Objects are identified by the same name
State trigger events areClass methods
Objects should correspond
Sequence messages are Class object methods
Actors are identified by the same name
1. States are identified by the same name2. Events, guard conditions and activities are identified the same
Actor HierarchyDiagram
Sequence Diagram
Class/Type Diagram
Activity Diagram State Diagram
Use Case TextualDescription
Use Case Diagram Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
Actors are identified by the same name
States should correspond State events should correspond with Sequence messages
1. Use Case steps should correspond with Sequence Diagram message2. Sequence begins in a state that satisfies the Use Case pre conditions3. Sequence ends in a state that satisfies the Use Case post conditions
Use Case Name is identicalDiagram Associations are described in DescriptionExtension PointscorrespondIncludes correspond
Use Cases correspond with Activities
Objects are identified by the same name
State trigger events areClass methods
Objects should correspond
Sequence messages are Class object methods
Actors are identified by the same name
1. States are identified by the same name2. Events, guard conditions and activities are identified the same
Correctness
Consistency Clarity
Completeness
Source: The Road to UML is paved with Good Intentions,Dion Johnson, StarEast 2002
9
www.nandbtesting.com © Newell & Budge, 2004
Working with developers - testing the design
Quality Planning
Opt A Opt B Opt C Opt D Opt E etc Attribute Impact Sum
Attribute 1
Attribute 2
Attribute 3, etc
60%+20%c=0.6
40%+ 10%c=0.2
Option Impact Sum
Costs
Benefit to cost ratio
Source: Tom Gilb, Principles of Software Engineering ManagementAddison Wesley, 1988
Attributes Options / Solutions
75%+ 10%c=0.8
www.nandbtesting.com © Newell & Budge, 2004
Working with developers - exploiting bad design
• Exploratory / Rapid Testing– Cem Kaner, James Bach– Simultaneous observation design
and execution of tests– Heuristics, Exploratory branches
• “Brain on, eyes open, test !”– 17 User Interface Attacks– 6 system Interface Attacks– 2 Software/OS Interface
Attack approaches
10
www.nandbtesting.com © Newell & Budge, 2004
Learning from Engineering
• What is good design?– Boeing 777
• Paperless design allowed simulated assembly of 777• Worked so well that only a nose mock-up was built
before assembly first flight vehicle
• Building Bridges• Design of experiments (DOE)• Information Feedback• The Importance of Context
www.nandbtesting.com © Newell & Budge, 2004
Learning from Engineering- what is good design?
• Useful– It works well and functions as promised. It does what
it is expected to and satisfies a minimum or appropriate level of performance
• Usable– It has appropriate ergonomics and user interface,
considering how, where, how often and who will be using it.
• Desirable• Producible• Profitable• Differentiated
http://www.betterproductdesign.net/index.htm
11
www.nandbtesting.com © Newell & Budge, 2004
A bridge to nowhere..
Spiral ramp of the larger 'bridge to nowhere', with a stub end where it should have been extended to the other (there were two!).
This end of the bridge has no guardrails at all, and the base of the spiral is protected by a high spiked fence.
The other end does have guardrails, but finishes abruptly with a 60 foot drop.
- M8 Motorway, Charing Cross district of Glasgow
www.nandbtesting.com © Newell & Budge, 2004
Learning from Engineering- design of experiments (DOE)
• DOE have been applied in Quality control in many engineering fields for several decades
• Aim : to maximise the amount of information gained in an experiment by optimising the combinations of independent variables
• In Software Testing Terms – Combination or Combinatorial Testing– E.g. “All Pairs”
Source: An Investigation of the Applicability of the Design of Experiments to Software Testing, D. Richard Kuhn & Michael J. Reilly, Proceedings, 27th NASA/IEEE Software Engineering Workshop, December 2002
Table 1.Number and percent of faults triggered by n-way Conditions
Conditions Browser Server Modules(values on n) (194 bugs) (171 bugs)
(percent) (cumulative (percent) (cumulativepercent) percent)
1 28.6 28.6 41.7 41.7
2 47.5 76.1 28.6 70.3
3 18.9 95.0 19.0 89.3
4 2.2 97.2 7.1 96.4
5 2.2 99.4 0.0 96.4
6 0.6 100.0 3.6 100.0
12
www.nandbtesting.com © Newell & Budge, 2004
Learning from Engineering - Information Feedback (design to operation)
Source: Mapping Function to Failure Mode during Component Development, Irem Y Tmer and Robert B. StoneResearch in Engineering Design 14 (2003)
www.nandbtesting.com © Newell & Budge, 2004
Learning from Engineering- context is everything
Source: James Bach, www.satisfice.com, © Satisfice Inc. 2003
Test TeamExpertiseLoadingCohesionMotivationLeadershipProject Integration
Test LabTest PlatformsTest ToolsTest LibraryProblem Tracking SystemOffice Facilities
Product MissionStakeholdersQuality CriteriaReference Material
RequirementsProductProject LifecycleProject ManagementConfiguration ManagementDefect PreventionDevelopment Team
Development
TestProcess
StrategyLogisticsWork-products
MissionFind Important ProblemsAssess QualityCertify to StandardFulfill Process MandatesSatisfy StakeholdersAssure Accountability
Advise about QAAdvise about TestingAdvise about QualityMaximize EfficiencyMinimize CostMinimize Time
13
www.nandbtesting.com © Newell & Budge, 2004
Some caveats• The need for good
requirements analysis isn’t diminished
• Economics– Practicality of design
proving– Cost of Quality C
osts
Quality Level
Failure Costs
Total Quality Costs
Costs forprevention
and detection
Minimum
Source: Juran’s Quality Control Handbook, 1988ISBN 0-070-33176-6
www.nandbtesting.com © Newell & Budge, 2004
Summary
• Get involved in design – For everyone’s benefit
• Consider using– Design Mapping– Use Cases, Scenario testing…
• They are powerful weapons in the war against faults
• Learn to exploit bad design– Rapid Testing / Exploratory Testing
• Get in early!