Upload
cosmic-common-software-measurement-international-consortium
View
84
Download
1
Tags:
Embed Size (px)
DESCRIPTION
IWSM Presentation
Citation preview
1
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Defect Density Measurements Using COSMIC
Thomas M. Fehlmann, ZürichEberhard Kranich, DuisburgEuro Project Office AGE: [email protected] H: www.e-p-o.com
2
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Dr. Thomas Fehlmann
1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD
Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG
1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD
Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG
Dr. Thomas Fehlmann
3
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Working at T-Systems International GmbH in Bonn, Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board
Eberhard KranichEberhard Kranich
Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Worked at T-Systems International GmbH in Bonn,
Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board
4
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
What is a Defect?
Defect = Behavior impacting expected or required functionality of software How many bugs? By counting the
size of defect repositories? By number of entries???
5
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Software Testing as a Game
Tester sees selected sequences in the UML sequence diagram
Tester can “walk” the data movements when planning or executing tests Functionality becomes visible to
the agile team Defects impacting functionality
become visible to testers
FunctionalProcess
OtherApplication
SomeDevice
8.// Move some data
9.// Move some data
10.// Move some data
11.// Move some data
OtherDevice
6
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Functionality, Defect Size, and Defect Density
What happens if data movements have defects?
Testers mark the data movement where a defect has been detected
Same Metric: ISO/IEC 19761 COSMIC
FunctionalProcess
OtherApplication
SomeDevice
8.// Move some data
Move some data
10.// Move some data
11.// Move some data
OtherDevice
Functional Size Number of Data Movements needed to implement all FUR
Test Size Number of Data Movements executed in Tests
Test Story Collection of Test Cases aiming at certain FURs
Defect Count Number of Data Movements affected by some defect detected in a test story
7
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Defects Density Prediction?
Now he counts the defects! And counts and adjusts test size By ISO/IEC 19761 COSMIC
FunctionalProcess
OtherApplication
SomeDevice
8.// Move some data
9.// Move some data
10.// Move some data
11.// Move some data
OtherDevice
How does he knowthat he found
all the defects?
8
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Research Questions
What is Defect Density? Defects per KDLOC?
What is Test Coverage? Code lines executed by some test case?
WHAT IF YOU DON’T HAVE ANY CODE?WHAT IF
USING SERVICES?
9
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
SW Testing and SW Metrics
Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”?
Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report?
A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!
10
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
SW Testing and SW Metrics
Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”?
Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report?
A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!
Revie
w C
om
ments
11
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Learn & Remember
Software Development the Six Sigma Way
Use COSMICISO/IEC 19761
Benchmark!
Count Defects basedon Reference Model
12
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit]
1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred destinations
they are valid for the Ticket Shop
I no longer have to pay fees when catched without tickets
2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether it's right
3) Q003 Process Payment Provider of transportation services
give user access to their preferred payment options
all payments are traceable in Ticket Shop
they can manage spending
4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees when catched without tickets
5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and travel range
I don't need to go into a dispute with a client
Functional User Requirements
Example: The Ticket Apps
Customer’s Voice: Give me a ticket subito!
Without much ado and questionsif I simply want to go home
13
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
The Ticket Apps
12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP
App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Local Time
1.// Add New Destination
Traveler
2.// Check whether Destination Exists
3.// Collect Matching Destinations
4.// Show Matching Destinations
5.// Select Exact Destination
6.// Record Selected Destinations
7.// Ask App for a Ticket
8.// Propose Destinations
9.// Proposed Destinations
10.// Select Destination
Prepare
11.// GPS Coordinates
12.// Date & Time
13.// Ask GIS
14.// Nearest Boarding Station
15.// Search Connection
16.// Next Connection
17.// Request Ticket
18.// Send Pricing Proposal
19.// Proposed Price
20.// Confirm Price
21.// Confirm Transaction
22.// Issue Ticket
23.// Show Ticket
24.// Record Selected Destination
14
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
The Ticket Apps – Objects of Interest
Description Type
Wants to buy a ticket to ride Device User
Store a list of destinations that are valid for "getting home" Functional Process
List of standard stations eligible as travel destination Persistent Data
Finds nearest origin station Other Application
Buy the ticket needed for that route Functional Process
External ticket shop able to identify the App’s user Other Application
Finds connections between stations at a given time & day Other Application
Provides GPS coordinates where phone is located Device User
Synchronized time service located in smartphone Device User
App User
Home Destinations
Ticket Purchase
Ticket Shop
Phone's GPS
Timetable Service
Saved Stations
GIS Application
Local Time
15
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
The Ticket Apps – Prepare Destinations
App User Ticket Shop
1.// Add New Destination
Prepare
2.// Check whether Destination Exists
3.// Collect Matching Destinations
4.// Show Matching Destinations
5.// Select Exact Destination
Home Destinations Saved Stations
6.// Record Selected Destination
Preparation: Store standard destination stations on Smartphone After checking with the Ticket Shop whether they exist or which stop to
select Involves Ticket Shop and local data ‘Standard Stations”
16
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
The Ticket App in the IFPUG Model
Boundary IFP=51
EI
EOEQ
ILF
EIF
EQ1 / 2
Select DestinationEI
1 / 3
Enter New Destination
EI2 / 2
Request Ticket
EI2 / 1
Confirm Purchase
EIF2 / 12
Ticket Shop
EIF1 / 3
GIS Application
EIF1 / 5
Timetable Service
EIF1 / 1
Local Time
EO2 / 2
Boarding StationEO3 / 3
Mobile TicketEO1 / 2
Show Matching Destinations
ILF1 / 2
Home Destinations
Less complicated than UML Sequence Diagrams Suits business better Better boundary
identification between different layers
17
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Prepare Destination in the IFPUG Model
It is not obvious what happens Two elementary process
seem unrelated Nevertheless, they update
an ILF [Uh, they forgot the
elementary process for deleting obsolete home destinations!]
Boundary IFP=19
EI
EOEQ
ILF
EIF
ILF1 / 2
Home Destinations
EO1 / 2
Show Matching Destinations
EI1 / 3
Enter New Destination
EIF2 / 12
Ticket Shop
18
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Buy Ticket in the IFPUG Model
Elementary processes stay The ILF ‘Home
Destinations’ becomes an EIF
Adding the two counts yields
IFPUG is useless for functional sizing when developing Apps or counting defects
IFPUG 4.3 is great for identifying system boundaries splitting apps!
Boundary IFP=42
EI
EOEQ
ILF
EIF
EIF2 / 12
Ticket Shop
EIF1 / 3
GIS Application
EIF1 / 5
Timetable Service
EIF1 / 1
Local Time
EIF1 / 2
Home Destinations
EI2 / 2
Request Ticket
EI1 / 1
Confirm Purchase
EO2 / 2
Boarding StationEO2 / 3
Mobile Ticket
EQ1 / 2
Select Destination
19
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Requirements for a Defect Measurement Reference Model
It must be additive The total size must be the sum of the components’ sizes.
It must be understandable Support UML Sequence Diagrams Works well with UML Use Cases Also with Business Process Model & Notation (BPMN) 2.0
It must fit into agile delivery Buglione-Trudel Matrix for managing agile projects Story Cards for Sprints
Story Card for Prepare Destinations Test isReady
Draft isReady
ReviewDone
Final-ized
Appro-ved
Func-tional
R001-01: Prepare Destinations6 x reworked
RefactoringCount:
5
0
0
Business Impact:
Functional Size:
Story Points:
App User Prepare Destinations Standard Stations Ticket Shop
1.// Add New Destination
2.// Check whether Destination Exists
3.// Collect Matching Destinations
4.// Show Matching Destinations
5.// Select Exact Destination
6.// Record Selected Destinations
The user prepares his ticket app by entering standarddestinations. The ticket shop is involved for resolvingstation names matches and avoid spelling errors
20
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Learn & Remember
Software Development the Six Sigma Way
Use COSMICISO/IEC 19761
Benchmark!
Count Defects basedon Reference Model
21
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
What is a Test?
A Software Test has Several Test Stories
• Weighted by Customer’s Priority for the Test Story, reflecting the value for the customer
Each Test Story has many Test Cases• For various kind of test data
A Test Size attribute • Number of data movements executed by test cases
A Test Coverage attribute • Percentage of data movements covered with test cases
An Outcome• Passed or Failed
– Passed: All responses according expectations– Failed: at least one test case didn’t yield the expected response
22
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Sample Test Stories
Eight Test Stories for the Ticket Apps Each Test Story has several Test Cases Each Test Case has defined Test Data The Expected Response is known as per Test Case
Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response
CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station
CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station
CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected
CT-B Ticketing CT-B.1 Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop
CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination
CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations
CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given
CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning
23
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
What is a Test Case?
A Test Cases has Entry Data (“Test Data”)
• Explaining the environment for the test case• Typically valid, invalid, borderline data• Normal and disturbed communication services
A known Expected Response• The response of the system is known in advance
A known sequence of data movements executed• Defining Test Coverage• Each Test Case has a Size
24
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
The sequence of data movements can be visualized in the sequence diagram per test case
By double-clicking therespectivedata movements
Find Route GIS Application Phone's GPS
8.// GPS Coordinates
9.// Date & Time
10.// Ask GIS
11.// Nearest Boarding Station
Local Time
What is a Test Case?
Test Story No. 1
User StoriesCT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected ResponseCT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station
CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station
CT-A.1.3 Enter malformatted GPS E006 Returns no station
CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning
CT-A.1.5 GPS not working E005,E006 E006 Returns warning
Test Story Contribution (CFP): 2 6 0 1 6 Test Size
Test Case Measurementsfor Test Story CT-A.1
25
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Defect Reporting
When a defect has been identified in a test case, it can be recorded by double-clicking the suspect sequence of data movements
User StoriesCT-B.1 Select Destination Q001: Propose Standard Destinations
CT-B.1.1 Station of origin known R001,X001,E005
CT-B.1.2 No destination selected R001,X001
CT-B.1.3 Valid boarding & destination W001,E005,R001,X001CT-B.1.4 List of destinations X003,R001,W001
Test Story Contribution (CFP): 12
Test Case Measurementsfor Test Story CT-B.1
Test Story No. 4
Expected Response CFP Name Label Description Name Label Asks for destination 6
Stops without contacting Ticket Shop 2
Returns next available connection 15All readable and visible 3 #003 Multiple References If a station contains more than one transport
mode, e.g., bus and train, both are recorded as separate destinations
W001 Record Selected Destinations
Test Size 26 1
Defects Observed Data Movements Affected
Defect Count
26
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP
App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service
1.// Add New Destination
Traveler
2.// Check whether Destination Exists
3.// Collect Matching Destinations
4.// Show Matching Destinations
5.// Select Exact Destination
6.// Record Selected Destinations
7.// Ask App for a Ticket
8.// Propose Destinations
9.// Proposed Destinations
10.// Select Destination
Prepare
11.// GPS Coordinates
12.// Date & Time
13.// Ask GIS
14.// Nearest Boarding Station
Extract Report
Select Data Movements forTest Case CT-B.1.4: List of destinationswhen executed in view of Q001: Propose Standard DestinationsThis Test Case identifies Fault #003: Multiple Referencesaffecting Data Movement 6.// Record Selected Destinations
Finish
Record Defects
27
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Count Defects
You can count One Defect
Per Data Movement
Identified by One Test Story
And never more…
©2012-2014 Rhafiel
28
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Compare
This means The more test stories you have, the more defects you can uncover Test size is equally important as functional size Defect density depends from test size
Test Benchmarking
Test Status Summary
Total CFP: 24Defects Pending for Removal: 3 Test Size in CFP: 185
Defects Found in Total: 3 Test Intensity in CFP: 7.7Defect Density: 13% Test Coverage: 79%
29
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Test Stories
Goa
l Tes
t Cov
erag
e
Find
Nea
rest
Sta
tion
Serv
ed S
tatio
ns o
nly
Ente
r New
Des
tinat
ion
Sele
ct D
estin
atio
n
Get
Tic
ket
Pric
e C
alcu
latio
n
Issu
e Ti
cket
Paym
ent T
ests
Ach
ieve
d C
over
age
CT-
A.1
CT-
A.2
CT-
A.3
CT-
B.1
CT-
B.2
CT-
B.3
CT-
B.4
CT-
B.5
Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55
Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45
Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59
Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23
Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31
Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap
0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.060.10 Convergence Range
0.20 Convergence Limit
Test StoriesDeployment Combinator
User Stories
Toggle Measured Controls
Measuring Test Coverage
AchievedResponse
Expected Response Number of
data movementsexecuted
30
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Learn & Remember
Software Development the Six Sigma Way
Use COSMICISO/IEC 19761
Benchmark!
Count Defects basedon Reference Model
31
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Conclusions
You need more than one model when measuring modern software ISO/IEC 19761 COSMIC is most appropriate for measuring defects Other methods might be suitable for cost estimations
However, in these days, what drives cost more than faulty software?
Stop the nonsense with counting defects in repositories Include defect density goals in contracts Do independent auditing as long as you don’t trust your measurement
equipment
Start Measuring Defects. Now.
32
CustomerOrientation
Lean Six Sigma
AgileProcesses
ProjectEstimations
TransferFunctions
Questions?