8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
1/89
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
2/89
SEI Webinar (28 Apr 09)
Speaker Biographies
Robert W. Stoddard currently serves as a Senior Member of the Technical Staff atthe Software Engineering Institute. Robert architected and designed several leading
measurement and CMMI High Maturity courses including: “Understanding CMMI
” “ ” , ,
“Designing Products and Processes using Six Sigma.”
us y oung curren y serves as e ppra sa anager a e . us y as +years S&SW development, management, and consulting He has worked in large and
small organizations; government and private industry. His recent focus has been
CMMI High Maturity.
Dave Zubrow currently serves as the Manager of the Software Engineering
easurement an na ys s n t at ve w t n t e o tware ng neer ngInstitute (SEI). Prior to joining the SEI, Dave served as Assistant Director of
Analytic Studies for Carnegie Mellon University. He is a SEI certified instructor
and appraiser, member of several editorial boards of professional journals, and
2Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
act ve n stan ar s eve opment.
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
3/89
NO WARRANTY
MATERIAL IS FURNISHED ON AN “ AS-IS" BASIS. CARNEGIE MELLON UNIVERSITYMAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO
ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FORPURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM
.
WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT,TRADEMARK, OR COPYRIGHT INFRINGEMENT.
Use of any trademarks in this presentation is not intended in any way to infringe on the.
This Presentation may be reproduced in its entirety, without modification, and freelydistributed in written or electronic form without requesting formal permission. Permission is
required for any other use. Requests for permission should be directed to the SoftwareEngineering Institute at [email protected].
This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the SoftwareEngineering Institute, a federally funded research and development center. TheGovernment of the United States has a royalty-free government-purpose license to use,
, , ,permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
4/89
SEI Webinar (28 Apr 09)
Permission to use Crystal Ball Screen Shots
Portions of the input and output contained in this modulemanual are rinted with ermission of Oracle formerl
Decisioneering). Crystal Ball 7.2.2 (Build 7.2.1333.0) is usedto capture screenshots in this module
The Web page for Crystal Ball is available athttp://www.crystalball.com
4Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
5/89
SEI Webinar (28 Apr 09)
Topics
Introduction
Review of Process Performance Models (PPMs)
Technical Process of Building PPMs
Questions
5Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
6/89
Review of CMMI
Models (PPMs)
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
7/89
SEI Webinar (28 Apr 09)
What is a PPM?
OPP SP 1.5
• PPMs are used to estimate or redict the value of a rocess-
performance measure from the values of other process, product,and service measurements
• PPMs t icall use rocess and roduct measurements collectedthroughout the life of the project to estimate progress towardachieving objectives that cannot be measured until later in the
project’s life
Glossary
• A description of the relationships among attributes of a process andits work roducts that is develo ed from historical rocess-
performance data and calibrated using collected process andproduct measures from the project and that is used to predict resultsto be achieved by following a process
7Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
8/89
SEI Webinar (28 Apr 09)
Purpose and Usage of Process Performance
Software Coding Software Unit Testing
Design Systems
Testing
Requirements
Requirements
Management
Integration Testing
Customer
Project
Forecasting
cceptance
Testing
ProjectStart Project
Project
Planning
8Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
9/89
SEI Webinar (28 Apr 09)
Healthy Ingredients of CMMI Process
Statistical, probabilistic or simulation in nature
Predict interim and/or final project outcomesUse controllable factors tied to sub-processes to conduct the prediction
Model the variation of factors and understand the predicted range or
variation of the outcomes
Enable “what-if” anal sis for ro ect lannin d namic re- lannin and
problem resolution during project execution
Connect “upstream” activity with “downstream” activity
Enable projects to achieve mid-course corrections to ensure projectsuccess
9Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
10/89
SEI Webinar (28 Apr 09)
Interactive Question #1
Do you now feel comfortable with your knowledge of thehealth in redients of a CMMI Process Performance Model
that were just presented?
es
10Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
11/89
Technical Process of
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
12/89
SEI Webinar (28 Apr 09)
Topics• ev ew o rocess er ormance o e s s
• Technical Process of Building PPMs
1. Identify or Reconfirm Business Goals
2. Identify the sub-processes/process
’.
4. Identify Controllable factors (x’s) to predict outcomes
5. Include Uncontrollable x factors
6. Collect Data
7. Assess Data Quality and Integrity
8. Identify data types of all y outcomes and x factors
9. Create PPBs
10.Select the proper analytical technique and/or type of regression equation
11.Create Predictions with both Confidence and Prediction Intervals
. -
13.Take Action Based on PPM Predictions
14.Maintain PPMs including calibration and reconfirming relationships
15.Use PPMs to assist in CAR and OID
• Questions
12Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
13/89
SEI Webinar (28 Apr 09)
1) Business Goal Flowdown (Y-to-x)
YYY Y High Level Business Goals(Balanced Scorecard)
g n o s t i c
yyy y y yy Subordinate Business Goals(e.g., $ Buckets,
o c e s s
- A
yyyy y y yy er ormance
High Level Process
P r
e d
xxx x x xx
(e.g., Organizational Processes)
Subordinate s - O r i e n
xxxx x x xxProcesses
(e.g., Down to a Vital xsub-process to be P
r o c e
13Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
14/89
SEI Webinar (28 Apr 09)
2) Identify the Sub-Process/Process
• Start with the Organization’s Business Objectives
•
Objectives (QPPOs)• For the QPPOs that can be Measured Quantitatively
• Perform Analysis to Determine which Sub-Process/Process Drivesthe Relevant Objective
• Determine if Sufficient Data is Available or can be Obtained toEstablish a Process Performance Baseline(s) and/or Build aProcess Performance Model(s)
14Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
15/89
SEI Webinar (28 Apr 09)
Identify the Sub-Process/Process
• Given Organizational Business Objectives:
• Im rove ualit
• Improve cycle time• Improve productivity
• rans a e o measurea e s
• Post-delivery defect density of less than 0.5 Defects/KSLOC
• Achieve 85% defect detection before System testing
• Ensure requirements duration is within 15% of plan
• Achieve a 5 % software productivity improvement
15Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
16/89
SEI Webinar (28 Apr 09)
Identify the Sub-Process/Process
• The QPPOs after analysis were determined to be driven bythe followin rocesses
• Peer Review• Test
• The processes were then decomposed to the followingrelated sub-processes to be statistically managed
• Inspection
•
• Requirements Analysis
16Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
17/89
SEI Webinar (28 Apr 09)
3) Identify Outcomes to Predict
DesignTransitionto Customer
Defectsinjected
Defectsinjected
SoftwareRequirements
Defectsinjected
Fielded
Defectsinjected
Implementation Integration
DesignReview
CodeReview
Defectsremoved
DefectsremovedReq.Review
Defectsremoved
Defectsremoved
Defectsremoved
Defectsremoved
Across business functions and disciplines!
Outcomes relatedto key handoffs of work durin ro ect
Outcomes relatedto interim milestonesand mgt reviews
Outcomes relatedto risks during the
to project completion
17Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
18/89
SEI Webinar (28 Apr 09)
Examples of Outcomes
Rework
Progress*
18Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
19/89
SEI Webinar (28 Apr 09)
4) Identify Controllable factors (x’s) to Predict
-
“ ”
influence over the factor prior to or during the projectexecution
A common misconception is that factors are not controllable.Requires out-of-the-box thinking to overcome this. Someorganizations employ individuals known as “assumption
”
19Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
20/89
SEI Webinar (28 Apr 09)
Identify Controllable factors (x’s) to Predict
-
As we view process holistically, controllable factors may bere ate , ut not m te , to any o t e o ow ng:
• People attributes
• Environmental factors
• Technology factors
• Tools (physical or software)
• Customers
• Suppliers
• Other Stakeholders
20Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
21/89
SEI Webinar (28 Apr 09)
5) Identify Uncontrollable Factors
• Normally these are constraints placed by the customer orconcrete terms of a contract or government regulation
• Can also be factors for which the project team truly has no
• Can be factors that are unchan in for a iven ro ect butcan be changed for future projects
• Often includes external factors or factors related to otherteams outside of the project
21
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
22/89
SEI Webinar (28 Apr 09)
Interactive Question #2
Of the steps presented so far, which step do you believewould be the most challen in for our or anization?
1) Business Goal formulation to drive PPMs
2) Identification of critical subprocesses supporting goals
3) Identify outcomes to predict with PPMs en y con ro a e ac ors o ma e pre c ons w
5) Identify uncontrollable factors to make predictions with
22
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
23/89
SEI Webinar (28 Apr 09)
6) Measurement and Analysis in Action
5 Decision-making
Corrective actionsto improve process
1
Data collection
4 Data & interpretationsreported
2Data stored
400
500
600
700
800
,interpreted, & stored
MeasurementReport
PrototypeRepository
DataRepository
0
100
200
300
1997 1998 1999 2000 2001E
23
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
easuremen s co ec e
23David Zubrow , October 2008© 2008 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
24/89
SEI Webinar (28 Apr 09)
Objective
Indicator Name/TitleDate
EstablishMeasurement
Documenting Measurement
Questions
Visual Display
80
100
Communicate
ect ves
Data Storage
, ,Measures
20
40
60
Perspective
Results
Algorithm
WhereHow
Security
Store Data &
Results
Data Elements
Definitions
Data Collection Specify
pec yMeasures
Interpretation
Assumptions
Probing Questions
Specify Analysis
Procedures
Form(s)
When/How Often
By Whom CollectData
DataCollectionProcedures Evolution
Analysis
Feedback Guidelines
Data
Responsibilityfor Reporting
Data Reporting
By/To Whom
How Often
CommunicateResults
-re erence
24
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
kaz6
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
25/89
Slide 24
kaz6 This slide seems out of place to me ... since the slides that follow cover some of the things noted in the template. the template.
Quite honestly, I would remove the slide since it is only repeating some of the criteria that is already being listed.Mark Kasunic, 10/7/2008
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
26/89
SEI Webinar (28 Apr 09)
7) Cost of Poor Data Quality to an Enterprise –
Typical Issues
• Inaccurate data [1-5% of data fields are erred]
• Inconsistencies across databases
•
Typical Impacts
• Lowered customer
satisfaction
• Increased cost
• Poorer decision making &
decisions take longer
• More difficult to im lement
• More difficult to set strategy
• More difficult to execute strategy
• Contribute to issues of data
• Lowered employeesatisfaction
data warehouses• More difficult to engineer
• Increased organizational
ownership• Compromise ability to align
organization
25
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
•ource: e man,
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
27/89
SEI Webinar (28 Apr 09)
Impacts of Poor Data Quality
Inability to• manage the quality and performance of software or
•
Estimate and plan realisticallyIneffective
• process change instead of process improvement• and inefficient testing causing issues with time to
market, field quality and development costsProducts that are painful and costly to use within real-lifeusage profiles
Bad Information leading to Bad Decisions
26
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
28/89
SEI Webinar (28 Apr 09)
Where do Measurement Errors come From1
Data Entry Errors
– Manual data entr
– Lack of integrity checks
Differing Operational Definitions
– , , ,
completion
Not a priority for those generating or collecting data
– omp e e e e or me s ee a e en o e mon
– Inaccurate measurement at the source
Double Duty
– Effort data collection is for Accounting not Project Management• Overtime is not tracked
• Effort is tracked only to highest level of WBS
27
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
29/89
SEI Webinar (28 Apr 09)
8) Types of Data
Examples
Defect types
Labor types
Categorical data where the order of thecategories is arbitraryNominal
Attribute(aka categorized or
anguages
A B C
Examples
Severity levelsNominal data with an ordering; may
have une ual intervalsSurvey choices 1-5
Experience categories< <Increasinginformation
content
ExamplesContinuous data that has equal intervals;
Continuous(aka variables
Ratio
Defect densities
Labor rates
Productivity
Variance %’s0
A B
may have decimal values
Interval data setthat also has
28
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
data) Code size SLOC
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
30/89
SEI Webinar (28 Apr 09)
Appropriate Analysis: Types of Hypothesis Tests
29
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
31/89
SEI Webinar (28 Apr 09)
9) Creating Process Performance Baselines
• Definition: A Process Performance Baselines (PPB) is adocumented characterization of the actual results achieved byfollowin a rocess
• Therefore a PPB needs to reflect actual project performance• CMMI-DEV OPP PA informative material:
• Establish a quantitative understanding of the performance of theorganization’s set of standard processes in support of objectives
• Select the processes that summarize the actual performance ofprocesses in projects in the organization
• Alternatively Practical Software and Systems MeasurementPSM recommends an or anization follow three basic ste s:
• Identify organization needs
• Select appropriate measures
30
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
• n egra e measuremen n o e process
SEI W bi (28 A 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
32/89
SEI Webinar (28 Apr 09)
Creating Process Performance Baselines
• We only need one baseline
•
• One data point constitutes a baseline• We can’t use the baseline until it is stable
• If the initial baseline is unstable we just remove the datapoints outside of the control limits and recompute the control
31
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI W bi (28 A 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
33/89
SEI Webinar (28 Apr 09)
10) Select the Proper Analytical Model
Statistical Modeling and Regression Equations
Monte Carlo Simulation
Probabilistic Modeling including Bayesian Belief Networks
Discrete Event Process Simulation
er vance o e ng ec n quesMarkov, Petri-net, Neural Nets, Systems Dynamics
32
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
34/89
SEI Webinar (28 Apr 09)
Statistical Regression Analysis
Y
ANOVA Chi-Square,
e t e
an ummy
Variable
og
Logistic D i s c
egress on egress on
Correlation i n u o u
& LinearRegression
Regression C o n
33
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
35/89
SEI Webinar (28 Apr 09)
Why Use Monte Carlo Simulation?
Use Monte Carlo simulation to do the following:
•
values instead of single value)
• Enable more accurate sensitivity analysis
•
(e.g., more realistic)
•
Aid buy-in and acceptance of modeling because user-provided
• Provide a basis for confidence in a model output (e.g., supports risk
management)
•
Increase the usefulness of the model in predicting outcomes
34
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
36/89
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
37/89
SEI Webinar (28 Apr 09)
Example: Adding Reality to Schedules-1
Process Durations
Step Best Expected Worst
2 45 50 1253 72 80 200
4 45 50 125
5 81 90 225
6 23 25 63
7 32 35 88
8 41 45 113
10 23 25 63
500
What would you forecast theschedule duration to be?
36
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
38/89
( p )
Adding Reality to Schedules-2
With 90% confidence,The ro ect is almost
under 817 daysduration.
guaranteed to missthe 500 days duration
100% of the time.
37
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
39/89
( p )
Adding Reality to Schedules-3
With only 50%conf idence, the projectwil l be under 731 daysduration.
38
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
40/89
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
41/89
Create Predictions with Both Confidence and
-
Confidence Intervals: The statistical range of behavior of aan average va ue compu e rom a samp e o u ure a a
points
Prediction Intervals: The statistical range of behavior ofindividual future data points
Note: Prediction Intervals are almost always much wider thanconfidence intervals because averages don’t experience the wide
swings that individual data points can experience (similar to howindividual grades in college compared to your grade point average)
40
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
42/89
Interactive Question #3
Based on what you now know, which analytical modelingtechni ue do ou believe would be most ractical and usefulin your organization?
tat st ca regress on equat ons
2) Monte Carlo simulation
4) Discrete Event Simulation
5 Other
41
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
12) St ti t i ll M S b /PPM
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
43/89
12) Statistically Manage Subprocesses w/PPMs
Review performance and capabili ty ofstatistically managed subprocesses for
progress to achieving objectives
QPM SP 2.2
QPM SP 2.3
OnDetermine actions needed totrack
Yes
No
Meet interimobjectives
address deficiencies such as change objectives
CAR (improve processreduce common cause,change mean etc.)
ado t new rocess ando
Suppliers on
technology(OID)
Revise PPM ,predict ifchanged wil l meet objectives
Yes
Yes
Updated PPM prediction
Yes
oMeetobjectives
No
Note: This isnot meant to be
anim lementat ion
trackYes
No
Identify and manage issuesand risks
YesTerminate No
End of
objectives
No
flowchart
42
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
44/89
13) Take Action Based on Results of PPM
If a PPM model predicts an unacceptable range of values,
more desirable range of outcome
Once a PPM model predicts a range of values for aparticular outcome, then actual values can be compared to
. ,be treated similarly to a point on a control chart fallingoutside of the control limits
Use PPM predictions to help inform process composition
43
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
45/89
What is Sub-optimization and how can PPMs
Sub-optimization is where one parameter is optimized at theex ense of other s
• Reduce delivered defects, but are late and over budget
• Meet the cost goal but don’t deliver desired functionality
• Gage the trade-offs amongst multiple goals
• Gage the effects of changes to multiple parameters
44
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
46/89
14) Validating and Maintaining PPMs
Initial estimation of a PPM typically yields
independent variables (x’s) and the dependent variable (y)
• An indication of the goodness-of-fit of the model to the data (e.g.,- -,
These do not necessarily indicate whether the modelprov es su c en prac ca va ue
• Track and compare predictions with actual results
• Failure to meet business criteria e. ., +/- 10% indicates need to
recalibrate (i.e, same variables with different data) or remodel (newvariables and data)
45
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
47/89
15) How PPMs Assist CAR
• Aid impact, benefit, and ROI predictions for
• Selectin defects for anal sis
• Selecting action proposals for implementation
• Use PPMs to identify potential sources of the problem or
• Use PPMs to understand the interactions among selected
im rovements; and the combined redicted im acts, costs,and benefits of the improvements (considered as a set)
• Compare the result versus the original PPM-based prediction
46
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
*The result of using a PPM to make a predict ion often takes
the form of a prediction interval as well as a single point .
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
48/89
How PPMs Assist OID
• Select process improvement proposals for implementationb aidin im act, benefit, and ROI redictions
• Identify opportunities for improvement
• Use PPMs to understand the interactions among selectedmprovements; an t e com ne pre cte mpacts, costs,and benefits of the improvements (considered as a set)
• • Confirm the prediction (provides input to maintaining PPMs)
47
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Q #
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
49/89
Interactive Question #4
Based on the information you saw in this mini-tutorial, wouldou be interested in attendin a full da tutorial on this
subject?
es
2) No
48
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
50/89
Questions?49
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Contact Information
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
51/89
Kevin Schaaff U.S. mail:
Email: [email protected]
Robert W. Stoddard
Email: [email protected]
Software Engineering Institute
Customer Relations
Rawdon Young
Email: [email protected]
Dave Zubrow
Pittsburgh, PA 15213-2612
USA
Email: [email protected]
World Wide Web: Customer Relationswww.sei.cmu.edu
www.sei.cmu.edu/contact.html
Email: [email protected]
Telephone: +1 412-268-5800. . .
sema/presentations.html SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257
50
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
52/89
Backup Slides
© 2009 Carnegie Mellon University
All Models (Qualitative and Quantitative)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
53/89
, ,
Statist ical or Probabilist ic Models
Anecdotal
Controllable x factors involved
samples
No
uncertaintyor variationmodeledOnly final
QQual
Model -With controllable x
factors tied to
outcomesaremodeledOnly
uncontrollable
factors are
Processes and/orSub-processes
modeledOnly phasesor lifecyclesare modeled
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
How are PPM Used? (OPP SP 1 5)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
54/89
How are PPM Used? (OPP SP 1.5)
The PPMs are used as follows:
• The or anization uses them for estimatin anal zin and redictin the process performance associated with the processes in the
organization’s set of standard processes• The or anization uses them to assess the otential return on
investment for process improvement activities
• Projects use them for estimating, analyzing, and predicting the
process performance for their defined processes• Projects use them for selecting processes or subprocesses for use
•
more information about the use of PPMs
53
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
How are PPMs Used? (QPM)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
55/89
How are PPMs Used? (QPM)
SP 1.4
• PPMs calibrated with obtained measures of critical attributes to estimate progress toward achieving the project’s quality and
process-performance objectives• PPMs are used to estimate ro ress toward achievin ob ectives
that cannot be measured until a future phase in the project lifecycle
SP 2.2
,establishing trial natural bounds are sometimes available from priorinstances of the subprocess or comparable subprocesses, process-erformance baselines, or PPMs
54
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
How are PPMs used? (OID)
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
56/89
How are PPMs used? (OID)
PPMs are used to quantify the impact and benefits ofinnovations.
SP 1.1
• PPMs provide insight into the effect of process changes on process
SP 1.2
•
PPMs can provide a basis for analyzing possible effects of changesto process elements
55
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Examples of Controllable People x factors
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
57/89
Examples of Controllable People x factors
TraitsInterruptions
Communication Mechanisms
Nature of Leadership
56
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Example of Controllable Environmental x
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
58/89
Example of Controllable Environmental x
Nature of work facilities
Accomodations for specific needs
Degree of Security Classification
57
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Example of Controllable Technology x Factors
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
59/89
Example of Controllable Technology x Factors
Mature tools
Availability of equipment, test stations Availabilit of Technolo
Newness of TechnoloProgramming Language Used
Technology TrendsTechnology Roadmap
58
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Example of Controllable Process x Factors
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
60/89
p
Efficiency of a work task
Com liance of a work task
ua y o ar ac s(Input to or Output from
a work task)
Resolution time of technical inquiries
Quality of a work task
Timeliness of a work task
Timeliness of Artifacts
Task Interdependence
Difficult of a work task
Readability of Arti facts
An of the criteria for
Measures of bureaucracy
Resource contention between tasks
Number of people involved with a work task
Degree of Job Aids, Templates, Instructions
good reqts statements
Any of the criteria for goo es gns
Code measuresStatic and D namic
Peer Review Measures
Test Coverage
Choices of subprocesses
Modifications to how work
59
Kevin Schaaff, Robert Stoddard
Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University
Measures Tasks are performed
SEI Webinar (28 Apr 09)
Example of Controllable Customer, Supplier
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
61/89
Example of Controllable Customer, Supplier
“ Maturity” assessment
Health of relationship
Degree of communication
Degree of Documentation
of Expectations
Speed of feedback loops
Trust
mage an ercep ons
Complexity of relationship
such as simultaneousl a
Degree of partnership, collaboration
competitor and partner and supplier Style
Culture
60
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
ra eo s, omprom ses, p m za on
SEI Webinar (28 Apr 09)
Criteria for Evaluation: Measurement Planning
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
62/89
Criteria for Evaluation: Measurement Planning
1
Measurement Objectives and Alignment
• business and project objectives
• prioritized information needs and how they link to the business,organizational, regulatory, product and/or project objectives
• necessary organizational and/or software process changes to implement
the measurement plan
•
criteria for the evaluation of the measurement process and qualityassurance activities
• schedule and responsibilities for the implementation of measurement plan
including pilots and organizational unit wide implementation
Adapted from ISO 15939.
61
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Measurement Planning Criteria2
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
63/89
g 2
Measurement Process
•
• Responsibility for data collection and sources of data
• Schedule for data collection (e.g., at the end of each inspection, monthly)
• Tools and procedures for data collection
• Data storage
•
Requirements for data validation and verification procedures• Confidentiality constraints on the data and information products, and
actions/precautions necessary to ensure confidentiality
• Procedures for configuration management of data, measurement experiencease, an a a e n ons
• Data analysis plan including frequency of analysis and reporting
Adapted from ISO 15939.
62
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Criteria for Evaluation: Measurement Processes
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
64/89
•
Availability and accessibility of the measurement process andrelated procedures
• e ne respons y or per ormance
• Expected outputs
• Interfaces to other processes
– a a co ec on may e n egra e n o o er processes
• Are resources for implementation provided and appropriate
• Is training and help available?
• s e p an sync ron ze w e pro ec p an or o er organ za ona
plans?
63
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
65/89
SEI Webinar (28 Apr 09)
Criteria for Evaluation: Data Collection
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
66/89
Is implementation of data collection consistent with definitions?
Reliabilit of data collection actual behavior of collectors
Reliability of instrumentation (manual/automated)
Ease/cost of collecting data
torage
- Raw or summarized
-
- Ease of retrieval
65
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Criteria for Evaluation: Data
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
67/89
Criteria for Evaluation: Data
Quality
• Data integrity and consistency
• u
– Performance variables
– Contextual variables
• ccuracy an va y o co ec e a a
• Timeliness of collected data
•
Precision and reliability (repeatability and reproducibility) ofcollected data
• Are values traceable to their source (meta data collected)
Audits of Collected Data
66
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Criteria for Evaluation: Data Analysis
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
68/89
y
Data used for analysis vs. data collected but not used
Appropriateness of analytical techniques used
- For data type
- For hypothesis or model
Analyses performed vs. reporting requirements
Data checks performed
Assumptions made explicit
67
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
69/89
SEI Webinar (28 Apr 09)
Interquartile Range: Example
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
70/89
333
5012
Upper outlier boundary*
4
40
30
27Q3
Interquartile Range30 – 16 = 14
.
22
20
18
Procedure
1. Determine 1st and 3rd
quarti les of data set: Q1,16
16
13
Q1
3
Q3
2. Calculate the difference:interquartile range or IQR
3. Lower outlier boundar =ower ou er oun ary
16 – 1.5*14 = -5
Q1 – 1.5*IQR
4. Upper outl ier boundary =Q3 + 1.5*IQR
69
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Tips About Outliers
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
71/89
Outliers can be a clue to process understanding
,
•
repair the erroneous data if possible• if it cannot be repaired, delete it
Charts that are particularly effective to flag possible outliersinclude: box plots, distributions, scatter plots, and control
Rescale charts when an outlier reduces visibilit into
variation.Be wary of influence of outliers on linear relationships
70
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Modeling and Outliers
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
72/89
Even if outliers are valid they can distort “typical”
So you might• delete the observations
• recode outliers to be more in line with the expected distribution
• for more information, research robust techniques
In any case, do so with caution
Run our models usin different techni ues to see if the
converge
71
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
73/89
Programmatic Aspects
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Topics
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
74/89
• ev ew o rocess er ormance o e s s• Technical Process of Building PPMs
• Programmatic Aspects of Building PPMs
• Skills needed to develop PPMs
•
Forming the PPM Development Team• Up front Critical Thinking Needed
• Barriers to Building PPMs
• Documentation needed when building PPMs
• Evidence from the building and usage of PPMs that may help SCAMPI teams
•
73
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Skil ls Needed to Develop PPMs
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
75/89
• Business Acumen
•
• Process Expertise
• Understandin of Measurement and Anal sisTechni ues
• Understanding of Advanced Statistical Techniques
• Understanding of Quantitative Management
74
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
76/89
SEI Webinar (28 Apr 09)
Forming the PPM Development Team
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
77/89
Basic statistical methods
• Using basic statistics to predict outcomes
• ANOVA; regression; multiple regression; chi-square; logistic regression;
•
Hypothesis testing• Design of experiments;
Monte Carlo simulation and optimization
• Using automated “what-if” analysis of uncertain factors and decisions in aspreadsheet
Process simulation
• Modeling process activities with a computer
• Discrete event process simulation
Probabilistic networks
• Using the laws of probability instead of statistics to predict outcomes
76
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Topics
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
78/89
• ev ew o rocess er ormance o e s s• Technical Process of Building PPMs
• Programmatic Aspects of Building PPMs
• Skills needed to develop PPMs
•
Forming the PPM Development Team• Up front Critical Thinking Needed
• Barriers to Building PPMs
• Documentation needed when building PPMs
• Evidence from the building and usage of PPMs that may help SCAMPI teams
•
77
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Considerations for Developing Models - 1
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
79/89
Think Process – what are the x’s and y’s
Be sensitive to scope of application – what works in onese ng may no wor n ano er
,
improvement or risk
78
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Considerations for Developing Models - 2
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
80/89
Three Criteria for Evaluating Models
• Truth : correctly explains and predicts behavior
• Beauty : simple in terms of underlying assumptions and number of
variables, and more broadly applicable
• Justice : implications for action lead to “a better world”
79
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Topics
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
81/89
• ev ew o rocess er ormance o e s s• Technical Process of Building PPMs
• Programmatic Aspects of Building PPMs
• Skills needed to develop PPMs
•
Forming the PPM Development Team• Up front Critical Thinking Needed
• Barriers to Building PPMs
• Documentation needed when building PPMs
• Evidence from the building and usage of PPMs that may help SCAMPI teams
•
80
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Barriers to Building PPMs
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
82/89
Lack of compelling outcomes to predict due to misalignment,
management sponsorship and involvement
- that direct changes in that process or sub-process can helpcause changes in predicted outcomes
Insufficient process and domain knowledge which isnecessary to identify the probable x factors to predict the
Insufficient training and practice with modeling techniques
81
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Documentation Needed when Building PPMs-1
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
83/89
Similar to the existing SEI Indicator Template but with someadditional information content:
1.Identity of associated processes and subprocesses
2.Identity of the outcome measure (y) and the x factors
3.Data type of all outcome (y) and x factors
4.Statistical evidence that the x factors are significant (e.g. p
5.Statistical evidence of the strength of the model (e.g. theadjusted R-squared value)
6.The actual prediction equation for the outcome (y)7.The performance baselines of the x factors
82
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Documentation Needed when Building PPMs-2
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
84/89
Similar to the existing SEI Indicator Template but with someadditional information content continued :
8.The resulting confidence interval of the predicted outcome
9.The resulting prediction interval of the predicted outcome
10.Use case scenarios of how the PPM is intended to be usedby different audiences for specific decisions
. , ,and calibrated
12.Description of how often the PPM is used to makepredictions with results shown to decision-makers
13.Description of which organizational segment of projects the
83
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Topics•
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
85/89
ev ew o rocess er ormance o e s s• Technical Process of Building PPMs
• Programmatic Aspects of Building PPMs
• Skills needed to develop PPMs
•
Forming the PPM Development Team• Up front Critical Thinking Needed
• Barriers to Building PPMs
• Documentation needed when building PPMs
• Evidence from the building and usage of PPMs that may help SCAMPI teams
•
84
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Its not the Data/Chart, it is How it is Used
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
86/89
wa u o con ro c ar s oes no ma e a eve• Who is using them for management
• How are they using them
• How timely is the use
– Retrospective vs. real-time
– As the events occur vs. end-of-phase
s ng na ura oun s
• Natural bounds vs. trial bounds
• Natural bounds vs. specification limits
( ----
-
--- )
LimitsInterval
PPM
85
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Evidence for Establish
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
87/89
Statistical analysis leading to which controllable anduncontrollable factors are selected
• ANOVA or any of the other basic statistical methods discussed
• Monte Carlo/Discrete Event simulation calibration vs past
• Hypothesis tests
• p values, R-squared, etc.
ags• Apparent p value chasing
• Inordinatel hi h R-s uared
Awareness• When unstable data used
86
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
• ource o a a
SEI Webinar (28 Apr 09)
Evidence for Maintain
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
88/89
Similar to establish, but analyzing changes to recalibratemodels
Rules for when models are recalibrated
• Process changes
• rocess r t
• New platforms, domains, etc.
• Voice of the Customer
• Changing business needs
87
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University
SEI Webinar (28 Apr 09)
Evidence for Usage
8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1
89/89
Composing projects defined process
behavior
Usage for evaluating alternative solutions whenprocess pro ect per ormance na equate to meetgoals/objectives
opportunities/causes
Usage to check if predicted performance is being achievedan no , w y
88
Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow
© 2009 Carnegie Mellon University