Probabilistic Schedule and Cost Analysis

Programmatic Risk Analysis 1/186

Probabilistic Schedule and Cost Analysis

Our goal is to develop a resource loaded, risk tolerant, Integrated Master Schedule (IMS), derived from the Integrated Master Plan (IMP) that clearly shows the increasing maturity of the program’s deliverables, through vertical and horizontal traceability to the program’s requirements.

Prepared for NNJ05111915R, by GB Alleman, December 2005


Source Material Disclaimer

§ All the materials in this briefing originated from publicly available sources, nothing here should be construed as proprietary or unique to an individual vendor or manufacturer.

§ All materials are used for informational and educational purposes ONLY and are not be reused outside the teaching process.

§ The Fair Use Copyright law provides use for research and teaching.

§ All references to materials used in this briefing are provided in the “resources” section.

§ No other reuse of this briefing should be performed outside of the specific learning objectives of “Probabilistic Schedule and Cost Analysis,” education on proposal and execution.

: Introduction


Programmatic Risk Management Involves Danger and Opportunity

§ The Chinese symbol for risk represents– Danger– Opportunity

§ Taken together this suggests that risk is a strategic combination of vulnerability and opportunity.

§ Viewed in this light, programmatic risk management is a tool for managing risk that enables the program to take advantage of value enhancing opportunities.

§ A missed strategic opportunity can result in a greater loss of (potential) value than an unfortunate incident or adverse change master schedule.

§ Many programs address risk in “silos.”§ Programs cannot take this approach – an integrated technical

and programmatic risk management strategy must be used.

: Introduction


Dealing With Our “Learning Opportunity” In The Presence Of The “Danger” Of Being Late

The task of space management is rocket science. It is terribly complicated. Launchers explode and spacecraft disappear. No one wants to fail. Good enough is not good enough for mechanisms within which thousands of components must work in tandem for a mission to succeed.– “Bureaucracy and the Space Program,” in Sadeh, ed., Space Politics and Policy, 2003.

§ When we add programmatic risk issues and their management to the technical risk issues, the integrated programmatic solution must track and support the integrated technical solution

§ This briefing provides the background for the use of Monte Carlo simulation in the construction of a risk tolerant IMS

: Introduction

§ The material presented here is an “in depth” look at Monte Carlo which is more than just a tool, it is the basis of a process

§ Because of this, a deeper understanding of the process is needed to properly apply Monte Carlo to build a risk tolerant IMS

§ So be prepared, this is not one of the overview tours, this will be work with the equivalent reward – a competitive IMS that can hold its own in front of the customer and the competition


This Briefing is an Overview of Programmatic Risk, but not a Handbook of

how to do it

§ Why the IMS must include a programmatic and schedule risk management strategy– Programmatic risk and technical risk are linked

§ Techniques for managing uncertainty in the IMS– Specific tasks for risk management

§ PERT and Monte Carlo methods– Why are we using Risk+ and not focusing on PERT

§ Models of a risk tolerant schedule– Examples of Risk+ – Examples of risk mitigation plans

: Introduction


One View of Project Management

§ Technocrats manage projects– Driven by the adherence to the

profession of engineering and the profession of managing engineering projects

§ This approach is very useful when we have reliable, predictable and operational projects

– Single point failures are managed within an “engineering” culture with built–in redundancy

§ The problem with this approach is:– Hierarchical structures try to ensure organizational control and accountability– Complex systems are prone to failure– Budget and schedule issues and to the technical complexity– Interactive failure modes abound for both technical and programmatic risk situations

§ The role of a Risk Tolerant IMS is to make visible these problems and provide the opportunity for their solution

: Introduction


What Kind of Risk are We Talking About in the IMS?

§ People are not generally good at analyzing risk

§ Most risk analysis is qualitative in nature

– Skills, time and resources are in short supply to undertake a quantitative risk analysis

– The result is usually a subjective assessment of the risk

§ However quantitative analysis is needed for high programmatic risks

§ A resource loaded IMP/IMS containing accurate predecessors and successor relationships is the starting point for quantitative risk assessment

: Introduction


Integrating Risk Management into the IMS is a Multi–step Process

§ Programmatic risk assessment must define in the IMS both the technical risks, their mitigations as well as the programmatic steps take during these mitigations.

§ These steps include:– Connection of

technical and programmatic risks

– Branching probabilities for alternative paths taken during mitigation

– Resources needed for these alternative paths

– Impacts on critical milestones resulting from the occurrence of a risk

: Introduction


I think you should be more explicit here in step two

Executive Overview

There is a large amount of material in this briefing. So this overview provides a “summary” of the concepts and the recommended approach to developing a credible IMS based on probabilistic risk analysis.But in the end the details must be “owned” before we can say we have a handle on Probabilistic Schedule and Cost Analysis.

: Executive Overview


We Usually Want To Know What Our Motivation In Order To Hold Our Interest

§ Noel Coward’s motivational speech– If you must have motivation, think of your pay check

on Friday§ Our motivation starts with DID 81650, which says:

– The IMS shall be used to verify attainability of contract objectives, to evaluate progress toward meeting program objectives, and to integrate the program schedule activities with all related components.

§ The construction of an IMS in this manner is necessary but not sufficient to show that the contract objectives can be attained.

– An IMS that is tolerant of both technical and programmatic risks is the next step

– Such an IMS contains both the known risk mitigation processes as well as the unknown risk mitigation opportunities

§ The BIG QUESTION is what are the units of measure of risk tolerance?


Noel Coward 1899 – 1973


Programmatic Risk Management in One Slide Starts with Abandoning PERT and CPM

§ The Critical Path Method (CPM) does not provide a realistic view of programmatic risk.

– Task durations are random variables drawn from an underling probability distribution

– Near Critical Paths biases the completion time of the program in ways not show by the CPM

– Correlations between multiple paths bias the risk analysis as well§ PERT estimates do not address the underlying probabilistic nature of

activity networks. PERT assumes…– Statistical independence of each activity – this is almost never the case– Symmetrical probability distributions that allow the addition of the “most

likely” (Mode) value of each activity to produce a total project duration§ Monte Carlo must be used and requires understanding of ...

– The probability distribution functions of the activity network– The branching probabilities for alternative paths created by risk mitigation– The influence of correlations between network activities and their impacts

on risk



Merging Technical and Programmatic Risk is the Core to Building a Risk Tolerant IMS

§ There are two types of “uncertainty” on a program

– Technical – uncertainty about the functional and performance aspects of the program’s technology that impacts the produceability of the product or creates delays in the schedule

– Programmatic – uncertainty about the duration and cost of the activities that deliver the functional and performance elements of the program independent of the technical risk

§ We’re interested in connecting the two in the schedule and cost model(s)

– When the technical uncertainty arises what is the impact on the schedule and cost?

– When the schedule or cost uncertainty arises what is the impact on the functional and performance aspects?


So much for our strategy of winning through technical dominance


The Meaning of Uncertainty in the Context of an Integrated Master Schedule

§ Uncertainty in plain English is about the “lack of certainty”– Uncertainty is about the “variability” in the performance measures

like cost, duration, or quality– Uncertainty is about the “ambiguity” associated with a lack of this

clarity § Discovering the known and unknown sources of bias and

ignorance helps define much effort it is worth to clarify the uncertainty – This is the underlying process driving uncertainty

§ As well, uncertainty arises from the basic processes of work– This is Deming uncertainty– It is the statistical “noise” built into the work process

§ Both of these sources of uncertainty impact cost and schedule– Trying to control the “noise” adds little value– Trying to control the “lack of certainty” arising from ambiguity and

lack of clarity does have value



Schedules Are Networks Of Random Variables not Collections of Deterministic Statements

§ Task completion durations are random variables not just dates:

– These random variables have underlying probability distributions

– These distributions can not be “added” to arrive at a project completion date

– Trying to force the work into a fixed duration does not increase the likelihood of completion

§ The PERT approach to estimating project duration contains several faulty assumptions:

– The assumed independence is rarely the case– Uniform distribution of completion times can not be

confirmed– The 3–Point estimates have built in optimistic bias

§ Monte Carlo Simulation provides more accurate estimates of project completion times:

– But only if the network topology is “well formed” – And if the interactions of the underlying probability

distributions are understood


Attempting to make the dryer “dry faster” is a loosing proposition.

The dryer’s capacity is constrained by its mechanics. Much like the capacity of the design team is constrained by availability and technical productivity.


Building a Credible IMS Means Managing the Numbers in Meaningful Ways

§ First step – build the numbers correctly– Build an IMS in “layers” of detail– Understand the probability distributions of each

layer– Identify schedule margin opportunities within

the IMS– Assess the margin’s impact on the probabilistic

outcome§ Continuous process improvement – build the IMS “many times”

– Gather historical and “expert” opinions of task duration– Build probability distribution functions from this data– Improve the Monte Carlo model using this data

§ Answer the question – many times, possibly continuously– “What are the units of measure for credibility?”– “What is the coupling and cohesion of the tasks in the IMS?”



The Right Effort Produces Results, But Interpreting the Results is Sometimes Difficult

§ Proposal Improvements – Credible schedule based on probabilistic model– Traceable data to probability distributions for each

class of task– Verifiable forecast of risk areas and project duration

§ Executable program Improvements– Model of “hot spots” in the IMS– Continuous assessment of schedule and cost risk– Increased visibility of probabilistic methods

§ Improvement in our understanding of probabilistic planning– “What is the critical path” requires more than a red line in a power point

chart– No point values are allowed, only statistically qualified estimates– Stochastic network models of schedule and cost are the minimum

deliverables for proposal and execution



The One Slide Describing Our Search for “Actionable Outcomes”

§ Building a risk tolerant IMS– Explicit technical risk mitigation must be embedded in the IMS– Explicit schedule margin must be embedded in the IMS

» Margin values identified through Monte Carlo simulations » Margin assigned in front gating events

– Technical risks connected to ARM in some form– Cost and Schedule risks connected to i–MAP (impact mapping from

Woods Analysis)§ Assessing the Risk Tolerant IMS – what does risk tolerant mean?

– Weekly status, monthly Earned Value, forecast of risk impacts– Weekly Monte Carlo assessment of confidence intervals and their

historical changes – are we getting better or worse?– Performance forecast based on likelihood outcomes from Monte

Carlo simulations, not just “adding up the numbers”


Programmatic Risk Analysis 18/186: Executive Overview

The Page Intentionally Left Blank


Risk Based Planning

The difference between failure and success is the difference between doing something almost right and doing something right.— Benjamin Franklin

: Risk Based Planning

Murray didn’t feel the first pangs of real panic until he pulled the emergency cord.


In the Risk Management Business, there is Simply Too Much Information for Our Needs



Risk Assessment and Management Techniques Vary with Maturity †

Add a Risk Factor or Percentage to the critical paths

A “bottom line” Monte Carlo or Range analysis

Detailed Monte Carlo for each WBS element

Expert Opinions in a Database with assessment

Detailed Bayesian Network Analysis

Increasing Detail and Difficulty

Incr

easi

ng P

reci

sion

and

Val

ue

§ There are several approaches to building a Risk Tolerant IMS– First recognize that we’re in the early stage of this effort– There is likely value in moving further up the curve

† Ron Coleman, Litton TASC, 33rd ADoDDCAS, Williamsburg, VA



Risk is Different from UncertaintyKnowing this Difference is Critical to Success

§ Reducible risk stems from known probability distributions

– An Estimating methodology risk resulting from improper models of cost + schedule

– Cost factors such as inflation, labor rates, labor rate burdens, etc

– Configuration risk (variation in the technical inputs)

– Unknown Cost, Schedule and Technical risk coupling

§ Irreducible risk stems from known statistical processes– Requirements change impacts– Budget Perturbations– Re–work, and re–test phenomena– Contractual arrangements (contract type, prime/sub relationships, etc)– Potential for disaster (labor troubles, shuttle loss, satellite “falls over”, war,

hurricanes, etc.)– Probability that if a discrete event occurs it will invoke a project delay



There are Two Types of Uncertainty Encountered in a Risk Tolerant IMS

§ Static uncertainty is natural variation and foreseen risks– Uncertainty about the value of a parameter

§ Dynamic uncertainty is unforeseen uncertainty and “chaos”– Stochastic changes in the underlying environment– System time delays, interactions between the network elements, positive

and negative feedback loops– Internal dependencies

Stochastic behavior of forecasted completion dates

Low WorkQuality

Poor WorkConditions

ExternalScope

Changes

UnintendedChanges

Upstream Hidden Change



The Multiple Sources of Schedule Uncertainty and Sorting Them Out is the Role of Planning

§ Unknown interactions drive uncertainty

§ Dynamic uncertainty can be addressed by flexibility in the IMS

– On ramps– Off ramps– Alternative paths– Schedule “crashing” opportunities

§ Modeling of this dynamic uncertainty requires simulation rather than static PERT based path assessment

– Changes in critical path are dependent on time and state of the network

– The result is a stochastic network



Schedule Risk Management is…

§ Schedule risk management seeks to anticipate and address uncertainties that threaten the goals and timetables of a project

§ Unmitigated risks lead rapidly to delays in delivery dates and budget overages that undermine confidence in the schedule and in the project manager

§ Schedule risk management is process oriented§ While any project accepts a certain level of risk, regular and

rigorous risk analysis and risk management techniques serve to defuse problems before they arise

§ Integrated Planning reflects the development phases and the hierarchical architecture of the system



What’s our Goal as Planners?

§ Construct an IMS that has integrity and credibility to show …

§ External assessors who may consider that the schedule …– Reflects the total scope of work– Is fully integrated

» Internally (task/milestone interdependencies)» Externally (other NASA facilities, contractor schedules, vendor deliveries,

etc.)– Has an established baseline– Is reasonable or even feasible at proposal submission– Does not provide for “What–if” analysis– Is capable of providing for multiple and varying Critical Path identification

or slack for all tasks and milestones§ … and how the schedule may…

– reflect an accurate model of planned implementation– reflect an accurate or complete status– provide the correct basis for resource planning



Some “Unpleasant” Questions Can Easily Occur If We Don’t Pay Attention To The Details

§ What is the degree of risk in our baseline? How do we measure this risk?

§ What are the branching probabilities for the critical path in the IMS? How are they derived?

§ How many “near critical” paths will become critical as the program proceeds? What drives these?

§ Have the “risk drivers” been identified and mitigations put in place through explicit tasks in the IMS to deal with each identified risk? If so, how are they shown in the IMS?

§ Do we understand the underlying task completion probability distributions? How are they derived?

§ How do these probability distributions change as the program proceeds? What is the analytical basis for this?


Remember Edsel Ford’s dream of the future?


Risk Management at NASA

§ Risk includes undesirable consequences (harm) and probability of occurrence of this harm

§ For the IMS, this harm is the failure to …– Identify risk mitigation tasks– Provide sufficient schedule margin at the right places

§ Risk consists of three elements:– What can go wrong? – define a set of scenarios– How likely is it? – an evaluation of the probabilities – What are the consequences? – an evaluation of the consequences

§ The identification of these risk scenarios is the most important result of our Risk Assessment process for the IMS:

– What are the failure scenarios for the program?– What are the mitigation strategies for these failure modes?– What resources are needed for each mitigation?



NASA IRMA Tool is Used at Johnson Space Center



NASA Risk Summary Card and Programmatic Risk



Implementing Programmatic Risk Assessment is a Straight Forward Process

Initiating Event Selection

Scenario Development

Scenario Logic Modeling

Scenario Frequency Modeling

Consequence Modeling

Risk Integration

§ Some simple steps to identifying risk opportunities in the IMS– Scenario based planning – “what if this happens?”– Event impact planning – “what inhibits success?”

§ Both must focus on the consequences in order to identify the mitigations



Continuous Risk Management (CRM) is the Basis of Programmatic Risk Management

§ NASA Guidance – OMB A–11– NASA NPG

7120.5A– NASA–SP–610S– NASA NPR 8000.4

§ DoD Guidance– DAU “Risk Management

Guide for DoD Acquisition– Air Force, “Acquisition

Risk Management”– Air Force “SMC Systems

Engineering Primer and Handbook”

CRM Activity IMS RepresentationIdentify Risk items with IMP/IMS #’s, CA/WP & resource assignmentsAnalyze Risk management responsibilities assigned

Plan Mitigation plans with durations and resource assignmentsTrack Status reported from Risk Management to IMS

Control Risk tasks reporting in weekly status processCommunicate IMS status reporting



Design v. Risk Evaluation are Two Sides of the Same Coin – Risk Tolerance

§ The IMS for the “planned” program can be considered the “reference mission” plan– Meets the SOW and DRD deliverable

plan with deterministic tasks– Critical paths defined– Explicit schedule margin assigned per

PMP and risk identification

§ Missing elements from the design evaluation that must be in the risk evaluation – Near critical paths impacting durations– Individual task risk assignment– Risk distribution curve for classes of activities– Cumulative risk probability distribution skewing– Branching probabilities not defined– Correlation between risk paths and off risk paths– Dynamic interactions that drive risk



Embedding Risk Management in the Integrated Master Schedule

§ The IMS should show the coupling between technical risk and programmatic risk– Technical risk activities in the IMS connected to Active Risk

Manager– Programmatic risk visible in the IMS

§ Technical estimates of task durations developed from subject matter experts– Past performance – Basis of estimate– Expert judgments– Parametric estimates– Dynamic models

§ Programmatic estimates developed through the win strategies– Convey the risk buy down and win theme support through the risk

mitigation activities: Risk Based Planning

All these estimates must be calibrated before being accepted into the IMS. Without this effort, these numbers are just as unreliable as raw estimates


Why Probabilistic Risk Analysis is Often Opposed by Management and IPT Leads

§ Many people do not understand the underlying statistics

– Education, practice, guidance§ Many planners lack the formal

probability and statistics training

– Education, practice, guidance§ Most planners perform

deterministic analysis of schedules and cost

– Risk is hard work

§ The fact the probabilistic risk analysis is built on uncertainty is seen as weakness in the planning process, not a strength

– Why can’t you know how long it will take or how much it costs?§ People tend to think that the “lack of data” is a reason not to perform

probabilistic schedule risk analysis– The exact opposite is true


Programmatic Risk Analysis



Managing Uncertainty in the IMS

A ship on the beach is a lighthouse to the sea.— Dutch Proverb

: Managing Uncertainty in the IMS


3 Troubles With Deterministic Schedule Estimating in a Traditional Manner

1. Single point estimates can be accuratea) Without stating the distribution

statistics, the number has no frame of reference

b) The median temperature in Cody Wyoming is a balmy 78º

c) Don’t be there in February in your shirt sleeves


2. There is no standard definition for the term “best” estimatea) A hoped for bestb) A planned for bestc) The actual best derived from the underlying statistical model

3. Rollup of the “most likelies” is not the same as Most Likely Total Duration

a) They are probability distributions not integersb) Probability distribution function (pdf) convolution is needed

The flaw with using averages aloneAverage depth is 3 feet


No Single Point Estimate Can Be Accurate, Since It Ignores The Underlying Statistics

§ Schedule durations are always vague in the beginning

– Existing technical capability often falls short of project needs

– Firm requirements, especially software requirements cannot be described in a simple list

– “Normal” schedule slippage of varying lengths result from integration problems and test failures

– Various other anticipated and unforeseen events resulting from the natural variation (Deming variation)

§ “Point” estimates can not be correct because…– Task point estimates of activity durations are not correct– Project point estimate is the sum of these “incorrect” activity estimates

§ “Actual” project duration will fall within some range around the point estimate– The best that can be done is to understand the uncertainty


1


The Traditional Roll Up Approach Starts with the “Best” estimates for the Most Likely

Durations for Tasks

§ Build a network of the project’s activities§ Determine the “best” estimate of the duration for each activity

in the network§ Compare the activities’ “best” estimates to find the critical path§ Sum all the “best estimate” durations of activities on the

critical path§ Define this sum of tasks to be “best” estimate of the project’s

schedule duration§ This will almost always be optimistically wrong§ Or it is pessimistically wrong§ Either way – it is wrong


1


The Problem is the Term “Best” Has No Standard Definition

§ For each activity the “best” estimate is …– The “most likely” duration – the mode of the distribution of

durations? (Mode is the number that appears most often)– It’s 50th percentile duration – the median of the distribution?

(Median is the number in the middle of all the numbers)– It’s expected duration – the mean of the distribution? (Mean is the

average of all the numbers)§ These definitions lead to values that are almost always different

from each other§ Rolling up the “best” estimate of completion is almost never

one of these.


2


Durations Are Probability Samples not Single Point Values

§ We know this because…– “Best” estimate is not the only possible estimate, so other

estimates must be considered “worse”– Common use of the phrase “most likely duration” assumes that

other possible durations are “less likely”– “Mean,” “median,” and “mode” are statistical terms characteristic

of probability distributions§ This implies activity distributions have probability distributions

– They are random variables drawn from the probability distribution function (pdf)

§ “Actual” project duration is an uncertain quality that can be modeled as a sum of random variables– The pdf may be known or unknown


2


Durations are Educated “Guesses,” but Rarely Have underlying Probability Distributions

§ Define the problem§ Identify the prediction variable§ Build the prediction model

– Develop a list of relevant factors– Consider the effects– Collect data– Plot each factor independently– Develop a prediction model using linear regression– Understand the model– Check the model

§ Make guesses with the model§ Take care with the results


2


One Way Of Producing A Guess Is With A “Twenty Questions” Game

§ The 20 questions approach– Ask an engineer how long it will

take to do a task and the answer might be “I can’t say.”

§ Planner– Will to take a year?

§ Engineer– Oh, of course not

§ Planner– Will it take a day?

§ Engineer– Oh, of course not

§ Planner– How about 6 months?

§ Engineer– Could be, but that’s too long

§ Planner– How about 2 months?

§ Engineer– No, that’s too short

§ Planner– How about 3 ½ months?

§ Engineer– Yea, that could work

§ In 5 question a first order estimate can be found to with 20%


2


Putting Guesses into a Schedule Requires Us to Sort Out Fact from Fiction

§ Rank all the tasks– 1 = scope known, duration known– 2 = scope known, duration unknown– 5 = scope unknown, duration unknown

§ Have a “planning session” where no one leaves the room until all Rank–5 tasks are turned into Rank–2 tasks– The reduction in rank comes for “information” about the

probability distribution which underlying the random variable representing the duration

§ Focus on “randomness” of the estimate is critical to success– Adjustments for confidence must become part of our vocabulary– This approach turns “guessing” into statistical estimating


2


Risk Drivers That Impact Uncertainty Must be Identified Before They are Used

§ Risk drivers include (but may not be limited to)– Beyond the state–of–the–art development– Unusual production requirements, either time or technique– Cost constraints, derived without consideration of technical or

programmatic processes– Software development issues– Multiple interface management– Subcontractor and supplier viability, variability or plain olde

“ability”– System integration and testing impacts from coupling and

cohesion of the tasks in the IMS– Unforeseen events


2


Estimating Accomplishment Criteria starts with defining the what “done” look like

§ When building the IMS, the first round should– Confirm the IMP (PE/SA/AC) structure overlays the topology of the

program– Ask for a duration estimate for each Accomplishment Criteria

which represent» Exit criteria» Deliverables for the maturing program» Incremental progress along the path to maturity

– Get this estimate as a “single” task with a duration§ Assign SA’s to an IPT Lead

– Develop IPT processes that support the SA– Identify AC’s from process deliverables or maturity improvements– Link vertical path to Program Events and horizontal paths across

IPTs


2


Modeling Duration Probability starts with experts but must include statistical estimates

§ Compile duration estimates from different sources and rank the estimates:– Subcontractors or IPTs estimate– Project manager’s estimate– “Independent” estimate– Risk–impacted estimate

§ Associate confidence levels with ranges between estimates, using information available from different situations and at different stages in the project development cycle– These can not be looked up in a book– Are not directly derived from historical data– Must be subjective, knowledge based consensus of technical

experts in a particular WBS– Should be a standard part of the risk mitigation plan


3


Statistics at a Glance as a starting point. But more details are needed

§ Probability distribution – A function that describes the probabilities of possible outcomes in a "sample space.”

§ Random variable – variable a function of the result of a statistical experiment in which each outcome has a definite probability of occurrence.

§ Determinism – a theory that phenomena are causally determined by preceding events or natural laws.

§ Standard deviation (sigma value) –An index that characterizes the dispersion among the values in a population.

§ Bias – The expected deviation of the expected value of a statistical estimate from the quantity it estimates.

§ Correlation – A measure of the joint impact of two variables upon each other that reflects the simultaneous variation of quantities.

§ Percentile – A value on a scale of 100 indicating the percent of a distribution that is equal to or below it.

§ Monte Carlo sampling – A modeling technique that employs random sampling to simulate a population being studied.


3


Siren Song of the Central Limit Theorem must be avoided in any robust estimating process

§ The probability distribution of the project’s total duration is obtained by statistically summing distributions of all activities along the schedule network critical path

§ Central Limit Theorem – if the number of critical path activities is “large,” the probability distribution of the total duration is “approximately” Gaussian.

§ Another theorem (not related to Mr. Gauss) – the sum of the means of activity duration equals the mean of the total duration

– But because the Gaussian distribution is symmetric, the total duration distribution results in mean = medium = mode

– Therefore, » Sum of the activity duration medians < total

duration median» Sum of the activity modes < total duration

mode


3


The Central Limit Theorem must be understood in order to be useful

§ The Central Limit Theory (CLT) consists of three statements– The mean of the sampling distribution of means equals the mean

of the population from which the samples were drawn– The variance of the sampling distribution of means is equal to the

variance of the population from which the samples were drawn divided by the size of the samples

– If the original population is distributed normally, the sampling distribution of means will also be normal. If the original population is not normally distributed, the sampling distribution of means will increasingly approximate a normal distribution as sample size increases

§ PERT assumes the duration and variance along the critical path are normally distributed and therefore the total duration follows the CLT

§ Departures from normal distribution and near critical path tasks becoming critical are usually ignored – to the peril of planning


3


Mr. Gauss’s Distribution is found in many text book examples, too bad is not applicable

§ The Gaussian distribution can be proved (by the Central Limit Theorem) in the situation that each measurement is the result of a large amount of small, independent error sources. These errors have to be of the same magnitude, and as often positive as negative.

§ When a physical item is measured and systematic errors are eliminated the measured values will spread around the average value.

§ The average value of a measured value is the “best value.”


3


Task “Most Likely” ≠ Project “Most Likely,” Must be Understood by Every Planner

§ PERT assumes probability distribution of the project times is the same as the tasks on the critical path

§ Because other paths can become critical paths, PERT consistently underestimates the project completion time 1 + 1 = 3


3


Probability Distribution Function is the Lifeblood of good planning

§ Probability of occurrence as a function of the number of samples

§ “The number of times a task duration appears in a Monte Carlo simulation”



Standard Deviation of a Probability Distribution

§ Which describes that “spread” of the random variables around the mean represented by the distribution

§ Standard deviation describes the width of the probability distribution function



Underlying Statistics and Confidence



Families of CDF’s Can Look Alike

§ Cumulative Distribution Functions (CDF) look similar for a variety of Probability Distribution Functions (pdf)

Cumulative Probability

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200%Bounds on Point Estimate

Beta Triangular Uniform

Probability Density


BetaTriangular Uniform

Cumulative Probability

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Beta Triangular Uniform Normal LogNormal

Probability Density


BetaTriangular Uniform Normal LogNormal


Programmatic Risk Analysis 58/186: Managing Uncertainty in the IMS



Approaches To Uncertainty

“It is moronic to predict without first establishing an error rate for a prediction and keeping track of one’s past record of accuracy”— Nassim Nicholas Taleb, Fooled By Randomness(where he argues people constantly delude themselves because they do not understand probability and are programmed to find reasons for optimism or pessimism where none exist.)

: Approaches to Uncertainty

If you hang them all, you’re sure to get the guilty ones.

– Judge Roy Bean, “The Law West of the Pecos”


There Is No Such a Thing as Risk Neutral Decision Making

§ Can we have a discussion of a probabilistic event without a discussion of associated risk? (confidence in the confidence interval)

§ The most severe problem with making decisions in the presence of risk is when the decision maker doesn’t have the faintest idea about what risks are to be incurred, yet thinks there is precision about the decision results.

§ Three features of decision making– Alternatives – which course of action might be taken?– Uncertainties – what uncontrolled elements exist?– Outcomes – Alternatives combined with Uncertainties = Outcomes


Natural Gas?Rotating equipment?Open flames?High voltage?


Types of Uncertainty Based Decisions

StrategyDescribes a collection of actions the decision maker makes

èGoal

A possible outcome of the strategy is a goal

DecisionActually performing a task is a decision è

PrioritizationDeciding which task to perform is a prioritization

AlternativeA set of choices that are allowable è

OptionAn option is an alternative that permits a future decision following the discovery of new information

Sensitivity of DecisionKnowing “what” to do

è Sensitivity of OutcomeKnowing “how” it will turn out

Direct valuesOutcomes traceable to bookable benefits to the project

èIndirect Values

Things the decision maker value but are unlikely to be visible in the project

Certain equivalent effectThe smallest sum of money for which the decision maker would be willing to sell rights to a risky product

è

Expected Net Present ValueThe hypothetical average NPV from numerous independent launches of identical projects



Approaches to Decisions with Uncertainty

§ Decision trees§ Line of balance§ PERT§ Monte Carlo



Decision Trees

§ Decision Trees are tools for helping choose between several courses of action.

§ They provide a effective structure within which to lay out options and investigate the possible outcomes of choosing those options.

§ They also help to form a balanced picture of the risks and rewards associated with each possible course of action.



Line Of Balance

§ The line–of–balance technique is based on the underlying assumption that the rate of production for an activity is uniform

§ Measurements are compared against a specific plan, all collected in one “balance” chart



PERT As The Starting Point For Probabilistic Schedule Management

§ PERT is a method to determine how long a project should take– Which activities are most critical– Deterministic PERT and Probabilistic PERT are common

§ PERT algorithms– Activity duration:

– Activity Standard Deviation:

– Activity Variance:

– Total Standard Deviation:

( )4 / 6et a m b= + +

( ) / 6et b as = -

( ) 22 / 6ev t b as é ù= = -ë û

( ) ( )2 21 2e e eT t ts s s= +



Deterministic PERT Uses The Three Point Estimates In A Static Manner

§ Durations are defined as three point estimates– These estimates are very subjective if captured individually by asking…– “What is the Minimum, Maximum, and Most Likely”

§ Critical path is defined from these estimates is the algebraic addition of three point estimates

§ Project duration is based on the algebraic addition of the times along the critical path

§ This approach has some serious problems from the outset

– Durations must be independent– Most likely is not the same as

the average



Probabilistic PERT Uses The Underlying Probability Distributions For Each Task

§ Any path could be critical depending on the convolution of the underlying task completion time probability distribution functions

§ The independence or dependency of each task with others in the network, greatly influences the outcome of the total project duration

§ Understanding this dependence is critical to assessing the credibility of the plan as well as the total completion time of that plan



Statistics V. Probability – We Need Both To Build A Risk Tolerant Schedule

§ In building a risk tolerant IMS, we’re interested in the probability of a successful outcome

– “What is the probability of making a desired completion date?”

§ But the underlying statistics of the tasks influence this probability

§ The statistics of the tasks, their arrangement in a network of tasks and correlation define how this probability based estimated developed.

Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.– H.G. Wells



The Problem with PERT

§ When the activity durations are random variables, each path of the project network is a likely candidate to be the critical path.

§ Every activity duration could result in a different longest path

§ Evaluating the distribution of the longest path, even under very specific distributional is not an easy problem



Simple Understanding of Statistics

The number of times a specific occurrence of a parameter occurs in a population.The number of time it snows before September 15th in Colorado indicates the confidence that it will snow on or before 9/15 this year



Inputs to the process have statistics too

§ The statistics of the events that impact the project must be understood as well – this second order impact is critical

§ These events form a stochastic process that drives the network

§ These drivers may or may not be random events

§ Correlations between the events can create nonlinear behaviors in the sensitivity of the model



The Program is a System, Just like the Spacecraft

§ The programmatic and planning dynamics act as a system§ The “system response” is the transfer function between input and

output§ Understanding this transfer

function may appear beyond our interest

– But it is part of the stochastic dynamic response to disruptions in our plans

– “What if” really means “what if” at this point in the response curve of the system


Inputs

Outputs


The Beta Distribution

§ The Beta distribution is given as:

§ Where B is the Beta function

( ) ( )

( ) ( )( )( )

111

0

, 1 ,

,

, .

B t t dt

B y

baa b

a ba b

b

--= -

G G=

G +

=

ò

( ) ( )( )

1 11 x xP x

B

b a

a b

- --=

-



What does the Beta Distribution Provide?

§ Beta can model events that are constrained to take place within an interval defined by a minimum and maximum value

§ Turns out the equation for task duration– Is a empirical approximation formulas not derived from the beta

distribution directly– The theoretical argument showing the relative weight of 4 on the

modal time is based largely on the Beta distribution where developed by the Pearson & Tukey based on the 5% points and the median

– Presumably, some experimentation was done in the early days, and some empirical basis was found for these forms

– At this point, we merely accept the formulas as the "traditional" way of doing PERT

§ Beta is used directly in Monte Carlo as a model of task duration

( )4 / 6et a m b= + +



Another Alternative, the Triangle Distribution

§ The triangle distribution was proposed in the late 90’s as a substitute of the Beta distribution

– The parameters of the triangle distribution have a one–to–one correspondence with the PERT parameters

§ The original “fitting” of PERT equation took place in 1959

§ Triangle distribution is better behaved in certain instances, but the “mean” is still greater than the “mode” (the most likely)

§ The result is overly optimistic durations for the tasks



Sensitivity of the PDF: Triangle

§ Triangle distribution – The “minimum” and the “maximum” are as influential on the mean

on the “most likely”– The “minimum,” “maximum” and the “modal” values capture

limited information about the underlying distribution– There can be an infinite number of distributions with these same

three values



Sensitivity of the PDF: Beta PERT

§ Manipulating the standard Beta distribution produces the BetaPERT (Vose, 2000, pp. 275)– BetaPERT combines the Beta distribution and the PERT formula

for the mean


Programmatic Risk Analysis 78/186: Approaches to Uncertainty



Gathering Risk and Mitigation Information

Getting viable estimates of the relative risk requires effort, patience and care not to over or under estimate the risk or the mitigation activities.Magic rabbits are not a reliable source of information.

: Gathering Risk and Mitigation Information


General Flow of Risk Management

§ Schedule and Cost risk management must be performed in a structured and rigorous manner

§ Each element below must appear in the IMS



Gathering Risk Analysis Information

§ Interviews with CAMs and IPT Leads about risk creates a “bias” toward optimism or pessimism – but never toward neutrality– Optimist says – The glass is ½ full– Pessimist says – The glass is ½ empty– An engineer says – The glass is twice as big as it needs to be

§ Asking for three point estimates (optimistic, most likely, pessimistic) is not the way to do this…– Gain some sense of the risk “ranking” from the owner of the task

or work package– Look to any historical data

» Durations» Causes of Over Target Baseline (OTB) and Over Target Schedule

(OTS)



The Dreaded 3–Point Estimates

§ Optimistic Estimate– The shortest duration – “It can’t be done in less time than

this”§ Most Likely Estimate

– The median time (middle most), not the mean time (the average)

– This builds in a symmetric distribution of the probability distribution function

§ Pessimistic Estimate– The additional time needed if things

go wrong– It is not the maximum time it would

take– It is not a “worst case scenario”

estimate



Classes of Project Risk

§ Delay– Slow decisions– Access – Lack of information– Time difference

§ Dependency– Interaction– Interface

difficulties– Interruption of

service§ Estimates

– Mis–estimation– Learning curves– Overly aggressive

deadlines

Pareto of Risk Causes



Thinking About Risk Classification

§ These classifications can be used to avoid asking the “3 point” question for each task

§ This information will be maintained in the IMS§ When updates are made the percentage change can be applied

across all tasks

Classification Uncertainty OverrunA Routine, been done before Low 0% to 2%

B Routine, but possible difficulties Medium to Low 2% to 5%C Development, with little technical difficulty Medium 5% to 10%D Development, but some technical difficulty Medium High 10% to 15%E Significant effort, technical challenge High 15% to 25%F No experience in this area Very High 25% to 50%



Steps in Characterizing Uncertainty in Task Duration Estimating Data

§ Use an “envelope” method to characterize the minimum, maximum and “most likely”

§ Fit this data to a statistical distribution§ Use conservative assumptions§ Apply greater uncertainty to less mature technologies§ Confirm analysis matches intuition

Remember Sir Francis Bacon’s quote about beginning with uncertainty and ending with certainty.If we start with a what we think is a valid number we will tend to continue with that valid number.When in fact we should speak only in terms of confidence intervals and probabilities of success



Some Sobering Observations

§ In 1979, Tversky and Kahneman proposed an alternative to utility theory. Prospect theory asserts that people make predictably irrational decisions. [45], [52]

§ The way that a choice of decisions is presented can sway a person to choose the less rational decision from a set of options.

§ Once a problem is clearly and reasonably presented, rarely does a person think outside the bounds of the frame.

§ Source:– “The Causes of Risk Taking By Project Managers,” Proceedings of

the Project Management Institute Annual Seminars & SymposiumNovember 1–10, 2001 • Nashville, Tenn

– Tversky, Amos, and Daniel Kahneman. 1981. The Framing of Decisions and the Psychology of Choice. Science 211 (January 30): 453–458



The Dark Side of PERTDuring the modeling process the “most likely” durations can not be added since they represent the moments of the underlying probability distribution (mean, mode, median, variance)As well, the phenomenon of “merge bias” at the merge points of parallel task streams move the completion probability point to the right in unpredictable ways.

The critical path is often referred to as the laugh track of the project.

: The Dark Side of PERT


Some Useful (and dreadful) History

§ The original paper (Malcolm 1959) states– The method is “the best that could be done in a real situation

within tight time constraints.”– The time constraint was One Month

§ The PERT time made the assumption that the standard deviation was about 1/6 of the range (b–a), resulting in the PERT formula.

§ It has been shown that the PERT mean and standard deviation formulas are poor approximations for most Beta distributions (Keefer 1983 and Keefer 1993).– Errors up to 40% are possible for the PERT mean– Errors up to 550% are possible for the PERT standard deviation



Critical Path and Most Likely

§ Critical Path’s are Deterministic– At least one path exists through the

network– The critical path is identified by

adding the “single point” estimates– The critical predicts the completion

date only if everything goes according to plan (we all know this of course)

§ Schedule execution is Probabilistic– There is a likelihood that some durations will comprise a path that

is off the critical path– The single number for the estimate – the “single point estimate” is

in fact a most likely estimate– The completion date is not the most likely date, but is a

confidence interval in the probability distribution function resulting from the convolution of all the distributions along all the paths to the completion of the project



Some (False) Assumptions of PERT

§ Three point estimates follow the Beta distribution– Using the simplified algebraic formula– But this formula has built in biases not revealed in normal use

§ The expected completion time and variance are calculated by summing the mean and variance of critical path activities– The central limit theorem suggests that if there are sufficient

activities on the critical path, the activity times are independent– If the activity times follow a probabilistic distribution with no one

activity time dominating, then the sum of activity times on the critical path follows approximately the normal distribution

§ The result are derived completion times using the normal distribution with a built–in optimistic bias



Merge Bias Must Be Addressed Upfront Before Management Gets Involved

§ The standard approach to schedule risk analysis

§ A “merge point” is where two tasks have a common successor

§ PERT naively assumes the pdf’s are identical for all tasks



The Architecture of Merge Bias Starts with Parallel Tasks Landing on a Single Node

§ The Cumulative Distribution Function (CDF) is biased by the merge points– The CDF is the source of samples for the Monte Carlo simulation

Effect of the Merge Bias on Schedule Risk

0%

20%

40%

60%

80%

100%

10/6 11/25 1/14 3/4

Date

Cum

ulat

ive

% OnePathThreePath



Merge Point Bias Can Be Very Confusing and Lead to False Optimism

§ It is misleading to consider only variances from single predecessors for each node in the critical path– Early start of a node depends on the maximum of finish (or start)

times of predecessors– This includes ALL the non–critical paths

§ Early Start is a random variable that is the maximum of all the non–independent random variables in the network

§ This effect is strongest if– There are more predecessors– Predecessors have equal or near equal duration– Low dependency among the predecessors

§ The result is an unrealistic optimism for the expected completion times, but most importantly the variance in the completion times



Notional View of Merge Bias

§ Near critical paths bias the completion time of the critical path

§ The finish (or start) times for all paths are random variables with individual probability distributions

§ These distributions are “joined” to form a probability distribution for the completion of the project



Notional Impact of Merge Bias

§ Activities with duration of 2 have s=0.707§ Activities with duration of 4 have s=1.414



Why Does This Happen?

§ The completion time of a task in the IMS can be considered a random variable– This is a mathematical random variable, independent of our ability

to manage to plan.– The distributions of random variables can be “added” by

convolving their probability distribution functions.– This is the case independent of the underlying distribution (Beta,

Triangle, Gauss, Poisson)

§ This is a critically important concept– No matter what our planning fidelity, the underlying processes of

task completion duration are random variables– These random variables have predictable and unpredictable

behaviors that impact the outcome is unfavorable ways



Conveying the Effect of Merge Bias is Sometimes Difficult, But it is Always There

§ Most projects have parallel paths, many times at crucial points in the schedule – PDR, CDR, ATLO

§ “Merge Bias” creates extra risk at these point but extending the probabilistic completion date

§ This may be the factual case or it may be the result of the statistics§ Either way the

answer will likely be unacceptable to those without the underlying knowledge

§ We must both educate and inform before proceeding



Conclusion of Merge Bias

§ The Critical Path is pretty much meaningless at the deal level of the project– Dynamic completion times must be modeled with Monte Carlo

§ Discussing the critical path requires that you look at your watch first to see what time it is– The critical path is not static– It is highly dependent on but stochastic behaviors of the task

completion time the emerge from the underlying probability distributions

– It is also dependent on the dynamics of the interactions of the network nodes

§ USE MONTE CARLO AND TRY TO AVOID SPEAKING ABOUT THE CRITICAL PATH IN ANY WORDS OTHER THAN VERY HIGH LEVEL A STATIC PATH THROUGH THE NETWORK



A Quick Look at Monte Carlo Simulation

Georges Louis Leclerc, Comte de Buffon, asked what was the probability that the needle would fall across one of the lines, marked here in green. That outcome will occur only if

sinA l q<

: A Quick Look at Monte Carlo Simulation


Monte Carlo Simulation Provides a Solution to Merge Bias and Other PERT Issues

§ Yes Monte Carlo is named after the country full of casinos located on the French Rivera

§ Advantages of Monte Carlo over PERT is that Monte Carlo…

– Examines all paths not just the critical path

– Provides an accurate (true) estimate of completion

» Overall duration distribution » Confidence interval (accuracy

range)

– Sensitivity analysis of interacting tasks– Varied activity distribution types – not restricted to Beta– Schedule logic can include branching – both probabilistic and conditional– When resource loaded schedules are used – provides integrated cost and

schedule probabilistic model



The Monte Carlo Tour Starts with WWII History

§ Any method which solves a problem by generating suitable random numbers and observing that fraction of the numbers obeying some property.

§ The Monte Carlo method provides approximate solutions to a variety of mathematical problems by performing statistical sampling experiments on a computer.

§ The method applies to problems with no probabilistic content as well as to those with inherent probabilistic structure.

§ The method is named after the city of Monte Carlo in the principality of Monaco, because of a roulette, a simple random number generator. The name and the systematic development of Monte Carlo methods dates from about 1944 and the Manhattan project.



The Monte Carlo Method Can Be Found in Many Scientific and Business Domains

§ Monte Carlo is a well developed discipline in many areas of science, finance and statistics

§ Probability distribution functions (pdf's) – the physical (or mathematical) system must be described by a set of pdf's.

§ Random number generator – a source of random numbers uniformly distributed on the unit interval must be available.

§ Sampling rule – a prescription for sampling from the specified pdf's, assuming the availability of random numbers on the unit interval, must be given.

§ Scoring (or tallying) – the outcomes must be accumulated into overall tallies or scores for the quantities of interest.



The Monte Carlo Method is Computationally Based

§ Tools like Risk+ are just a sample of the many approaches to simulating physical system

§ Error estimation – an estimate of the statistical error (variance) as a function of the number of trials and other quantities must be determined.

§ Variance reduction techniques –methods for reducing the variance in the estimated solution to reduce the computational time for Monte Carlo simulation

§ Parallelization and vectorization – algorithms to allow Monte Carlo methods to be implemented efficiently on advanced computer architectures.



Monte Carlo is Fundamentally a Sampling Approach

§ The large the number of “runs” the higher the fidelity of the simulation

§ Size does matter

§ The “sample space” of random numbers is defined by the probability distribution function of the task completion times

§ Knowing this pdf is important for quality answers to “how long will it take?”



Using Monte Carlo Starts with the Three Point Estimates, but Goes Far Beyond

§ Conceptually simple approach– The normal PERT 3–Point estimate can be used (reused)– There is no need for special assumption about the underlying

probability distributions of the completion times§ The Criticality Index is provided from the simulation

– Which tasks are critically impacting the completion time of the project

§ It is computationally expensive– But desktop computing can keep up (3GHz with 2GB of memory is

baseline machine for any serious Monte Carlo work)§ Scalable analysis quality

– Small runs for testing assumptions: £ 1,000 runs– Larger runs for validation of assumptions: ³ 10,000 runs



Risk+ Quick Overview

Task to “watch”(Number3)

Most Likely(Duration3)

Pessimistic(Duration2)

Optimistic(Duration1)

Distribution(Number1)



Risk+ Quick Overview

§ The height of each box indicates how often the project complete in a given interval during the run

§ The S–Curve shows the cumulative probability of completing on or before a given date.

§ The standard deviation of the completion date and the 95% confidence interval of the expected completion date are in the same units as the “most likely remaining duration” field in the schedule

Date: 9/26/2005 2:14:02 PMSamples: 500Unique ID: 10Name: Task 10

Completion Std Deviation: 4.83 days95% Confidence Interval: 0.42 daysEach bar represents 2 days

Completion Date

Freq

uenc

y

Cum

ulat

ive

Prob

abilit

y

3/1/062/10/06 3/17/06

0.1

0.2

0.30.4

0.5

0.60.7

0.8

0.91.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16 Completion Probability Table

Prob ProbDate Date0.05 2/17/060.10 2/21/060.15 2/22/060.20 2/22/060.25 2/23/060.30 2/24/060.35 2/27/060.40 2/27/060.45 2/28/060.50 3/1/06

0.55 3/1/060.60 3/2/060.65 3/3/060.70 3/3/060.75 3/6/060.80 3/7/060.85 3/8/060.90 3/9/060.95 3/13/061.00 3/17/06

Task to “watch”

80% confidence that task will complete by 3/7/06



A Well Formed Risk+ Schedule

§ A good critical path network– No constraint dates– Lowest level tasks have predecessors

and successors– 80% of relationships are finish to start

§ Identify risk tasks – These are “reporting tasks”– Identify the preview task to watch during

simulation runs

§ Enter probability distribution profile for each task– Bulk assignment is an easy way to start– 1 – 5 ranking is another approach– Individual risk profile assignments is best but tedious



Analyzing the Risk+ Simulation

§ Risk + will generate one or more of the following outputs:

– Earliest, expected, and latest completion date for each reporting task

– Graphical and tabular displays of the completion date distribution for each reporting task

– The standard deviation and confidence interval for the completion date distribution for each reporting task

– The criticality index (percentage of time on the critical path) for each task

– The duration mean and standard deviation for each task – Minimum, expected, and maximum cost for the total project – Graphical and tabular displays of cost distribution for the total project – The standard deviation and confidence interval for cost at the total project

level


Programmatic Risk Analysis 110/186: A Quick Look at Monte Carlo Simulation



Building a Robust IMS

Never undertake a project unless it is manifestly important and nearly impossible.

— Edwin Land

: Building a Robust IMS

Edwin Land 1909 – 1991


Schedule Contingency Analysis

§ The schedule contingency needed to make the plan credible can be derived from the Risk+ analysis

§ The schedule contingency is the amount of time added (or subtracted) from the baseline schedule necessary to achieve the desired probability of an under run or over run.

§ The schedule contingency can be determined through– Monte Carlo simulations (Risk+)– Best judgment from previous experience– Percentage factors based on historical experience– Correlation analysis for dependency impacts


Is This Our ContingencyPlan ?


Schedule Quality and Accuracy

§ Accuracy range– Similar for each estimate class

§ Consistent with estimate– Level of project definition– Purpose– Preparation effort

§ Monte Carlo simulation– Analysis of results shows quality attained versus the quality

sought (expected accuracy ranges)§ Achieving specified accuracy requirements

– Select value at end points of confidence interval– Calculate percentages from base schedule completion date,

including the contingency



Technical Performance Measures

§ Technical Performance Measures are one method of showing risk by done

– Specific actions taken in the IMS to move the compliance forward toward the goal

§ Activities that assessing the increasing compliance to the technical performance measure can be show in the IMS

– These can be Accomplishment Criteria



Elements of a Risk Tolerant Program

CAIVAPB

ScheduleRisk

CostRisk

TechnicalRisk

• Critical Path• Monte Carlo• IMS Maturity

• Systems Engineering• Simulation & Test• Design• TPM

• WBS• Cost Estimates• EVMS

CAIV – Cost as an Independent VariableAPB – Allocated Program Baseline

When the three independent variables of the program are FIXED in the beginning.Building in risk tolerance now becomes a real challenge



Inserting Risk Tolerant in the IMS

§ Technical risk management using ARM– Identification– Classification– Triage

§ Programmatic risk management– Mitigation tasks– Maturity enhancement– Technical Performance Measures

§ Tracking and Communication– Risk ID embedded in the IMS– Resource assignment– Cost model cross connections



Moving forward

§ The previous approaches to the development of an IMS was to build the IMP and then the IMS and all the supporting tasks.

§ The tasks define the work effort to deliver the ACs and the product

§ Simple RISK+ metrics run at the end

§ Explicit risk mitigation provides another view– Risks identified in ARM– Mitigation tasks for those risks are made explicit in the IMS– Branching probabilities for each mitigation initiation events– Monte Carlo analysis of the main stream and mitigation effort



Integrating Risk and Schedule

§ Probabilistic completion times change as the program matures

§ The efforts that produce these improvements must be traceable in the IMS

§ The “error bands” on the events must include the risk mitigation activities as well

§ IMS activities show how the “error band” narrows over time. – This is the basis of a “programmatic risk tolerant” IMS– The probabilistic interval becomes more reliable as risk mitigations and

maturity assessments add confidence the to IMS

Baseline Plan

80%

Mean

Missed Launch Period

Launch Period

Ready Early

Oct 07

Nov 07

Dec 07

Jan 08

Feb 08

Mar 08

Apr 08

May 08

Jun 08

PlanMargin

Current Planwith risks is the stochastic schedule

CD

R

PD

R

SR

R

FRR

ATL

O

20%

Aug 05 Jan 06 Aug 06 Mar 07 Dec 07 Feb 08

Current Planwith risks is the deterministic schedule

Risk Margin



Event Driven Risk Management

§ Water fall risk management integrated with the IMP or high level IMS

§ We’re missing this connection at the moment

§ But making it visible in the simplest manner is the first step in building awareness

§ Technical risk management and programmatic risk management must be integrated into the IMS



Steps in the Process of a Risk Tolerant IMS

1. Identifying the risksa) ARM and risk trace numberb) Mitigation tasksc) Duration and resources for the mitigation

2. Integrate risk in the IMSa) Embedded tasks with IMP/IMS numbersb) Network tasks c) Assess Monte Carlo impact on completion dates

3. Evaluate outcomea) Events and SAs within acceptable confidence interval?b) If not, risk mitigation activities need improvementc) If so, baseline mitigation and narrative supporting the strategy

4. Connect Risk Management (ARM) and IMS at the tracking number level



Branching Probabilities – Simple Approach

§ Plan the risk alternatives that “might” be needed

– Each mitigation has a Plan B branch

– Keep alternatives as simple as possible (maybe one task)

§ Assess probability of the alternative occurring

§ Assign duration and resource estimates to both branches

§ Turn off for alternative for a “success” path assessment

§ Turn off primary for a “failure” path assessment

30% Probabilityof failure

70% Probabilityof success

Plan B

Plan A Current Margin Future Margin

80% Confidence for completion with current margin

Duration of Plan B Plan A + Margin£



Managing Margin in the Risk Tolerant IMS requires the reuse of unused durations

§ Programmatic Margin is added between Development, Production and Integration & Test phases

§ Risk Margin is added to the IMS where risk alternatives are identified

§ Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives

– This enables us to buy back schedule margin for activities further downstream

– This enables us to control the ripple effect of schedule shifts on Margin activities

5 Days Margin

5 Days Margin

Plan B

Plan A

Plan B

Plan AFirst Identified Risk Alternative in IMS

Second Identified Risk Alternative in IMS

3 Days Margin Used

Downstream Activities shifted to left 2 days

Duration of Plan B < Plan A + Margin

2 days will be added to this margin task to bring schedule back on track



Simulation Considerations of Monte Carlo need to be understood to use the data

§ Multiple Critical Paths– Microsoft Project™ lacks the capability to:

» Identify off critical path impacts on the critical path» Identify effects of branching

– This can be manually compensated for by:» Examine each activity’s criticality associated wit more than one

critical path» Path analysis may give substantially different vie of the schedule’s

uncertainties» Use bar chart to illustrate multiple critical paths

§ Monte Carlo examines ALL paths not just the critical path– Near critical paths are exposed



Simulation Considerations

§ Schedule logic and constraints– Simplify logic – model only paths which, by inspection, may have a

significant bearing on the final result– Correlate similar activities– No open ends– Use only finish–to–start relationships with no lags– Model relationships other than finish–to–start as activities with

base durations equal to the lag value– Eliminate all date constraints– Consider using branching for known alternatives



Simulation Considerations

§ Selection of Probability Distributions– Develop schedule simulation inputs concurrently with the cost

estimate» Early in process – use same subject matter experts» Convert confidence intervals into probability duration distributions

– Number of distributions vary depending on software– Difficult to develop inputs required for distributions– Beta and Lognormal better than triangular; avoid exclusive use of

Normal distribution



Sensitivity Analysis describes which tasks drive the completion times

§ Concentrates on inputs most likely to improve quality (accuracy)

§ Identifies most promising opportunities where additional work will help to narrow input ranges

§ Methods– Run multiple simulations– Use criticality index– “Tornado” or Pareto graph



Models of the Schedule

All models are lies. Some models are useful.– George Box

: Models of the Schedule

Concept generator from Ramon Lull’s Ars Magna (C. 1300)


Graphical Interpretations provide information that numbers alone can’t

§ Graphical view of confidence, contingency and target management assessment



What Can Confidence Intervals Tell Us about the validity of the IMS?

§ As the program proceeds so does– Increasing

accuracy– Reduced

schedule risk– Increasing

visual confirmation that success can be reached Current Estimate Accuracy



Confidence of schedule dates



Sensitivity Analysis

§ The schedule sensitivity of a task measures the closeness with which change in the task duration matches change in the project duration over the simulation.

§ This closeness is called correlation– Correlations can be derived from the sampling processes

§ A task with high schedule sensitivity is more likely to be a major driver of the project duration than a lower ranked task



Task Criticality Analysis

§ A measure of the frequency that an activity in the project schedule is critical (Total Float = 0) in a simulation

§ If a task is critical in 500 of the 1,000 iterations of the simulation, it has a Criticality Index of 0.5

§ The higher the criticality index, the more certain it is that the task will always be critical in the project



Schedule Cruciality describes how important each tasks is the risk tolerance

§ Cruciality = Schedule Sensitivity X Criticality§ Schedule Sensitivity can be statistically misleading:

– A task with high sensitivity may not be on or near the critical path– Thus a reduction in that task’s duration may have little effect on

the project duration§ Cruciality sharpens the analytical focus:

– It highlights critical or near–critical activities with high§ Schedule Sensitivity

– These tasks are most likely to drive project duration


Programmatic Risk Analysis 134/186: Models of the Schedule



Examples Of Monte Carlo

A simple overview of Risk+ shows how to produce an estimate of a project completion date. Interpreting this information takes thought and practice and most importantly reading and rereading the manual and the resources provided at the end of this presentation

: Examples of Monte Carlo

Sewall Wright’s probabilistic

network notation (1921)


The Monte Carlo Process starts with the PERT 3 point estimates

§ Estimates of the task duration are still needed, just like they are in PERT– Three point estimates could be used– But risk ranking and algorithmic generation of the “spreads” is a

better approach§ Duration estimates must be parametric rather than numeric

values– A geometric scale of parametric risk is one approach

§ Branching probabilities need to be defined– Conditional paths through the schedule can be evaluated using

Monte Carlo tools– This also demonstrate explicit risk mitigation planning to answer

the question “what if this happens?”



Expert Judgment is required to build a Risk Management approach

§ Expert judgment is typically the basis of cost and schedule estimates– Expert judgment is usually the weakest area of process and

quantification– Translating from English (SOW) to mathematics (probabilistic risk

model) is usually inconsistent at best and erroneous at worst§ One approach

– Plan for the “best case” and preclude a self–fulfilling prophesy– Budget for the “most likely” and recognize risks and uncertainties– Protect for the “worst case” and acknowledge the conceivable in

the risk mitigation plan§ The credibility of the “best case” estimates if crucial to the

success of this approach



Guiding the Risk Factor Process requires careful weighting of each level of risk

§ For tasks marked “Low” a reasonable approach is to score the maximum 10% greater than the minimum.

§ The “Most Likely” is then scored as a geometric progression for the remaining categories with a common ratio of 1.5

§ Tasks marked “Very High” are bound at 200% of minimum.

– No viable project manager would like a task grow to three times the planned duration without intervention

§ The geometric progress is somewhat arbitrary but it should be used instead of a linear progression

Min Most Likely

Max

Low 1.0 1.04 1.10Low+ 1.0 1.06 1.15

Moderate 1.0 1.09 1.24Moderate+ 1.0 1.14 1.36

High 1.0 1.20 1.55High+ 1.0 1.30 1.85

Very High 1.0 1.46 2.30Very High+ 1.0 1.68 3.00



Progressive Risk Factors

§ A geometric progression (1.534) of risk can be used§ The phrases associated with increasing risk have been shown

at the Naval Research Laboratory to correlate with an engineers “sense” of increasing risk



Risk Factor Attributes

§ The “narrative” for each risk factor needs to be developed

§ Each description may be dependent on…

– Discipline– Program stage– Complexity– Historical data– Current “risk state” of the program

§ This approach is similar to NASA’s Technology Readiness Level

§ This is currently missing from our efforts to quantify schedule and cost risk



Extreme Serial Task Example

§ Task completion distributions are “added” with little effect on the project completion estimate


Completion Std Deviation: 2.51 days95% Confidence Interval: 0.22 daysEach bar represents 1 day

Completion Date

Freq

uenc

y

Cum

ulat

ive

Prob

abilit

y2/10/062/1/06 2/23/06

0.1

0.2

0.30.4

0.5

0.60.7

0.8

0.91.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16



0.55 2/13/060.60 2/13/060.65 2/13/060.70 2/14/060.75 2/14/060.80 2/14/060.85 2/15/060.90 2/16/060.95 2/17/061.00 2/23/06



Extreme Parallel Task Example


Completion Std Deviation: 4.83 days95% Confidence Interval: 0.42 daysEach bar represents 2 days

Completion Date

Freq

uenc

y

Cum

ulat

ive

Prob

abilit

y3/1/062/10/06 3/17/06

0.1

0.2

0.30.4

0.5

0.60.7

0.8

0.91.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14



0.55 3/1/060.60 3/2/060.65 3/3/060.70 3/3/060.75 3/6/060.80 3/7/060.85 3/8/060.90 3/9/060.95 3/13/061.00 3/17/06

§ Task completion distributions are “added” with a large unfavorable effect on the project completion estimate distribution



A “Real World” Schedule Analysis

One should expect that the expected can be prevented, but the unexpected should have been expected— Augustine Law XLV

: “Real World” Schedule Analysis


The Baseline Schedule

§ Sample construction project plan



PERT Assessment

Most Likely

Min MaxPERT Adjusted Duration

PERT Adjusted Date

Original Target Date: 2/8/06



Risk+ Assessment

Most Likely

Min Max

Date: 10/4/2005 1:58:06 PMSamples: 23Unique ID: 143Name: Construction Schedule Margin

Completion Std Deviation: 1.55 days95% Confidence Interval: 0.63 daysEach bar represents 1 day

Completion Date

Freq

uenc

y

Cum

ulat

ive

Prob

abilit

y

2/9/062/3/06 2/14/06

0.1

0.2

0.30.4

0.5

0.60.7

0.8

0.91.0

0.05

0.10

0.15

0.20

0.25

0.30



0.55 2/9/060.60 2/9/060.65 2/9/060.70 2/10/060.75 2/10/060.80 2/10/060.85 2/10/060.90 2/10/060.95 2/10/061.00 2/14/06

Target Date

80% confidence



What is the Purpose of Project Risk Analysis?

What do users want from a project risk analysis?How accurate must we be to provide value to the program?Can we confirm this accuracy and integrity to build confidence in the projected completion date?

: What is the Purpose of Project Risk Analysis?

Risk appears in all aspects of spaceflight


Accuracy

§ Given a specified final cost or project duration, what is the probability of achieving this cost or duration?

§ Frequentist approach– Over many different projects, four out of five will cost less or be

completed in less time than the specified cost or duration§ Bayesian approach

– We would be willing to bet at 4 to 1 odds that the project will be under the 80% point in cost or duration

§ Accuracy is needed to plan reserves§ Accuracy is needed when comparing competing proposals



Structured Thinking

§ All estimates will be in error§ Trying to quantify these errors will result in bounds too wide to

be useful for decision making§ Risk analysis should be used to

– Think about different aspects of the project– Try to put numbers against probabilities and impacts– Discuss with colleagues the different ideas and perceptions

§ Thinking things through carefully results in– Which elements of the programmatic and technical risk are

represented in the IMS– The process becomes more valuable than the numbers


Programmatic Risk Analysis 150/186: What is the Purpose of Project Risk Analysis?



Basic Principles of Probabilistic Cost

Now that the schedule can be produced using probabilistic methods, it’s time to talk about the cost.Cost does not have a linear relationship with schedule unfortunately

: Basic Principles of Probabilistic Cost


Basic Principles with Probabilistic Cost Estimating

§ Cost estimates usually involve many CERs– Each of these CERs has uncertainty (standard error)– CER input variables have uncertainty (technical uncertainty)

§ Must combine CER uncertainty with technical uncertainty for many CERs in an estimate

– Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples

» Add and multiply probability distributions rather than numbers» Statistically combining many uncertain, or randomly varying, numbers

– Monte Carlo simulation» Take random sample from each CER and input parameter, add and multiply

as necessary, then record total system cost as a single sample» Repeat the procedure thousands of times to develop a frequency histogram

of the total system cost samples» This becomes the probability distribution of total system cost



The Cost Probability Distributions as a function of the weighted cost drivers

$

Cost Driver (Weight)

Cost = a + bXc

CostEstimate

Historical data point

Cost estimating relationship

Standard percent error boundsTechnical Uncertainty

Combined Cost Modeling and Technical

Uncertainty

Cost Modeling Uncertainty



Basic Principles of connected Cost with the IMS involve three steps

§ Step 1: Define “likely–to–be” program– Using deterministic inputs from the Independent Technical

Assessment (ITA)§ Step 2: Quantify the probability distributions describing the

modeling uncertainty of all CERs, cost factors, and other estimating methods– Specifically, the type of distribution (normal, triangular, lognormal,

beta, etc.)– The mean and variance of the distribution

§ Step 3: Quantify the correlation between all WBS elements that are estimated using CERs and other methods– If unknown, assess whether No correlation, Mild correlation, or

High correlation, for example:» None: r = 0, Mild: r = ±0.2, High: r = ± 0.6

– Correlation affects the overall cost variance



Basic Principles

§ Step 4: Set up and run the cost estimate in a Monte Carlo framework (e.g., Crystal Ball, @RISK), resulting in a “baseline” estimate– This will provide a probability distribution of the cost based on

cost estimating model uncertainty only– Report the MEAN as the baseline expected cost

§ Step 5: Now incorporate technical uncertainty and discrete risks– Step 5a: Set up a new estimate which also contains any “discrete

risk” events that are to be guarded against» Quantify appropriate modeling uncertainties and correlations, as in

Steps 2 and 3, for these discrete risks– Step 5b: Define the probability distributions for all CER input

variables» Also may need to quantify correlation between CER input variables



Basic Principles

§ Step 6: Re–run the Monte Carlo simulation with random CER input variables and discrete risk events, resulting in a final “risk–adjusted” estimate– Results in a new risk–adjusted cost probability distribution.– Wider and shifted to the right

Baseline vs. Risk-Adjusted Estimates

0 50 100 150 200 250 300 350FY$M

Like

lihoo

d



Basic Principles

§ Step 7: Assess “risk dollars”– Difference between the “risk–adjusted” mean and the “baseline”

mean represents the estimate of “risk dollars”– Risk dollars can be allocated downward to any level of WBS using

a variety of simple approachesRisk-Adjusted Estimate

0 100 200 300 400 500 600 700 800FY$M

Likeli

hood

Risk-AdjustedEstimate $346M

BaselineEstimate $270M

Risk Dollars Relative to BaselineEstimate $76M



Basic Principles

§ Step 8: Assess “budget risk”– Area under the PDF to the right of the budget represents budget

riskRisk-Adjusted Estimate

0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d

Risk-Adjusted Estimate

0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d

Risk-AdjustedEstimate$346M


Budget Risk= 51%

Budget Risk= 51%


0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d


0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d



Budget Risk= 51%

Budget Risk= 45%

Note: Assumes budget is setat the risk-adjusted estimate

expected value.

Note: Assumes budget is setat the risk-adjusted estimate

expected value.Budget = $346M

Budget = $346M



Basic Principles

§ Step 9: Assess “management reserve”– Difference between mean of risk–adjusted estimate and budget

represents management reserveRisk-Adjusted Estimate

0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d


0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d



Budget$405M

Budget$405M

UnencumberedMargin $59M




Budget$405M

Budget$405M




0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d


0 100 200 300 400 500 600 700 800FY$M

Like

lihoo

d



Budget$405M

Budget$405M





Budget$405M

Budget$405M


ManagementReserve $59M



Budget$405M

Budget$405M





Budget$405M

Budget$405M


ManagementReserve $59M

Note: If budget is set at risk-adjusted expected value, thenmanagement reserve is ZERO.

Note: If budget is set at risk-adjusted expected value, thenmanagement reserve is ZERO.



The Risk Adjusted Cost Estimate Connected To The IMS Is The Basis Of Risk Tolerance

§ In the risk–adjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs

§ Since the input distributions tend to be right–skewed, the expected cost tends to be larger than the baseline estimate

§ In addition, the risk–adjusted cost distribution tends to be wider than the baseline estimate

§ The difference between the expected cost of the risk–adjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of RISK dollars included in the risk–adjusted estimate



Baseline versus Risk Adjusted Cost Estimates Almost Always Show an Increase In Cost

Baseline vs. Risk-Adjusted Estimates

0 50 100 150 200 250 300 350FY$M

Like

lihoo

d

Baseline:Mean = $102.6MStd Dev = $29.8M

Risk–Adjusted:Mean = $122.6MStd Dev = $42.8M



The S–Curve for Cost Modeling

Cumulative Distribution Function

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

$60 $80 $100 $120 $140 $160 $180 $200

FY00$M

Cum

ulat

ive

Prob

abili

ty

Baseline EstimateMean $102.6M

50th percentile$114.7M

Risk–adjustedEstimate Mean$122.6M

80th percentile$153.5M



The Real Question Always Returns to…“But How Much Does It Cost? Really?”

§ This is impossible to answer precisely§ Decision–makers and cost analysts should always think of a

cost estimate as a probability distribution, NOT as a deterministic number

§ The best we can provide is the probability distribution – If we think we can be any more precise, we’re fooling ourselves

§ It is up to the decision–maker to decide where he/she wants to set the budget

§ The probability distribution provides a quantitative basis for making this determination– Low budget = high probability of overrun– High budget = low probability of overrun


Programmatic Risk Analysis 164/186: Basic Principles of Probabilistic Cost



Summary

At this point there is too much information. Processing of this information will take time, patience, and most of all practice with the tools and the results they produce.

But there are some fundamental conclusions that can be applied to our problem at hand –Phase II

: Summary

I’ll be happy to give you innovative thinking. What are my guidelines?

With our new found knowledge we need to break out of the habits of the past and start applying probabilistic cost and schedule analysis to our program


Conclusions

§ Project schedule status must be assessed in terms of a critical path through the schedule network

§ Because the actual durations of each task in the network are uncertain (they are random variables following a probability distribution function), the project schedule duration must be modeled statistically

: Summary


Conclusions

§ Quality (accuracy) is measured at the end points of achieved confidence interval (suggest 80% level)

§ Simulation results depend on:– Accuracy and care taken with base schedule logic– Use of subject matter experts to establish inputs– Selection of appropriate distribution types– Through analysis of multiple critical paths– Understanding which activities and paths have the greatest

potential impact

: Summary


Conclusions

§ Cost and schedule estimates are made up of many independent elements. – When each element is planned as best case – e.g. a probability of

achievement of 10% – The probability of achieving best case for a two–element estimate

is 1% – For three elements, 0.01%– For many elements, infinitesimal– In effect, it is zero.

§ In the beginning no attempt should be made to distinguish between risk and uncertainty– Risk involves uncertainty but it is indeed more– For initial purposes it is unimportant– The effect is combined into one statistical factor called “risk,”

which can be described by a single probability distribution function

: Summary


Points To Remember When we Build The Risk Tolerant IMS

§ Good project management is good risk management§ Risk management is how adults manage projects§ The only thing we manage is project risk§ Risks impact objectives§ Risks come from the decisions we make while trying to achieve

the objectives§ Risks require a factual condition and have potential negative

consequences that must be mitigated in the schedule

: Summary

Programmatic Risk Analysis 170/186: Basic Principles of Probabilistic Cost



More Opportunities to Improve our Processes

Here and elsewhere, we shall not obtain the best insights into things until we actually see them growing from the beginning.— Aristotle

: Summary


Improvement Opportunities

§ Statistical dependence of risk related activities§ Construction of probability distributions from actual project

data§ Connecting cost and schedule risk into a “program” risk

assessment model§ Causal models versus regression model

: Summary


The End

This is actually the beginning, since building a risk tolerant, credible, robust IMS requires constant “execution” and corrects to the plan.

A planning algorithm from Aristotle’s De Motu Animalium c. 400 BC

: Summary


Resources for Probabilistic Risk Analysis

Knowledge is of two kinds; we know a subject ourselves, or we know where we can find information upon it.— Samuel Johnson

: Resources


Resources

1. “The Parameters of the Classical PERT: An Assessment of its Success,” Rafael Herrerias Pleguezuelo, http://www.cyta.com.ar/biblioteca/bddoc/bdlibros/pert_van/PARAMETROS.PDF

2. “Advanced Quantitative Schedule Risk Analysis,” David T. Hulett, Hulett & Associates, http://www.projectrisk.com/index.html

3. “Schedule Risk Analysis Simplified,” David T. Hulett, Hulett & Associates, http://www.projectrisk.com/index.html

4. “Project Risk Management: A Combined Analytical Hierarchy Process and Decision Tree Approach,” Prasanta Kumar Dey, Cost Engineering, Vol. 44, No. 3, March 2002.

5. “Adding Probability to Your ‘Swiss Army Knife’,” John C. Goodpasture, Proceedings of the 30th Annual Project Management Institute 1999 Seminars and Symposium, October, 1999.

6. “Modeling Uncertainty in Project Scheduling,” Patrick Leach, Proceedings of the 2005 Crystal Ball User Conference

7. “Near Critical Paths Create Violations in the PERT Assumptions of Normality,” Frank Pokladnik and Robert Hill, University of Houston, Clear Lake, http://www.sbaer.uca.edu/research/dsi/2003/procs/237–4203.pdf

: Resources


Resources

8. “Teaching SuPERT,” Kenneth R. MacLeod and Paul F. Petersen, Proceedings of the Decision Sciences 2003 Annual Meeting, Washington DC, http://www.sbaer.uca.edu/research/dsi/2003/by_track_paper.html

9. “The Beginning of the Monte Carlo Method,” N. Metropolis, Los Alamos Science, Special Issue, 1987. http://www.fas.org/sgp/othergov/doe/lanl/pubs/00326866.pdf

10. “Defining a Beta Distribution Function for Construction Simulation,” Javier Fente, Kraig Knutson, Cliff Schexnayder, Proceedings of the 1999 Winter Simulation Conference.

11. “The Basics of Monte Carlo Simulation: A Tutorial,” S. Kandaswamy, Proceedings of the Project Management Institute Annual Seminars & Symposium, November, 2001.

12. “The Mother of All Guesses: A User Friendly Guide to Statistical Estimation,” Francois Melese and David Rose, Armed Forces Comptroller, 1998, http://www.nps.navy.mil/drmi/graphics/StatGuide–web.pdf

13. “Inverse Statistical Estimation via Order Statistics: A Resolution of the Ill–Posed Inverse problem of PERT Scheduling,” William F. Pickard, Inverse Problems 20, pp. 1565–1581, 2004

: Resources


Resources

14. “Schedule Risk Analysis: Why It Is Important and How to Do It, “Stephen A. Book, Proceedings of the Ground Systems Architecture Workshop (GSAW 2002), Aerospace Corporation, March 2002, http://sunset.usc.edu/GSAW/gsaw2002/s11a/book.pdf

15. “Evaluation of the Risk Analysis and Cost Management (RACM) Model,” Matthew S. Goldberg, Institute for Defense Analysis, 1998. http://www.thedacs.com/topics/earnedvalue/racm.pdf

16. “PERT Completion Times Revisited,” Fred E. Williams, School of Management, University of Michigan–Flint, July 2005, http://som.umflint.edu/yener/PERT%20Completion%20Revisited.htm

17. “Overcoming Project Risk: Lessons from the PERIL Database,” Tom Hendrick , Program Manager, Hewlett Packard, 2003, http://www.failureproofprojects.com/Risky.pdf

18. “The Heart of Risk Management: Teaching Project Teams to Combat Risk,” Bruce Chadbourne, 30th Annual Project Management Institute 1999 Seminara and Symposium, October 1999, http://www.risksig.com/Articles/pmi1999/rkalt01.pdf

: Resources


20. Project Risk Management Resource List, NASA Headquarters Library, http://www.hq.nasa.gov/office/hqlibrary/ppm/ppm22.htm#art

21. “Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition Quarterly, Spring 1999, http://www.dau.mil/pubs/arq/99arq/raymond.pdf

22. “Continuous Risk Management,” Cost Analysis Symposium, April 2005, http://www1.jsc.nasa.gov/bu2/conferences/NCAS2005/papers/5C_–_Cockrell_CRM_v1_0.ppt

23. “A Novel Extension of the Triangular Distribution and its Parameter Estimation,” J. Rene van Dorp and Samuel Kotz, The Statistician 51(1), pp. 63 –79, 2002. http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/TheStatistician2002.pdf

24. “Distribution of Modeling Dependence Cause by Common Risk Factors,”J. Rene van Dorp, European Safety and Reliability 2003 Conference

Proceedings, March 2003, http://www.seas.gwu.edu/~dorpjr/Publications/ConferenceProceedings/Esrel2003.pdf

Resources

: Resources


Resources

25. “Improved Three Point Approximation To Distribution Functions For Application In Financial Decision Analysis,” Michele E. Pfund, Jennifer E. McNeill, John W. Fowler and Gerald T. Mackulak, Department of Industrial Engineering, Arizona State University, Tempe, Arizona,http://www.eas.asu.edu/ie/workingpaper/pdf/cdf_estimation_submission.pdf

26. “Analysis Of Resource–constrained Stochastic Project Networks Using Discrete–event Simulation,” Sucharith Vanguri, Masters Thesis, Mississippi State University, May 2005, http://sun.library.msstate.edu/ETD–db/theses/available/etd–04072005–123743/restricted/SucharithVanguriThesis.pdf

27. “Integrated Cost / Schedule Risk Analysis,” David T. Hulett and Bill Campbell, Fifth European Project Management Conference, June 2002.

28. “Risk Interrelation Management – Controlling the Snowball Effect,” Olli Kuismanen, Tuomo Saari and Jussi Vähäkylä, Fifth European Project Management Conference, June 2002.

29. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, David Salsburg, W. H. Freeman, 2001

: Resources


Resources

30. “Triangular Approximations for Continuous Random Variables in Risk Analysis,” David G. Johnson, The Business School, Loughborough University, Liecestershire.

31. “Statistical Dependence through Common Risk Factors: With Applications in Uncertainty Analysis,” J. Rene van Dorp, European Journal of Operations Research, Volume 161(1), pp. 240–255.

32. “Statistical Dependence in the risk analysis for Project Networks Using Monte Carlo Methods,” J. Rene van Dorp and M. R. Dufy, International Journal of Production Economics, 58, pp. 17–29, 1999. http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Prodecon1999.pdf

33. “Risk Analysis for Large Engineering Projects: Modeling Cost Uncertainty for Ship Production Activities,” M. R. Dufy and J. Rene van Dorp, Journal of Engineering Valuation and Cost Analysis, Volume 2. pp. 285–301, http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/EVCA1999.pdf

34. “Risk Based Decision Support techniques for Programs and Projects,” Barney Roberts and David Frost, Futron Risk Management Center of Excellence, http://www.futron.com/pdf/RBDSsupporttech.pdf

: Resources


Resources

35. Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners, Office of Safety and Mission Assurance, April 2002. http://www.hq.nasa.gov/office/codeq/doctree/praguide.pdf

36. “Project Planning: Improved Approach Incorporating Uncertainty,” Vahid Khodakarami, Norman Fenton, and Martin Neil, Track 15 EURAM2005: “Reconciling Uncertainty and Responsibility” European Academy of Management. http://www.dcs.qmw.ac.uk/~norman/papers/project_planning_khodakerami.pdf

37. “A Distribution for Modeling Dependence Caused by Common Risk Factors,” J. Rene van Dorp, European Safety and Reliability 2003 Conference Proceedings, March 2003.

38. “Probabilistic PERT,” Arthur Nadas, IBM Journal of Research and Development, 23(3), May 1979, pp. 339–347.

39. “Ranked Nodes: A Simple and effective way to model qualitative in large–scale Bayesian Networks,” Norman Fenton and Martin Neil, Risk Assessment and Decision Analysis Research Group, Department of Computer Science, Queen Mary, University of London, February 21, 2005

: Resources


Resources

40. “Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition Review Quarterly, Spring 1999, pp. 147–154

41. “The Causes of Risk Taking by Project Managers,” Michael Wakshull, Proceedings of the Project Management Institute Annual Seminars & Symposium, November 2001.

42. “Stochastic Project Duration Analysis Using PERT–Beta Distributions,” Ron Davis

43. “Triangular Approximation for Continuous Random Variables in Risk Analysis,” David G. Johnson, Decision Sciences Institute Proceedings 1998. http://www.sbaer.uca.edu/research/dsi/1998/Pdffiles/Papers/1114.pdf

44. “The Cause of Risk Taking by Managers,” Michael N.Wakshull, Proceedings of the Project Management Institute Annual Seminars & Symposium November 1–10, 2001, Nashville Tenn, http://www.risksig.com/Articles/pmi2001/21261.pdf

45. “The Framing of Decisions and the Psychology of Choice,” Tversky, Amos, and Daniel Kahneman. 1981, Science 211 (January 30): 453–458, http://www.cs.umu.se/kurser/TDBC12/HT99/Tversky.html

: Resources


Resources

46. “Three Point Approximations for Continuous Random Variables,” Donald Keefer and Samuel Bodily, Management Science, 29(5), pp. 595 – 609

47. “Better Estimation of PERT Activity Time Parameters,” Donald Keefer and William Verdini, Management Science, 39(9), pp. 1086 – 1091.

48. “The Benefits of Integrated, Quantitative Risk Management,” Barney B. Roberts, Futron Corporation, 12th Annual International Symposium of the International Council on Systems Engineering, July 1–5, 2001, http://www.futron.com/pdf/benefits_QuantIRM.pdf

49. “Sources of Schedule Risk in Complex Systems Development,” Tyson R. Browning, INCOSE Systems Engineering Journal, Volume 2, Issue 3, pp. 129 –142, 14 September 1999, http://sbufaculty.tcu.edu/tbrowning/Publications/Browning%20(1999)––SE%20Sch%20Risk%20Drivers.pdf

50. “Sources of Performance Risk in Complex System Development,” Tyson R. Browning, 9th Annual International Symposium of INCOSE, June 1999, http://sbufaculty.tcu.edu/tbrowning/Publications/Browning%20(1999)––INCOSE%20Perf%20Risk%20Drivers.pdf

: Resources


Resources

51. “Experiences in Improving Risk Management Processes Using the Concepts of the Riskit Method,” Jyrki Konito, Gerhard Getto, and Dieter Landes, ACM SIGSOFT Software Engineering Notes , Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering SIGSOFT '98/FSE-6, Volume 23 Issue 6, November 1998.

52. “Anchoring and Adjustment in Software Estimation,” Jorge Aranda and Steve Easterbrook, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering ESEC/FSE-13

: Resources


Web Resources

§ Decision Sciences Institute, http://www.decisionsciences.org/§ Tyson R. Browning, http://sbufaculty.tcu.edu/tbrowning/index.htm§ The Risk Doctor, http://www.risk-doctor.com/§ Canadian Treasury Board Policy for the Management of Projects,

http://www.tbs-sct.gc.ca/pm-gp/site/home_accueil-eng.aspx

: Resources


Thank You for Your Patience

: Resources

Documents

Probabilistic Schedule and Cost Analysis