IS SIG Meeting, April 8, 2008 Understanding Technology Project Risks and Predicting Project Performance Dr. Andrew Gemino Simon Fraser University [email protected]

IS SIG Meeting, April 8, 2008

Understanding Technology Project Risks and Predicting Project

Performance

Dr. Andrew GeminoSimon Fraser University

[email protected]

IS SIG Meeting, April 8, 2008 2

Agenda

• Introduction• Part 1: IT Project Performance• Part 2: Elements of Performance

• Discussion: How does your company evaluate performance

• Part 3: Predicting IT Project Performance• Part 4: PM Practice and Performance


Motivation for My Research

• Application of Information Technology plays an important role in Canada’s capacity to innovate• The Info. And Comm. Technology Sector

• Contributes $61 bil to Canadian GDP (in 1997 constant dollars)• Comprises 5.8% of Canadian GDP• Employs over 560,000 Canadians• Responsible for 38% of all private sector R&Dhttp://strategis.ic.gc.ca/epic/site/ict-tic.nsf/en/h_it06143e.html

• Our focus is to advance our ability to understand and influence IT project performance.


The Team

• Blaize Horner Reich, Simon Fraser University• Chris Sauer, Oxford University, UK• Andrew Gemino, Simon Fraser University

• Website: www.PMPerspectives.org

• Funding: • Social Sciences and Humanities Research Council

(SSHRC) • Initiative for the New Economy (2003-2006)• Research Grant funding (2007-2010)• INE Communication Grant

• Natural Sciences and Engineering Research Council (NSERC)

• Research Grant Funding


Some Recent Output

• Sauer, C., Gemino, A, and Reich, B.H. “Managing Projects for Success: The Impact of Size and Volatility on IT Project Performance”, Communications of the ACM, 60:11, Nov. 2007, pp. 79-84.

• Gemino, A., Horner-Reich, B. and Sauer, C. "A Temporal Model for IT Project Management", Journal of MIS, Winter 2007–8, Vol. 24, No. 3, pp. 9–44 (awaiting publication).

• Gemino, A., Reich, B. and Sauer, C. "Beyond Chaos: Examining IT Project Performance" Proceedings of the 2nd Annual International Research Workshop for the Special Interest Group for Information Technology Project Management (SIGITProjMgmt), Montreal, Canada, Dec. 8, 2007.

• Reich, B.H., and Sauer, C. “Myths About Information Technology Project Performance”, Proceedings of the Administrative Sciences Association of Canada (ASAC) 2007, Ottawa, Canada, June 2-5, 2007.

• Gemino, A., Reich, B.H. “Estimating Risk in Information Technology Projects”, Proceeding if the Americas Conference on Information Systems, Keystone, Colorado, Aug. 9-12, 2007.

5


PART 1: IT Project Performance


The Standish Report

• The Chaos Report (1994)• Many have heard the numbers

• On time, budget and scope – 16%• Challenged 53%• Abandoned 31%

• Average Performance• Cost overruns 189%• Time overruns 222%• % of Original Specs 61%

• http://www.standishgroup.com/sample_research/chaos_1994_1.php


Standish Surveys Show Improvement

0%

10%

20%

30%

40%

50%

60%

Succeeded Failed Challenged

1994

1996

1998

2000

2002

2004

2006

(complied from press releases from www.standishgroup.com )(complied from press releases from www.standishgroup.com )


Some Questions about Standish Figures

• “Most such academic papers and guru reports cite the same source for their crisis concern—a study published by the Standish Group more than a decade ago”

Robert Glass, CACM, 2006

• Reasons to doubt the Standish figures• Inherent bias due to perspective on “failure” • Tough test for “success” – all targets at 100%• Sampling method is very unclear

• what projects are in the sample?


Survey Research

Study 1: Computer Weekly UK

• Web based survey of• Readers and PM’s

• Asked IT project managers about their last project (completed or abandoned)

• 421 full responses• Average 17 years industry

experience• Average 9 years as project

manager

Study 2: PMI Chapters in Ohio

• Web based survey• 3 PMI chapters in Ohio

• Asked IT project managers about their last project (completed or abandoned)

• 194 full responses• Average age, 43• Average years experience,

15• Average PM training: 34

days

10


What we found

• 65% of projects in the UK and 66% US samples had good performance• Delivered within a reasonab;le contingency (approx

7%) of ALL targets. • Compare with 63% challenged or abandoned in

Standish

• What’s the difference? • Experienced Project Managers• Data collection

• We asked for actual variances from original goal

• Cluster Analysis• Didn’t use hard line (99% is not challenged)• “let the data speak for themselves”

11


IT Project Types: UK Study

PerformanceVariance

Type 1:

Abandoned

n=39

Type 2:

BudgetChallenged

n=22

Type 3:

ScheduleChallenge

dn=76

Type 4:

Good

n=250

Type 5:

Stars

n=31

Performance Variances (Actual as % of Originally Planned) – 100%

Schedule N/A +18% +46% +2% +1%

Budget N/A +40% +16% +7% -24%

Scope N/A -12% -16% -7% +15%


UK - Project Characteristics

Size Characteristi

c

Type 1:Abandon

Type 2:Budget

Challeng

Type 3:ScheduleChalleng

Type 4:Good

Type 5:Star

Budget(Median, in ₤

000’s)₤ 1,000 ₤ 625 ₤ 500 ₤ 450 ₤ 2,000

Effort(Average

Person Months)

798 557 212 89 170

Duration (Average

Elapsed Time in Months)

17.4 20.0 13.0 11.2 15.3

Team Size(Effort/

Duration) 35.7 17.7 12.9 7.3 9.8


IT Project Types: US Study

Performance Variance

Type 1:Abandoned

Projectsn=16

Type 2:

Challengedn=13

Type 3:Schedule

Challengedn=36

Type 4:

Goodn=87

Type 5:

Starsn=32

Performance Variances (Actual as % of Originally Planned – 100%)

Schedule N/A +41% +107% +4% +1%

Budget N/A +25% +43% +3% -12%

Scope N/A -12% -4% -9% +3%


US - Project Characteristics

Size Characteristi

c

Type 1:Abandon

Type 2:Budget

Challeng

Type 3:ScheduleChalleng

Type 4:Good

Type 5:Star

Budget(Median, in

US $ 000’s)$2,000 $700 $1,042 $670 $600

Effort(Average

Person Months)

N/A 126.4 163.2 255.1 66.5

Duration (Average

Elapsed Time in Months)

N/A 16.7 16.5 15.8 9.5

Team Size(Effort/

Duration) N/A 6 11.0 16.4 6.0


Surprises:Some IT Projects Exceed

Expectations1. The IT Performance story is not all bad

1. 2/3rds of projects are performing well2. Some IT projects exceed expectations

1. 7% of UK2. 17% of US projects

• These projects that exceed expectations are not mentioned in Standish Group Reports • We found them because we did not constrain

“success” in the data collection

16

IS SIG Meeting, April 8, 200817

IT Project Performance

The Impact of Size and Volatility


Risk Associated with Project Size

1(a) Risk of Underperformance due to Effort

25%31%

36% 36% 38%

50%

77%

100%

0%

33%

67%

100%

25 or less 25-50 50-100 100-200 200-500 500-1000 1000+ 2400+

Effort (person months)



1(b) Risk of Underperformance due to Duration

24%21%

34%

25%

36%

48%

0%

33%

66%

< 3 months 3-6 months 6-9 months 9-12 months 12-18months

18+ Months

Duration (months)



1(c) Risk of Underperformance due to Team Size

28%

38%

26%

33%

58%

0%

33%

67%

<5 FTE 5-10 FTE 10-15 FTE 15-20 FTE >20 FTE

Team Size (Effort / Duration)


Risk Associated with Volatility

2(a) Risk of Underperformance due to Governance Volatility

22%33%

60%53%

74%82%

0%

33%

67%

100%

0 changes 1 change 2 changes 3 changes 4 changes 5+changes

Governance Volatility (changes in PM or Sponsor)


Risk Associated with Volatility

2(b) Risk of Underperformance Due to Project Target Volatility

11%

25%

34%

57%

0%

33%

67%

< 3 changes 3-6 changes 6-9 changes 9+ changes

Project Target Volatility (changes to project targets)


Recommendations from UK Study

What level of performance can organizations expect?

Guideline From experienced project managers, 2 out of 3 projects should be implemented within small variances of budget, scope and schedule.

Rationale Two thirds of our sample were better-performers. On average these projects come within 7% of all targets.



What levels of risk are normal for IT projects?

Guideline For any project, the baseline risk of underperforming is 25%.

Rationale Even the smallest and shortest projects in the sample had on average a 25% chance of underperforming.

Guideline To keep projects below a 50% risk of underperforming, limit them to -less than 500 person months of effort.-under 18 months duration and -team size less than 20

Rationale Size is an issue because large teams increase coordination costs; long projects increase the risk of external change and lose momentum.

The best single measure of risk from size has proved to be effort. However, risk cannot be managed by reducing one dimension of size at the expense of others.



What can organizations do to reduce IT project risk?

Guideline Develop and retain or hire experienced project managers

Rationale Experienced project managers achieved better results than have been previously reported.

Guideline Keep projects short and manageable without being obsessively minimalist

Rationale We found little difference in risk between 3 and 12 month projects.

Guideline Plan and budget for continuity in leadership during the project. This may involve incentives for the project manager or understudies for key roles.

Rationale Changes in key personnel occur for personal, performance and strategic reasons. A PM change costs the project 15% of targets, so resources put towards avoiding this risk are a good investment.

Guideline Change targets only for strategic reasons (e.g. changes to increase business value of the project)

Rationale In the aggregate, changes in targets are disruptive and reduce performance.


PART 2: Group Discussion Elements of Performance: Let’s hear

from you –

How does your company measure Project Performance?


Are Budget and Schedule variances good measures of IT Project

Performance?

27

http://www.codinghorror.com/blog/images/software_engineering_explained.gif


How should performance be measured?

• Two basic outcomes in project• Process Outcomes

• On time• On budget• On specs

• Product Outcomes• Value delivered• Benefits• Quality

PM’s are often measured here

But are expected to deliver here


Question

• Does your company consider both product and process when considering evaluating performance?

• Should this be done? And if so, how?


PART 3: Predicting IT Project Performance

What factors determine IT Project Performance?


What Affects Performance

• Risk Factors and Resources• Initial (A-priori) Factors

• Knowledge resources Team. PM, Sponsor, Clients

• Structural Factors Technical complexity, budget, duration, effort

• Emergent Factors• Organizational Resources

Top Management Support, User participation• Volatility

Governance changes, Target Changes, Environment changes


What Affects Performance

• Project Management Practice• Expertise Coordination

• Who are knowledge leaders• How can they be accessed

• Horizontal Communication• Across team and clients

• PM Methods and Tools• Traditional PM


US Study: Results

Knowledge Resources

Structural Risk

(Size & Tech Complexity)

OrganizationalSupport

Rsources

Project Management

Practices

VolatilityRIsk

ProjectProcess

Performance

ProjectProduct

Performance

0.292**

0.290**

0.502**

0.298**

0.359**

R2 = 0.361

R2 = 0.269

R2 = 0.076

R2 = 0.219

R2 = 0.385

-0.321**

0.396**

0.154*

-0.209*

A-Priori Emergent Performance


Results

• With this model we can explain• Approximately 40% of the variance in Process

performance• Approximately 22% of the variance in Product

performance

• We can also show the strength of the relationships between risk and resource factors associated with IT Projects. • Strength and direction is given by the number

associated with each line.


Results

• Two Broad Forces Acting on Projects 1. Forces of Evil

• Structural risk is strongly related to volatility and volatility is negatively related to process performance

• Large projects often have a bumpy ride and are challenged in regards to being on time and budget.

35

Structural Risk

(Size & Tech Complexity)

VolatilityRIsk

ProjectProcess

Performance

0.502**

R2 = 0.361

R2 = 0.385

-0.321**


Results

Knowledge Resources

OrganizationalSupport

Rsources

Project Management

Practices

ProjectProcess

Performance

ProjectProduct

Performance

0.292**

0.290**

0.298**

0.359**

R2 = 0.269

R2 = 0.076

R2 = 0.219

R2 = 0.385

0.396**

0.154*

2. Forces of Good• Increased knowledge is

related to higher organizational support and higher level of PM Practices.

• Increased PM Practice is related to better process and PRODUCT performance


Summary

• IT Project performance is a critical issue in Canada’s ability to innovate.

• We have a good idea of the factors that affect

process performance. More work needs to be done. • It is important to consider initial factors as well as

emergent factors when considering performance.

• We do not yet have a good understanding of how these factors affect product performance• More work required


Questions?

If you are interested in more information please visit: www.PMPerspectives.org

Documents

IS SIG Meeting, April 8, 2008 Understanding Technology Project Risks and Predicting Project Performance Dr. Andrew Gemino Simon Fraser University [email protected]