Www. Gilb.com Slide 1 1 "Why software projects fail, how to diagnose potential failure, and how to make your projects succeed. With Focus on “12 Tough

www.Gilb.com Slide 1

1

"Why software projects fail, how to "Why software projects fail, how to diagnose potential failure, and howdiagnose potential failure, and how

to make your projects succeed.to make your projects succeed.With Focus on “12 Tough Questions”With Focus on “12 Tough Questions”

By Tom GilbBy Tom [email protected]@Gilb.comwww.Gilb.comwww.Gilb.com

Duration: 90 minutesDuration: 90 minutes

Univ. of Westminster, March 11 2009, 6-8PMUniv. of Westminster, March 11 2009, 6-8PM


2

• Letters to the Editor ("The Times"), August 19, 2002, No easy task to start IT projects– From Professor Martyn Thomas,

• Sir, Your leading article (August 14) rightly focuses attention on the unacceptable rate of failure of IT projects, although contrary to what you suggest the problems are not necessarily worse in the public sector.

• A survey of 1,027, mainly private sector, IT projects published in the 2001 British Computer Society Review showed that only 130 (12.7 per cent) succeeded.

• SoftwareSoftware development projects were worst development projects were worst: : of more than 500 projects surveyed, only of more than 500 projects surveyed, only three were regarded as having succeededthree were regarded as having succeeded.

• As you report (Business, August 14) the problems are similar in the US. I have audited many software projects in both public and private sector organisations and in my experience the many and complex reasons for failures boil down to this :

– most customers and most suppliers fail to recognise that complex software development is an most customers and most suppliers fail to recognise that complex software development is an engineering task. engineering task.

• Every step needs engineering rigour, based on sound computer science and supported by formal quality control.

– Companies who work this way produce software much closer to their planned timescales and at lower cost. – Their customers are asked detailed questions to fix the requirements early in the project, and every subsequent change

can be quickly analysed to see what it will cost, in money and time. – Their software contains so few errors that maintenance costs are negligible.

• Most of the worldwide software industry treats software development as a craft that requires only technician-level skills.

– Until that changes, projects will continue to suffer massive delays and cost overruns.

• Regards,MARTYN THOMAS,Visiting Professor of Software Engineering,Oxford University Computing Laboratory,The Wolfson Building,Park Road, Oxford OX1 3QD.August 14, 2002, [email protected]

3 of 500 Sw projects Succeed


3Talk Summary1. • 90% of all software projects, according to

several surveys, are partly or totally failed. 2. This is due to the project management methods we

have been using, mainly from the Anglo-American culture.

3. There are a large number of systematic causes of project failure.

4. This talk will list some fundamental failure causes and their cures.

5. It will not deal with all possible causes. 6. The first part is methods for diagnosing failure:

knowing you are on the wrong path. 7. The second part is avoiding or preventing failure

and 8. the third part is creating and controlling successful

projects.


410 principles of failure analysisThings you can do to ‘sense’ failure during a project

1. You must decide and enforce rules of requirement specificationRule: All quality requirements must be quantified.

2. You must decide the level of specification-defects (rule violations) that is acceptable (max. 1 Major Defect/Page?)

3. You must focus on the top level requirements, and they must be quantifiedTop 10 critical requirements: drive entire project, and can be tracked continuously

4. You must define key stakeholders, and their values, to derive complete requirements35 Key Stakeholders, 1 or more requirements each: not just ‘users’

5. You must define top level strategies that measurably meet your objectives within your constraints - and estimate that they do

Top 10 strategies for meeting top requirements: estimate their numeric impact

6. You must prioritize early partial delivery of ‘highest stakeholder value’, firstEstimate value at each step. Choose step content with highest value of all possible steps now.

7. You must control costs, together with benefits, on a continuous basisEstimate benefits/costs in weekly steps, measure benefits/costs end of week, cumulative

8. You must plan to measure stakeholder results on an early, frequent, and continuous basisSpend 1 hour ±30 minutes per week to test and measure value delivered to stakeholders

9. You must plan to change all plans, based on current and cumulative experience feedbackBe prepared to change, requirements, designs, time estimates, cost estimates, work processes

10. You must plan to reward your team for benefits actually delivered - not effort.Reward teams for actual measured required results testably delivered.


5TWELVE TOUGH QUESTIONS1. Why isn't the improvement

quantified?2. What is degree of the risk

or uncertainty and why?3. Are you sure? If not, why

not?4. Where did you get that

from? How can I check it out?

5. How does your idea affect my goals, measurably?

6. Did we forget anything critical to survival?

7. How do you know it works that way? Did it before?

8. Have we got a complete solution? Are all objectives satisfied?

9. Are we planning to do the 'profitable things' first?

10. Who is responsible for failure or success?

11. How can we be sure the plan is working, during the project, early?

12. Is it ‘no cure, no pay’ in a contract? Why not?

There is a detailed paper on these “12 Tough Questions” at www.Gilb.com


6Questions to Get Management Involvedin avoiding project failure

The 12 Tough questionsAre a way to get managers interested in metrics

The Tough Questions lead directly• to quantifiedquantified management objectives• to quantifiedquantified project and product requirements

• to the use of impact estimation tables–For evaluating alternative solutions

quantitatively


7•Figure 1: Real (NON-CONFIDENTIAL version) example of an initial draft of setting the objectives that engineering processes must meet.


8Strategy Impact Estimation


91. NUMBERS

Why isn’t the improvement quantified?


102. RISK

What’s degree of the risk or

uncertainty and why?


113. DOUBT

Are you sure? If not, Why

not?


12

4. SOURCE

Where did you get that from?

How can I check it out?


13

5. IMPACT

How does your idea affect my

goals, measurably?


146. ALL CRITICAL FACTORS

Did we forget anything critical

to survival?


157. EVIDENCE

How do you know it works that way? Did it before?

"The most formidable weapon against errors of every kind is reason."

--Thomas Paine

"Facts are stubborn things; and whatever may be our wishes, our inclinations, or the dictates of our passions, they cannot

alter the state of facts and evidence.” --John Adams

(Argument in Defense of the [British] Soldiers in the Boston Massacre Trials [December 1770]


168. ENOUGH

Have we got a complete solution? Are all objectives

satisfied?


179. PROFITABILITY FIRST

Are we planning to do the profitable

things first?


1810. COMMITMENT

Who’s Responsible for failure or

success?


1911. PROOF

How can we be sure the plan is

working, •during the project, •early?


20

Is it ‘no cure, no pay’, in the contract? Why not?

12. NO CURE

No cure

No pay!

See paper: No Cure No Pay (2006, Tom Gilb). Roots 2005


2110 Principles of Preventing Project Failure

1. Keep in touch with critical stakeholders, early, continuously2. Do the critical top requirements quantitatively and with rich

background information (justification, authority, benchmarks etc.)3. Drive majors defects per page of specs down from over 100 to <10 in six

months; and then in a year or so to less than one4. Make sure you have real-project time process improvement at the

project level. Learn fast what works.5. Measure real benefits delivered, not tasks and effort6. Plan to deliver critical benefits very early, and cumulate them, and lock

them in7. Reward success, and punish failure, continuously and early8. Give project team power to find the means; focus on controlling the

results, not the means9. Make sure all strategies (designs, architectures) are estimated with Impact

Estimation, using evidence, uncertainty and sources.10. Make sure suppliers are on an evolutionary contract, and paid by results

only


22The Simplest and Best Agile Project Method; ‘XE’

Process Description 1. Gather from all the key stakeholders the top few (5 to 20) most

critical goals that the project needs to deliver. Give each goal a reference name (a tag).

2. For each goal, define a scale of measure and a ‘final’ goal level. For example: Reliable: Scale: Mean Time Before Failure, Goal: 1

month.3. Define approximately 4 budgets for your most limited resources

(for example, time, people, money, and equipment).4. Write up these plans for the goals and budgets

(Try to ensure this is kept to only one page).5. Negotiate with the key stakeholders to formally agree the goals

and budgets.6. Plan to deliver some benefit

(that is, progress towards the goals) in weekly (or shorter) increments (Evo steps).

7. Implement the project in Evo steps. Report to project sponsors after each Evo step (weekly, or shorter)

with your best available estimates or measures, for each performance goal and each resource budget.

On a single page, summarize the progress to date towards achieving the goals and the costs incurred.

8. When all Goals are reached: ‘Claim success and move on’ a. Free remaining resources for more profitable ventures

1

2

4

6

7

8

5

3


23Agile project Management; XE Policy

Policy1. The project manager, and the project, will be judged

exclusively on 1. the relationship of progress towards achieving the goals 2. versus the amounts of the budgets used. 3. The project team will do anything legal and ethical to deliver the

goal levels within the budgets.

2. The team will be paid and rewarded for 1. benefits delivered 2. in relation to cost.

3. The team will find their own work process and their own design.

4. As experience dictates, the team will be free to suggest to the project sponsors (stakeholders) adjustments to ‘more realistic levels’ of the goals and budgets.


2410 Principles for Project Success. How to make sure you deliver what is really valued

1. Keep in intimate contact with key stakeholders (weekly basis, Evo delivery)2. Quantify, and verify, and keep updating - the critical few stakeholder values3. Derive project requirements from the stakeholder values on the basis of

most profitable stakeholder values are supported by specific project requirements

4. Build systems evolutionarily, using existing systems and environments as departure basis, for delivering results - even if longer vision is radically different.

Prove concepts and deliver real results.

5. Do your project from a systems engineering point of view - not a ‘programmers view’. Plan and do all the things it takes to successfully deliver real results to real stakeholders.

6. Line up rewards system so that stakeholder view, management view, project view is pulling in same direction and rewarded for same thing

7. Teach all employees good planning and specification practices, using formal rules, formal sampling measures of rule conformance, and minimum quality level exit control: don’t even let ‘junk specs’ out your process door!

8. Avoid all pressure in the direction of bureaucratic development processes (like CMM, CMMI, Waterfall Model) - focus on results.

9. Consider changing and improving your entire organizational climate, a CEO Job, to be more receptive to change.

10. Get top management (CEO, MD, President, COO, CTO) to establish and follow up simple results oriented policy.

(no majors in specs, measurable benefits early, highest benefit to costs first. That’s all.)


25

Last Slide

[email protected]+47 920 66 705

+44 77 1 267 0707


26Backup Slides

•To answer questions•To illuminate in more detail• In case there is sudden additional time to the 30 minutes


27Sensing Problems in Existing Projects

How would you know that a project is on a failure path?Early?

Quantitative sampling of requirements specs show high Major Defect Density (like 100 majors per page)

Early Planned Evolutionary result delivery steps fail to deliver planned value, or take far more time to do so than planned.

Quantitatively? See aboveCredibly? See above


28

Example of Evo Step Tracking

Courtesy Trond Johansen, FIRM AS, Norway.Paper: “From waterfall to EVO in a medium size Norwegian software house. The path and

experiences”. 2004

Step 9 is a weekly step


29

FIRM EVO week


30Real Inspection of System Requirements Specification (SRS) of 82 pages. Major US Corporation.

This presentation shows how to carry out a short specification quality control with senior/middle managers.

The purpose is to make managers aware that they play a key-role in creating project delays by approving poor quality of requirements documents.

The inspection results shown in this real-life example successfully predicted a project delay of at least 2 calendar years.

Poor quality marketing requirements documents prove time and again to be a good predictor of project delays.

The clue is that requirements documents with a high defect density are an indicator of a truly unprofessional engineering culture.


31Framework

Demonstration of power of Inspection8 Managers2 hours4 real requirements specifications


32“Rules”: Best Practice Strong Advice

Three Rules for Requirements:

1. Unambiguous to intended Readership

2. Clear enough to test.

3. No Design specs (= ‘how to- be good’) mixed in

Mixed up in the Requirements

(Reqts. = ‘how good - to be’) not how to be good!

MARK Design as “D”

Except if it is a conscious Design Constraint (which is a requirement type)

Introduce the following three rules for inspecting a requirements document:


33Specification Defect

A Spec Defect is a violation of a Rule

Note: If there are 10 ambiguous terms in a single requirement then there are 10 defects!9 of 10 removed, leaves 1 defect

Explain the definition of a Defect:


34Severity

Major: a defect severity where there is potential of high (10 lost engineering hours) losslater downstream (test, field).

Explain:• the definition of Major Defect• the checkers must focus on finding Major Defects


35Exit?

Exit Conditions: (when Requirements can go to Design, Test etc with little risk)

Maximum 1 Major Defect/ (Logical) Page

Logical Page = 300 Non commentary words.

Agree with the management team on a numeric exit condition: Is 1,000 Majors per page OK? 100, 10, 1?


36The JobI gave the 8 managers

(4 did page 81, 4 did page 82)

You have up to 30 minutes for checking 1 requirements page from an 82 page document

Count all potential Rule Violations = Defects

Classify defects as Major or minor


37Report for Page 81 (reported inspection results on requirements document, 4 managers)

24= Total, 15=Majors, 5 = Design44, 15, 1955, 20, 422, 4, 2

Defect Density estimation for this page =Total for group 20 x2 = 40 Majors assume are uniqueIf 33.333% effective, total in page = 3x 40=120Of which 2/3 or 80 were not yet found.If we fix all we found (40), then the estimated remainder of Majors

would be 80 (not found) +8 not fixed for real = 88 Majors remaining.


38Report for page 82

(reported inspection results on requirements document, 4 other managers)

Total, Majors, DefectsM+m M Design (Design = part of Total and M&m) ---------------------------------41, 24, D=133, 15, D=5

44, 30, D=10

24, 3, D=5

Team would (we estimate) log unique Majors about 2x30=60 actually foundWhich is 1/3rd of total existing , so estimated total this page is about 180 Majors

About 2/3, = 120 Majors , were not found by the team during the agile spec QC process.Not all 180 spec defects (only 25%-35%) would actually become product defects, before testing.

We don’t know which ones in advance: it depends on human interpretation, on the day.

Many (only 25%-35%) of these Major spec defects would cause product defects (bugs in product, not just spec) These would be found later in test,and product use as bugs or major defects

The product defect rework would cause serious project (product development) delay

No human review process is better, even when continuously improved for years, than 60% (source code record) to 88% (the ‘logic spec’ record) at defect identification. <- IBM MN.

If we attempt to fix the 60 defects we logged, and correctly fix 5/6 of them; then 10 defects are ‘failed fixes’, so:The total remaining after inspection and editing = 10+120 =130 Majors per page.

10 failed fixed +120 not detected major defects= 130 Major defects per page ( 300 non commentary words) remaining after the review and fixing process - before text.

Conclusion:Human defect removal by Inspections/reviews/SQC is a hopeless cause: not worth itSpec QC can be used, in spite of imperfect effectiveness, to accurately estimate major defect level per page.This measurement can be used to motivate engineers to dramatically (100x! Over about 7 learning cycles) reduce their defect insertion ( rule violation) to a

practical exit level ( like < 1.0 Major/page)

18060120


39Extrapolation to

Total Majors in Whole Document

Page 81: 120 majors/pPage 82: 180 Majors/pAverage 150 Majors/page x 82 page = 12,300 Majors in

the document.-----------------If a Major has 1/3 chance of causing lossAnd each loss is avg 10 hours then total project Rework

cost is about 41,000 hours loss.(This project was over a year late)

1 year = 2,000 hour 10 people

Documents

Www. Gilb.com Slide 1 1 "Why software projects fail, how to diagnose potential failure, and how to make your projects succeed. With Focus on “12 Tough