Chief Skeptical Officer – A New C-Level Role
Alan McSweeney
October 18, 2011 2
Why Does This Happen?
The Big Idea
The Big FailureAnd Continue To Happen Time and
Again?
And What Can Be Done to Stop It?
Reduced Functionality
Requiring Manual
Workarounds
Not What Is Wanted Or What Was Required/ Envisaged
Too Late
Too Expensive
High Cost Of Operation
Not Scalable And/Or Poor Performance
Lack Of Integration
Requires Rework Or
Replacement
October 18, 2011 3
Spectrum Of Failures
Complete Project Success:
On-time, On-budget and Delivering
Specified Benefits
Project Late and/or Over
Budget
More Expensive to Operate Than
Planned
Performance and/or Operational
Problems
Specified Business Benefits and Savings Not
Delivered
Functionality Delivered Does not
Meet Business Requirements
Significant Rework
Required
Solution Largely Unused And/Or
Unusable
Complete Project Failure: Cancelled, Unused, Rejected
Increasing Frequency/ Probability of Occurrence
October 18, 2011 4
What Is The Role Of The Chief Skeptical Officer?
• Role with end-to-end view and responsibility from idea definition to delivery
• Provide positive skepticism around idea definition and solution specification
• Be an independent voice
• Be an advocate for and champion of standards
• Maintain a repository of learning and standards
• Embed learning into the organisation
October 18, 2011 5
The Key Functions Of The Chief Skeptical Officer
Provide Independent Oversight of Decisions
Ensure Rational Decision-Making
Remove Arbitrariness, Relativity And
Subjectivity
Identify and Eliminate Behavioural And Other
Biases On Decision-Making
Ensure Evidence-Based Judgements
Require Assertions To Be Well Supported By
Evidence
Track End-To-End View From Concept To
Delivery
Question Assumptions, Attitudes Of
Knowledge, Opinions and Beliefs Stated As
Facts
Avoid Systematic Distortion and
Misrepresentation
October 18, 2011 6
The Journey From Idea To Successful Operation
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
October 18, 2011 7
The Journey From Idea To Successful Operation
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
But Promised Benefits All Too Frequently Not
Delivered
October 18, 2011 8
Frequently Too Many Handoffs On The Journey From Idea To Successful Operation
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
Handoff and Information Loss
Handoff and Information Loss
Handoff and Information Loss
Handoff and Information Loss
Handoff and Information Loss
“Chinese Whisper”Effect as Initial
Concept Moves to Implementation
October 18, 2011 9
What Causes the Disconnect From Idea To Delivery
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
Too Many Handoffs Between Stages, Too Often Separate Teams,
Miscommunication, Issues Left Unresolved, Assumptions M
ade, No One With End-
To-End View or End-To-End Responsibility
October 18, 2011 10
What Can Go Wrong Along The Journey From Idea To Successful Operation
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
Poorly Defined Need, Benefits Exaggerated
Inadequate Analysis, Poor Or Incomplete
Design
Overly Optimistic Initial and Operational Cost
Estimates
Poor Requirements Analysis, Poor Design,
Poor Management
Slow, Unreliable, Expensive To Operate,
Not Integrated, Requires Manual
Workarounds
Promised/Expected Benefits Not Delivered
October 18, 2011 11
Cost of Fixing Errors During Project Lifecycle
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Benefits Realised
Idea, Business Need, Business
Benefits
Solution Operation
Errors/ gaps/ omissions become significantly more expensive to fix at later stages of the
solution
Relative cost to remediate errors at the end can be 50-100 (or more) times more
expensive than at the start
October 18, 2011 12
Chief Skeptical Officer’s Role Is To Take End-To-End View To
Ensure That The Start Delivers And Can Deliver The End
Role Of The Chief Skeptical Officer Is To Take End-To-End View, Enforce Discipline and Perform Reviews
Idea, Business Need, Business
Benefits
Process Definition and
Solution Design
Costing -Implementation and Operational
Solution Implementation
and Delivery
Solution Operation
Benefits Realised
Getting It Right Here …
… Reduces/ Avoids Problems
Here
October 18, 2011 13
What Is The Chief Skeptical Officer Trying To Protect You Against?
• Cognitive Bias – Poor or inaccurate judgements, illogical interpretations and decisions, characterised by patterns of behaviour
• Strategic Misrepresentation – Deliberate misrepresentation in budgeting caused by distorted incentives
• Planning Fallacy – Systematic tendency to underestimate how long it will take to complete a task even when there is past experience of similar tasks over-running
• Optimism Bias – Systematic tendency to be overly optimistic about the outcome of actions
• Focalism – Systematic tendency to become inwardly focussed and to lose situational awareness and appreciation of wider context during times of stress
October 18, 2011 14
Cognitive Bias - Types
• Many classifications and types of cognitive bias
• Can be very difficult to avoid because of their embedded nature and emotional/irrational basis
−Decision-Making and Behavioural Biases - affecting belief formation and business decisions
− Probability and Belief Biases - affecting way in which information is gathered and assessed
−Attributional Biases - affecting the determination what was responsible for an event or action
• Cognitive biases are very real and can have damaging effects
October 18, 2011 15
Decision-Making And Behavioural Biases
• Relying too heavily on one piece of information when making a decision -Anchoring
• Believing things because many others believe the same – Bandwagon
• Assigning greater weight to apparently dominant factors – Attention Bias
• Interpreting information so as to that confirms preconceptions – Confirmation
• Seeing oneself as less biased than others –Blind Spot
• Strong preference for immediate payoffs relative to later ones – Hyperbolic Discounting
• Greater preferences just because of familiarity - Exposure Effect
• Paying more attention and giving more weight to the negative rather than the positive – Negativity Bias
• Looking for information even when it cannot affect action – Information Bias
• Placing too much importance on one aspect - Focusing Effect
• Looking to reduce a small risk to zero rather than a greater reduction of a larger risk – Zero-Risk Bias
• Rejecting new evidence that contradicts an established paradigm – Semmelweis Effect
• Making decisions based to what is pleasing to imagine instead basing decisions on evidence and rationality – Wishful Thinking
• Assigning a higher value to disposal/loss compared with cost of acquisition – Sunk Cost Effect
• Viewing a harmful action as worse than an equally harmful omission or inaction –Omission Bias
• Justifying increased investment based on the cumulative prior investment despite new evidence suggesting that the decision was wrong - Irrational Escalation
October 18, 2011 16
Probability And Belief Biases
• Excessive or inflated belief one's performance, ability – Overconfidence Effect
• Belief gaining plausibility through increasing repetition - Availability Cascade
• Assigning greater weight to initial or recent events more than subsequent or later events – Serial Position Effect
• Assigning a lower probability to the whole than the probabilities of the parts – Subadditivity Effect
• Avoidance of risk or the negative by pretending they do not exist – Ostrich Effect
• Judging future events in a more positive light than is warranted by actual experience – Optimism Bias
• Perceiving patterns where none exist – Clustering
• Selecting an options for which the probability of a favorable outcome is known over an option for which the probability of a favorable outcome is unknown - Ambiguity Effect
• Considering information to be correct if it has any personal meaning or significance – Subjective Validation
• Overestimating the likelihood of positive rather than negative outcomes – Valence Effect
• Failure to examine all possible outcomes when making a judgment –Attentional Bias
October 18, 2011 17
Attributional Biases
• Where skilled underrate their abilities and unskilled overrate their abilities – Dunning–Kruger Effect
• Defending the status quo – System Justification
• Overestimation of agreement – False Consensus Effect
October 18, 2011 18
Strategic Misrepresentation
• Deliberate misrepresentation in planning and budgeting caused bydistorted incentives
• Response to how organisations structure rewards and give rise tomotivations
• Systematic (and predictable) misrepresentation
− Deliberately understand costs to gain acceptance with understanding that costs will increase
− Not willing to face reality of high costs
− Overstatement or understatement of requirements
− Competition for scarce funds or jockeying for position
− Inclusion of ideology into planning
• Underlying system and processes need redesign to eliminate
October 18, 2011 19
What Are The Key Disciplines The Chief Skeptical Needs To Promote
• Disciplines and Skills
− Benefits Management and Realisation
− Capital Planning and Investment Management
− Business and Process Analysis
− Solution and Process Analysis and Architecture Design
• Practices
− Reference Class Forecasting
−Audits and Checklists
October 18, 2011 20
Reference Class Forecasting
• Technique to improve accuracy in plans and projections by basing them on actual performance in a reference class of comparable actions
• Based on knowledge about actual performance
−Analyse distributions and probability of cost overruns
− Compare the proposed project with the reference class distribution to establish the most likely outcome
• Used to avoid Planning Fallacy, Strategic
Misrepresentation and Optimism Bias
October 18, 2011 21
Benefits Management And Realisation
• To increase the likelihood of success from IT and other investments, organisations must identify the different causes of benefits before developing any project implementation plan
• Types of IT projects with very different approaches to benefits management− Fixing or improving something that exists
• Resolve problem
• Improve integration
− Implementing a new initiative • New system
• New processes
• Need to focus on the changes needed to achieve results and take full advantage of new facilities offered rather than on just IT features
• Effective change management is crucial to achieving benefits
October 18, 2011 22
Achieving Benefits – Some Questions To Ask
• Why is there a need to improve?
• What improvements are needed? What improvements are possible or achievable? Have the improvements been agreed by all stakeholders?
• What benefits will be realised by each stakeholder if the business objectives are achieved? How can each benefit be measured?
• Who owns each of the benefits and will be accountable for its delivery?
• What business changes are needed to achieve each benefit? Have the explicit links between each benefits and required business changes been identified?
• Who will be responsible for ensuring the business changes are made successfully?
• How and when can the changes be made? Who will make the changes? Does the business have the ability and capacity to make the changes?
October 18, 2011 23
Benefits Management And Benefits Dependency Network
• Benefits Dependency Network (BDN) is an approach to linking:
− Information Technology Enablers, Changes and New Capabilities – enabling technologies and functions and facilities needed to support the realisation of the identified benefits and to allow the necessary changes to be undertaken
− Business Changes – business activities and new ways of working that are required to ensure that the desired benefits are realised
− Enabling Changes - prerequisites for achieving the business changes or that are essential to bring the new system into effective operation
− Business Objectives - high level priorities in relation to the drivers, outcomes and improvements to be delivered on completion of the project
− Business Benefits - outcomes of a change that are deemed to be positive by a stakeholder and that are valuable to the organisation and are measureable
− Stakeholders - individuals or groups who will benefit from the project and areeither affected by or directly involved in making the changes needed to realise the benefits
October 18, 2011 24
Types Of Changes
• Business Changes - Permanent changes to working practices, processes, procedures, interactions and relationships that will cause the benefits to be delivered−Generally need new IT system to be in place
−May require enabling changes to be implemented
• Enabling Changes - Typically one-time changes that are pre-requisites for making the business changes or are necessary to bring the new system into effective operation− Can be made in advance of system implementation
− Training
− Processes redesign
−New work practices
− Changes to job roles and responsibilities
October 18, 2011 25
Benefits Dependency Network
Information Technology
Enablers, Changes and
New Capabilities
Enabling Changes
BusinessChanges
BusinessBenefits
BusinessObjectives
Require Allow EnableProvide for Delivery of
• BDN provides a framework for explicitly linking the overall investment objectives and the requisite benefits with the business changes which are necessary to deliver those benefits and the essential IT functionality to both drive and enable these changes to be made.
• BDN forms part of the benefits realisation plan
• Helps keep the focus on benefits realisation during the program execution
• Allows variations of the project or program to be assessed for their impact on benefits realisation
October 18, 2011 26
Benefits Dependency Network
Information Technology
Enablers, Changes and
New Capabilities
Enabling Changes
BusinessChanges
BusinessBenefits
BusinessObjectives
Means to Achieve Changes
Ways to Achieve Changes Results of Changes
October 18, 2011 27
Benefits Dependency Network
What Information Technology Capabilities
and Features Offer
What Users Do With
Information Technology Capabilities
What Information Technology Capabilities
Achieve
What the Business
Gains From Information Technology Capabilities
Why the Business Uses The
Information Technology Capabilities
Understand Why the Business Will Use the Information Technology System
What Information Technology Capabilities
Are Required
What the Business Has
to Do
What the Business Needs to Provide
What the Business
Needs To See
What the Business Wants
Understand What Information Technology System is Needed to Deliver on Requirements
October 18, 2011 28
Benefits Dependency Network
• Shows the link from the solution, through business activities, outcomes and benefits to the organisation's overall drivers
• Used to confirm that the solution being introduced will actuallyprovide the results you are seeking
• Any function within the proposed solution that is not linked to a benefit is potentially of doubtful value
• Functions with many link or links to key benefits can be identified and given extra attention
• BDN is a complex approach that requires benefit identification, assessment, validation and realisation maturity within the organisation
• Imposes a rigour on the organisation in analysing benefits from projects
October 18, 2011 29
Capital Planning And IT And Other Investment Core Requirements
• Determine the scale, scope, and sources of funding for IT and other areas
• Assign financial resources to competing activities within the IT portfolio
• Establish a balance between capital expenditure (new projects) and operating expenditure (running systems delivered by past projects)
• Optimise the total cost of ownership
• Manage IT and other portfolios for value and not just cost
• IT and other areas need to implement a process for justifying its costs and be seen to be taking these steps
October 18, 2011 30
Capital Planning And IT and Other Investment Management
• Aligns IT and other investments to organisation strategy (scoring)
• Prioritises investments (ranking)
• Provides strategic criteria for investment analysis
• Conduct annual IT and other portfolio management reviews
• Provides recommendation to stop, slow, maintain or accelerate program funding
• Identifies redundant/inefficient systems
• Integrates IT and other architectures within investments
• Ensures compliance with funding standards
October 18, 2011 31
Characteristics Of Credible Cost Estimates
• Clear identification of requirements of the ultimate deliverable
• Broad participation in preparing estimates
• Availability of valid data for performing estimates – historical, experience, benchmarks
• Standardised and comprehensive estimate structure that includes all possible sources of cost
• Provision for uncertainties – include known costs explicitly and allow for unknown costs
• Recognition of inflation
• Recognition of excluded costs
• Independent review of estimates for completeness and realism
• Revision of estimates for significant changes in requirements
October 18, 2011 32
Challenges Of Developing Good Cost Estimates
• Requires detailed, stable, agreed requirements
• Agreed assumptions
• Access to detailed documentation and historical data for comparison
• Trained and experienced analysts
• Risk and uncertainty analysis
• Identification of a range of confidence levels
• Adequate contingency and management reserves
October 18, 2011 33
Reasons for Good And Bad Cost Estimates
Detailed, Stable, Agreed Requirements Agreed Assumptions
Detailed Documentation and Historical Data
Effective Risk and Uncertainty Analysis
Trained and Experienced Analysts
Identification of a Range of Confidence LevelsAdequate Contingency and
Management Reserves
Complex Project or Technology
Unrealistic Project Savings
New Processes Untrained and Inexperienced Analysts
No or Limited Comparison Data Available
Project Instability
Unrealistic Assumptions Overoptimism
Unrealistic or Unreliable Data
Unfamiliar Technology or First-Time UseProblems Getting Access to Data Unreasonable
Project Baseline
Ineffective Risk and Uncertainty Analysis
• Lost of reasons for and causes of inaccurate cost estimates
October 18, 2011 34
Sources of Risk and Uncertainty In Estimating Costs
• Lack of understanding of the project requirements
• Shortcomings of human language and differing interpretations of meaning of project
• Behaviour of parties involved in the cost estimation process
• Haste
• Deception
• Poor cost estimating and pricing practices
October 18, 2011 35
IT Investment
• IT is commonly seen as failing to deliver value for money
• Benefits and value must be actively managed for
• Realising and assessing business benefits from IT-enabled investments involves more than simply assessing Total Cost of Ownership for IT-related projects and managing the IT budget
• Key requirements− Ability to get lifetime costs right− Ability to define benefits correctly and effectively− Ability to manage the benefits management process− Ability to increase and sustain benefits management maturity
• Use existing methodologies and frameworks to implement key requirements quickly− ITIM− Benefits Dependency Network− ValIT− Organisational change and commitment
• Effective benefits management enables organisations to clearly and consistently articulate IT’s contribution to achievement of business objectives
October 18, 2011 36
Business and Process Analysis And Design Builds Bridge From Business To Solution
ProblemRequirement
Current State
Business Analysis and Solution
Design
Solution
Desired Future State
Business Analysis:
Elicit RequirementsAnalyse
CommunicateValidate
Solution Design:
Translate Requirements into
Solution
• Business analysis is a key driver of business value
• Solution delivery start with business analysis
October 18, 2011 37
Weaknesses In Business Analysis Capabilities And Competencies At The Root of Many Project Failures
Business Needs Not Met
Opportunities Lost
Investment Wasted
Inadequate Business Case, Undefined Problem/Need
Business Benefits Not Measured
Poor Analysis Practices
Business Requirements Not Captured
Poor Requirements
Poor Strategic Alignment
Poor Focus on Business Needs
Inadequate Resource Allocation and Prioritisation
Inadequate Business Involvement
Poor Solution Design
Inadequately Explored Solution Options
Solution Design Not Aligned to Business Needs
Large Project, Complex, Difficult
Changes and Processes
Large Project Team and Multiple Stakeholders
Size/Capacity/Complexity
Uncertainly/Ambiguity
Unproven Technology
Dynamic, Changing Environment
October 18, 2011 38
Analysis-Related Causes Of Failures
Business Needs Not Met
Opportunities Lost
Investment Wasted
Inadequate Business Case, Undefined Problem/Need
Business Benefits Not Measured
Poor Analysis Practices
Business Requirements Not Captured
Poor Requirements
Poor Strategic Alignment
Poor Focus on Business Needs
Inadequate Resource Allocation and Prioritisation
Inadequate Business Involvement
Poor Solution Design
Inadequately Explored Solution Options
Solution Design Not Aligned to Business Needs
Large Project, Complex, Difficult
Changes and Processes
Large Project Team and Multiple Stakeholders
Size/Capacity/Complexity
Uncertainly/Ambiguity
Unproven Technology
Dynamic, Changing Environment
October 18, 2011 39
Taking A Complete View of Systems And Processes Is Essential to Solution Success
• Overall solution operates with a mix of automated and manual processes in a structured or ad-hoc manner o deliver the required results
• Understanding the overall set of processes and their operation is crucial to successful results
• Need to see the entire picture to understand how a solution should operate
− Systems/applications are just one part of this universe
• Unambiguous definition of processes is required
• Processes that are to be automated define the scope of the development and implementation work
October 18, 2011 40
Complete View of Systems and Processes Is Essential to Solution Success
System Component
System Component
System Component
External Component
External Component
External Component
Automated Process
Automated Process
Automated Process
External Manual
Interaction
External Manual
Interaction
Manual Process
Manual Process
External Manual
Interaction
External Manual
Interaction
Manual Process
Manual Process
October 18, 2011 41
Solution Design and Implementation Sequence
Business Plan
Business Need
Business Benefits
Requirements Definition
Process Design
Solution Architecture and Design
Technical and Detailed Design
Implementation
Defines where the business wants to go
Business need identifies solutions that will allow
delivery of plan
Defines the benefits to be achieved by the solution
Defines the detailed requirements of the solution
Defines the processes that will be implemented by the solution
Defines the solution design to implement the processes
Creates a detailed technical design for implementation
Implements the detailed design
October 18, 2011 42
Consistent Approach to Business And Process Analysis
• Adopt a consistent and robust framework in business analysis − Enables effective benefits realisation through a solution which meets the
business need
• Key elements− Establish enterprise standards, procedures and governance
− Standardise on infrastructure, analysis methods and operational procedures
− Develop competence and skills
• Key benefits− Implement solutions that meet business needs
− Increase the ability of the business to adapt quickly to changes
− Reduce risk, complexity, redundancy and support complexity
− Align business and IT
− Enable re-use and faster time-to-market
− Present one face to the business (customer)
− Increase business value
Recommended
