KPIs for Agile Project Managers and Business Analysts

Title Here KPIs for Agile Project Managers

and Business Analysts

Powering Business Value

John E. Parker, CEO Enfocus Solu6ons Inc.

www.EnfocusSolu6ons.com

February 19, 2014

© Copyright 2014 Enfocus Solu6ons Inc. All Rights Reserved.

1 © Copyright 2014 Enfocus Solutions Inc. All Rights Reserved.

John E. Parker •  Chief Executive Officer of Enfocus Solutions Inc. •  Previous Positions

o  Chief Visionary Officer of Enfocus Solutions Inc o  EVP and CTO, MAXIMUS Inc. o  Outsourced CIO for HSHS (Large Healthcare System) o  EVP and Cofounder, Spectrum Consulting Group o  KPMG Partner

•  Expertise o  IT Strategic Planning o  Business Analysis o  Recovering Troubled and Challenged Projects o  Enterprise Architecture o  Development Methodologies (Agile, Waterfall, RUP,

Design First, FDD, TDD) o  Financial and Cost Benefit Analyses o  Business Process Improvement, Reengineering, and

Management

Contact: •  http://enfocussolutions.com •  [email protected]


What the CEO Expects from IT Source: CIO Magazine’s 2014 State of the CIO Survey

1. Help the Company Reach a Specific Revenue Goal.

2. Successfully complete a Major Enterprise Project.

3. Support Customer Acquisi6on and Reten6on.

4. Simplify IT.

5. Lead a Product Innova6on Effort


2014 CIO Agenda Source: CIO Magazine’s 2014 State of the CIO Survey

1. Improve the Use of Data and Analy6cs.

2. Support Business and Marke6ng Goals.

3. Improve IT Project Delivery.

4. Developing Skills for Emerging Technology and Innova6on.

5. Reorganize or Retrain IT to Focus on Business Outcomes and Innova6on


Changing View of Project Success

•  Delivering a project “on-time and on-budget?” is no longer adequate.

•  In today’s environment, the key question should be: “Did the project deliver value to the business?”

•  Value is largely determined by how the system is embraced by the user community in the months and years after it is deployed.


Creating All Deliverables on Time and on Budget Does not Make a Successful Project

•  Does not guarantee that benefits outlined in business case were achieved

•  Does not guarantee user adoption

•  Does not guarantee that expected ROI was achieved

•  Does not guaranteed a satisfied customer

•  Does not guarantee that solution addresses the customer need

•  Does not guarantee that sales were in line with forecasts

•  Does not guarantee that there will be market demand for the product


•  Achieved a 55% ROI

•  Increased production throughput by 26%

•  Consolidated applications from 29 to 11

•  Achieved a total savings of $14 million in 9 months

•  Increased the number of sales leads by 30%

•  Increased labor efficiency by 32%

•  Cut procurement costs by 21%

•  Reduced Inventory by 37%

Business Outcomes The success of a project should be measured based on business outcomes:


Transforming Project Management Using Business Metrics to Manage Projects

•  Project professionals are historically skilled and focused on managing the triple constraints of budget, time, and scope, but this is not enough.

•  To deliver optimized project results that deliver long lasting benefits, project teams need to incorporate relevant value metrics into their project approach that are focused on business outcomes.

•  Defining and using KPIs should become a standard skill for all project managers and business analysts.


Key Performance Indicators (KPIs) are an excellent method to

measure business value on projects.

Executing a project without KPIs is akin to going on road trip to a

place you have never been without a map.

However, executing a project

with the wrong KPIs will take to you a place you probably do not

want to be.


Using Metrics on Projects

Project Measurement and Status •  Time: How the project is going against schedule? •  Cost: How the project is going against budget? •  Resources: How much time is being spent on this project? •  Scope: Is the project's scope kept in line with expectations? •  Quality: Are quality problems being reviewed and fixed? •  Actions: Are there any outstanding action items on the project?

Business Value Delivered •  Average Size per Sale •  Number of Leads generated •  Production Throughput •  Process Cycle Efficiency •  Cash Flow •  Cost Savings •  ROI

Business Change •  User Adoption •  Stakeholder Engagement •  Business Process Efficiency •  Customer Satisfaction


KPIs are Misunderstood

•  Key Performance Indicators (KPI’s) are one of the most over-used and little understood terms in business.

•  They are too often taken to mean any metric or data used to measure business performance.

•  The role KPI’s play is much bigger and more important. In fact, KPI’s are one on the most important guideposts for any business. Every business should have them.

•  KPI’s are an actionable scorecard that keeps your strategy on track. They enable you to manage, control and achieve desired business results.


KPI Implementation Using KPIs takes Preparation

•  There are many companies where KPIs are treated as a "necessary evil.”

•  Poor implementation and design and poor communications of KPIs to employees are common reasons for failure.

•  Help managers use KPIs for business management, not just for measurement;

•  Ensure that the project team and stakeholders buy-into KPIs as a business tool.


Why Metrics are Important

•  Without properly establishing quantitative metrics, it is very difficult to achieve measurable business success on a project.

•  Well defined metrics allow team members and stakeholders to work together towards common goals

•  Stakeholders are less likely to second guess the necessity of the project.

•  If you can’t measure it, you can not mange it.


Metrics Gone Bad Continental Airlines Recovering from Bankruptcy in the 90s

•  Continental had to cut costs, and because fuel costs were high, management created a metric to track its usage.

•  The metric was used to reward pilots for reducing fuel consumption.

•  As expected, pilots acted to earn their reward -- they skimped on air conditioning and flew more slowly.

•  Management got what it seems they wanted, and fuel consumption fell.

•  Unexpectedly however, their customer satisfaction and on-time performance also fell, and their most valuable frequent flyer customers moved on to competitors.


Metrics Gone Bad Programming Example

•  Software organizations reward productive programmers who o  Write lots of code and o  Fix lots of bugs.

•  The best way to get ahead in an organization like this is to check in lots of buggy code and fix it all, rather than taking the extra time to get it right in the first place.

•  When you try to fix this problem by penalizing programmers for creating bugs, you create a perverse incentive for them to hide their bugs or not tell the testers about new code they wrote in hopes that fewer bugs will be found. You can't win.


Difference Between CSF and KPI

•  Some people use CSF and KPI interchangeably or confuse them, but they’re two totally different concepts.

•  A Critical Success Factor (CSF) is what is necessary for an organization or project to achieve its mission.

•  A Key Performance Indicator ( KPI) is a metric that is tied to a target to determine if we have met our CSF.

•  CSFs are the cause of your success, whereas KPIs are the effects of your actions.


Difference Between Metric and KPI

•  A metric is a measurement.

•  A KPI is a metric, but not all metrics are KPIs.

•  A KPI is an indicator (a metric) that you have chosen, and agreed with your partners (whether internal to IT or with customers), that will determine whether you are meeting your critical success factors (CSF).

•  So when is a metric a KPI?

•  KPIs are metrics that are:

o  Outcome-Oriented -- tied to an objective

o  Target-Based -- have at least one defined time-sensitive target value

o  Graded -- have explicit thresholds that grade the difference (or gap) between the actual value and the target


Outcome Oriented

•  KPI should be explicitly tied to an objective

•  If you can't describe the business goal it's monitoring, it's not a KPI -- it's a metric.

•  Metrics that track just inputs or outputs are just metrics not KPIs


Target Based

•  KPIs are more than just numbers.

•  KPIs are an indicator of progress toward organizational goals.

•  Targets represent the value you would like a KPI to be at a specific moment in time.


Graded

•  KPIs are used to calculate the gap between actual and target values.

•  Effective KPI should also rate or grade the size of this gap.

•  Associating a rating system with KPIs provides a quick and easy-to-understand reading of whether a particular KPI status is on track or off track.

o  A, B, C, D, F

o  Green, Yellow, and Red

o  0- 100


Grading Color Coding makes Problems More Apparent


KPIs are Tied to Objectives

Defining objec6ves is cri6cal to ensure that the solu6on meets the business need.

•  Objec6ves are specific, measurable, and have a defined comple6on date. They are more specific and outline the “who, what, when, where, and how” of reaching the goals.

•  Clear project objec6ves are crucial because your project's success will be determined by how closely you meet them.

•  For objec6ves to be effec6ve, it is important that the objec6ves be approved by the project sponsor and effec6vely to communicated to all project stakeholders.

Features should be mapped to objec6ves.


Defining KPIs

•  Align with Business Goals and Objectives – Ensure that KPIs are aligned with business goals and objectives

•  Drive Outcomes - Need to drive towards the benefits that your project is expected to deliver once its output is being used

•  Keep them Simple - Keep the metric simple and easy to understand. •  Good Enough - Complicated or overthought metrics often require excessive

work, usually confuse people, and do not get used. •  Use as Indicators The KPIs are designed to alert you that CSF attainment

might be in jeopardy, that is all. •  Not Too Many - Too many metrics, even if they are effective, can overwhelm

a team. (Usually 8 to 10 per project) •  Beware of Metric Traps – Be sure you are measuring the right thing.

Measuring the wrong will result in bad outcomes. •  Frequently Measure - Need to regularly measure and report progress. •  Optimize Whole – Make sure that the metrics work together to optimize the

whole. Distracting and competing metrics can be very disruptive.


Working with Stakeholders to Define KPIs

Start with a business context, not with a KPI. Define what your business objectives are. 1 Define KPIs for monitoring performance of processes and outcomes and tracking progress on project. 2 Simplify KPIs and ensure that are measuring what needs to be measured.. 3 Communicate the KPIs to the Team and All Stakeholders 4 Measure and report results using a KPI visualization tool. 5


Understand the Hawthorne Effect Use it to Your Advantage and Not Disadvantage

•  The Hawthorne effect is a psychological phenomenon that produces an improvement in human behavior or performance as a result of increased attention from superiors, clients or colleagues.

•  In a collaborative effort, the effect can enhance results by creating a sense of teamwork and common purpose.

•  You can exploit this in a positive way.

•  However, many metrics have a negative Hawthorne effect.


Don’t Measure People

•  Effective project metrics do not measure people -- they measure teams and processes. The real purpose of metrics is to help you make better decisions and not to hold people accountable, or punish them.

•  Good metrics focus on the attainment of process objectives and goals


Examples of Good and Bad Metrics Measure the Right Thing

Avoid Metrics Such As …. •  Lines of Code •  Story Points Delivered

(Velocity) •  Hours Worked •  Number of Defects

Try Metrics Such As… •  Number of Features Accepted •  % of Features Completed •  User Satisfaction •  Feature Cycle Time •  Defects discovered after

deployment •  Customer LTV •  Revenue/Feature


Project Critical Success Factors

Benefits Realiza?on Business Change User Adop?on

Solu?on Delivery Collabora?on


Project Critical Success Factors

Stakeholder Engagement

Team Sa?sfac?on

Stakeholder Sa?sfac?on

Quality Velocity Waste

Benefits

Percep?on

Collabora?on

Solu?on Delivery

Benefits Realiza?on ROI

Business Change People Process Technology

Learning User

Adop?on Applica?on

Costs


Measuring Project Success

Benefits Realiza?on


Metrics - ROI

•  ROI •  Time to Break Even •  Deviation of Planned ROI •  Deviation of Planned Payback •  Deviation of Planned NPV •  NPV •  Actual ROI / Estimated ROI •  Time to Market

ROI Descrip?on

Defini6on Net Financial Benefits/Cost

Target Year End 2014 -‐ 10% Year End 2015 -‐ 40% Year End 2016 -‐ 75%

Grade Green -‐ Meets or Exceed Yellow -‐ Within 25% Red -‐ Variance more than 25%

Benefits Realiza?on


Metrics - Benefits Bernard Marr’s 75 Key Metrics Every Manger Should Know

Financial Performance: 1.  Net Profit 2.  Net Profit Margin 3.  Gross Profit Margin 4.  Opera6ng Profit Margin 5.  EBITDA 6.  Revenue Growth Rate 7.  Total Shareholder Return (TSR) 8.  Economic Value Added (EVA) 9.  Return on Investment (ROI) 10.  Return on Capital Employed (ROCE) 11.  Return on Assets (ROA) 12.  Return on Equity (ROE) 13.  Debt-‐to-‐Equity (D/E) Ra6o 14.  Cash Conversion Cycle (CCC) 15.  Working Capital Ra6o 16.  Opera6ng Expense Ra6o (OER) 17.  CAPEX to Sales Ra6o 18.  Price Earnings Ra6o (P/E Ra6o)

Customers: 19.  Net Promoter Score (NPS) 20.  Customer Reten6on Rate 21.  Customer Sa6sfac6on Index 22.  Customer Profitability Score 23.  Customer Life6me Value 24.  Customer Turnover Rate 25.  Customer Engagement 26.  Customer Complaints

Marke?ng 27.  Market Growth Rate 28.  Market Share 29.  Brand Equity 30.  Cost per Lead 31.  Conversion Rate 32.  Search Engine Rankings (by

keyword) and click-‐through rate 33.  Page Views and Bounce Rate 34.  Customer Online Engagement Level 35.  Online Share of Voice (OSOV) 36.  Social Networking Footprint 37.  Klout Score


Opera?onal Performance: 38.  Six Sigma Level 39.  Capacity U6liza6on Rate (CUR) 40.  Process Waste Level 41.  Order Fulfillment Cycle Time 42.  Delivery In Full, On Time (DIFOT) Rate 43.  Inventory Shrinkage Rate (ISR) 44.  Project Schedule Variance (PSV) 45.  Project Cost Variance (PCV) 46.  Earned Value (EV) Metric 47.  Innova6on Pipeline Strength (IPS) 48.  Return on Innova6on Investment

(ROI2) 49.  Time to Market 50.  First Pass Yield (FPY) 51.  Rework Level 52.  Quality Index 53.  Overall Equipment Effec6veness (OEE) 54.  Process or Machine Down6me Level 55.  First Contact Resolu6on (FCR)

Employees Performance: 56.  Human Capital Value Added

(HCVA) 57.  Revenue Per Employee 58.  Employee Sa6sfac6on Index 59.  Employee Engagement Level 60.  Staff Advocacy Score 61.  Employee Churn Rate 62.  Average Employee Tenure 63.  Absenteeism Bradford Factor 64.  360-‐Degree Feedback Score 65.  Salary Compe66veness Ra6o

(SCR) 66.  Time to Hire 67.  Training Return on Investment

Environmental and Social Sustainability Performance: 68.  Carbon Footprint 69.  Water Footprint 70.  Energy Consump6on 71.  Saving Levels Due to

Conserva6on and Improvement Efforts

72.  Supply Chain Miles 73.  Waste Reduc6on Rate 74.  Waste Recycling Rate 75.  Product Recycling Rate

Metrics - Benefits Bernard Marr’s 75 Key Metrics Every Manger Should Know


Metrics - Cost

•  Percent of budget expended •  Budget variance •  Dollars remaining •  % Cost expended /% Project Completion •  Testing as % of Total Cost •  Discovery as % of Total Cost •  Development as % of Total Cost •  Cost per Function Point Delivered •  Earned Value Management Metrics

o  Actual Cost of Work Performed (ACWP) o  Planned Value (PV) o  Earned Value (EV)

Benefits Realiza?on



Business Change


Business Change

People Process Technology

Data Rules


Metrics – Business Change

•  Number of Impacts Defined by Type •  Number of Impacts without Change Owner •  Impact Risk Index •  Number of Features without Related Impacts •  % Completion •  Number of Impacts without Risk Assessment •  Number of impacts w/o Gap Analysis •  Number of Impacts w/o Transition Requirements •  Organizational Readiness Index •  Number of users trained •  Reduction in process cycle time •  % improvement in SLA •  Number of Change Requests •  Staff Turnover •  Number of training hours per user



User Adop?on


User Adoption - Perception

•  User satisfaction index •  Intent to use •  % of users who view solution will resolve problem •  % of users who have participated in survey •  % of users who have been notified •  Number of user complaints •  Number of users submitting recommendations •  % of users submitting recommendations •  % favorable rating by users in survey •  Number of trouble tickets/users

User Adop?on


User Adoption - Learning

•  Number of users trained on new solution •  % of users trained on new solution •  Number of users certified •  % of users certified •  %Users with required skills

User Adop?on


User Adoption - Application

•  % of users using the solution •  % increase in user productivity •  % reduction in error rates •  % reduction in cycle time •  User throughput •  No. of users that have never logged in •  % of Users that have logged in •  Average number of transactions/User •  Number of accounts/user

User Adop?on



Solu?on Delivery


Manage the IT Project Value Stream Lean, Fast, Quality, and Value

Objec?ves Features Bundles Releases Projects

Ensure that each project has a viable business case and is aligned with organiza6onal goals.

Ensure that clear objec6ves have been defined for each project. Iden6fy required business changes using impacts.

Break down the project into small independent components that can be delivered independently. Map each feature to objec6ves. Eliminate components that provide linle or no value.

Allocate requirements from validated feature to bundles. Manage lifecycle events to validate delivery of value.

Plan deployment of func6onality in Releases ensuring high user adop6on and anainment of planned benefits. Conduct Benefit retrospec6ves to ensure benefits realiza6ons.


Remove Delays Obtaining Approvals Is Usually the Biggest Source of Delays

Use Metrics such as Process Cycle Efficiency To Minimize Waste in the Process.


Where to Focus

Process Cycle Efficiency

Which Provides a BeSer Return? •  Gepng Bener at What you Do or •  Elimina6ng Delays Between What you Do Ideas for improvements •  Improve PCE from current 14.9% to > 75% •  U6lize metrics to measure velocity, cycle 6me, and quality •  Establish service levels for business case review and approval. •  Reduce wait 6mes for review/approval by moving to con6nuous review process •  Eliminate paper requirement documents: Manage data not documents •  Implement bener discovery process to reduce number of development itera6ons •  Iden6fy defects earlier in project lifecycle •  Consider test driven development


Metrics – Project Lifecycle

•  Process Cycle Efficiency •  Number of Projects with Paper Requirement

Documents •  Average number of development iterations to

acceptance •  Average number of days for review/approval of

requirement documents (Delays) •  Rework from poor requirements •  Defects discovered in UAT •  Defects discovered in Production •  % of Software Functionality that is not used •  % of Requirements validated by QA •  % of Requirements validated by DEV •  % of Requirements validated by Business •  Number of Hours in Meetings

Solu?on Delivery


Metrics - Planning

•  Portfolio o  % of Projects without an approved business case o  Average time to approve project o  % of Projects without a set of approved objectives o  % of Descoped Features to Total Features o  % of Features without mapped business objectives o  Number of cancelled projects after approval o  Number of projects approved

•  Project o  Average cycle time to define and approve project o  Average cycle time to define and approve feature

Solu?on Delivery


Metrics- Discover

•  Portfolio o  Number of Projects without Stakeholder Needs o  Number of Projects without Impact Records o  Average cycle time to approve Feature

•  Project o  Cycle time to complete a Feature o  Cycle time to deliver a Feature o  Stakeholder Satisfaction o  Stakeholder Participation o  Number of requirement defects identified after

approving features

Solu?on Delivery


Metrics- Development

•  Portfolio o  Quality – Number of defects discovered in UAT o  Team Satisfaction o  Predictability – Number of points completed/number

of points committed

•  Project o  Sprint Goals – Number of points completed /

Number of points committed o  Velocity - Number of Story Points completed per

sprint o  Quality - Number of Defects discovered in UAT o  Resources- % of team that is part time o  Team turnover o  Average iterations to acceptance

Solu?on Delivery


Metrics - Deployment

•  Number of Story Points Deployed •  Number of Features Deployed •  Number of Requirements Delivered •  Number of incidents caused by release •  Number of urgent releases •  Number of Releases •  % of successful software upgrades •  Average cycle time between urgent releases •  Average cost of releases •  Number of release aborted •  Number of releases deployed to production •  Number of releases that had to be reversed •  % of releases that were deployed automatically •  Number of releases that failed to meet SLA

Solu?on Delivery



Collabora?on


Business Value Metrics

•  Deviation of Planned ROI •  Deviation of Planned Payback •  Deviation of Planned NPV •  Business Process Improvement Increase •  Business Process Cycle Time Reduction

Collabora?on


Stakeholder Engagement Metrics

•  Number of review comments/stakeholders •  Number of Stakeholders that wrote needs or

scenarios •  Number of Stakeholders that wrote comments •  % of Stakeholders that drafted comments •  % of Stakeholders that defined their own needs •  Number of stakeholder participating in demos •  Number of stakeholders participating in UAT •  Number of User Logins to StakeholderPortal •  Number of Logins per week per user to

StakeholderPortal •  Number of StakeholderPortal users who did not

log in this week •  Number of Views by Stakeholders Per Week •  Number of Views, Creates, and Updates per week

for Stakeholders •  Number of Action Items by Stakeholder •  Number of Defects Identified by Stakeholders

Collabora?on


Stakeholder Engagement Monitoring Stakeholder Engagement


Stakeholder Activity


Team Satisfaction

•  Team Satisfaction •  Scrum Team Turnover

Collabora?on



Project and PorUolio Management


Business Analysis Metrics

•  % of Rework attributable to requirements •  % of Projects with prioritized features •  % of Features fully implemented •  % of Features Developed •  % of Approved Requirements not Implemented •  Developer Requirements Satisfaction Index •  Project Stakeholder Satisfaction Index •  QA Requirements Satisfaction Index •  % of Requirements Tested •  Number of Missing Requirements


Project Management Metrics

•  Deviation of Planned Budget •  Deviation of Planned Time (Days) •  % of Milestones missed •  % of benefits received from business case •  % of milestones late


Project Portfolio Metrics

•  % of initiated projects without business case •  % of projects delivered on time •  % of project achieving benefits in business case •  % of projects on budget •  % of project on time and on budget •  % of challenged or failed projects •  % of projects with pre-defined KPIs •  % of projects that achieved ROI •  % of milestones missed •  % of overdue project status reports •  % of projects with testing plan •  % of projects with Vision statement •  % of projects with scope changes •  % of projects with Executive Sponsor •  % of projects with risk mitigation plan •  Average number of days open for issues •  Resource utilization %


Agile Delivery Metrics

•  Actual Stories completed/Actual Stories Committed •  Technical Debt management •  Team Velocity - Number of Story Points per Sprint •  Changes to active sprint •  Number of defects in UAT •  Number of defects in Production •  Number of backlog items validated •  Number of boomerangs •  Cost per Story Point •  Number of iterations from Story to Delivery •  Number of Stories completed •  Number of Story Points Completed •  Cycle time for Customer Request to Delivery •  Estimating Accuracy •  % of Features Delivered •  % of Time spent in Non Value Added Activities •  % of Time grooming backlog •  Team Satisfaction •  Overtime Hours worked by Team (Sustainable Pace)


Project Dashboards


Summary

•  Delivering projects on-time and on-budget is no longer adequate. Projects are now being expected to deliver business value.

•  Metrics should be used to ensure the project deliver business value.

•  Metrics, if properly defined, can be a unifying force between IT and business focusing both on improving business outcomes.

•  Defining metrics is not easy. However, it is key skill that should be learned by both BAs and PMs.

•  Metrics must be defined very carefully. Using the wrong metrics can lead to unfavorable outcomes.

•  KPIs are metrics that are outcome oriented, target based, and graded.

•  Metrics should also linked to business objectives.

•  Metrics should be visualized in a dashboard.


We Provide Services to Help You Deliver More Value

Workshops Empowering Business Value

Consul?ng & Mentoring

SaaS Business Analysis as a

Service


Enfocus Solutions Inc. Our Project Success Metrics

KPI Expected Benefit

ROI from investment in Enfocus Requirements Suite and Services

100% or Higher

Reduce Rework 10-‐15% Reduc6on in Rework

Reduce Project Costs 10-‐20% Cost Savings

Catch Defects Earlier 10-‐20% Reduc6on of Produc6on Defects

Higher Project Success Rates 5-‐10% Reduc6on in Project Porrolio Costs

Shorter Time to Benefits Reduce 6me to benefit by 25%

Greater Benefits Realiza6on Increase benefits realiza6on by 30%

ROI on Projects Increase in Average ROI by 50%

Please contact us to see how we can help you.

Contact: •  http://enfocussolutions.com •  [email protected]


Thank You Learn how to deliver more business value on your projects Please request a consulta6on.

www.enfocussolu6ons.com