Dawn Stevens: Adapting Traditional Metrics to Measure, Monitor, and Achieve Agile Success

© 2016 Comtech Services, Inc. © 2016 Comtech Services, Inc.

Adapting Traditional Metrics to Measure, Monitor, and Achieve Agile Success Dawn Stevens Vice President Comtech Services

From Waterfall to Agile:

© 2016 Comtech Services, Inc. 2

Comtech and CIDM

About me

•  Over 30 years providing consulting services to information development organizations in all industries around the world

•  27 years in information development o  11 years managing a

combined documentation and training organization

o  16 years at Comtech

Myself


Audience participation:"About you • How many are currently in

an Agile environment?

• How many years have you been in an Agile environment?

• How many are considering adopting Agile?


Audience participation: "Current measurement practices

• Do you gather any metrics today?

• What are you measuring?

• Why are you measuring them?

• What do you do with them?


Why measure?

•  To judge the performance of your team and the individuals on the team X


Why measure?

•  To demonstrate that promises were met

•  To validate or correct assumptions

•  To alert the team to irregularities

•  To track progress toward a goal


Audience participation:"Agile expectations • What promises have been

made about your Agile implementation?

• What assumptions do you have about what Agile brings?

• What might go wrong with an Agile implementation?

• How will you know when you have reached Agile success?


CIDM survey:"The promises of Agile

12.2

24.4

31.7

31.7

43.9

46.3

63.4

73.2

0 10 20 30 40 50 60 70 80

Mo-vateemployees

Decreasecostofinforma-ondevelopment

Increasecustomersa-sfac-on

Increasetechnicalcompetence

Improvequality

Receivemorecustomerfeedback

Decreasedevelopment-me

Increasecollabora-on


The Agile difference

That is, while there is value on the items on the right, we value the items on the left more.

Responding to change Customer collaboration

Working software Individuals and interactions

over following a plan over contract negotiation

over comprehensive documentation over processes and tools We value:


Audience participation: "The Agile difference Process/performance vs. Result


Traditional�metrics


• Budget

•  Scope

•  Productivity

•  Process

• Value

• Quality

• Capability

• Adaptability and collaboration

Agile �metrics


Budget


•  How much can we / did we spend?

•  How many resources can be assigned / do we need?

•  What tools can we acquire to complete this project?

•  Is it needed? •  Is it important to the

customer? •  What will customers pay

for it? •  What affect will it have on

company reputation? •  What difference does it

make on our market share?

Value


Audience participation: "Measuring value

What metrics might indicate the value of your:

• Agile implementation?

• product(s)?

• department?


My thoughts: "Measuring value Agile implementation

• Number of projects cancelled before launch

• Number of launched products without expected customer acceptance


Product

My thoughts: "Measuring value

•  Actual sales and usage statistics o  Revenue o  # of customers o  Market share

• Growth rate •  Net promoter scores • % time spent on customer

priorities

• Given $100, how much would customers assign to: o  product itself o  documentation o  training o  customer support

•  Reduction in service calls over time

•  Amount of content never accessed

Department


Scope


• What is the size of what we are creating?

• What specific features must it include?

• How complex is it?

• Does it meet the need?

• Are customers satisfied with it?

•  Is it usable?

• Are there any defects?

Quality


Audience participation: "Measuring quality

• What measurements provide information about the quality of your work?

• How do your customers measure quality?

• How do your managers measure quality?


My thoughts: "Measuring quality • % of topics with positive customer feedback

•  Time required for users to locate needed information

• # of support calls on topics not covered in content

• # of errors found / reported post release

• Amount of time spent on rework after initial development effort


Productivity


• How many man hours will it / did it take us to complete?

• How much calendar time time will it / did it take?

• How much can the team accomplish within the time constraints?

• Does the team have the required skills?

•  Is the team able to sustain the production rate?

Capability


Audience participation: "Measuring team capability What measurements provide information about the capabilities of your team?


My thoughts: "Measuring team capability •  Team velocity -- # of storypoints / sprint

• On-time delivery rate

•  Experience of team working together

• % work requiring specialized skill


Beware: "Velocity ≠ Productivity

Which team is more productive?


Beware: " The observer effect

•  People adjust behavior based on the metrics they are held against

•  What you measure is often not only what you get, but all you get

•  To change behavior, change what you measure


Process


• How will we get this done?

• Did we check off all the boxes?

• What do the rules say?

• How can we get this done?

• What should we do next?

• What needs to change?

• How can we work together better?

Adaptability / Collaboration


Audience participation: "Measuring Adaptability / Collaboration

How can you measure team adaptability and collaboration?


My thoughts: "Measuring adaptability / collaboration •  # of changes required during sprint •  Time required to adjust to changes •  # of changes / innovations suggested by

team members • % team members expressing

satisfaction with communication during sprint

• % team members expressing that they are respected on the team

•  Time required for team members to get answers or help


Beware:"The streetlight effect

What you can measure

What you can’t measure

What you should

measure

Ask yourself: • What decisions will be made based

on this measure?

•  Is this the RIGHT thing to measure or an easy thing to measure?

• What might be an unintended consequence of this measure?

•  How can we guard against unintended consequences?


A final caution:"Interpreting your measurements Don’t collect metrics if you aren’t going to use them or believe them


Questions?

If we didn’t measure things we wouldn’t know how good we were at measuring the things that we’re measuring.

[email protected]

Leadership & Management

Dawn Stevens: Adapting Traditional Metrics to Measure, Monitor, and Achieve Agile Success