Dawn Stevens: Adapting Traditional Metrics to Measure, Monitor, and Achieve Agile Success

© 2016 Comtech Services, Inc. © 2016 Comtech Services, Inc.

Adapting Traditional Metrics to Measure, Monitor, and Achieve Agile Success Dawn Stevens Vice President Comtech Services

From Waterfall to Agile:

© 2016 Comtech Services, Inc. 2

Comtech and CIDM

About me

•  Over 30 years providing consulting services to information development organizations in all industries around the world

•  27 years in information development o  11 years managing a

combined documentation and training organization

o  16 years at Comtech

Myself

© 2016 Comtech Services, Inc. 3

Audience participation:"About you • How many are currently in

an Agile environment?

• How many years have you been in an Agile environment?

• How many are considering adopting Agile?

© 2016 Comtech Services, Inc. 4

Audience participation: "Current measurement practices

• Do you gather any metrics today?

• What are you measuring?

• Why are you measuring them?

• What do you do with them?

© 2016 Comtech Services, Inc. 5

Why measure?

•  To judge the performance of your team and the individuals on the team X

© 2016 Comtech Services, Inc. 6

Why measure?

•  To demonstrate that promises were met

•  To validate or correct assumptions

•  To alert the team to irregularities

•  To track progress toward a goal

© 2016 Comtech Services, Inc. 7

Audience participation:"Agile expectations • What promises have been

made about your Agile implementation?

• What assumptions do you have about what Agile brings?

• What might go wrong with an Agile implementation?

• How will you know when you have reached Agile success?

© 2016 Comtech Services, Inc. 8

CIDM survey:"The promises of Agile

12.2

24.4

31.7

31.7

43.9

46.3

63.4

73.2

0 10 20 30 40 50 60 70 80

Mo-vateemployees

Decreasecostofinforma-ondevelopment

Increasecustomersa-sfac-on

Increasetechnicalcompetence

Improvequality

Receivemorecustomerfeedback

Decreasedevelopment-me

Increasecollabora-on

© 2016 Comtech Services, Inc. 9

The Agile difference

That is, while there is value on the items on the right, we value the items on the left more.

Responding to change Customer collaboration

Working software Individuals and interactions

over following a plan over contract negotiation

over comprehensive documentation over processes and tools We value:

© 2016 Comtech Services, Inc. 10

Audience participation: "The Agile difference Process/performance vs. Result

© 2016 Comtech Services, Inc. 11

Traditional�metrics

The Agile difference

• Budget

•  Scope

•  Productivity

•  Process

• Value

• Quality

• Capability

• Adaptability and collaboration

Agile �metrics

© 2016 Comtech Services, Inc. 12

Budget

The Agile difference

•  How much can we / did we spend?

•  How many resources can be assigned / do we need?

•  What tools can we acquire to complete this project?

•  Is it needed? •  Is it important to the

customer? •  What will customers pay

for it? •  What affect will it have on

company reputation? •  What difference does it

make on our market share?

Value

© 2016 Comtech Services, Inc. 13

Audience participation: "Measuring value

What metrics might indicate the value of your:

• Agile implementation?

• product(s)?

• department?

© 2016 Comtech Services, Inc. 14

My thoughts: "Measuring value Agile implementation

• Number of projects cancelled before launch

• Number of launched products without expected customer acceptance

© 2016 Comtech Services, Inc. 15

Product

My thoughts: "Measuring value

•  Actual sales and usage statistics o  Revenue o  # of customers o  Market share

• Growth rate •  Net promoter scores • % time spent on customer

priorities

• Given $100, how much would customers assign to: o  product itself o  documentation o  training o  customer support

•  Reduction in service calls over time

•  Amount of content never accessed

Department

© 2016 Comtech Services, Inc. 16

Scope

The Agile difference

• What is the size of what we are creating?

• What specific features must it include?

• How complex is it?

• Does it meet the need?

• Are customers satisfied with it?

•  Is it usable?

• Are there any defects?

Quality

© 2016 Comtech Services, Inc. 17

Audience participation: "Measuring quality

• What measurements provide information about the quality of your work?

• How do your customers measure quality?

• How do your managers measure quality?

© 2016 Comtech Services, Inc. 18

My thoughts: "Measuring quality • % of topics with positive customer feedback

•  Time required for users to locate needed information

• # of support calls on topics not covered in content

• # of errors found / reported post release

• Amount of time spent on rework after initial development effort

© 2016 Comtech Services, Inc. 19

Productivity

The Agile difference

• How many man hours will it / did it take us to complete?

• How much calendar time time will it / did it take?

• How much can the team accomplish within the time constraints?

• Does the team have the required skills?

•  Is the team able to sustain the production rate?

Capability

© 2016 Comtech Services, Inc. 20

Audience participation: "Measuring team capability What measurements provide information about the capabilities of your team?

© 2016 Comtech Services, Inc. 21

My thoughts: "Measuring team capability •  Team velocity -- # of storypoints / sprint

• On-time delivery rate

•  Experience of team working together

• % work requiring specialized skill

© 2016 Comtech Services, Inc. 22

Beware: "Velocity ≠ Productivity

Which team is more productive?

© 2016 Comtech Services, Inc. 23

Beware: " The observer effect

•  People adjust behavior based on the metrics they are held against

•  What you measure is often not only what you get, but all you get

•  To change behavior, change what you measure

© 2016 Comtech Services, Inc. 24

Process

The Agile difference

• How will we get this done?

• Did we check off all the boxes?

• What do the rules say?

• How can we get this done?

• What should we do next?

• What needs to change?

• How can we work together better?

Adaptability / Collaboration

© 2016 Comtech Services, Inc. 25

Audience participation: "Measuring Adaptability / Collaboration

How can you measure team adaptability and collaboration?

© 2016 Comtech Services, Inc. 26

My thoughts: "Measuring adaptability / collaboration •  # of changes required during sprint •  Time required to adjust to changes •  # of changes / innovations suggested by

team members • % team members expressing

satisfaction with communication during sprint

• % team members expressing that they are respected on the team

•  Time required for team members to get answers or help

© 2016 Comtech Services, Inc. 27

Beware:"The streetlight effect

What you can measure

What you can’t measure

What you should

measure

Ask yourself: • What decisions will be made based

on this measure?

•  Is this the RIGHT thing to measure or an easy thing to measure?

• What might be an unintended consequence of this measure?

•  How can we guard against unintended consequences?

© 2016 Comtech Services, Inc. 28

A final caution:"Interpreting your measurements Don’t collect metrics if you aren’t going to use them or believe them

© 2016 Comtech Services, Inc. 29

Questions?

If we didn’t measure things we wouldn’t know how good we were at measuring the things that we’re measuring.

DawnStevensdawn.stevens@comtech-serv.com