The TPI: Measuring Task Performance on Cisco · Top Tasks Management was first introduced at Cisco in 2010 with a “top tasks” survey. The tasks selected were based on a Top Tasks

Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 1

The TPI:Measuring Task Performance on Cisco.comGerry McGovern & Bill SkeetSWTG Customer Experience

Webinar: March 2014

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

Accenture 2013 Customer

Satisfaction Survey


Temkin Group 2013

IT Decision Maker Survey


First

Channel

Outcome

Phone

18% of Contact

Volume

First

Channel

Choice

Website

44% of Contact

Volume

Web Chat

3% of Contact

Volume

Second

Channel

Choice

E-mail

20% of Contact

Volume

Resolved81%

Recontact11%

Give up8%

Community

8% of Contact

Volume

Resolved53%

Recontact31%

Give up16%

Resolved71%

Recontact24%

Give up5%

Resolved50%

Recontact46%

Give up4%

50%

20%

0%

30%

0%

Resolved61%

Recontact34%

Give up5%

27%

38%

2%

23%

10% 10%

30%

40%

20%

0%

16%11% 9%

56%

9%5% 5%

0%

15%

74%


Identify Top

Tasks

Measure Top

Tasks

Task Based

Support


Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 7Cisco Confidential 7© 2013 Cisco and/or its affiliates. All rights reserved.


Top Tasks Management is a model of management for measuring usability of a site by scoring the performance of top tasks.

CSAT (Maritz)

Advisory Boards

Metrics Reports (Unica)User Testing (TPI)

Site Feedback

(OpinionLab)

Radian 6

Behavior

Voice of

Customer

Listen

What

Happens

If…?

Measure

What

Happened

Inquire

Entity Feedback

Multivariate testing (A/B)

Session Monitoring (TeaLeaf)

ForeSee

Omniture, SiteCatalyst, Etc

Representative, Moderated

audience

Comprehensive,

Unmoderated audience

Representative, Moderated

audience

Comprehensive,

Unmoderated audience

R E F L E C T I O N S

E X P E R I M E N T S F O R E N S I C S

I M M E D I A T E

#1


A task performance measurement study asks 15-18 subjects to perform 10-12 tasks.

#2


Top task questions are designed to represent the most important, typical tasks.

#3

“Your 7600 router crashed after entering the “show mplsforwarding with owner …” command. Determine if there are any workarounds for this known issue.”

Sample task: Find Bug Information


Testing is performed remotely on the live site. Remote testing is:

1. Faster: It’s easier to get participants and faster to carry out

2. Cheaper: It’s less expensive to set up. All you need is Webex

3. Better: It’s more real. The participant is

in their own environment.

#4


Top Tasks Management was first introduced at Cisco in 2010 with a “top tasks” survey.

The tasks selected were based on a Top Tasks study.

#5Download software, firmware,

drivers, patches, updates

Configure / set up

a product

Troubleshooting

(bug fixes, Diagnostics)

Top 25%

26-50%

51-75%


Top Tasks Management was first introduced at Cisco in 2010 with a “top tasks” survey.

The tasks selected were based on a Top Tasks study.

#5Tasks

491

Voters

983

Voters

1474

Voters

1966

Voters

1 Download software, firmware, drivers, patches, updates 8% 8% 9% 9%

2

Configure / set up a product (tech guides, notes, how-

to's) 9% 8% 8% 8%

3 Troubleshooting (bug fixes, diagnostics, guides) 7% 8% 8% 7%

4 Install / upgrade / migrate (tech guides, notes, examples) 6% 6% 6% 5%

5 Support (TAC) service requests (case creation, tracking) 6% 5% 5% 5%

6 Maintain / operate (tech guides, notes, examples) 4% 4% 4% 4%

7 Network design (tech guides, notes, examples) 5% 4% 5% 4%

8 Service, support options available for my product 3% 4% 3% 4%

9

Product and technical communities (troubleshooting,

developer, learning) 3% 3% 3% 3%

10 Support alerts (new drivers, patches, software, security) 2% 3% 3% 3%

11

Networking certification requirements (CCIE, CCNP,

CCNA) 2% 2% 2% 3%

12 Talk / chat online with a support person 3% 2% 2% 2%

13

Compatibility / interoperability (software, hardware,

version) 2% 2% 2% 2%

14 Training (courses, calendar, locations) 2% 2% 2% 2%

15

Products by specific technical requirements (capacity,

bandwidth, dimensions) 2% 1% 2% 2%

16

Pre-purchase technical information (datasheets,

whitepapers, requirements) 1% 2% 2% 2%

17

Solutions to match business needs (work remotely,

improve productivity, secure my business) 2% 2% 1% 1%

18 Networking certification status, maintenance and renewal 1% 1% 1% 1%

19 Warranties 1% 1% 1% 1%

20

Configure a product or service for the purpose of

researching or ordering 1% 1% 1% 1%

0

500

1000

1500

2000

2500

3000

1 6 11 16 21 26 31 36 41 46 51 56 61 66

Vo

te

Carewords


Top Tasks Management is very easy to understand.

#6

TASK PERFORMANCE

SCALE

HOSPITALIZED [55-64]

HEALTHY (85-94)

MONITORING (75-84)

MEDICATED [65-74]

ICU [45-54]

LAST RITES [0-34]

ABLE TO EXCEL [95-100]

LIFE SUPPORT [35-44]


Top Tasks Management is very easy to understand.

#6May 2013


Top Tasks is about monitoring performance on an ongoing basis

#7

Bug Task Success Rate Comparisons 2012-2013

Oct 2012 May 2013 July 2013 Nov 2013Feb 2012 Jun 2012 Feb 2013

Ran a pilot test to simulate a

possible solution. It worked!

TPI 6 TPI 6TPI 15 TPI 17 TPI 39TPI 23TPI 8


Top Tasks Management diagnoses what is wrong and clearly indicates how to fix it.

#8 JUNE 2012 Bug Task

Jun 2012

TPI 235 Succeed

6 Give up

18 Subjects

6 Time out or

partial answer

1 Disaster


• 13/18 failed or disaster

• 5/18 successful

• All 13 failures were due to the user’s attempt to search:

Cisco.com site search (8 failures)

Google (5 failures)


#8 Minor: Users were frustrated when, after they had got to the

product page (product as hub strategy), they clicked on the

Bug Toolkit and discovered that the link was not context

sensitive. They had to re-select the product all over again.

Recommend: Provide a context sensitive link from the

product page, similar to the link provided from the product

download software page.


#8Minor: Search does not support looking for bug related information very well. None of the first page results was

related to the task, even though the words used in the search phrase are identical to those used in the title of the

bug details page (shown lower right). When people added quotes to the command they got no results and the

suggestions were not specific enough.

Recommend: Since many people try to search, consider a filter for Bug Reports. Instead of a generic link to

Support Tools, provide a set of links to other search related tools, like Bug Look Up or Error Message Decoder, etc.


• 13/18 failed or disaster

• 5/18 successful

• All 13 failures were due to the user’s attempt to search:

Cisco.com site search (8 failures)

Google (5 failures)

• If we fix SITE SEARCH, 13/18 (5+8) should succeed

• Team Commitment: Raise Task Task Performance Score to 70

Assumptions:

Increase success rate to 13/18 participants

Limiting the additional incremental penalties (e.g. excessive success time and disaster confidence) to be ~2-3


Top Tasks scoring is predictable and statistically reliable.

#9

Hospitalized (55-64)

Healthy (85-94)

Monitoring (75-84)

Medicated (65-74)

ICU (45-54)

Last Rites (0-34)

Able to Excel (95-100)

Life Support (35-44)

Original Score = 23

70

FY13

Task Performance

Target =


November 2012 : Pretend you have forgotten the password for the Cisco account and take whatever actions are required to log in.

Success/Failure Rates Completion Time

Password recovery = Minor Issues (50-59) TPI = 59


Major: As in other testing, people using Internet Explorer or Chrome typically saw

an unformatted page, like that shown. Unfortunately, it became a major issue for

this task because if people requested to see all content then the page was

reloaded. Reloading the page violated the security requirement that the URL could

only be opened once so people had to start all over. The only way to complete the

task was to leave the page unformatted.

Recommend: Make all content secure so people don’t have to contend with

annoying dialogs and difficult to understand error conditions..

Comment: Clicking Show all content leads to

the page shown in the overlay.


Minor: Several people

encountered multiple input

errors due to the constraints on

a valid password.

Recommend: Simplify the

rules for password

composition. Provide a

password strength indicator

instead.

Major: After clicking the Submit

button, the system would not

respond for a minute or more,

prompting some people to

abort and try all over again.

Recommend: Address system

performance issues.


May 2013: Pretend you have forgotten the password for the Cisco account and take whatever actions are required to log in.

Success/Failure Rates Completion Time

Password Recovery = Fair (60-69) TPI = 68


Strength: The Password Strength help

mechanism worked very well in helping people

to get an acceptable password at the time of

input, rather than having to wait for verification

after they had submitted the entire form.

People still did not like the excessive number

of constraints which had to be met.



The Top Tasks approach helps you become a more effective manager.

#10 Support Goal: TPI 67 70

Enable customers and partners to optimally plan, build and run their networks:

prevent and solve, inform and educate, connect and collaborate

Network

Lifecycle

Management

Quick Task

Completion

Personalized &

Customized

Experience

Comprehensive

and High

Quality Content

Extended

Access &

Usage of

Assets

Develop a stronger single

customer feedback process

Redefine content as an

enabler

Improve content by

establishing Knowledge

Operations Management

Deliver Smart Software

Selection

Learn from customer

interactions by Predictive

Operational Analytics/BI

Create rapid, efficient

access to content, tools,

and services

Continue to improve Cisco

Service Request

Management App

Improve the BTK

application and web

service

Continue to refine personas

Improved search

experience

Create a contextual user

interaction / journey

Increase mobile and

Ratings & Review adoption

Deliver 3 new APIs to

partners

Decrease 1st Search

Abandon Rate

Operational Excellence & Focused Innovation

Refreshed Configuration

and Operate tools portfolio

55DQI

(Data Quality Index)

49 (+1)

11%NLM

Overall Completeness

7%

70SWTG TPI Task Performance

67 (+2)

4.0SWTG CSAT ( Pure, 3 moving avg.)

4.10 (+.03)

40N&X

(Engage & Expansion )

44 (+19)

Provide indispensable value and concrete differentiation through

the Industry’s leading virtual services experience

Thank you.Thank you.


Measuring usability by scoring the top tasks.

#1Tasks represent the most important, typical tasks.

#2Studies use 15-18 subjects to performing 10-12 tasks.

#3

It is a monitoring tool.

#7It is a diagnostic tool.

#8It is predictable.

#9

Testing is performed remotely, on the live site.

#4Was first performed at Cisco in 2010 with a “top tasks” survey.

#5Top Tasks is very easy to understand.

#6

Documents

The TPI: Measuring Task Performance on Cisco · Top Tasks Management was first introduced at Cisco in 2010 with a “top tasks” survey. The tasks selected were based on a Top Tasks