1 Training Objectives Prepare you to plan for data collection and analysis by using the ASA Data Collection Plan Discuss various ways to display and analyze

1

Training Objectives

• Prepare you to plan for data collection and analysis by using the ASA Data Collection Plan

• Discuss various ways to display and analyze data

• Describe how to construct various graphs as they apply to the ASA Template

• Review the basics of creating charts using Excel

2

ASA Data Collection Plan

3

PLAN before you act!

• Data collection can be time consuming

• Need to figure out the Who, What, Where, When, Why for each performance measure

• Distribute the data collection responsibility among your ASA Team Members

• Garbage in = Garbage out

• Plan now to prevent crisis down the road

4

• Takes ASA Template information and spells out the specifics of gathering data – Performance objective– Performance measure– Methodology– Point of Contact (POC)– Source of Data– Frequency– Target

• Recommend EACH ASA Team complete ONE Data Collection Plan– Road map for the team’s data collection efforts– Distributes responsibilities among team members– Allows the team to communicate their methods to others

• ASA Consultant, OQM, your Division/Office Director may request to review your Data Collection Plan

ASA Data Collection Plan

5

• Step 1: Review Common Performance Objectives and Measures

• Step 2: Add Unique Performance Objectives and Measures

• Step 3: Review/Determine Methodology

• Step 4: List Points of Contact (POC)

• Step 5: List Source of Data

• Step 6: List Data Collection Frequency

• Step 7: List Target

Completing the ASA Data Collection Plan

6

• Common performance objectives and performance measures are already listed

• Need to add Discrete Services as appropriate

• Need to add unique performance objectives and measures as appropriate

• Boxes with text can be edited if desired

• Shaded areas typically do not need to be filled in

Completing the ASA Data Collection Plan (cont.)

7

• Review common measures in each perspective• If measurement is occurring at the Discrete Service level

– Enter your Discrete Services on the form

– Plan provides for 3 Discrete Services for common measures

– Add/delete rows for Discrete Services as necessary

• If measurement will occur at the Service Group level– Change shaded boxes to white background on row that lists the

common measure

– Enter information in that row

– Delete the rows for the Discrete Services

Step 1: Review Common Performance Objectives and Measures

8

Step 1: Review Common Performance Objectives and Measures (cont.)

Common performance measures from ASA Template are listed for each of the four perspectives.

Methodology POC Source of Data Frequency Target

Once / FY02 One chart



Customer segmentation of Discrete Services

ASA Data Collection Plan - 2002

Customer Performance Objective: Increase Understanding of Customer Base

Service Group:

Customer Perspective

Performance Measure

DS2

The ASA Team will meet to decide what makes most sense as the variable to use in segmenting our customers (e.g., NIH IC, customer location, type of customer). From there we will gather customer data from appropriate IT systems on products/services that will be needed to complete the customer segmentation chart. We will relate these product/service data to Discrete Services. We will then create graphs of our customer segments for each Discrete Service using Excel. Training on how to create the charts will be obtained in Data Analysis training. The charts will be included in our ASA Final Presentation.

DS1

DS3

Shaded areas usually require no entry

9


Type in Discrete Services for common measures as appropriate.





Customer segmentation of Discrete Services

ASA Data Collection Plan - 2002

Customer Performance Objective: Increase Understanding of Customer Base

Service Group:

Customer Perspective

Performance Measure

DS2

The ASA Team will meet to decide what makes most sense as the variable to use in segmenting our customers (e.g., NIH IC, customer location, type of customer). From there we will gather customer data from appropriate IT systems on products/services that will be needed to complete the customer segmentation chart. We will relate these product/service data to Discrete Services. We will then create graphs of our customer segments for each Discrete Service using Excel. Training on how to create the charts will be obtained in Data Analysis training. The charts will be included in our ASA Final Presentation.

DS1

DS3

10


• To add rows – Click left mouse button on DS3

– Choose “Insert”, “Row”

– Select Columns A and B in new row– Choose “Format”, “Cells”, “Alignment”, “Merge Cells”– Click “OK”– Re-label and edit rows as appropriate




DS2

DS1

DS3




DS2

DS1

DS3

11


• To delete rows– Click left mouse button on DS3

– Choose “Edit”, “Delete”– Click “Entire Row”– Click “OK”




DS2

DS1

DS3



DS1

DS2

12

Step 2: Add Unique Performance Objectives and Measures

Type in Unique Performance Objectives and Measures on Template for each perspective. Add Discrete Services as needed.

Methodology POC Source of Data Frequency TargetThe ASA Team will meet to discuss the kinds of products/services we deliver to customers. Consideration will be given to the categories of products/services, the frequency of delivery of those products/services, and whom they are delivered to in planning for data collection. Customer segmentation data will be reviewed to identify possible groups to survey. Methods to distribute and collect the surveys will be determined. Prior to distribution of any customer surveys the methodology must be reviewed by OQM. OQM will provide our team with the results. We will interpret the results and include our assessment in our ASA Final Presentation.

Baseline


Performance Measure

Customer Performance Objective: Performance Measure

Customer Performance Objective: Increase Customer Satisfaction

Customer satisfaction ratings from the ORS Customer Scorecard

13

Step 3: Review/Determine Methodology

• Methodology is a summary of how you plan to gather and analyze the data– For common measures, review the suggested methodology and edit

as appropriate

– For unique measures, decide on appropriate methodology and type on template

14

Step 3: Review/Determine Methodology (cont.)

Edit or develop methodology for data collection and analysis.


Baseline


Performance Measure




15


• To resize rows – Position mouse at bottom of row number at left of screen– A crosshair appears

• Edit methodology as appropriate

Methodology POC Source of Data Frequency TargetThe ASA Team will meet to discuss the kinds of products/services we deliver to customers. Consideration will be given to the categories of products/services, the frequency of delivery of those products/services, and whom they are delivered to in planning for data collection. Customer segmentation data will be reviewed to identify possible

Baseline


Performance Measure




16


• Click left mouse button to display row height• Move mouse up or down to increase or decrease row height


Baseline


Performance Measure




Height: 236.25

17


• To format cells for word-wrap– Select cell(s) with mouse– Choose “Format”, “Cells” , “Alignment”– Click “Wrap text”, “Merge cells”– Click “OK”


BaselinePerformance Measure



18

Step 4: List Points of Contact (POC)

• Decide who will have primary responsibility for data collection and analysis for each performance measure– Serve as focal point regarding the data collection/analysis if several

people are involved

– Can provide Team Leader with updates on how data collection and analysis is proceeding

– POC will ensure that data collection for that measure occurs in a timely fashion

19

Step 4: List Points of Contact (POC) (cont.)

Type in Point of Contact for each measure as appropriate.

Methodology POC Source of Data Frequency TargetThe ASA Team will determine meaningful performance measures for each of the Discrete Services. Deployment process maps will be utilized to help pinpoint useful process measures. Our consultant will help define the measures, the data to be collected, the source of the data, the frequency of data collection, and the methodology. Guidance on how to analyze and display the data will be obtained in Data Analysis training and Process Behavior Chart training. Our assessment will be included in our ASA Final Presentation.

100% of Discrete Services will identify at least one unique internal business process measure


Internal Business Performance Objective:

Performance Measure

DS3

DS2

DS1

Identify and report on process measures for Discrete Services

Performance Measure

Internal Business Performance Objective: Identify Methods to Measure Processes

20

Step 5: List Source of Data

• Identify where you will actually get the data for each measure• Possible sources of data include:

– Ordering systems (ADB)

– Budgeting systems (OROS)

– Studies that have been conducted (Rate Studies)

– Hard copy order forms

– Tally of customer requests

– Email messages

– Phone calls

– Observations

21

Step 5: List Source of Data (cont.)

• If no known source exists – Need to develop methods to collect data– Check sheets useful for all kinds of data collection

» Constructed to whatever size, shape, format appropriate for you

» Easy to compile data in a way that can be readily graphed

» Record data for later analysis using bar, pareto, and run charts

» Provide historical record of the process over time

» Can be used to introduce data collection to workers who may not be familiar with it

• May want to use simple check sheet to summarize data already collected but not tallied

22

ExampleCheck Sheet

Errors May June July Aug Sept Total

Wrong form used 11 1 1 11 6

No CAN

1 1 2

No POC

1 1 1 3

Incomplete information

111 11111 11 1111 1111 18

23

ExampleCheck Sheet

--NIAID

12111111111111Outside Agency

2111111

11111111

11111111

HHS

41111Other NIH Institute

6111111NHLBI

30111111111111111111111111111111NCI

18111111111 111111111Clinical Center

811111111NIH OD

1211111111 1111Other ORS Div/Office

VisitPhoneEmailVisitPhoneEmailVisitPhoneEmail

TotalSeptAugJuly

Customer Contacts

24


• For more information on designing data collection forms:– Memory Jogger for an example of a Check Sheet (p.31)

– Statistical Methods for Quality Improvement (pp. 7 - 16)

– Guide to Quality Control (pp. 30-41)

– Basic Tools for Process Improvement Module 7: Data Collection (pp. 12-23)

» http://www.odam.osd.mil/qmo/pdf/datacoll.pdf

– Process Improvement Notebook for Data Collection Sheet and Check Sheet (pp. 66, 68)

» http://www.odam.osd.mil/qmo/pdf/pin.pdf

25


Type in Source of Data for each measure you have listed.





Performance Measure

DS3

DS2

DS1


Performance Measure


26

Step 6: List Data Collection Frequency

• Identify the frequency of data collection for each measure• Examples include:

– Ongoing (e.g., ordering systems)

– Weekly (e.g., count of number of complaints)

– Monthly (e.g., utilization statistics)

– Quarterly (e.g., financial reports)

– Once per fiscal year (e.g., customer survey)

• Some Common Measures are filled in for you

27

Step 6: List Data Collection Frequency (cont.)

Type in Frequency of Data Collection for each measure.





Performance Measure

DS3

DS2

DS1


Performance Measure


28

Step 7: List Target

• Identify the target performance level for this FY if possible• Targets are usually identified after it is understood what the

process is capable of doing– Consult Service Level Agreements (SLA) if they exist

– Examples include:

» Response time within 3 business days

» 95% of reports error free

» Actual asset utilization within 10% of plan

» Reduce use of vendors by 5%

• If performance measure is being defined for the first time this year– True process capability is being defined for first time

– Type in “Baseline” under “Target”

• Targets for some Common Measures are filled in for you

29

Step 7: List Target (cont.)

Type in Target performance for each measure listed on your Plan.





Performance Measure

DS3

DS2

DS1


Performance Measure


30

Methods of Analyzing Data With Graphs

31

There are Many Ways to Analyze Data

• Two general types of data– Quantitative

– Qualitative

• Ways to analyze quantitative data– Through visual displays - graphs

– Through process behavior charts

– Through statistical analyses

» Chi-square, t-tests, ANOVA, correlation, regression analyses, factor analysis

– Through predictive modeling

» LISREL

32

Common Graphs for Analysis

• Pie charts

• Bar charts– Pareto charts

• Radar charts

• Line graphs– Run charts

– Process behavior charts

• Scatter diagrams– Correlation

– Gap analysis

33

Pie Charts

• Often used to summarize categorical data

• Show the proportional size of items that make up a whole

• Convey the relative contribution of different categories to the total

• Usually used with percentages

• Good for simple descriptions, quick snapshots of some kinds of data

34

ORS ExamplePie Chart

Conference Services Survey Respondents

N=564

35

Bar Graphs

• Useful for comparing different categories by contrasting heights of various bars

• Helpful in making comparisons among items

– Frequency, size, importance, satisfaction, dollars, etc.

• Often used to show comparisons on more than one dimension

– Categories of products ordered

36

ORS ExampleBar Graph

Conference Services: Scheduling Actions that Occurred

N=564

37

Categories of Photography Products Ordered by Year

ORS ExampleBar Graph

38

Pareto Charts• Type of bar chart• Display bars in descending order• Help to focus efforts on areas that offer the greatest potential for

improvement– Based on the Pareto principle– Most problems are due to a minority of categories of causes

• For more information:– The Memory Jogger for more information (p. 95)– Building Continual Improvement (pp. 38-44)– Statistical Methods for Quality Improvement (pp. 17-24)– Basic Tools for Process Improvement Module 8 - Pareto Charts

»http://www.odam.osd.mil/qmo/pdf/pareto.pdf– The Process Improvement Notebook for Pareto Chart of Causes of

Quality form (pp. 70-73)

»http://www.odam.osd.mil/qmo/pdf/pin.pdf

39

ExamplePareto Chart

Improvement Ideas Supported by Customers

N=250

40

Radar Charts

• Used to compare actual values on a series of categories to ideal values

– Allows comparison among data points

– Encourages identifying strengths and weaknesses

• See The Memory Jogger for more information (p. 137-140)

41

ExampleRadar Chart

FY02 Customer Service Ratings: ORSIndex = 6.9

0.0

2.0

4.0

6.0

8.0

10.0Availability

Responsiveness

ConvenienceCompetence

Handling ofProblems

Data obtained from 3,000 customers

Ideal Value

Actual Value

42

Line Graphs

• Used to study data for patterns

• Helpful in making comparisons over time– Show changes in numerical amounts

– Identify sequences/changes in data

– Demonstrate performance before and after an intervention

– Can be used to predict future performance

• Run charts and process behavior charts are types of line graphs

• For more information:– The Memory Jogger (pp. 141-144)

– Building Continual Improvement

– The Process Improvement Notebook for Run Chart and Control Chart forms (pp. 82-98)

» http://www.odam.osd.mil/qmo/pdf/pin.pdf

43

ExampleLine Graph

Ratings of Responsiveness to Customer Complaints by Year

N=125

44

ORS ExampleLine Graph

NIH ID Cards Issued by Year

45

Scatter Diagrams

• Demonstrate the relationship between two variables

• Values on two variables are plotted on a graph to visually show the relationship

• For more information:– The Memory Jogger (pp. 145-149)

– Statistical Methods for Quality Improvement (pp. 67-89)

– The Process Improvement Notebook for Scatter Diagram Worksheet (pp. 78-81)

»http://www.odam.osd.mil/qmo/pdf/pin.pdf

46

ExampleScatter Diagram

Gap Analysis of Customer Ratings of Satisfaction and Importance

FY02 Service Group Importance and Satisfaction Ratings

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00

Importance Mean Rating

Sa

tis

fac

tio

n M

ea

n R

ati

ng

Not Important, Satisf ied

Not Important, Not Satisf ied

Important, Satisf ied

Important, Not Satisf ied

Cost

ResponsivenessTimeliness

Convenience

Reliability

Quality

Competence

Availability

Handling of Problems

Each symbol indicates both the importance and satisfaction rating for a variable, such as timeliness.

47

Analyzing Data With Graphs

Customer segmentation charts are good examples of these.

Purpose Examples Pie Bar Pareto Radar LineRun

Control ScatterPercentage Relative to Whole Percentage of customers by type X

Revenue by customer

Number of errors by type

Number of complaints by type

Frequency of help desk calls by typeComparison of Actual Values to Ideal Values Customer satisfaction ratings

X

Number of defects in product by day

Average number of days to respond to customer requests by monthAverage holding time of animals by month

% bills with errors by weekAge of equipment and unscheduled repairs

Availability of product and request turn-around time

Problem resolution speed and % team fully trained

% error free bills and customer satisfaction ratings

Gap Analysis of survey data

Identify Relationships between Variables

Measure Chart Type

Relative Frequency, Size, Importance, Dollars

X X

Monitor Performance over Time

X X

X

48

Analyzing Data With Graphs (cont.)

Customer satisfaction results will be provided to Service Groups using radar charts.



Revenue by customer




X









Measure Chart Type


X X


X X

X

49


Internal Business Process measures often require run charts or control charts. Control charts are a special type of run chart. Training on process behavior charts will be available (http://www.nih.gov/od/ors/od/oqm/asa/asa_training.htm)



Revenue by customer




X









Measure Chart Type


X X


X X

X

50


Scatter charts depict the correlation between 2 variables and can be used to investigate hunches you may have about relationships.



Revenue by customer




X









Measure Chart Type


X X


X X

X

51

Tips for Graph Analysis

• Realize that analyzing data is a skill – With experience you will get better

– Analysis is both a science and an art

• Common methods to interpret graphical data– Compare size of categories to each other

– Compare self to target

– Compare self to others who are similar

– Compare self to industry standards

– Compare self over time

• General things to look for in data– Highlight similarities and differences

– Identify trends or patterns

– Notice if anything is missing (e.g., customer groups)

– Look for themes

52

Tips for Graph Analysis (cont.)

• Make conclusions based on graphs – Summarize what you learned from the data

– Diagnose problems identified by data

– Identify any potential solutions suggested by data

• Generate potential actions based on what you have learned from the data– Can you make changes to address what you learned from data?

– How might you implement those actions?

– State actions in terms of recommendations

53

Summarizing Analyses in your ASA Presentation

• Organize your graphs and conclusions to tell the story of your ASA– Who are your customers (i.e., customer segments)

– Are you customers satisfied with your products/services (i.e., customer satisfaction)?

– What have you learned about your internal business processes from the process maps and measures?

– Learning and Growth analyses and conclusions

– Financial findings regarding unit costs, asset utilization

– Recommendations for improvement based on data gathered and analyzed

• ASA Presentation Template and tips will be available this summer via the ASA web page

54

Summarizing Analyses in Your ASA Presentation (cont.)

• Beware of excess detail– Do NOT place all graphs in main section of presentation

» Select only the most informative graphs for main portion of ASA Presentation

» Insert narrative slides that include your conclusions of the graphs, findings from the data

» Place graphs not used in main presentation in Appendices to your main presentation

– Try to see the “forest through the trees”– Look for themes and major findings as a result of the ASA – Include recommendations and proposed follow-on actions

» Make realistic recommendations that could actually be implemented

» If have power to implement suggested changes, do so

55

Constructing Graphs from the ASA Template

56

Customer Perspective Graphing and Analyzing Customer Segmentation Data

57

Customer Segmentation

1. Select customer characteristic(s) to segment• NIH IC

• Location– On-campus, Off-campus– Building

• Type of customer – NIH employees, contractors, visitors

2. Select product/service dimension• New equipment sales

• Trouble reports

• Requests for service

• Frequency of use

3. Determine time frame• Current fiscal year, several fiscal years, current month,

several months

58

Customer Segmentation (cont.)

4. Generate chart(s)

5. Review and interpret chart(s)

Which segments are your primary customers?

Is there any pattern to your chart(s)?

Which NIH customers are not currently your customers? Why?

What does the information say about who is and who is not your customer?

59

Pie Chart Customer Segmentation of Discrete Services

• Discrete Service = Manage concession services program, contracts, and use agreements

• Customer characteristic = Customer Type

• Product/Service dimension = Frequency of Use

• Time frame = January through April, 2001

• Methodology

– Surveys handed out to all willing participants at point of sale at 8 Dining Halls on campus for one day during the data collection period

60

Pie Chart Customer Segmentation of Discrete Services (cont.)

1. Collect data and enter in Excel

List Customer Type Horizontally

Label Frequency of Use Measure

Enter Data

Federal Government Employees

Fellows/Visiting Researchers Contractors

Visitors/ Conference Attendees Other

# Sales 13250 5500 2500 1500 2250

61


2. Generate Chart• Select all labeled cells and data using mouse

• Choose “Insert”, “Chart”, “Pie”

• Click “Finish”

# Sales

Federal GovernmentEmployees

Fellows/VisitingResearchers

Contractors

Visitors/ Conference Attendees

Other

62


3. Format Chart• Click on chart

• Click right mouse button

• Choose “Chart Options”

• Choose “Titles”, “Chart titles”: Type in chart title (e.g., Concession Service Customers)

• Choose “Data Labels”: Click ”Show percent”

• Click “OK”Concession Service Customers

10%

6%

53%

22%

9% Federal GovernmentEmployees


Contractors


Other

63


4. Add important notes to chart• Ensure that “Drawing” toolbar is available to you

– Choose “View”, “Toolbar”, “Drawing”

• Click on chart

• Click on “Text Box” from Drawing Toolbar

• Place text box on chart

• Type in note and format text

Concession Service Customers

53%

22%

10%

6%9% Federal Government

Employees


Contractors


OtherData obtained from 564 customers at all

concessions during January - April, 2001.

64

Bar GraphCustomer Segmentation of Discrete Services

• Discrete Service = Conduct collaborative bioengineering and physical science research

• Customer characteristic = NIH IC

• Product/Service dimension = Number of Collaborations

• Time frame = FY01

• Data Collection

– Data collected on all collaborations during FY01

65

Bar Graph Customer Segmentation of Discrete Services (cont.)



Type Measure Label

Enter Data

NCI NIA NIAID NIMH NINDS

# Collaborations 16 1 7 5 18

66



• Choose “Insert”, “Chart”, “Column”


# Collaborations

02468

101214161820


# Collaborations

67





• Choose “Titles”, “Chart title”: Type in chart title (e.g., # Collaborations with NIH ICs)

– Choose “Category (X) Axis”: Type in Customer Type (e.g., NIH IC)– Choose “Category (Y) Axis”: Type in Measure Label (e.g., #

Collaborations)

• Choose “Legends”: Click “Show legend”

• Choose “Data Labels”: Click “Show value”

• Click “OK”

# Collaborations WIth NIH IC

16

1

75

18

0

5

10

15

20


NIH IC

# C

oll

abo

rati

on

s

68


3. Format Chart (cont.)• Click on gray area on chart (e.g., plot area)


• Choose “Format Plot Area”

• In the “Area” section, click on the white box.

• Click “OK”

• Click on gridlines in center of chart


• Choose “Clear” # Collaborations With NIH ICs

16

1

75

18

0

5

10

15

20


NIH IC

# C

olla

bo

rati

on

s

69




• Click on chart




# Collaborations With NIH ICs

16

1

75

18

0

5

10

15

20


NIH IC

# C

olla

bo

rati

on

s

Data based on 47 collaborations in FY01

70

Pareto Chart Customer Segmentation of Discrete Services


• Customer characteristic = NIH IC• Product/Service dimension = Number of Collaborations• Time frame = FY01• Data Collection

– Data collected from all DBEPS collaborators on the number of collaborations during FY01

Note: Data is same as bar graph example, but pareto chart will re-order NIH ICs from the greatest to the fewest number of collaborations.

71

Pareto Chart Customer Segmentation of Discrete Services (cont.)

1. Collect data and enter in Excel (see slide 65)

• Click “Data”, “Sort”

• Ensure correct row (e.g., row with data) is displayed in “Sort By” box. If not, select correct row

• Click “Descending”

• Click “OK”



Select IC and Data Cells using Mouse

NINDS NCI NIAID NIMH NIA


Data is resorted by IC in descending order

72

Pareto Chart Customer Segmentation of Discrete Services (cont.)

2. Generate Chart• See Slide 66

3. Format Chart• See Slides 67-68

# Collaborations With NIH ICs

1816

75

1

0

5

10

15

20

NINDS NCI NIAID NIMH NIA

NIH IC

# C

oll

ab

ora

tio

ns

Data based on 47 collaborations in FY01

73

Bar Graph with Added Dimension Customer Segmentation of Discrete Services


• Customer characteristic = NIH IC• Product/Service dimension = Number of Collaborations

• Time frame = FY01 and FY02• Data Collection

– Data collected on all collaborations during FY01 and FY02

Added Dimension

74

Bar Graph with Added Dimension Customer Segmentation of Discrete Services (cont.)



Type Measure Labels

Enter Data


FY01 16 1 7 5 18

FY02 18 3 12 8 18

75



• Choose “Insert”, “Chart”, “Column”


0

2

4

6

8

10

12

14

16

18

20


FY01

FY02

76





• Choose “Titles”, “Chart title”: Type in chart title (e.g., # Collaborations With NIH ICs by Fiscal Year)

– Choose “Category (X) Axis”: Type in Customer Type (e.g., NIH IC)– Choose “Category (Y) Axis”: Type in Measure Label (e.g., # Collaborations)

• Choose “Data Labels”: Click “Show value”• Click “OK”

# Collaborations With NIH ICs by Fiscal Year

16

1

75

1818

3

12

8

18

0

5

10

15

20


NIH IC

# C

oll

abo

rati

on

s

FY01

FY02

77






• Click “OK”



• Choose “Clear”# Collaborations With NIH ICs by Fiscal Year

16

1

75

1818

3

12

8

18

0

5

10

15

20


NIH IC

# C

oll

abo

rati

on

s

FY01

FY02

78




• Click on chart




# Collaborations With NIH ICs by Fiscal Year

16

1

75

1818

3

12

8

18

0

5

10

15

20


NIH IC

# C

olla

bo

rati

on

s

FY01

FY02

Data based on 47 collaborations in

FY01 and 59 in FY02

79

Customer Perspective Graphing and Analyzing Customer Satisfaction Ratings

80

Customer Satisfaction Ratings

1. Review methodology with OQM before distributing surveys to any customers

2. Completed surveys will be analyzed by OQM and summary charts will be provided

3. Review the radar charts and interpret the findings• Compare rating dimensions on each chart

• What is highest? Why?

• What is lowest? Why?

4. Review the scatter diagram (gap analysis)• What do customers feel is most important?

• What was their satisfaction on the most important dimensions?

• What actions can be taken to address the situation?

5. Focus first improvement efforts on areas most important to customers with lower satisfaction ratings

81

Radar Chart Customer Satisfaction Ratings

FY02 Product/Service Ratings: ORSIndex = 7.4

0.0

2.0

4.0

6.0

8.0

10.0Cost

Quality

Timeliness

Reliability


FY02 Product/Service Ratings: Service Group Index = 7.7

0.0

2.0

4.0

6.0

8.0

10.0Cost

Quality

Timeliness

Reliability

Data obtained from 200 customers

Mean Product/Service satisfaction ratings provided for ORS overall on the left and the Service Group on the right.

82

Radar Chart Customer Satisfaction Ratings (cont.)

Mean customer service ratings provided for ORS overall on the left and the Service Group on the right.

FY02 Customer Service Ratings: ORSIndex = 6.9

0.0

2.0

4.0

6.0

8.0

10.0Availability

Responsiveness


Handling ofProblems


FY02 Customer Service Ratings: Service Group Index = 7.3

0.0

2.0

4.0

6.0

8.0

10.0Availability

Responsiveness


Handling ofProblems

Data obtained from 200 customers

83

Scatter Diagram: Gap Analysis Customer Importance and Satisfaction Ratings

Mean satisfaction and importance ratings for the Service Group.

FY02 Service Group Importance and Satisfaction Ratings

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00

Importance Mean Rating

Sa

tis

fac

tio

n M

ea

n R

ati

ng

Not Important, Satisf ied

Not Important, Not Satisf ied

Important, Satisf ied

Important, Not Satisf ied

Cost

ResponsivenessTimeliness

Convenience

Reliability

Quality

Competence

Availability

Handling of Problems

84

Internal Business Process Perspective Graphing and Analyzing Process Measures

85

Internal Business Process Measures

1. Identify process measures using deployment process maps, known problem areas, customer feedback

2. Implement methodology to collect data

3. Collect data

4. Generate chart(s)

5. Review and interpret chart(s)• What do the process measures tell us?

• Can we determine why problems are occurring?

• Is process improving or declining? Why?

• What can be done to impact process?

6. Identify and report on areas for improvement

86

Run ChartInternal Business Process Measures

• Unique Measure = Reduce number of bills processed with errors

• Methodology = Count number of bills returned by customer noting an error

• Point of Contact = Billing Supervisor

• Source of Data = Log of bills returned classified by customer and type of error

• Frequency = Quarterly

• Target = 5% reduction in bills returned

87

Run ChartInternal Business Process Measures (cont.)


List Frequency Intervals Horizontally

Label Data Being Collected

Enter Data

Q1 FY01 Q2 FY01 Q3 FY01 Q4 FY01 Q1 FY02 Q2 FY02 Q3 FY02 Q4 FY02

# Bills Processed with Errors 58 62 54 68 57 48 25 22

88



• Choose “Insert”, “Chart”, “Line”


# Bills Processed with Errors

01020304050607080

Q1FY01

Q2FY01

Q3FY01

Q4FY01

Q1FY02

Q2FY02

Q3FY02

Q4FY02

# Bills Processed withErrors

89


3. Format Chart• Click on chart• Click right mouse button• Choose “Chart Options”• Choose “Titles”, “Chart title”: Type in title (e.g., # Bills Processed

With Errors by Quarter)• Choose “Legend”: Click “Show legend”• Choose “Data Table”: Click “Show data table”• Click “OK”

# Bills Processed with Errors by Quarter

0

20

40

60

80

# Bills Processed w ith Errors 58 62 54 68 57 48 25 22

Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

90


3. Format Chart (cont.)• Click on gray area on chart (e.g., plot area)• Click right mouse button• Choose “Format Plot Area”• In the “Area” section, click on the white box. • Click “OK”• Click on gridlines in center of chart• Click right mouse button• Choose “Clear”


0

20

40

60

80


Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

91




• Click on chart





0

20

40

60

80


Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

Note: Process changed in Q2 FY02

92


• Run Chart shows that the change in process in Quarter 2 of FY02 appears to have resulted in fewer errors in billing

• Training on process behavior charts will be available. Check the ASA web page (http://www.nih.gov/od/ors/od/oqm/asa/asa_training.htm)

• Process behavior charts (i.e., control charts) use formulas to help determine whether the difference is significant


0

20

40

60

80


Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

Note: Process changed in Q2 FY02

93

Scatter DiagramRelationship Between Two Measures

• After completing data analyses on other measures, a scatter chart may be generated

– Further examine relationships between 2 measures– Relationship may be suggested by other analyses– Example - examine the relationship between two unique measures

» Unscheduled repairs and age of equipment

• Graph data and study relationship– Positive in nature– Negative in nature

94

Scatter DiagramRelationship Between Two Measures (cont.)


Repairs Age of Equipment1 = Routine Maintenance2 = Unscheduled Repair

1 11 21 31 42 52 81 11 11 22 102 42 251 22 41 22 122 52 72 62 51 11 1

Round off age of equipment to nearest year

List Categories and Definitions Horizontally

Enter Data

95

Scatter DiagramRelationship Between Two Measures (cont.)

2. Generate Chart• Select all data cells using mouse

• Choose “Insert”, “Chart”, “XY (Scatter)”

• Click “Next”

• Choose “Data Range”. Click “Columns”


0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5

Series1

96

Scatter Diagram Relationship Between Two Measures (cont.)

3. Format Chart• Click on chart• Click right mouse button• Choose “Chart Options”• Choose “Titles”, “Chart title”: Type in chart title (e.g., Repair Calls by Age of

Equipment)– Choose Value (X) Axis: Type in X axis title (e.g., Type of Repair Call)– Choose Value (Y) Axis: Type in Y axis title (e.g., Age of Equipment)

• Choose “Legend”• Click “Show legend”• Click “OK”

Repair Calls by Age of Equipment

0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5

Type of Repair Call

Ag

e o

f E

qu

ipm

ent

97

Scatter Diagram Relationship Between Two Measures (cont.)

4. Format Chart (cont.)• Click on chart

• Double click left mouse button on values on X axis

• Choose “Scale”– Choose “Minimum”: Type in 0– Choose “Maximum”: Type in 3– Choose “Major Unit”: Type in 1– Choose “Minor Unit”: Type in 1


0

5

10

15

20

25

30

0 1 2 3

Type of Repair Call

Ag

e o

f E

qu

ipm

ent

98

Scatter ChartRelationship Between Two Measures (cont.)





• Click “OK”



• Choose “Clear”Repair Calls by Age of Equipment

0

5

10

15

20

25

30

0 1 2 3

Type of Repair Call

Ag

e o

f E

qu

ipm

ent

99




• Click on chart





0

5

10

15

20

25

30

0 1 2 3

Type of Repair Call

Ag

e o

f E

qu

ipm

ent

1 = Routine Maintenance2 = Unscheduled Repair

100


6. Run Regression Analysis on Data• Choose “Tools”, “Data Analysis”, “Regression”

• Click “OK”

• Click inside “Input Y Range” box

• Select all Y axis data cells (e.g., repair data cells)

• Click inside “Input X Range” box

• Select all X axis data cells (e.g., age of equipment cells)

• Click “Output Range”. Click in Output Range box

• Click on blank cell on data sheet. Click “OK”

101


6. Run Regression Analysis on Data (cont.)

• Correlation = number between 0 (no relationship) and 1 (1 to 1 relationship)

• Significance level = .05 indicates 95% confidence level

Correlation = .61

Significance Level = <.002

SUMMARY OUTPUT

Regression StatisticsMultiple R 0.61241

R Square 0.37505Adjusted R Square 0.3438Standard Error 0.41456Observations 22

ANOVAdf SS MS F Significance F

Regression 1 2.062755747 2.062756 12.00238 0.002448

Residual 20 3.437244253 0.171862Total 21 5.5

Number of Observations = 22

102

Learning and Growth Perspective

Graphing and Analyzing Turnover, Sick Leave, Complaints, Awards

103

Turnover, Sick Leave, Complaints, Awards

1. Data is being obtained from ORS HR IT systems

2. OQM will graph data and provide to ASA Teams

3. ASA Teams meet to review and discuss the data

4. Highlight conclusions from the discussion in the final ASA presentation

104

Learning and Growth Perspective

Analysis of Readiness Index

105

Qualitative Analysis of Readiness Index

1. The Readiness Index will be available this summer via the ASA web page

2. ASA Teams meet to review and discuss the questions on the Readiness Index

3. Prepare a short narrative summary of conclusions included as Appendix in final ASA presentation

4. Highlight conclusions in the final ASA presentation

106

Financial Perspective

Graphing and Analyzing Unit Cost

107

Unit Cost

1. Obtain guidance from OBSF on the definition of unit cost measures for your Discrete Services

2. Determine how you can obtain your unit cost data

3. Gather your unit cost data

4. Graph your unit cost data and interpret the findings

• Are your costs going up? Why?

• Are your costs going down? Why?

108

Run ChartUnit Cost (cont.)

Data might be displayed using a run chart (shown) or control chart.

Unit Cost by Quarter

Discrete Service 1: Unit Cost

5

6

7

8

9

10

11

12

13

14

15

Quarter

Do

llar

s

Unit Cost $10.15 $10.52 $10.22 $10.98 $11.25 $11.35 $11.88 $11.13


109

Financial Perspective

Graphing and Analyzing Asset Utilization

110

Asset Utilization

1. Obtain guidance from OBSF on how to define assets for your Discrete Services

2. Determine how you can obtain your planned and actual asset utilization data

3. Gather your data

4. Graph your asset utilization data and interpret the findings

111

Run ChartAsset Utilization

Data might be displayed using a run chart showing % deviation from plan.

Asset Utilization: % Deviation From Plan

-25%

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

Quarter

% D

evia

tio

n b

y Q

uar

ter

(Bar

)

-25%

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

Cu

mu

lati

ve %

Dev

iati

on

(L

ine)

Deviation (Quarter) -13% -3% -6% -5% -8% -4% -2% -3%

Deviation (Cumulative) -13% -8% -7% -7% -7% -6% -6% -5%


112

Run ChartAsset Utilization

1. Calculate the % deviation from plan by quarter

2. Calculate the % deviation from plan cumulatively

Quarterly Quarter 1 Quarter 2Actual 80 87Planned 90 90

% Deviation From Plan (80 - 90)/80 = -13% (87 - 90)/87 = -3%

Cumulatively Quarter 1 Quarter 2Actual 80 (80 + 87) = 167Planned 90 (90 + 90) = 180

% Deviation From Plan (80 - 90)/80 = -13% (167 - 180)/87 = -8%

113

Run ChartAsset Utilization (cont.)


List Data Collection Intervals Horizontally

Label Deviation Measures Enter Deviation Percentages




114



• Choose “Insert”, “Chart”, “Custom Types”, “Line - Column on 2 Axes”


-14%

-12%

-10%

-8%

-6%

-4%

-2%

0%

Q1 FY01Q2 FY01Q3 FY01Q4 FY01 Q1 FY02Q2 FY02Q3 FY02Q4 FY02

-14%

-12%

-10%

-8%

-6%

-4%

-2%

0%

Deviation (Quarter)

Deviation (Cumulative)

115





• Choose “Titles”, “Chart title”: Type in chart title (e.g., Asset Utilization: % Deviation From Plan)

– Choose “Category (X) Axis”: Type in data collection interval (e.g., Quarter)

– Choose “Value (Y) Axis”: Type in title (e.g., % Deviation by Quarter (Bar))

– Choose “Second value (Y) Axis”: Type in title (e.g., Cumulative % Deviation (Line))

• Choose “Legend”, Click “Show legend”

• Choose “Data Table”, Click “Show data table”

• Click “OK”

116



-14%

-12%

-10%

-8%

-6%

-4%

-2%

0%

Quarter

% D

evia

tio

n b

y Q

uar

ter

(Bar

)

-14%

-12%

-10%

-8%

-6%

-4%

-2%

0%

Cu

mu

lati

ve %

Dev

iati

on

(L

ine)



Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

117





• In the “Area” section, click on the white box. Click “OK”

• Double Click left mouse button on left “Y” axis

• Choose “Scale”– In “Minimum” box type -.25– In “Maximum” box type .25– In “Major unit” box type .05– Click “OK”

• Repeat for right “Y” axis

118



-25%-20%

-15%-10%-5%0%

5%10%15%

20%25%

Quarter

% D

evia

tio

n b

y Q

uar

ter

(Bar

)

-25%-20%

-15%-10%-5%0%

5%10%15%

20%25%

Cu

mu

lati

ve %

Dev

iati

on

(L

ine)



Q1 FY01

Q2 FY01

Q3 FY01

Q4 FY01

Q1 FY02

Q2 FY02

Q3 FY02

Q4 FY02

O% = perfect planning (no deviation)

Bars show deviation each quarter

Solid line shows cumulative deviation

119

All Perspectives

Graphing and Analyzing Unique Measures

120

Unique Measures

• Use any of the graph types already discussed as appropriate for unique measures you have added– Refer to slides 47-50 to select the type of graph best suited to the

data

– Try other graph types to see if different displays highlight different results

• Interpret the data• Identify potential actions to address problems areas, issues,

improvement opportunities• Make recommendations for improvement and plan actions

accordingly• Include in Final ASA Presentation

– In main part of presentation if major finding

– In Appendices if back-up material

121

Summary

• Data collection can be time consuming– PLAN before you act– Each ASA Team should complete a Data Collection Plan

• Analyzing data is both a science and an art– Use graphs to summarize data

»Pie, bar, pareto, radar, line, run, scatter»ASA Team review of the graphs is basis of the data analysis

and interpretation• Organize your graphs and conclusions to tell the story of your ASA

– Look for themes and major findings – Identify potential actions to address problems areas, issues,

improvement opportunities– Make recommendations for improvement and plan actions

accordingly

122

Resources

• Brassard, M., & Ritter, D. (1994). The memory jogger. Salem, NH: GOAL/QPC.

• Culbertson, A., Houston, A., Faast, D., White, M., Aguirre, M., & Behr, C. (1997). The Process Improvement Notebook (TQL 97-01). Washington, DC: Department of the Navy. http://www.odam.osd.mil/qmo/pdf/pin.pdf

• Ishikawa, K. 1982. Guide to quality control. Tokyo, Japan: Asian Productivity Organization.

• Kume, H. (1989). Statistical methods for quality improvement. Tokyo, Japan: Association for Overseas Technical Scholarship.

• Navy Total Quality Leadership Office. (1996). Basic Tools for Process Improvement. Washington, DC: Department of the Navy. http://www.odam.osd.mil/qmo/pdf/pareto.pdf

• Wheeler, D. J. (2000). Understanding variation: The key to managing chaos. Knoxville, TN: SPC Press.

• Wheeler, D. J., & Poling, S. R. (1998). Building continual improvement. Knoxville, TN: SPC Press.

Documents

1 Training Objectives Prepare you to plan for data collection and analysis by using the ASA Data Collection Plan Discuss various ways to display and analyze