ABCs of Satisfaction & Impact Surveys

ABCs of Satisfaction & ABCs of Satisfaction & Impact Surveys Impact Surveys

Elizabeth Harris, Ph.D.Elizabeth Harris, Ph.D.Evaluation, Management & Training Evaluation, Management & Training

Associates, Inc.Associates, Inc.

May 21May 21stst , 2008 , 2008

ABCs of Satisfaction & Impact ABCs of Satisfaction & Impact SurveysSurveys

ObjectivesObjectives– Understand the importance and relevance of science-Understand the importance and relevance of science-

based surveyingbased surveying– Understand method of developing telephone and Understand method of developing telephone and

Internet surveys to measure inquirer satisfaction and Internet surveys to measure inquirer satisfaction and key outcomes important to contact centerskey outcomes important to contact centers

– Leave the workshop with the tools necessary to Leave the workshop with the tools necessary to develop surveys tailored to the needs of each contact develop surveys tailored to the needs of each contact center center

Workshop exerciseWorkshop exercise

““Lessons LearnedLessons Learned” ” from surveying for: from surveying for:

Centers for Disease Control & PreventionCenters for Disease Control & Prevention– Consumer Response Services Center (CDC-INFO)Consumer Response Services Center (CDC-INFO)– Monitor calls & e-mail responses Monitor calls & e-mail responses

First 5 LAFirst 5 LA– Countywide Parent Helpline Countywide Parent Helpline

We will discuss:We will discuss:

Survey: Survey: – DesignDesign

To Do’sTo Do’s

Common PitfallsCommon Pitfalls

– Administration Administration OptionsOptions

– Data UseData Use

Satisfaction & Impact Satisfaction & Impact

Satisfaction:Satisfaction:– ImmediateImmediate– Caller’s opinion about Caller’s opinion about

Overall ServiceOverall Service

Various Aspects of the Service (Drill Down)Various Aspects of the Service (Drill Down)

Impact:Impact:– What happened as a result of the call?What happened as a result of the call?

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 1Step 1

What is important to us when we consider What is important to us when we consider the caller’s perspective and experience? the caller’s perspective and experience?

What questions are our key stakeholders What questions are our key stakeholders asking that we need to answer? asking that we need to answer?

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 2Step 2

Determine method of administration Determine method of administration – Interactive Voice ResponseInteractive Voice Response– Live SurveyorLive Surveyor– Web-Based Web-Based

FeasibilityFeasibility

Balance between cost & knowledge gainBalance between cost & knowledge gain

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 3 Step 3

Develop questions Develop questions Response options Response options – ClosedClosed

Yes/NoYes/NoLikert ScaleLikert Scale

– e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very SatisfiedSatisfied

– Allow a middle option? Neither Satisfied or Dissatisfied Allow a middle option? Neither Satisfied or Dissatisfied

– Open Open ProPro

– Information Information

ConCon– Labor considerationLabor consideration

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 3Step 3

Develop questions Develop questions Handout example of closed and open-Handout example of closed and open-ended questions ended questions

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 4Step 4

Design Survey Design Survey – Question wordingQuestion wording

NeutralNeutral

– Response order effect Response order effect Carefully consider which question is first and which Carefully consider which question is first and which is last is last

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 4Step 4

Design SurveyDesign Survey– IVR RulesIVR Rules

Brief and to the pointBrief and to the point

No more than six response optionsNo more than six response options

– Consider lengthConsider lengthLonger = more respondents drop off Longer = more respondents drop off

– Language options Language options

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 5Step 5

Determine Survey ProtocolDetermine Survey Protocol– Who is surveyed-Immediate SatisfactionWho is surveyed-Immediate Satisfaction

Every callerEvery caller– Works well if no automated transfer optionWorks well if no automated transfer option

TrainingTraining

BiasBias

Random selectionRandom selection– Works well with a robust systemWorks well with a robust system

program algorithm that automatically chooses program algorithm that automatically chooses

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 5Step 5

Who is surveyed—Impact (Call Back) Who is surveyed—Impact (Call Back) – Again consider:Again consider:

Purpose of the surveyPurpose of the survey– Caller preferenceCaller preference– Part of service follow up effortPart of service follow up effort

How the information will be used How the information will be used

– Other considerations:Other considerations:Labor allocation/resourcesLabor allocation/resources

– To surveyTo survey– To interpret/make use of information To interpret/make use of information

Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 6Step 6

Pilot Testing Pilot Testing – Works out the bugsWorks out the bugs– Improves introductory instructionsImproves introductory instructions– May revise questions and responses based May revise questions and responses based

on pilot test results on pilot test results

What to Expect: Response Rate What to Expect: Response Rate

Immediate Satisfaction SurveyImmediate Satisfaction Survey– 22% to 33%22% to 33%

Follow Up SurveyFollow Up Survey– Will reach half of those you try to callWill reach half of those you try to call

If you try more than onceIf you try more than once

Vary time/day called Vary time/day called

Results: Results:

Over 90% give the First 5 LA Parent Helpline the Over 90% give the First 5 LA Parent Helpline the highest satisfaction rating for overall service!highest satisfaction rating for overall service!

Over half who received a referral successfully Over half who received a referral successfully accessed servicesaccessed services

Results: Results:

Over 75% give CDC-INFO the highest satisfaction rating for Over 75% give CDC-INFO the highest satisfaction rating for overall serviceoverall service

Over 75% report learning something new from their callOver 75% report learning something new from their call

Over half report that what they learned makes them want to Over half report that what they learned makes them want to change a behavior change a behavior

Questions? Questions?

Exercise: Exercise:

See handoutsSee handouts

Select one question you would like to answer Select one question you would like to answer with a satisfaction survey with a satisfaction survey

Exercise—Break Out: Exercise—Break Out:

Develop a survey question and responses:Develop a survey question and responses:– Doable/feasible to implement in your setting Doable/feasible to implement in your setting

Exercise—Debriefing: Exercise—Debriefing:

Share examples from each group Share examples from each group

Questions?Questions?

Contact information:Contact information:

Elizabeth Harris, Ph.D.Elizabeth Harris, Ph.D.

818-990-8301818-990-8301

eharris@emt.orgeharris@emt.org