Designing a Survey Instrument

Developing A Survey Instrument -‐ An Overview

Sam Klaidman Principal Adviser

Middlesex Consul?ng Sam@Middlesexconsul?ng.com

@Sklaidman 508.877.1924

© 2009 Middlesex Consulting

. All Rights Reserved

© 2009 Middlesex Consulting . All Rights Reserved

How Much is Enough? Responses Needed for 95% Confidence with ±5% Margin of Error

First Steps

•  Iden?fy the execu?ve sponsor –  Plan her involvement

•  Determine objec?ve of the survey –  Who is the audience?

•  Iden?fy the intended survey community –  Size –  Roles –  Level

•  Think about how the results will be used –  Ac?on –  Informa?on –  Baseline


First Steps (con?nued)

•  Get input from your customers –  What’s on their mind? –  How much ?me can they spend on survey –  Preferred media

•  Plan the mix of qualita?ve vs. quan?ta?ve ques?ons •  One ?me or ongoing survey? •  Medium -‐ web, phone, mail •  Incen?ves? •  Process


Now we can start planning the survey instrument

Survey Types -‐ Transac?on (event driven) •  A_er installa?on •  A_er a call to Tech Support, Inside Sales, AR, Service •  A_er an interview (HR process evalua?on) •  A_er customer uses your product or service

–  Intruder detected and police no?fied –  Accident claim submiaed –  First delivery received with no other company interac?on (ship to end

user)


• Typically web

• Typically 2 to 5 ques?ons

• 1 to 3 days a_er the event

Survey Types -‐ Rela?onship (mid-‐level)

•  Used to monitor how customers feel about your business •  Very useful to determine:

–  Sta?s?cal impact of each touch point on Sa?sfac?on and Loyalty –  Gap between Importance and Sa?sfac?on

•  Broad focus as compared to the narrower Transac?on survey •  Used to plan changes and monitor their impact on a long term basis


• Typically 15 to 30 ques?ons

• Scheduled (monthly, quarterly, etc.)

Survey Types -‐ Key Accounts •  Like the rela?onship survey but for decision makers/recommenders/end-‐

users •  Find strengths and weaknesses •  Use with similar employee surveys -‐ ensures alignment •  Must be face-‐to-‐face with telephone as fallback •  These are special people and deserve special treatment


• No more than 35 to 40 ques?ons • Should take <30 minutes to complete • As o_en as possible without annoying par?cipants

Survey Types -‐ Special Purpose Surveys

•  Market research •  Lost business •  Employees •  Suppliers •  Partners


Typical Areas to Rate

•  Reliability -‐ Deliver on your promise? •  Responsiveness -‐ Helpful? •  Assurance -‐ Trust and confidence •  Empathy -‐ Treat customers as individuals •  Tangibles -‐ Making the intangible “real”



Qualita've/Quan'ta've Research

• Random samples • Closed ques?ons • Limited response op?ons • Can’t get the story behind the story

• Provides hard data that can be extrapolated to a larger audience

•  Convenience samples •  Polls with comments •  Online Communi?es •  Focus Groups •  Provide the Story behind the story

•  Can’t extrapolate to a larger audience

Quan?ta?ve Research

Qualita?ve Research

Quan?ta?ve Ques?ons •  Should have granularity

–  Yes/no is vague –  1-‐5, 0-‐10 are typical and more specific

•  Should have “balanced” anchors –  Extremely sa?sfied / extremely dissa?sfied –  High value / low value

•  Must be self explanatory and unambiguous •  Must rate only 1 item per ques?on

–  Professionalism and courtesy are 2 items usually lumped together •  Should be impersonal

–  Rate processes and policies not individuals •  Should have an escape selec?on

–  No opinion –  No comment


Qualita?ve Ques?ons

•  Must not suggest answers •  Must be easy to understand •  Must provide enough room to answer completely •  Ideal for text mining

–  Depending on number of surveys could have different people with individual biases, summarizing results

–  Lose emo?ons –  Very ?me consuming

•  Help explain quan?ta?ve answers •  Add credibility to survey results


Choosing Media •  Web survey

–  Lowest comple?on rate but easiest to automate –  Least intrusive to customer –  Reports exactly what the customer wants to say

•  Telephone survey –  Higher comple?on rate –  Can be pre-‐scheduled –  Poten?al for “edi?ng” –  High tech/high touch

•  Mail/Fax –  So 5 minutes ago but may be necessary where English is not the primary

language.


Some Examples From a Market Research Telephone Survey


Example #1


Please rate the following service delivery attributes in terms of their importance to you, using a scale from 10 to 1, where 10 is the most important and 1 is least important : Question 03 Please rate how important it is to easily get your inspection and/or service visits scheduled. 01) 10 04 02) 9 03) 8 04) 7 05) 6 06) 5 07) 4 08) 3 09) 2 10) 1 11) No Opinion

This is part of a group of ques?ons asking customers to rate the importance of various service aaributes.

Example #2


Question 13 What features of your current service agreement(s) are of most value to you?

01) Comments / What? 14 GL3 02) No Comment

Question 14 What features that you don’t currently have in your service agreement(s) would you like to have?

01) Comment / What? 15 GL3 02) None 03) No Comment Notice the “escape” selections

and the invitation to comment

Example #3


Question 17 Assuming you were allowed, how likely would you be to recommend COMPANY to colleagues or others? Would you be…

01) Extremely Likely 1 8 02) L ikely 03) Neutral / Why? G L 3 04) Un likely / Why? G L 3 05) Extremely Unlikely / Why? DK) No Comment

No?ce the symmetry of the choices

Any Ques?ons?


Thank You

Business

Designing a Survey Instrument