What your customers really think about you (parts 1 & 2)

#RelateLive

What Your Customers Really Think About You

Part 1: Do’s and Don'ts of Survey Design

Lori Gauthier, Ph.D.ZendeskDirector of Marketing Research

@datadocgauthier

Know What You Need from Your Data

DestinationInformation Construct Question

What Are You Measuring?Are You Sure?

“I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”

- Unknown

Define What You Need to Measure

Words Mean ThingsSearch definitions, synonyms, antonyms.

Source: snappywords.com

Define What You Need to Measure

Words Mean ThingsSearch definitions, synonyms, antonyms.

Use the language and tone appropriate for your population.

Result: Respondents answer the question you think you’re asking.

What Questions Should You Ask? What Response Options Should You

Provide?Understanding Construct Polarity and Scale Sensitivity

Which Way Do We Go?Construct polarity

Unipolar Construct Bipolar ConstructVery common; typically specific; often descriptive Very rare; typically global; occasionally comparative

Measures absence to maximum: not at all likely to extremely likely

Measures maximum negative to maximum positive:disapprove a great deal to approve a great deal

Midpoint represents half of construct Midpoint represents ambiguity or neutrality

5-point scale is ideal 7- or 9-point scale is ideal

How likely are you to vote in a primary this year? Do you approve or disapprove of negative campaigning?

Examples: likelihood, frequency, duration, intensity Examples: bad/good, dis/satisfied, dis/like, worse/better

common labels: not at all, slightly, moderately, very, extremely

none, a little, a moderate amount, a lot, a great deal

common labels (mirrored sides):extremely, very, moderately, slightly, neither/nor …a great deal, a lot, a moderate amount, a little, neither/nor …

zero????

Ideal scale sensitivity (example 1)How Many Scale Points Should You Use?

unipolar

not a

t all

extre

mely

moder

ately

sligh

tly very

1000 5025 75

bipolar

neith

er/no

r

extre

mely

moder

ately

sligh

tly very

1000 5025 75

sligh

tlyvery

extre

mely

moder

ately

-25-75-100 -50

Ideal scale sensitivity (example 2)How Many Scale Points Should You Use?

unipolar

not a

t all

a gre

at de

al

a mod

erate

amou

nt

a litt

lea l

ot1000 5025 75

bipolar

neith

er/no

r

a gre

at de

al

a mod

erate

amou

nt

a litt

lea l

ot1000 5025 75

a litt

lea l

ot

a gre

at de

al

a mod

erate

amou

nt-25-75-100 -50

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

unipolar

not at alllikely

extremelylikely

moderatelylikely

slightlylikely

verylikel

y

1000 5025 75

????not likely likely

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

dislikea little

dislikea great

deal

dislike a moderate

amount

1000 33 67-33-67-100

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

1000 50-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

dislikea great

deal

dislike a moderate

amount

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

1000-100

neitherlike nordislike

like a great

deal

dislikea great

deal

A step-by-step approach to designing sound surveysWhat Have We Learned So Far?

start at your destination

define your construct

scale your construct

draft your question

Is Measurement Error Destroying Your Data?

Done with the Do’s. Let’s get to the Don’ts.

Stewie DataLook at him go!

Stewie DataLook at him go!

Random ErrorBad survey design can introduce data-destroying random error, making your data — and decisions — bounce all over the place.

Rooting Out Random ErrorSo long, Stewie!

no!nooo!

noo!

double barreled questionunexpected scale direction

insensitive scaleoverly sensitive scale

scale without midpointscale without verbal labels

overlapping scale labelsnon construct-specific scale

confusing question or scale

true|false, yes|no, agree|disagree scale

Tower of Pisa DataOne way or another, it’s gonna getcha!

Systematic ErrorBad survey design can introduce data-destroying systematic error, leading you to make biased decisions.

Banishing BiasArrivederci, Pisa!

worstever!!

!thingunbalanced scale

leading question

true|false, yes|no, agree|disagree scale

missing extreme endpoints bipolar scale without midpoint

order effectscontext effects

unbalanced question

question formatted as statement

That’s A Lot of Stuff to Remember. Let’s Recap.Phew!

A step-by-step approach to designing sound surveysWhat Have We Learned So Far?

start at your destination

define your construct

scale your construct

check for random error

check for systematic error

collect good data

bing!bing

!bing

!draft your question

Q&A

Let Me Know What YOU Think!

Share your thoughts about Part 1 of today’s workshop.

Two minutes, a few taps in your Relate Live app, and I’ll know what you think.

Thank you!

Your finger here!

#RelateLive

#RelateLive

What Your Customers Really Think About You

Part 2: Critique and Create Survey Questions

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

Response Effect

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

RE

Response Effect

semantic confusion ups volatility

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Problem RE/SE

Response Effect

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Problem

statement as question

RE/SE

SE

Response Effect

acquiescence bias inflates ratings

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Critique Two Questions in EIGHT MinutesGroup Work

Review Question CritiquesGroup Work

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

SE

SE

RE

SE

RE

Response Effect

STM bias inflates ratings

DS/NN Rs pick 1, inflating ratings

semantic confusion ups volatility

zero sat Rs pick 1, inflating ratings

midpoint Rs pick?, upping volatility

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Problem

incorrectly defined construct

leading/unbalanced question

confusing scale

scale missing N/N midpoint

missing scale extremes

RE/SE

—

SE

RE

RE

SE

Response Effect

won’t measure CSAT

STM bias inflates ratings

misinterpretations up volatility

ambig Rs pick?, upping volatility

“all the time” Rs pushed inward

What’s wrong with this question?Measuring Customer Satisfaction

What do you think about Acme’s customer support? Are you happy with it?

no, most of the time

no, some of the time

yes, some of the time

yes, most of the time

no yes

Problem

incorrectly defined construct

awkward question

confusing scale

missing low-end scale point

scale missing actual midpoint

RE/SE

—

RE

RE

SE

RE

Response Effect

won’t measure org-created effort

misinterpretations up volatility

“neutral” misinterps up volatility

zero Rs pick low, inflating ratings

mod Rs pick?, upping volatility

What’s wrong with this question?Measuring Customer Effort

How much effort did you personally have to put forth to get your issue resolved?

Very low effort Very high effortNeutral High effortLow effort

Question source: The Effortless Experience

Problem

statement as question

A/DA scale

non construct-specific scale

A/DA scale

confusing scale

RE/SE

SE

SE

RE

RE

RE

Response Effect

acquiescence bias inflates ratings

acquiescence bias inflates ratings

mismapping ups volatility

misinterpretations up volatility

moderately A/DA Rs pick?

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Create One New Question in FOUR MinutesGroup Work

Review New QuestionsGroup Work

A methodologically sound questionMeasuring Customer Satisfaction

Overall, how satisfied or dissatisfied are you with Acme’s customer support?

moderatelydissatisfied

slightlydissatisfied

neithersatisfied nor dissatisfied

slightlysatisfied

moderatelysatisfied

extremelydissatisfied

extremelysatisfied

7-point, fully labeled, construct-specific,

bipolar scale

measures what we want to measure: satisfaction with customer support

“overall” appropriate for global-level measure

balanced question

ambivalent midpoint

Measuring Customer EffortA methodologically sound question

How easy was it to get the help you needed from us today?

not at alleasy

extremely easy

moderatelyeasy

veryeasy

slightly easy

measures what we want to measure: effort needed to get company’s help “today” appropriate for

transaction-level measure

5-point, fully labeled, construct-specific,

unipolar scale

Measuring Customer EffortWhat is driving customer effort?

Content source for drivers of effort: The Effortless Experience

How did we make it difficult? (Check all that apply)

You didn’t solve the problem I had to contact the company multiple timesI felt like I was talking to a robotI had to repeat myselfI had to use a channel I don’t like (phone, web form, chat, email, FAQ)I was transferred from person to personSome other reason (Please specify)

don’t assume resolution

pick list Q measures freq of known responses

open-ended option capturesunknown responses

limit list to 7-9 options

random rotate pick list

Workshop Recap

What Your Customers Really Think About You

start at your destination

define your construct

scale your construct

check for random error

check for systematic error

collect good data

bing!bing

!bing

!draft your question

Remember!Use this step-by-step approach for designing sound surveys

Thank You!Questions? Contact me at lgauthier@zendesk.com or

@datadocgauthier.

Let Me Know What YOU Think!

Your finger here!

Share your thoughts about Parts 1 + 2 of today’s

workshop.

Two minutes, a few taps in your Relate Live app, and I’ll

know what you think.

Thank you!

#RelateLive