Ray poynter advances in quant - 2011

A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event

19 September, 2012

Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material

For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org

Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University

Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1

Agree/Disagree scales in online surveys LeK, right, top, and middle thoughts

Ray Poynter Director Vision Cri=cal University

@ Vision Cri=cal


Agree/Disagree Scales Sometimes called Likert Scales

– More strictly Likert Items

Typical format – Agree strongly – Agree – Neither agree nor disagree – Disagree – Disagree strongly

Many variants exist


Issues

Acquiescence bias

Left versus right

Top versus bottom

Mid-points


Acquiescence Bias Saris, Krosnick, and Shaeffer http://goo.gl/LvAwS

Too many people agree!

Traditional solution –  Include each factor several times, with slightly

different wordings – Phrasing half positive, half negative

Not practical in modern market research – Ask positively and only once


Item Order Response Bias Many sources (e.g Krosnick, Chan, Friedman etc)

Items that appear first are more likely to be accepted

A Agree Neither Agree nor Disagree Disagree B Disagree Neither Agree nor Disagree Agree

In A there will be more Agrees In B there will be more Disagrees


Left to Right To what extent do you agree or disagree with the statement Diet Coke, Orange juice, Milk and Red wine are healthy drinks?

A

B

Cells approx. 500, Vision Cri=cal Springboard UK Omni, 2012


Left to Right Diet Coke Healthy

5%

10%

23%

35%

27%

2%

11%

24%

33% 31%

Agree strongly Agree Neither Nor Disagree Disagree Strongly

Diet Coke A Diet Coke B +/-‐ 3% = sig dif


Left Versus Right

% Agree Strongly Agree on Left Agree on Right Diet Coke 5% 2% Orange Juice 21% 16% Milk 38% 30% Red Wine 6% 4%

% Disagree Strongly Disagree on

Right Disagree on

Left Diet Coke 27% 31% Orange Juice 1% 1% Milk 1% 1% Red Wine 5% 9%

People (in this context, in the UK) are more likely to pick the le` side of the scale


Real but not big!

Healthy? %

Agree Mean

(5 points) Milk A 89% 4.2 Milk B 83% 4.1 Orange Juice A 76% 3.9 Orange Juice B 72% 3.8 Red Wine A 36% 3.1 Red Wine B 33% 3.0 Diet Coke A 15% 2.3 Diet Coke B 12% 2.2

Agree = Agree + Strongly Agree


Observations so far 1.  Acquiescence bias leads to higher

agreement

2.  Putting agree on left increase the bias

3.  Putting agree on right reduces the bias

4.  The effect is not very large

5.  So, DON’T MIX AND MATCH


Top to Bottom A B


Top to Bottom Water Healthy

83%

16%

1% 0% 0%

66%

29%

2% 2% 2%

Agree strongly Agree Neither Nor Disagree Disagree Strongly

Water A Water B +/-‐ 3% = sig dif


Top Versus Bottom

% Agree Strongly Agree at Top Agree at Bottom Water 83% 66% White Wine 3% 1% Apple Juice 19% 13% Coke 2% 1%

% Disagree Strongly Disagree at

Bottom Disagree at

Top Water 0% 2% White Wine 5% 11% Apple Juice 0% 2% Coke 39% 46%

People (in this context, in the UK) are more likely to pick the top items


Don’t Mix and Match

Horizontal -‐ standard Ver*cal -‐ visual % Agree + Agree Strongly

Agree Le`

Agree Right

Agree Top

Agree Bofom

Water 95% 94% 98% 95% White Wine 19% 14% 26% 18% Apple Juice 66% 66% 77% 70% Coke 5% 6% 6% 5%


Response Order Thoughts

There are order effects

Putting Agree on the left/top compounds acquiescence

AOTBE put Disagree left/top

Don’t mix and match


Mid-points


Adding a Don’t Know

DKs: Water 0%, Red Wine 2%, Milk 0%, Coffee 0%, Diet Coke 0%


Moving the Neither Nor



0%

10%

20%

30%

40%

50%

60%

70%

80%

Disagree Strongly

Disagree Neither agree nor ..

Agree Agree Strongly

Water -‐ Mid Water -‐ End



0% 5%

10% 15% 20% 25% 30% 35% 40% 45%

Disagree Strongly



Red Wine -‐ Mid Red Wine -‐ End



0%

10%

20%

30%

40%

50%

60%

Disagree Strongly



Milk -‐ Mid Milk -‐ End



0%

10%

20%

30%

40%

50%

60%

Disagree Strongly



Coffee -‐ Mid Coffee -‐ End



0% 5%

10% 15% 20% 25% 30% 35% 40% 45% 50%

Disagree Strongly



Diet Coke -‐ Mid Diet Coke -‐ End


Neither Nor

In this sort of scale – Where the ends are opposites – And where people who do not agree or

disagree are likely to be in the middle – Neither agree nor disagree works well as a

mid-point in a five point scale Even if it does not always make semantic sense!


Concluding Thoughts 1.  The way we ask questions shapes the

answers

2.  We can’t remove bias –  Recognise it –  Hold it constant

3.  Most standard formats work reasonably well

4.  But don’t mix and match!


Thank you Ray Poynter

Director Vision Cri=cal University @ Vision Cri=cal


Q & A

Sue York NewMR

Ray Poynter Vision Critical University


Ray Poynter Vision Cri*cal University @Vision Cri*cal www.visioncri*caluniversity.com ray.poynter@visioncri*cal.com @RayPoynter

www.linkedin.com/in/raypoynter

A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event

19 September, 2012

Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material

For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org

Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University

Marketing

Ray poynter advances in quant - 2011