Embed Size (px)
Citation preview
Question Order Effects
in Long Question Lists
May 19, 2012
67th Annual Meeting of the
American Association for Public Opinion Research
Jamie Marincic Martha Kovac Hong Zhang
Attitude formation
Context effects
Purpose of current study
Data and methods
Findings
Outline
2
Attitudes are context-dependent evaluative
judgments
Relationship between target and standard (Schuman and Presser 1981)
Attitude Formation
3
STANDARD
TARGET
STANDARD
TARGET
PART-PART PART-WHOLE
Inclusion/Exclusion model (Schwarz and Bless 1992)
Saliency
– Influence of entire context preceding a question
Fatigue
– Questions at end of long list or long questionnaire
answered less thoughtfully than those at beginning
Context Effects
4
STANDARD
TARGET
STANDARD
TARGET
CONSISTENCY CONTRAST
1. To what extent are part-part consistency
effects moderated by increasing context?
2. To what extent does the entire context
preceding a question affect response
patterns?
3. To what extent do fatigue effects occur in long
question lists?
Purpose of Current Study
5
Computer-assisted telephone interviewing
(CATI) survey of national stratified sample of
health care professionals
Data and Methods (1)
6
Date
Number of Items
in Perceptual
List AAPOR RR 1
October 2010 –
January 2011 9
788 of 1,023
77.0%
Data and Methods (2)
I’m going to read a list of barriers that facilities may face in improving
their performance. As I read each one, please tell me whether each is
currently a major barrier, a minor barrier, or not a barrier for this facility.
The facility lacks enough staff trained in quality improvement
Resource constraints, other than staffing, limit improvement strategies
Lack of staff interest or involvement
Documentation of the care that is actually given is a major problem
Physicians at the hospital disagree with selection of the measure or its
definition
The facility has other higher priorities
The facility is unsure of how to improve performance
The facility has no incentive to improve
Insufficient senior management leadership and support
7
Data and Methods (3)
8
● ● ●
Question Order Group
a
b
c
d
e
f
g
h
i
i
a
b
c
d
e
f
g
h
h
i
a
b
c
d
e
f
g
g
h
i
a
b
c
d
e
f
f
g
h
i
a
b
c
d
e
b
c
d
e
f
g
h
i
a
1st
2nd
3rd
4th
5th
6th
7th
8th
9th
1 2 3 4 5 9
Qu
es
tio
n O
rde
r
Influence of Question Order (1)
9
84
51
82
49
0
20
40
60
80
100
1st 2nd 3rd 4th 5th 6th 7th 8th 9th Marginal(M)
Perc
en
tag
e
Position of Question in List
(a) Target
No Yes
Influence of Question Order (2)
10
80
51
81
49
0
20
40
60
80
100
1st 2nd 3rd 4th 5th 6th 7th 8th 9th Marginal(M)
Perc
en
tag
e
Position of Question in List
(i) Target
No Yes
Question order matters.
Conclusion
11
Part-Part Consistency Effects
12
● ● ●
Question Order Group
a
b
c
d
e
f
g
h
i
i
a
b
c
d
e
f
g
h
h
i
a
b
c
d
e
f
g
g
h
i
a
b
c
d
e
f
f
g
h
i
a
b
c
d
e
b
c
d
e
f
g
h
i
a
1st
2nd
3rd
4th
5th
6th
7th
8th
9th
1 2 3 4 5 9
Qu
es
tio
n O
rde
r
Part-Part Consistency Effects: Findings (1)
13
57
11 16
81
44
78 82
0
20
40
60
80
100
i a i a i a i a i a i a i a i a i a i a
—|1 1|2 2|3 3|4 4|5 5|6 6|7 7|8 8|9 M
Perc
en
tag
e
Position of Question Pair in List
(i) Standard / (a) Target
No Yes
Part-Part Consistency Effects: Findings (2)
14
79
57 64
16 24
78 81
0
20
40
60
80
100
a b a b a b a b a b a b a b a b a b a b
—,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 M
Perc
en
tag
e
Position of Question Pair in List
(a) Standard / (b) Target
No Yes
The influence of an
immediately preceding
question does not diminish as the question
pair is preceded by
increasing context.
Part-Part Consistency Effects: Conclusion
15
Saliency Effects
16
● ● ●
Question Order Group
a
b
c
d
e
f
g
h
i
i
a
b
c
d
e
f
g
h
h
i
a
b
c
d
e
f
g
g
h
i
a
b
c
d
e
f
f
g
h
i
a
b
c
d
e
b
c
d
e
f
g
h
i
a
1st
2nd
3rd
4th
5th
6th
7th
8th
9th
1 2 3 4 5 9
Qu
es
tio
n O
rde
r
Saliency Effects: Findings (1)
17
57 49
0
20
40
60
80
100
a i a h a g a f a e a d a c a b a a
1st 2nd 3rd 4th 5th 6th 7th 8th 9th M
Pe
rce
nta
ge
Position of Question in List
(First Item) Standard / (a) Target
No Yes
Saliency Effects: Findings (2)
18
79 64
9
49
0
20
40
60
80
100
b a b i b h b g b f b e b d b c b b
1st 2nd 3rd 4th 5th 6th 7th 8th 9th M
Perc
en
tag
e
Position of Question in List
(First Item) Standard / (b) Target
No Yes
The influence of the first
question in a list is short-
lived.
Saliency Effects: Conclusion
19
Exceptionally high item response rates
At most one “don’t know” response or refusal
per question order group
Fatigue Effects
20
Question order matters.
The influence of an immediately preceding
question does not diminish as the question
pair is preceded by increasing context.
The influence of the first question in a list is
short-lived.
Fully randomize long lists of
attitudes or perceptions.
Conclusions
21
Mathematica® is a registered trademark of Mathematica Policy Research.
Please contact:
– Jamie Marincic
For More Information
22
