Survey Research and Methodology
UNL-GALLUP Research Center
Culturally-Related Response Styles
for Attitude Questions:
A Comparative Analysis of Chinese and
American Respondents
1
Presenter: Mia (Mengyang) Wang
Mia (Mengyang) Wang
Allan L. McCutcheon
GALLUP Research Center
Survey Research and Methodology Program
University of Nebraska-Lincoln
Survey Research and Methodology
UNL-GALLUP Research Center
OBJECTIVE
• Under the Cross-National Context:
– To detect the biasing effect of stylistic responding
– To examine whether variable response styles for attitude
questions can be explained by the presence of an
underlying construct related to cultural difference
– Specifically, to study how differences between American
and Chinese cultures influence stylistic responding
2
Survey Research and Methodology
UNL-GALLUP Research Center
CULTURALLY-RELATED RESPONSE STYLES
• Cross-cultural researchers often rely on surveys for collecting
data because of ease, convenience, and cost considerations.
• Unfortunately, survey responses can be influenced by individual
response styles (T Hamamura et al., 2008).
• Research also indicates that particular response styles may differ
across cultural groups (Chen et al., 1995).
• Researchers conducting cross-cultural surveys are particularly
challenged in drawing valid conclusions from survey data when
respondents demonstrate culturally-based response patterns that
are systematically different, though unrelated to content
(Dolnicar and Bettina, 2007).
3
Survey Research and Methodology
UNL-GALLUP Research Center
CULTRUAL EFFECTS
• Chinese Culture vs. American Culture
In China, the ‘Doctrine of the Mean’ is a central aspect
of Chinese culture. Unlike American culture, Chinese
principles encourage people to express neutral, instead
of polarized, attitudes.
4
Survey Research and Methodology
UNL-GALLUP Research Center
STYLISTIC RESPONSES
5
STYLISTIC RESPONSE THEORETICAL DEFINITION
Extreme Response StyleThe tendency to report either strongly disagree or strongly agree to
scalar questions.
Midpoint Response StyleThe tendency to use the neutral middle category of a scale regardless of
content.
Mild Response StyleThe tendency to moderately agree or moderately disagree.
No Opinion Response StyleThe tendency to endorse the “don’t know” or “don’t apply” categories.
Acquiescence Response StyleThe tendency to report agreement with items regardless of content.
Disacquiesence Response StyleThe tendency to report disagreement with items regardless of content.
*Reference: Hans Baumgartner and Jan-benedict E.M. Steenkamp 2001
Survey Research and Methodology
UNL-GALLUP Research Center
THE BIG IDEA
• Cultural differences related to response style have
the potential to impact implicit survey response
patterns, which may manifest itself in questionnaire
reporting behaviors.
• This study compares Chinese and American survey
responses on Likert scales in order to investigate
cultural effects in underlying response patterns.
6
Survey Research and Methodology
UNL-GALLUP Research Center
RESEARCH QUESTIONS
• Do American respondents demonstrate significantly different
response patterns compared to Chinese respondents?
– Is the occurrence of extreme response styles lower among Chinese
respondents?
– Are Chinese respondents more likely to endorse the midpoints of
scales?
– Are Chinese respondents more likely to report moderate agreement or
moderate disagreement?
7
Survey Research and Methodology
UNL-GALLUP Research Center
METHOD
• To examine whether variable response styles for attitude
questions can be explained by the presence of an
underlying construct related to cultural difference,
Latent Class Analyses are employed in this research.
• Specifically, exploratory latent class models are used:
1. To investigate the existence of the underlying stylistic responses
for each country
2. To examine the frequency of different types of respondents in
terms of their underlying stylistic responses within each country.
3. To detect the differences in response patterns of Chinese and
American respondents.
8
Survey Research and Methodology
UNL-GALLUP Research Center
DATA
• The Gallup World Poll
Probability-based
Multinational
• Data Collected (2013)
CHINA – Sample Size = 4,244
USA – Sample Size = 1,005
9
Survey Research and Methodology
UNL-GALLUP Research Center
CONCEPTUAL ILLUSTRATION
Cultural
Difference
Health
Life
Emotion
Energy
Stylistic Responses on
Attitude Questions
10
Survey Research and Methodology
UNL-GALLUP Research Center
CROSS-CULTURAL SURVEY QUESTIONS
11
EXTREME
MILD
NEUTRAL
1 - Strongly
disagree
5 - Strongly
agree
2 - Disagree
4 - Agree
3 - Midpoint
Survey Research and Methodology
UNL-GALLUP Research Center12
COMPARATIVE TENDENCY OF STYLISTIC RESPONSES
38%
55%
51%
44%
21%
33%29%
18%
Life Health Energy Emotion
EXTREME RESPONSES
USA CHINA
Survey Research and Methodology
UNL-GALLUP Research Center13
COMPARATIVE TENDENCY OF STYLISTIC RESPONSES
40%
31% 31%
36%
46% 45%43%
44%
Life Health Energy Emotion
MILD RESPONSES
USA CHINA
Survey Research and Methodology
UNL-GALLUP Research Center14
COMPARATIVE TENDENCY OF STYLISTIC RESPONSES
22%
14%
17%
20%
32%
22%
26%
37%
Life Health Energy Emotion
MIDPOINT RESPONSES
USA CHINA
Survey Research and Methodology
UNL-GALLUP Research Center15
COMPARATIVE TENDENCY OF STYLISTIC RESPONSES
0.0%
0.3%0.4%
0.3%
0.9%
0.5%
2.0%
0.9%
Life Health Energy Emotion
“DON’T KNOW” RESPONSES
USA CHINA
Survey Research and Methodology
UNL-GALLUP Research Center
EXPLORATORY LATENT CLASS MODEL – USA
Exploratory latent class analyses are used to investigate the
existence and the frequency of different types of respondents in
terms of their underlying stylistic responses for American
respondents.
Model X2 P Value AIC BIC DF
One-Class 1062.2625 0.000 587.3202 234.3951 72
Two-Class 176.1059 0.000 23.8267 -284.9827 63
Three-Class 112.2962 0.000 -5.5802 -270.2740 54
Four-Class 69.5729 0.1067 -25.9386 -241.6980 45
Five-Class 46.8363 0.0108 -21.0745 -202.4374 36
Six-Class 39.5284 0.0567 -11.6503 -143.9972 27
16
Survey Research and Methodology
UNL-GALLUP Research Center
MODEL SELECTION – USA
• The response styles of American respondents for scalar
attitude questions can be classified by four classes.
• Each respondent from the entire USA sample was assigned to
one of four latent classes:
1. Extreme Responder
2. Mixed-Style Responder
3. Midpoint Adverse Responder
4. Mild Responder
17
Survey Research and Methodology
UNL-GALLUP Research Center
CONDITIONAL PROBABILITY AND RELATIVE FREQUENCY– USA
RESPONDENT TYPE – USA
Manifest Questions Response TypeMixed-Style
Responder
Extreme
Responder
Midpoint
Adverse
Responder
Mild
Responder
Learn or Do
Something
Interesting
EXTREME RESPONSE 0.245 0.807 0.283 0.007
MILD RESPONSE 0.437 0.159 0.453 0.682
MIDPOINT RESPONSE 0.318 0.035 0.263 0.310
Encourages to be
Healthy
EXTREME RESPONSE 0.293 0.980 0.715 0.055
MILD RESPONSE 0.354 0.000 0.255 0.843
MIDPOINT RESPONSE 0.353 0.020 0.030 0.102
Positive Energy
EXTREME RESPONSE 0.133 0.892 0.785 0.000
MILD RESPONSE 0.443 0.081 0.215 0.802
MIDPOINT RESPONSE 0.424 0.027 0.000 0.198
Felt Active and
Productive
EXTREME RESPONSE 0.340 0.877 0.323 0.106
MILD RESPONSE 0.332 0.124 0.421 0.734
MIDPOINT RESPONSE 0.328 0.000 0.256 0.160
Relative Class
Frequency 30% 29% 27% 14%
18
Survey Research and Methodology
UNL-GALLUP Research Center
RELATIVE DISTRIBUTION – USA
Extreme
Responder
29%
Mixed-Style
Responder
30%
Midpoint Adverse
Responder
27%
Mild Responder
14%
Relative Distribution of American Respondents
Exhibiting Each Response Style
19
Survey Research and Methodology
UNL-GALLUP Research Center
MODEL RESULTS – USA
• Mixed-Style Responder30% of respondents were in the largest class, which consistently
showed a tendency not to exhibit a particular response style.
• Extreme Responder29% of respondents tended to select the most extreme response
options.
• Midpoint Adverse Responder27% of respondents avoided endorsing the midpoint, or neutral,
response option.
• Mild Responder14% of respondents exhibited a tendency to select the mild
response options.
20
Survey Research and Methodology
UNL-GALLUP Research Center
EXPLORATORY LATENT CLASS MODEL – CHINA
Exploratory latent class analyses are used to investigate the
existence and the frequency of different types of respondents in
terms of their underlying stylistic responses for Chinese
respondents.
Model X2 P Value AIC BIC DF
One-Class 17587.156 0.000 5210.238 4757.102 72
Two-Class 1657.722 0.000 1144.741 30067.991 63
Three-Class 462.558 0.000 344.234 4.382 54
Four-Class 228.514 0.000 130.513 -152.697 45
Five-Class 93.426 0.006 22.786 -203.782 36
Six-Class 59.333 0.004 3.537 -166.389 27
21
Survey Research and Methodology
UNL-GALLUP Research Center
MODEL SELECTION - CHINA
• The response styles of Chinese respondents for scalar attitude
questions can be classified by five classes.
• Each respondent from the entire Chinese sample was assigned
to one of five latent classes:
1. Extreme Adverse Responder
2. Mixed-Style Responder
3. Neutral Responder
4. Mild Responder
5. Extreme Responder
22
Survey Research and Methodology
UNL-GALLUP Research Center
CONDITIONAL PROBABILITY AND RELATIVE FREQUENCY - CHINA
RESPONDENT TYPE – CHINA
Manifest
Questions
Extreme Adverse
Responder
Mixed-Style
Responder
Mild
Responder
Extreme
Responder
Neutral
Responder
Learn or Do
Something
Interesting
EXTREME RESPONSE 0.124 0.368 0.005 0.946 0.000
MILD RESPONSE 0.587 0.362 0.765 0.045 0.125
MIDPOINT RESPONSE 0.289 0.270 0.230 0.009 0.875
Encourages to
be Healthy
EXTREME RESPONSE 0.000 0.614 0.000 0.973 0.022
MILD RESPONSE 0.486 0.304 0.987 0.019 0.265
MIDPOINT RESPONSE 0.514 0.082 0.013 0.008 0.713
Positive Energy
EXTREME RESPONSE 0.027 0.507 0.042 0.935 0.000
MILD RESPONSE 0.465 0.322 0.958 0.024 0.146
MIDPOINT RESPONSE 0.508 0.171 0.000 0.042 0.854
Felt Active and
Productive
EXTREME RESPONSE 0.070 0.245 0.000 0.896 0.000
MILD RESPONSE 0.524 0.414 0.827 0.104 0.000
MIDPOINT RESPONSE 0.406 0.341 0.173 0.000 1.000
Relative Class
Frequency33% 22% 20% 13% 12%
23
Survey Research and Methodology
UNL-GALLUP Research Center
RELATIVE DISTRIBUTION- CHINA
Extreme Adverse
Responder
33%
Mixed-Style
Responder
22%
Mild Responder
20%
Extreme
Responder 13%
Neutral
Responder
12%
Relative Distribution of Chinese Respondents
Exhibiting Each Response Style
24
Survey Research and Methodology
UNL-GALLUP Research Center
• Extreme Adverse Responder33% of respondents avoided endorsing the extreme response options, instead
tending to endorse the mild or midpoint options.
• Mixed-Style Responder22% of respondents tended not to exhibit any particular response style.
• Mild Responder
20% of respondents exhibited a tendency to select mild response options.
• Extreme Responder13% of respondents consistently showed a tendency to select the most extreme
response options.
• Neutral Responder12% of respondents exhibited a tendency to select the midpoint, or neutral,
response option.
MODEL RESULTS – CHINA
25
Survey Research and Methodology
UNL-GALLUP Research Center
RESULTS
• There were four different types of respondents identified for
Americans and five different types of respondents identified for
Chinese.
• American and Chinese respondents exhibited some overlap in
response styles.
• However, these styles were exhibited in different frequencies in the
US as compared to China:30% 29%
14%
22%
13%
20%
Mixed-Style Responder Extreme Responder Mild Responder
USA CHINA
26
Survey Research and Methodology
UNL-GALLUP Research Center
RESULTS
• In addition, American and Chinese respondents
exhibited some unique response styles
• In the US: Midpoint Adverse Responder (27%)
• In China: Extreme Adverse Responder (33%) and Neutral
Responder (12%)
27
Survey Research and Methodology
UNL-GALLUP Research Center
CONCLUSIONS
• These results indicate that there are cultural differences in
the response styles of American respondents as compared to
Chinese respondents. The findings of this study suggest that
Chinese respondents do tend to exhibit less extreme
response styles than Americans, which has the potential to
impact survey findings.
• The existence of culturally-related response patterns
represents a serious threat to the correct conclusions and
interpretations drawn from research comparing the United
States and China.
28
Survey Research and Methodology
UNL-GALLUP Research Center
FUTURE RESEARCH
• Compare frequencies of acquiescence response style
and disacquiesence response style across cultures.
• Explore methods to control for cultural effects
between Chinese and American respondents when
performing or analyzing cross-national studies.
29
Thank You!Contact Author: [email protected]
Survey Research and Methodology
UNL-GALLUP Research Center
CROSS-CULTURAL SURVEY QUESTIONS
• Question Stem:
“Thinking about your life in general, please rate your level of agreement with each of the
following using a five-point scale, where 5 means you STRONGLY AGREE and 1 means you
STRONGLY DISAGREE. You may choose any of the numbers 1, 2, 3, 4, or 5.” (Gallup World Poll,
2013)
31
Topic Questions
Learn or Do Something Interesting You learn or do something interesting every
day
Encouragement to be HealthySomeone in your life always encourages you to
be healthy.
Positive Energy Your friends and family give you positive energy
every day.
Felt Active and ProductiveIn the last seven days, you have felt active and
productive every day.