Embed Size (px)
DESCRIPTION
Reducing Bias in Public Opinion Polls - By Autumn CarterPresentation Delivered at Stanford University, Summer Research College, August 2009Presentation of some of my results from my research. Over the course of the project, I analyzed 7 survey experiments across 9 unique vendors, who conducted the survey via phone or the internet and using either a probability sample or a non-probability sample. In all, about 10000 unique individuals were surveyed. This presentation highlights the question bias of Agree/Disagree type questions and the response bias associated with the Acquiescence Effect.I conducted my research through Stanford\'s Political Psychology Research Group. pprg.stanford.edu
Citation preview
Reducing Bias in Public Opinion Polls
Autumn CarterProfessor Jon Krosnick
Summer Research College 2009
Satisfy Your Constituents...
Or Suffer the Consequences.
So, how do we effectively determine what Americans
are feeling?
Public Opinion Polling
But Are the Polls Accurate?
?=
Critics Don't Think So...
“The dirty little secret of the polling industry is that, all too often, its findings are based on flawed methodology and dubious assumptions.”
-Arianna Huffington, Huffington Post Founder
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree
Disagree
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree
Disagree
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree: 56.8%
Disagree: 43.2%Acquiescence Effect: 16.4%
What is Acquiescence?
“...a presumed tendency for respondents to agree with attitude statements presented to them.”
-Schuman and Presser (1981)
Why do Respondents Acquiesce?
Norms of Conduct: Be polite and agreeable
Status Differential: Defer to higher status
Satisficing: Choosing the easiest response because it requires less thinking
Can we reduce acquiescence by
administering the survey via the Internet instead of
by telephone?
Would Internet Administration Reduce Acquiescence?
Norms of Conduct: Be polite and agreeable Do you have to be polite to a computer?
Status Differential: Defer to higher statusIs there a status difference without a human interviewer?
Satisficing: Choosing the easiest response because it requires less thinking
Are people less likely to satisfice on Internet surveys? (Chang & Krosnick, in press)
The Present Research
Replicate Schuman & Presser (1981)
Mode effects: Telephone vs. the Internet
Sample effects: Representative vs. Non-Representative
How Do You Survey a Representative Sample Via the Internet?
Knowledge Networks randomly dialed phone numbers and invited people to join a panel (of about 50,000 adults).
Provided Internet access to homes without it.
Periodically emails about 1,000 panelists to complete surveys and receive incentives.
A Classic Acquiescence Experiment
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree
Disagree
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree
Disagree
Replication in a National Survey
Telephone survey of a representative sample:
Internet survey of a representative sample:
* p < 0.05+ p <0.10
11.5%*
7.1%+
TelephoneXInternet Interaction: n.s.
What about Internet surveys of people who are not representative of the
national population?
Non-Probability Sample Internet Surveys
Volunteers click on banner ads and “opt-in” into the panel.
You've all seen these...
Non-Probability Sample Internet Surveys
Volunteers click on banner ads and “opt-in” to the panel.
Participants receive money or prize.
Perhaps they acquiescence less because they like taking surveys and have more practice.
In fact, previous research has found less acquiescence vs. a telephone survey (Chang and Krosnick, in
press)
Acquiescence BiasT
elep
hone
Pro
babi
lity
Web
Non
-Pro
b W
eb 1
Non
-Pro
b W
eb 2
Non
-Pro
b W
eb 3
Non
-Pro
b W
eb 4
Non
-Pro
b W
eb 5
Non
-Pro
b W
eb 6
Non
-Pro
b W
eb 7
0.0
5.0
10.0
15.0
20.0
25.0
Acquiescence Bias
Survey
Acq
uies
cenc
e B
ias
(in p
erce
nt)
*
*
**
*
†*
† †
*
If anything, non-probability samples increase acquiescence.
So we should be cautious when interpreting these
types of surveys.
What if we just didn't ask questions that ask
respondents to agree or disagree with a
statement?
Balanced Forced-Choice Questions
Form 1
Which do you think is more responsible for crime and lawlessness in this country: individuals or social conditions?
Individuals
Social Conditions
Form 2
Which do you think is more responsible for crime and lawlessness in this country: social conditions or individuals?
Social Conditions
Individuals
Response Order EffectsT
elep
hone
Pro
babi
lity
Web
Non
-Pro
b W
eb 1
Non
-Pro
b W
eb 2
Non
-Pro
b W
eb 3
Non
-Pro
b W
eb 4
Non
-Pro
b W
eb 5
Non
-Pro
b W
eb 6
Non
-Pro
b W
eb 7
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
8.0
Survey
Diff
eren
ce b
etw
een
For
ms
(in p
erce
nt)
ns nsnsnsnsnsnsnsns
Conclusions
Acquiescence bias is real and can undermine public opinion polls and possibly the democratic process.
Internet administration did not significantly reduce acquiescence.
Non-probability sample surveys showed, if anything, more acquiescence.
BUT WAIT! You can simply ask the questions using the balanced forced choice format, eliminating order effects.
