Upload
leo-cunningham
View
224
Download
4
Tags:
Embed Size (px)
Citation preview
Survey ResearchFrankfort-Nachmias & Nachmias (Chapter 10 – Survey Research, Chapter 11 – Questionnaire Construction)
Applications:James H. Kuklinski; Michael D. Cobb; Martin Gilens, “Racial Attitudes and the "New South." The Journal of Politics Vol. 59, No. 2 (May, 1997), pp. 323-349.
Canache, Damarys, Mondak, Jeffery J., & Seligson, Mitchell A. (2001). “Meaning and measurement in cross-national research on satisfaction with democracy.” Public Opinion Quarterly, 65, 506-528.
Survey Research
Two types of error – Sampling error– Response error
Error in Surveys
Survey experts agree – BY FAR the largest amount of error in survey research comes from response error
Sources of Response Error Survey Administration Question Order Question Type Question Wording
Survey Administration
There are three typical methods used to administer surveys
– Interview– Telephone– Mail
Personal/Telephone Interviewing
Advantages– High response rate– Flexibility/control in questioning– Observation (personal interviewing)– Supervision of interviewers
(telephone)
Personal/Telephone Interviewing
Disadvantages– Expensive– Lack of anonymity – Characteristics of interviewer may
introduce bias
Mail Surveys
Advantages– Cheap!!!– Often avoid bias due to interviewer
characteristics / social desirability– Allows respondent time to answer
Mail Surveys
Disadvantages– Low response rates– Cannot probe– Cannot clarify– Cannot be sure who completes
questionnaire
Dillman’s Total Design Method for Mail Surveys First Mailing
– Typically, when you send a survey out by mail, about 24% of those you sent it to will respond.
Why do we care about the response rate so much???
Dillman Method
After you send the first survey, wait about two weeks and then send a follow-up postcard
The postcard should be sent only to non-respondents to save money
Disadvantage: Loss of anonymity
Dillman Method
The postcard should say something like, “In the past week or so you received a survey from me about XYZ. I know you are a very busy person, but it is important for me to include your opinions in my research. If you still have the survey, could you please complete it and send it to me in the stamped addressed envelope I included? If it got misplaced, I am happy to send you a replacement survey. Please call or email me at ABC and I will send a replacement to you right away. Thank you so much for helping me with this important project. Sincerely, Jane Doe.”
The postcard follow-up typically increases response rate to about 42%.
Dillman Method
About one to two weeks after you send the postcard, send the first replacement questionnaire
Attribute the lack of response to the first survey getting lost in the mail, accidentally misplaced or discarded, or lack of time, etc. Stress the importance of that person’s response to your project.
The first replacement survey will usually get your response rate up to about 59%.
Dillman Method
Send a second replacement survey by certified mail about two weeks after the first replacement survey was sent. This will increase the response rate to about 72%
Dillman Method
Other pieces of advice to maximize response rate– Keep it simple– Make it look like personal mail– Get a sponsor– Offer inducements
The New Frontier in Survey Administration:
The Internet Email surveys Web surveys
Question Order
The order in which you place your questions can cause the error of a response set.
A response set may occur when a series of questions have the same answer choices. This is most likely to happen if questions are on related subjects.
Question OrderExample: Response Set (from the
GSS) “Listed below are various areas of
government spending. Please indicate whether you would like to see more or less government spending in each area. Remember that if you say ‘much more,’ it might require a tax increase to pay for it.
A. The environment. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE
Question OrderExample: Response Set (from the
GSS)
B. Health Care. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE
Question OrderExample: Response Set (from the
GSS)
C. Education. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE
Question OrderExample: Response Set (from the
GSS)
D. Unemployment Benefits. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE
Question Order
Etc. What is the danger in a response
set and what can we do about it???
Acquiescence Response-set A type of response-set pattern in
which the respondent tends to “agree” with a series of statements due to their desire to please the interviewer (or respond with the “right” answer)
Acquiescence Response-set and the California F-scale
Authoritarianism personality – “implicit antidemocratic tendencies and fascist potential”
F-scale: http://www.anesi.com/fscale.htm
THE GENERAL ATTITUDE TOWARDS INSTITUTIONAL AUTHORITY SCALE
For each of the following 16 opinions, please state your personal reaction by crossing (x) the appropriate answer for each question as: "agree strongly", "uncertain", "disagree strongly", "agree but not strongly", "disagree but not strongly"
1. The Police are pretty trustworthy 2. A Person should obey only those laws that seem reasonable 3. The Army develops initiative 4. It is unreasonable to say that as a rule teachers work in the best interests of their students 5. The Police are quite unfair in their treatment of certain groups of society 6. The law is the embodiment of justice and equality 7. I disagree with what the Army stands for 8. The Police have a hard job which they carry out well 9. Laws are so often made for the benefit of small, selfish groups that one cannot respect the law 10. The Army reduces men to robots 11. Teachers do not respect the individual personalities of the students 12. Policemen like to bully people 13. Teachers are usually ready to take seriously whatever students feel earnest about 14. Obedience to the law constitutes a value indicative of the highest citizenship 15. In this day and age students should not be expected to call a teacher "Sir" 16. People should feel proud to serve in the Army
Question Order
The order in which you place your survey questions can cause the error of a saliency effect
Specific mention of an issue in a survey may cause a respondent to associate the issue in connection with a later question.
Fox News and Public Opinion on Health Care Reform
“Based on what you know about the health care reform legislation being considered right now, do you favor or oppose the plan?”
– Fox – 35% support, 49% oppose– Other polls – 43% support, 45%
oppose
– Why the difference?
Fox News and Public Opinion on Health Care Reform
3. Do you think Barack Obama's travel and speaking schedule makes him look more like he is a candidate on the campaign trail or more like he is the president of the United States?
4. Do you think President Obama apologizes too much to the rest of the world for past U.S. policies?
5. Do you think the Obama administration is proposing more government spending than American taxpayers can afford, or not?
6. Do you think the size of the national debt is so large it is hurting the future of the country?
7. Would you rather: [ROTATE OPTIONS 1 and 2]Cut spending now so future generations don't have to payKeep spending at current levels and let future generations pay
20. When Barack Obama was a candidate campaigning for the presidency, he spoke of the urgent need to finish the fight in Afghanistan, which he called the central front on the war on terrorism. Do you think that, as president, Obama is doing what it takes to win in Afghanistan?
Question Order
Consistency is another problem we might encounter due to question order.
Respondents may strive to appear consistent in their answers: an answer to a question may be constrained by an answer given earlier.
Question OrderExample: Consistency
“And what about the federal government, does it have too much power or too little power?”
FAR TOO MUCH TOO MUCH RIGHT AMNT OF POWER TOO LITTLE POWER FAR TOO LITTLE POWER CANT CHOOSE
What if this question preceded the government spending questions?
Question Order
A final concern about question order is fatigue
Respondents may give perfunctory answers to questions late in the survey especially if it is long
In addition, response sets are often a problem in lengthy surveys toward the end
Question Order
Some final thoughts on question order– The first few questions should “break the
ice” – don’t jump in with a big question.– Move from general questions to more
specific questions.– Questions on personal or sensitive topics
(such as income, race, age) should be asked at the end with a big disclaimer about why you are asking those questions
Question Type
Closed-ended questions are ones that have response options provided by the researcher.
What are some of the advantages and disadvantages to closed-ended questions?
Question Type
Advantages of Closed-Ended:
– Easy to answer– Answers are easy to compare– Respondents willing to provide more
personal information– May help clarify the question
Question Type
Disadvantages of Closed-Ended:
– Forces people to accept the categories or puts too many responses into the “other” category.
– Choices listed may communicate the kind of response wanted.
Question Type
Open-ended questions allow the respondent to write in an answer.
What are some of the advantages and disadvantages to open-ended questions?
Question Type
Advantages of Open-ended:
– May better indicate respondents thoughts/beliefs/opinions.
– Better when the list of responses might be excessive.
– Might get unanticipated answers.
Question Type
Disadvantages of Open-ended:
– Respondent may say too much or too little.
– Difficult to code all the various answers and analyze them.
– Can be expensive and time-consuming.
Question Type
Single-sided vs. Two-sided questions.
Single-sided: a question that asks respondents to agree or disagree with a single substantive statement.
Two-sided: a question that offers the respondent two substantive choices.
Question Type
Single-sided example:
Do you agree or disagree with the idea that the government should see to it that every person has a job with a good standard of living?
Question Type
Two-sided example:
Do you think that the government should see to it that every person has a job with a good standard of living, or should it let each person get ahead on his or her own?
Question Type
With a single-sided question a larger percentage of respondents tend to agree with the statement given.
Question Type
44% of respondents agreed with the single-sided statement.
Only 30.3% of respondents agreed government should guarantee employment and standard of living in the two-sided question.
Two-sided questions reduce the proportion of respondents who give no opinion.
Question Wording
Double-Barreled Questions are two questions in one. Example:
Do you agree with the statement that the situation in Iraq is deteriorating and that the United States should increase the number of troops in Iraq?
What is wrong with how this question is worded?
Question Wording
Ambiguous Question: one that contains a concept that is not clearly defined. Example:
– What is your income?
What is the problem with this survey question???
Question Wording
The choices you provide on a frequency scale can influence how respondents answer. Consider this question:
How much tv do you watch per day?
Question Wording
When the response options were:A) Up to .5 hoursB) .5 to 1 hourC) 1-2 hoursD) 2-2.5 hoursE) 2.5+ hours
16% responded they watched 2.5+ hours.
Question Wording
When the response options were:A) Up to 2.5 hoursB) 2.5 to 3 hoursC) 3-4 hoursD) 4-4.5 hoursE) 4.5+ hours
38% responded they watched 2.5+ hours.
(From Schwarz et al., 1985)
Question Wording
Leading Questions: Encourages respondents to choose a particular response because the question indicates the researcher expects it.
“Don’t you think that global warming is a serious environmental problem?”
Question Wording
Polls conducted by political organizations and politicians often include leading questions.
1980 poll for the Republican National Committee read as follows on the next slide (a dated, but classic example that is widely used).
Question Wording
“Recently the Soviet armed forces openly invaded the independent country of Afghanistan. Do you think the U.S. should supply military equipment to the rebel freedom fighters?”
Question Wording
Word Choice may lead respondents to an answer as well. You have to really think about what words you use to describe people, things, and events.
Question Wording
Helping “the needy” vs. those “on welfare”.
“Socialized Medicine” vs. “National Health Insurance”
“Fight Against Terrorism” vs. “Going to War”
Question Wording
Linking personalities or institutions to issues can also affect responses.
“Would you say that Governor Beshear’s program for promoting economic development has been very effective, fairly effective . . .”
A respondent’s feelings about Governor Beshear might affect his or her response to the question about economic development.
Question Wording
Push Polls: These are relatively new and conducted by campaigns, parties, and political organizations.
Interviewers feed respondents false and damaging information about a candidate or cause under the guise of asking a question.
Question Wording
“Do you agree or disagree with Candidate X’s willingness to tolerate terrorism in our country?”
The goal is NOT to conduct research but to use innuendo to spread rumors and lies.
This is a politically risky strategy and if it’s too blatant, it can back-fire.
Question Wording
Confusing Questions should be avoided. Yes, this seems so obvious, but even the experts make mistakes sometimes. Double-negatives are especially confusing.
The following question was asked of a national random sample by the Roper Center, a respected polling organization.
Question Wording
Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?
A) Impossible it never happenedB) Don't knowC) Possible it never happened
Question Wording
The responses were as follows:
Impossible it never happened: 65%Don't know: 12%Possible it never happened: 22%
Question Wording
Does it seem possible to you that the Nazi extermination of the Jews never happened or do you feel certain that it happened?
A) Certain it happenedB) Don't knowC) Possible it never happened
Question Wording
The responses were as follows:
Certain it happened: 91%Don't know: 8%Possible it never happened: 1%
Question Type
Social Desirability – a form of response error in which respondents are not truthful due to the fact that they perceive their answer to be out of step with the mainstream of society
Race Relations and the “New South”
Race Relations and the “New South”
Experimental Design– Random assignment of individuals
into two groups (national random sample of white Americans)
Race Relations and the “New South”
Race Relations and the “New South”
Race Relations and the “New South”
Result– South – 42% angry– Non-South – 11% angry
“Equlivalence” in Cross-National Research What is equivalence?
“Satisfaction with Democracy” Canache, Mondak and Seligson “On the whole, are you very
satisfied, fairly satisfied, not very satisfied, or not at all satisfied with the way democracy works in [country]?”
“Satisfaction with Democracy”Canache, Mondak and Seligson
“Satisfaction with Democracy”Canache, Mondak and Seligson