Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Survey and List Experiments
Thomas Pepinsky Department of Government
Cornell University [email protected]!
Prepared for the Workshop on Cu#ng Edge Mixed Methods Research Techniques for Analy;cal Work on Local Governance and Service Delivery
Jakarta 24-‐26 June 2014
Overview
• Typology • MoJvaJons • Examples (many) • PracJcal and Conceptual Concerns • New DirecJons
6/18/14 Survey and List Experiments 2
Typology 1. Survey VigneSes
Short descrip;ons of social situa;ons highligh;ng relevant decision-‐ or judgment-‐making processes
2. Survey Experiments a) “Mini MulJple Survey” Experiments
Randomize parts of ques;ons, compare across answers b) Priming Experiments
… or “cueing,” “informaJon,” “vigneSe” experiments Randomize prompts, compare across answers
3. List Experiments Randomize number of possible answers, compare number of answers chosen
6/18/14 Survey and List Experiments 3
MoJvaJons
1. Biased self-‐reporJng (social desirability, etc.) – How racist are you? – Did you sell your vote?
2. Difficult quesJons – Do you prefer that Pheu Thai/Democrats endorse/
oppose/nothing the Red Shirts/Yellow Shirts/King/some of them/all of them?
3. Confounding – Ethnicity, income, status -‐> preferences/behaviors
6/18/14 Survey and List Experiments 4
VigneSes
Goal: “standardize the social sJmulus across respondents and at the same Jme makes the decision-‐making situaJon more real” (Alexander and Becker 1978) Here, non-‐experimental
6/18/14 Survey and List Experiments 5
VigneSes
Arsovska (2012): What do people think about revenge killing in Albania?
6/18/14 Survey and List Experiments 6
Mini-‐MulJple Survey Experiment
Pepinsky, Liddle, and Mujani (2012): does economic policy explain support for Islamic par;es in Indonesia?
6/18/14 Survey and List Experiments 7
Mini-‐MulJple Survey Experiment
Pepinsky, Liddle, and Mujani (2012): does economic policy explain support for Islamic par;es in Indonesia?
6/18/14 Survey and List Experiments 8
Priming Experiment (1)
Adida, Lo, and Verink (2013): Does coethnicity affect support for President Boni Yayi in Parakou, Benin?
6/18/14 Survey and List Experiments 9
Priming Experiment (2)
Malhotra and Kuo (2008): do par;san cues lead people to accord blame for Hurricane Katrina differently?
6/18/14 Survey and List Experiments 10
Priming Experiment (3)
Benjamin, Choi, and Strickland (2010): Does ethnicity affect economic norms and preferences?
6/18/14 Survey and List Experiments 11
Priming Experiment (3)
Benjamin, Choi, and Strickland (2010): Does ethnicity affect economic norms and preferences?
6/18/14 Survey and List Experiments 12
List Experiment (1)
Kuklinski, Cobb, and Gilens (1997) Are people upset by black families moving in to their neighborhoods?
6/18/14 Survey and List Experiments 13
List Experiment (1)
Kuklinski, Cobb, and Gilens (1997) Are people upset by black families moving in to their neighborhoods?
6/18/14 Survey and List Experiments 14
List Experiment (2)
Ahlquist, Mayer, and Jackman (2014) How frequent is voter impersona;on in U.S. elec;ons?
6/18/14 Survey and List Experiments 15
List Experiment (2)
Ahlquist, Mayer, and Jackman (2014) How frequent is voter impersona;on in U.S. elec;ons?
6/18/14 Survey and List Experiments 16
List Experiment (2)
Ahlquist, Mayer, and Jackman (2014) How frequent is voter impersona;on in U.S. elec;ons?
6/18/14 Survey and List Experiments 17
List Experiment (3)
Imai, Park, and Greene (2014) How prevalent is vote-‐selling in Mexico?
6/18/14 Survey and List Experiments 18
List Experiment (3)
Imai, Park, and Greene (2014) How prevalent is vote-‐selling in Mexico?
6/18/14 Survey and List Experiments 19
PracJcal and Conceptual Concerns
1. How do respondents interpret quesJons? 2. RepresentaJveness of sample populaJon 3. Small effect sizes 4. Floor/ceiling effects 5. InterpreJng “primes” 6. Bundled treatments 7. External (or “ecological”) validity
6/18/14 Survey and List Experiments 20
New DirecJons
1. More efficient analysis of list experiments (Blair and Imai 2011)
2. Heterogeneous treatment effects (Green and Kern 2012)
3. Endorsement experiments (Bullock, Imai, and Shapiro 2011
4. Lab-‐in-‐the-‐field experiments
6/18/14 Survey and List Experiments 21
Final Thoughts
Main strengths for policy work – Empirical evidence on hard-‐to-‐measure phenomena
– Precise control over counterfactuals – Credible esJmates of treatment effects
Main limitaJon for policy work – Mapping esJmable treatment effects to acJonable policy (c.f. field/natural experiments)
6/18/14 Survey and List Experiments 22
Further Reading METHODS – Alexander & Becker (1978) “The Use of VigneSes in Survey Research” Pol Opinion Quarterly – Arsovska (2012) “Researching Difficult PopulaJons” Handbook of Survey Methodology for the Soc Sci – Blair & Imai (2012) “StaJsJcal Analysis of List Experiments” Pol Analysis – Bullock, Imai, Shapiro (2011) “StaJsJcal Analysis of Endorsement Experiments” Pol Analysis – Corstange (2009) “SensiJve QuesJons, Truthful Answers?” Pol Analysis – Druckman, Green, Kuklinski, Lupia (2011) Handbook of Experimental Poli;cal Science – Gaines, Kuklinski, Quirk (2007) “Logic of the Survey Experiment Reconsidered” Pol Analysis – Green & Kern (2012) Modeling Heterogeneous Treatment Effects Pub Opinion Quartery – Imai, Park, Greene (2014) “Using the Predicted Responses from List Experiments” W.P., Princeton
APPLICATIONS – Adida, Lo, Verink (2013) “Measuring the Effect of Coethnicity” EGAP Pre-‐analysis Plan – Ahlquist, Mayer, Jackman (2014) “Alien AbducJon and Voter ImpersonaJon” W.P., Wisconsin – Benjamin, Choi, Strickland (2010) “Social IdenJty and Preferences” AER – Kuklinski, Cobb, Gilens (1997) “Racial antudes and the ‘New South’” JOP – Malhotra & Kuo (2008) “ASribuJng Blame” JOP – Pepinsky, Liddle, Mujani (2012) “TesJng Islam’s PoliJcal Advantage” AJPS
6/18/14 Survey and List Experiments 23