How to measure what we'd rather not say: List experiments and anti-immigrant sentiment Mathew J Creighton UMass, Boston Mathew J. Creighton - U54 SSMC

1

How to measure what we'd rather not say:

List experiments and anti-immigrant sentiment

Mathew J CreightonUMass, Boston

Mathew J. Creighton - U54 SSMC - 2015

2

Motivation

In surveys, as in life, there are things that you would prefer keep to yourself.



3

Objectives

What is the list experiment?

How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.

Throughout, we will explore some (many) drawbacks to the list experiment.

4

Examples

Four examples of applications of the list experiment will be explored

Example 1: Does religion shape our attitudes toward immigrants in the US?

Example 2: Has the US become more hostile toward immigration after the economic crisis?

Example 3: Is opposition to Muslim immigrants masked in the Netherlands?

Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?


5

Examples: The 1st and 2nd are a reasonable introduction to the technique

Four examples of applications of the list experiment to measure (in)tolerance







6

Objectives





7

Example 1: Social Identity Theory and Citizenship for Muslim Immigrants in the US

Social Identity Theory (SIT)*: Individuals express greater affinity for members of their own “group”.

Ingroup bias (Tajfel 1970, Tajfel et al. 1971; Tajfel and Turner 1986)

Outgroup hostility

*For thorough review of the literature see Brown (2000)


8

Social identity theory (SIT) applied to immigrant reception and religion

Religion is a group identity Co-religious immigrants benefit from ingroup

bias Immigrants from other religions do not benefit

from ingroup bias

Example 1: Social Identity Theory, religion and immigration


9

Example 1: Hypotheses

H1: Explicit opposition to citizenship for legal Muslim immigrants is greater than explicit opposition to citizenship for legal Christian immigrants.

H2: Implicit opposition to citizenship for legal Muslim immigrants is no greater than implicit opposition to citizenship for legal Christian immigrants.


10

Social desirability bias

Affirmative action (Kuklinski et al. 1997b)

Same-sex marriage (Janus 2011)

Closing the US border (Janus 2011)

Immigration policy preferences (Knoll 2013)

Nativism (Knoll 2013)

Religious service attendance (Brenner 2011, 2012a, 2012b)

Race attitudes (Kuklinsky et al. 1997)


11

Time-Sharing Experiments in Social Science (TESS) and Knowledge Networks (KN)

Random-digit dialing (RDD) and Address-based sampling (ABS)

Provision of free computers or WebTV’s

Sample of adults 18+

Representative of 97% of U.S. households

Response rate of 66.6%


12

Control:Direct

QuestionsN = 788

Treatment 1:Opposition to Citizenship for Legal Muslim Immigrants

n = 804

Treatment 2:Opposition to Citizenship for Legal Christian

Immigrantsn = 774

Treatment 3:Support for

Closed Bordersn = 836


13

Example 1: The control (explicit opposition)

Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.

1. the federal government increasing assistance to the poor

2. professional athletes making millions of dollars per year

3. large corporations polluting the environment

Do you support or oppose granting citizenship to a legal immigrant who is Muslim?

Do you support or oppose granting citizenship to a legal immigrant who is Christian?


14

Example 1: The treatment (implicit opposition)

Below you will read four things that sometimes people oppose or are against. After you read all four, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.




4.(Treatment 1)

granting citizenship to a legal immigrant who is Muslim

4.(Treatment 2)

granting citizenship to a legal immigrant who is Christian


15

Standard Estimation Strategy: Implicit Opposition

Difference in Means:

Recent developments: Maximum likelihood (Imai, 2011; Graeme and Imai, 2011)

R Package list (Graeme and Imai, 2011)


16

Measuring social desirability bias

Proportion opposed to when asked directly

(i.e., explicit opposition)

Proportion opposed when asked indirectly

(i.e., implicit opposition)


17

Christian immigrants: Implicit vs. explicit

0.28 [±0.07]

0.17 [±0.07]

0.11 [±0.02]

Source: TESS/Knowledge Networks® 2010


18

Muslim immigrants: Implicit vs. explicit

0.33 [±0.06] 0.03 [±0.08]0.30 [±0.03]

Source: TESS/Knowledge Networks® 2010


19

Example 1: Findings

Opposition to citizenship for legal Muslim immigrants is not greater, only more openly expressed.

Religious ingroup membership for Christian immigrants does not always lend itself to ingroup protections. What it does offer is protection from anti-immigrant sentiment’s explicit expression.

20

Examples








21

Objectives





22

Example 2: Labor Market Competition Theory, the economic Crisis and immigration

Labor Market Competition Theory (LMCT)*

“As rational actors, natives pursuing their own wellbeing develop unfavorable attitudes in order to

legitimate their social positions when competing with foreigners over jobs…, especially during times of

economic recession” (Ceobanu et al. 2010: 317)

*For thorough review of the literature see Ceobanu et al. (2010).


23

Example 2: Evidence is mixed

Of 21 peer-reviewed studies over 15 years*

4 are strongly supportive 8 are weakly supportive 9 find no support whatsoever

Main evidence is derived from measures of education or unemployment

*For thorough review of recent findings see Malhorta et al. (2010).


24


25

Control:Direct

QuestionsN = 788

Treatment 1:Opposition to Citizenship for Legal Muslim Immigrants

n = 804

Treatment 2:Opposition to Citizenship for Legal Christian

Immigrantsn = 774

Treatment 3:Support for

Closed Bordersn = 836


26

Example 2: The control (explicit opposition)

Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.




Do you support or oppose cutting off all immigration to the United States?


27

Example 2: The treatment (implicit opposition)

Below you will read four things that sometimes people oppose or are against. After you read all four, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.




4.(Treatment 3)

Cutting off all immigration to the United States


28

Example 2: Hypotheses

H1: After the financial crisis, opposition to immigration increased. H2: After the financial crisis, opposition to immigration increased more among the less educated than the more educated. H3: After the financial crisis, opposition to immigration increased more among the unemployed than the employed. H4: After the financial crisis, opposition to immigration increased more among those with a lower household income than those with a higher household income.


29

Example 2: Hypothesis 1

H1: After the financial crisis, opposition to immigration increased. H2: After the financial crisis, opposition to immigration increased more among the less educated than the more educated. H3: After the financial crisis, opposition to immigration increased more among the unemployed than the employed. H4: After the financial crisis, opposition to immigration increased more among those with a lower household income than those with a higher household income.


30

Example 2: Overall pre-crisis to post-crisis trend

SocialDesirabilityBias

SocialDesirabilityBias

60%


31

Example 2: Pre-crisis to post-crisis trend

Explicit opposition Implicit opposition

32

Examples








33

Objectives





34

Example 3: Converting direct questions from the ESS to a list experiment in the Netherlands

One of the primary drawbacks of a list experiment is that the response must be a binary.

Typically, direct questions offer more options and/or a neutral category

This example is an effort to convert a scaled response (i.e., some, few, many) to a binary response (i.e., none, some).


35

Example 3: These are the original ESS questions (first asked in 2003)


36

Example 3: The binary conversion


37

Example 3: The binary conversion


38

Example 3: The list experiment


39

Example 3: The Data


40

Control:Direct

QuestionsAnd 3-

question list

Treatment 1:ESS questions

Treatment 2:List experiment

(poor immigrants)


(same race)Treatment 4:

List experiment (different race)


(Muslim immigrants)


41

Example 3: Preliminary results

*Binary conversion analysis is pending. Check in after ESRA Conference

42

Examples: The 1st and 2nd are a reasonable introduction to the technique

Four examples of applications of the list experiment to measure (in)tolerance







43

Objectives





44

Example 4: Longitudinal approach

List experiments are almost always cross-sectional

The stability of social desirability bias over time is hard to determine.

If, for some questions, the bias is stable, corrections are easier to make.


45

Example 4: Stability over time

Wave 1

A control and three treatments are asked about certain immigrant groups in the UK

Wave 2

The treatment and controls are randomized for a subset in the second wave.

T1 T2


46

Example 4: The baseline list


47

Example 4: The Innovation Panel

Special module of the UK Household Longitudinal Study

Intention is to test innovative experimental designs at the population level that focus on substantive questions

Our experiment is in beta testing and should enter the field soon.


48

Control:Direct

Questions and control

listW1C1

Treatment 1:Caribbean immigrants

W1T1

Treatment 2:Eastern

European immigrants

W1T2

Treatment 3:Muslim

immigrantsW1T3

Wave 1:


49

Control:Direct

Questions and control

listW2C1

(W1C1 + Random sub-set)

Treatment 1:Caribbean immigrants

W2T1(W1T1 + Random sub-

set)

Treatment 2:Eastern

European immigrants

W2T2(W1T2 + Random sub-

set)

Treatment 3:Muslim

immigrantsW2T3

(W1T3 + Random sub-set)

Wave 2:


50

Example 4: Preliminary results


51

Recap

List experiment allows total and permanent anonymity.

When compared to a direct question, social desirability bias can be directly assessed and it’s magnitude quantified.

Dichotomous response patterns are not your only option, but converting from a more complicated direct question should, at the very least, involve a conversion process in the experiment itself.

Longitudinal measures of social desirability bias can be done with repeated cross-sections or with a true longitudinal set-up. The advantages of the latter remain to be seen.


52

Acknowledgements

Time-Sharing Experiments in Social Science (TESS) Jeremy Freese (Northwestern) Penny Visser (University of Chicago)

Social, Behavioral and Economic Sciences Directorate of the National Science Foundation


53

Acknowledgements

LISS Panel (The Netherlands)

Innovation Panel of the UK Household Longitudinal Study


54

Thank you

<there are bonus slides after this to fill time if nobody asks questions>


55

Step 2: Pre-crisis to post-crisis trend: high school

SocialDesirabilityBias Social

DesirabilityBias


56

Step 2: Pre-crisis to post-crisis trend: some college


DesirabilityBias


57

Step 2: Pre-crisis to post-crisis trend: college or more


DesirabilityBias


58

The Maximum Likelihood Estimator (Imai 2011)

Documents

How to measure what we'd rather not say: List experiments and anti-immigrant sentiment Mathew J Creighton UMass, Boston Mathew J. Creighton - U54 SSMC