Designing, implementing, and analyzing Leverage Saliency Theory … · about [INSERT “NOT TOO IMPORTANT” OR “NOT IMPORTANT” TOPIC FROM Q39] that was conducted face to face

icf.com

Designing, implementing, and analyzing Leverage Saliency Theory experiments

Wed, April 17, 2018

FedCASIC 2019 | Washington, DC

Matt Jans, Julia Sumner, Rachael Forando, John

Boyle, James Dayton, Lew Berman

ICF

ICF proprietary and confidential. Do not copy, distribute, or disclose.

Overlapping Research Steps

TheoryResearch Questions

Protocol Statistics Findings

24/17/19 - FedCASIC 2019 - BLS

Designing Implementing Analyzing


What is Leverage Salience(y) Theory?

34/17/19 - FedCASIC 2019 - BLS


Leverage Saliency Theory (LST) Principals (Groves, Singer, & Corning, 2000)

1. Salience: Whether a design feature is noticed and how important it is

2. Leverage: The amount of force the feature has

3. Valence: Does the feature nudge toward or away from participation?

44/17/19 - FedCASIC 2019 - BLS


LST Example

People interested in birding will be more likely to participate in a “Survey

of Birding” than people who are not

Stated survey topic (a design feature) influences participation and thus, birding activities (and related

characteristics) are over-represented

Incentives will have less effect on people with high topic interest

Thus incentives help minimize nonresponse bias by increasing response from those with low topic interest

Interest in a topic leads people to think it’s important that they respond

54/17/19 - FedCASIC 2019 - BLS


Our Research Questions

64/17/19 - FedCASIC 2019 - BLS


LST Research Questions in the Dimensions of Survey Participation Study

RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an

increase in stated willingness to participate?

74/17/19 - FedCASIC 2019 - BLS





RQ #2: Duration: Increasing duration form 1 hour to 2 hours will reduce stated

willingness to participate?

84/17/19 - FedCASIC 2019 - BLS







RQ #3: Interactions (moderation): Do incentive and duration interact to predict

participation willingness?

94/17/19 - FedCASIC 2019 - BLS







RQ #3: Interactions (moderation): Do incentive and duration interact to predict

participation willingness?

RQ #4: Generalizability of effects (LST): Do these main effects vary across …

4a: Survey topics and topic importance

4b: Types of respondents

- Demographics

- Survey experience

- Civic involvement and general interests

104/17/19 - FedCASIC 2019 - BLS


Implementing our Experiment

114/17/19 - FedCASIC 2019 - BLS


Experimental Factors and Measures

All respondents asked about… Importance/Unimportance of several topics (RQ #4b)

Respondents randomized factorially to… Incentive level $25 v. $50 (RQ #1)

Interview duration 1 hr. v. 2 hrs (RQ #2)

124/17/19 - FedCASIC 2019 - BLS

Important Topics

1 hour

$25

1 hour

$50

2 hours

$25

2 hours

$25

Unimportant

Topics

Questionnaire Flow


Topic Importance

Q39 “If you were asked to participate in government surveys on the

following topics, how important do you think it would be for you to

participate?” Very important

Somewhat important

Not too important

Not important at all

Topics

Education and schools

Child care and problems of parents

Medicare and aging

Voting and elections

Health and disease prevalence

Nutrition and physical activity

Issues facing the nation

134/17/19 - FedCASIC 2019 - BLS


Participation Likelihood/willingness

“Imagine you were contacted this week to participate in a government survey

about [INSERT “VERY IMPORTANT” OR “SOMEWHAT IMPORTANT” TOPIC

FROM Q39] that was conducted face to face in your home at a time of your

choice, took one hour [OR two hours], and you received $25 [OR $50],

How likely would you be to participate in a survey like this?”

Definitely would

Probably would

Might or might not

Probably would not

Definitely would not

144/17/19 - FedCASIC 2019 - BLS


Participation Likelihood/willingness

“Imagine you were contacted this week to participate in a government survey

about [INSERT “NOT TOO IMPORTANT” OR “NOT IMPORTANT” TOPIC

FROM Q39] that was conducted face to face in your home at a time of your

choice, took one hour [OR two hours], and you received $25 [OR $50],

How likely would you be to participate in a survey like this?”

Definitely would

Probably would

Might or might not

Probably would not

Definitely would not

154/17/19 - FedCASIC 2019 - BLS

Previous question with unimportant

topics filled.


Analyzing Results

164/17/19 - FedCASIC 2019 - BLS


Principal #3: Consider (i.e., plan) the analysis approach before collecting data

Easy for experimental studies

174/17/19 - FedCASIC 2019 - BLS

Incentive Amount

Inte

rvie

w

Du

rati

on

$25 $50

1 hour % willing to participate % willing to participate

2 hours % willing to participate % willing to participate


Analysis Methods for Our LST Experiment – Main Effects and Interaction

svy:logistic Q40_important_will_bin ///

int_dur_bin##inc_amt_bin

contrast int_dur_bin##inc_amt_bin

margins int_dur_bin#inc_amt_bin

marginsplot…

184/17/19 - FedCASIC 2019 - BLS

Incentive main effect (F = 11.35, p = .0008)

Duration main effect (F = 3.68, p = .0551)

Interaction btwn

Incentive and Duration(F = 5.86, p = .0155)


Exploring the Effect of R characteristics –Residence location

svy:logistic Q40_important_will_bin ///

int_dur_bin##inc_amt_bin##town_large_city_bin

contrast int_dur_bin##inc_amt_bin@town_large_city_bin

margins int_dur_bin#inc_amt_bin#town_large_city_bin

marginsplot…

194/17/19 - FedCASIC 2019 - BLS


Duration main effect

(F = 3.54, p = .0602


Interaction btwn

incentive and duration(F = 6.38, p = .0116)


Exploring the effect of R characteristics - Age

204/17/19 - FedCASIC 2019 - BLS


LST Design and Analysis Checklist

214/17/19 - FedCASIC 2019 - BLS


Checklists Flow from the Framework

Principal #1: Start with problem/question/theory-based research

questions Define the problem statement or hypothesis you want to test

Which design features do you want to manipulate? How many levels (and what values) will they have?

Will you use a “captive audience“ and hypothetical response, or “field survey”

What respondent (or sample member) sub-groups do you want to compare? What indicators of topic interest

do you have?

Write specific research questions or hypotheses, including the direction of expected effect

224/17/19 - FedCASIC 2019 - BLS



Principal #2: Experimental factors flow from research questions Define the factors you want to study (i.e., comparisons you want to make)

Consider counter-factuals (i.e., comparison and control groups)

If you want to make causal inferences, make them experimental manipulations via random assignment

Confirm that the experimental factors will answer your research questions (or adjust research questions)

234/17/19 - FedCASIC 2019 - BLS



Principal #3: Consider (i.e., plan) the analysis approach before collecting

data Choose modeling method (e.g., ANOVA, linear regression, logistic regression)

Define how effects and interactions will be tested and reported (e.g., odds ratios v. coefficients)

Determine visualizations that you will create before or during modeling and testing

244/17/19 - FedCASIC 2019 - BLS

ICF proprietary and confidential. Do not copy, distribute, or disclose. 25

Matt Jans: [email protected]

4/17/19 - FedCASIC 2019 - BLS

Thank You!

Documents

Designing, implementing, and analyzing Leverage Saliency Theory … · about [INSERT “NOT TOO IMPORTANT” OR “NOT IMPORTANT” TOPIC FROM Q39] that was conducted face to face