Upload
maci-tilly
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Survey Experiment Survey Experiment ModalitiesModalities
Some Pros and Cons of Differing Sampling Sources and Some Pros and Cons of Differing Sampling Sources and MethodsMethods
Matthew A. BaumHarvard University
&Leonie Huddy
Stonybrook University
OutlineOutlineReview of prominent Internet playersReview of prominent Internet players
Amazon Mechanical TurkAmazon Mechanical Turk
Knowledge NetworksKnowledge Networks
Yougov/PolimetrixYougov/Polimetrix
SurveyMonkey AudienceSurveyMonkey Audience
Comparing survey modalitiesComparing survey modalities
USA vs. Swedish Phone/Internet Usage USA vs. Swedish Phone/Internet Usage StatisticsStatistics
ConclusionsConclusions
Amazon Mechanical Turk Amazon Mechanical Turk (AMT)(AMT)
OverviewOverviewBegan in 2005Began in 2005““Workers” sign up to participate in tasks for pay Workers” sign up to participate in tasks for pay (called “Human Intelligence Tasks”, or “HITs”)(called “Human Intelligence Tasks”, or “HITs”)
Global workforce (~100,000 workers)Global workforce (~100,000 workers)
Salary: ~$3-$8/hour (maybe $10/hr for high-experience Salary: ~$3-$8/hour (maybe $10/hr for high-experience workers)workers)
““Requestors recruit “workers” on internal message Requestors recruit “workers” on internal message board describing taskboard describing task
Can specify “qualifications” for workers (e.g., Can specify “qualifications” for workers (e.g., experience, quality ratings, nationality, etc.)experience, quality ratings, nationality, etc.)
10% fee to Amazon10% fee to Amazon
AMT provides survey building tools, or Requestors AMT provides survey building tools, or Requestors can include links to external surveyscan include links to external surveysHITs range from 1-second marketing surveys (“Rate HITs range from 1-second marketing surveys (“Rate appeal of this photo on 1-10 scale”) to elaborate 30+ appeal of this photo on 1-10 scale”) to elaborate 30+ minute experimental surveysminute experimental surveys
Compensation Amount
Short survey(5 min)
Medium survey (10
min)
Long survey(30 min)
$.02 5.6 5.6 5.3
$.10 25.0 14.3 6.3
$.50 40.5 31.6 16.7
Effects of Compensation Amount and Task Length on Participation Rates(Submitted Surveys per Hour of Posting Time)
Source: Buhrmester, Kwang, and Gosling (2011)
My AMT Experience:•paid $1.00 per completed HIT for ~22 minute survey•Listed as “News and Politics Survey”•Received 1933 completed surveys in ~one month• rapidly diminishing returns• 1825 valid responses
Amazon Mechanical Turk Amazon Mechanical Turk (AMT)(AMT)
% Female % Non-White
% Non-American
Age
AMT 55 36 31 32.8
Internet 57 Fewer (?) ~31 24.3
Buhrmester, Kwang, and Gosling (2011)
Amazon Mechanical Turk Amazon Mechanical Turk (AMT)(AMT)
DemographicsDemographics
Paolacci, Chandler, and Ipeirotis (2010)
Category (N=1000 AMT workers)
% Female 64.9
Average Age 36
% Earning below $60k/year
66.7
Education “higher than general pop.”
% Non-American 53
Amazon Mechanical Turk Amazon Mechanical Turk (AMT)(AMT)
My AMT Survey (N=1824)
US National Averages (From Pew 2010 Survey)
Average Age 32.6 51.0
% Non-White 12.2 20.3
% Female 58.2 56.9
% Democrats 37.4 31.9
% Republicans 17.7 28.0
% Independents 44.3 40.1
% Liberal 36.7 17.5
% Moderate 46.9 43.9
% Conservative 16.3 38.6
DemographicsDemographics
Amazon Mechanical Turk Amazon Mechanical Turk (AMT)(AMT)
Follow-Up SurveyFollow-Up SurveyContacted all 1825 workers who completed valid HITs in Contacted all 1825 workers who completed valid HITs in original survey approximately 4 months lateroriginal survey approximately 4 months later
Offered $.50 for ~12 minute surveyOffered $.50 for ~12 minute survey426 valid responses (24%)426 valid responses (24%)
2nd batch of invites
TradeoffsTradeoffs
Source: Paolacci, Chandler, & Ipeirotis (2010)
Some Conclusions re Some Conclusions re AMTAMT
““Our analyses of demographic characteristics Our analyses of demographic characteristics suggest that MTurk participants are at least as suggest that MTurk participants are at least as diverse and more representative of noncollege diverse and more representative of noncollege populations than those of typical Internet and populations than those of typical Internet and traditional samples. Most important, we found that traditional samples. Most important, we found that the quality of data provided by MTurk met or the quality of data provided by MTurk met or exceeded the psychometric standards associated exceeded the psychometric standards associated with published research.”with published research.”
Buhrmester et al. (2011)Buhrmester et al. (2011)
““Our theoretical discussion and empirical findings Our theoretical discussion and empirical findings suggest that experimenters should consider suggest that experimenters should consider Mechanical Turk as a viable alternative for data Mechanical Turk as a viable alternative for data collection.”collection.”
Paolacci et al. (2010)Paolacci et al. (2010)
Knowledge NetworksKnowledge Networks2007 Survey (via TESS)2007 Survey (via TESS)
Participants recruited via residential address Participants recruited via residential address searchessearches
Respond online (Internet access provided to recruits Respond online (Internet access provided to recruits who don’t have it)who don’t have it)
Pretty good samples, but imperfect (selection effects Pretty good samples, but imperfect (selection effects not completely purged) and expensive not completely purged) and expensive
DemographicsDemographicsPost-Survey Matched Sample
(N=1014)
% Female 50
% Non-white 26.7
% Liberal 25
% Moderate 41
% Conservative 33
Average Age 46
Ideological Intensity (0-3 scale)
.99
Political Knowledge (0-1 scale)
.487
Recent (2012) KN Recent (2012) KN ProposalProposal
• Design: 4-wave study, longitudinal sample, all waves occurring within 1 yearDesign: 4-wave study, longitudinal sample, all waves occurring within 1 year
• Pretest: N=25 interviews each wavePretest: N=25 interviews each wave
• Sample: General population adults, age 18+, English-language survey-takersSample: General population adults, age 18+, English-language survey-takers
• Number of completed interviews: N=2,000 wave 1, with about 70%-80% of Number of completed interviews: N=2,000 wave 1, with about 70%-80% of wave 1 respondents completing each of the later waveswave 1 respondents completing each of the later waves
• Median survey length: 20 minutes each waveMedian survey length: 20 minutes each wave
• Multimedia/graphics: NoneMultimedia/graphics: None
• Incentives: $5 for each of waves 1-3, $10 for wave 4Incentives: $5 for each of waves 1-3, $10 for wave 4
• KN will provide standard deliverables (self-documented data file with all the KN will provide standard deliverables (self-documented data file with all the survey data, general demographic profile data, and field report documenting survey data, general demographic profile data, and field report documenting all sampling and data collection procedures, codebook, and panel recruitment all sampling and data collection procedures, codebook, and panel recruitment methodology)methodology)
• Price: $255,550 (No, that’s not a typo!)Price: $255,550 (No, that’s not a typo!)
Yougov/PolimetrixYougov/PolimetrixOpt-in samplingOpt-in sampling
Random draw from target population matched with Random draw from target population matched with most comparable available panelists to create most comparable available panelists to create representative population samplesrepresentative population samples
Demographics + attitudinal/behavioral factorsDemographics + attitudinal/behavioral factors
Post-Survey Matched Sample
My Survey (2007, N=1200)
CCAP (2008, 6-wave panel)*
Average Age 40 44.2
% Female 51 52
% Non-white 31.5 16.7
% Democrat 38.0 36.1
% Republican 39.8 30.6
% Independent 22.2 33.3
% Liberal 26.4 22.8
% Moderate 36.8 34.5
% Conservative 36.8 35.3
*Notes:(1)Overweights battleground states 2-fold.(2)Demo weights based on age, race, gender, educ, marital status, kids, income, state, metro area, employment, citizenship(3)Attitude/Behavior weights based on religion, church attendance, evangelical status, news interest, PID, ideology.
Comparing Modalities: KN vs. Comparing Modalities: KN vs. Polimetrix vs. Natural Survey: Polimetrix vs. Natural Survey:
Barabas & Jerit (2010)Barabas & Jerit (2010)
Polimetrix vs. Pew vs. NES Polimetrix vs. Pew vs. NES (Partisan Distribution)(Partisan Distribution)
Source: Hill, Lo Vavreck & Zaller (2007)
Polimetrix vs. Pew vs. NES Polimetrix vs. Pew vs. NES (Ideologial Distribution)(Ideologial Distribution)
Source: Hill, Lo Vavreck & Zaller (2007)
Demographic ComparisonsDemographic ComparisonsSource: Hill, Lo Vavreck & Zaller (2007)
Comparing Modalities: KN vs. Comparing Modalities: KN vs. Polimetrix vs. Natural Survey: Polimetrix vs. Natural Survey:
Barabas & Jerit (2010)Barabas & Jerit (2010)
Conclusions:“The results presented here should be encouraging to anyone devoted to the scientific study of politics because they suggest that what occurs in survey experiments resembles what takes place in the real world.”
“Although there was a discrepancy between the size of survey treatment effects and the general population in our natural experiment, we observed correspondence exactly where one would expect to find it—among those who were most likely to be exposed to media messages about the two government announcements.”
SurveyMonkey AudienceSurveyMonkey AudienceNew enterprise for SMNew enterprise for SM
still figuring it out!still figuring it out!
Recruit participants from survey respondentsRecruit participants from survey respondents
Currently U.S. only, but likely to expandCurrently U.S. only, but likely to expand
$3 per finished response; $5 for rush project (3 business $3 per finished response; $5 for rush project (3 business days)days)
Custom create demographic profilesCustom create demographic profilesGender, age, income, location, education, race, industry of Gender, age, income, location, education, race, industry of employment, job function, marital status, employment status, home employment, job function, marital status, employment status, home ownership, vehicle ownership, smartphone ownership, exercise habits.ownership, vehicle ownership, smartphone ownership, exercise habits.
No political attitude selectors yet (though I’ve lobbied them!)No political attitude selectors yet (though I’ve lobbied them!)
Similar to Polimetrix, except don’t start with random draw from Similar to Polimetrix, except don’t start with random draw from populationpopulation
Much less expensive!Much less expensive!
AAPOR 2010 Task Force on AAPOR 2010 Task Force on Online Panels Online Panels
Focus on nonprobability samplesFocus on nonprobability samples
Informative, but inconclusive…Informative, but inconclusive…
ConclusionsConclusionsIf research objective includes accurate estimate of population If research objective includes accurate estimate of population values, avoid nonprobability online panels.values, avoid nonprobability online panels.
Results differ significantly from probability-based methods (like Results differ significantly from probability-based methods (like RDD telephone) on range of behaviors and attitudes, with latter RDD telephone) on range of behaviors and attitudes, with latter being more accurate.being more accurate.
Nonprobability online panels sometimes appropriate, when precise Nonprobability online panels sometimes appropriate, when precise estimates of population values not criticalestimates of population values not critical
More research needed on evaluating and testing techniques used More research needed on evaluating and testing techniques used across disciplines to make population inferences from across disciplines to make population inferences from nonprobability samples.nonprobability samples.
Ansolabehere & Schaffner Ansolabehere & Schaffner (2011)(2011)
Comparison of Survey ModalitiesComparison of Survey Modalities
3-mode study conducted in 20103-mode study conducted in 2010
1.1. Opt-in Internet panelOpt-in Internet panel
2.2. Live telephone interviews (using Live telephone interviews (using national RDD sample of landlines and national RDD sample of landlines and cell phones)cell phones)
3.3. Mail (using national sample of Mail (using national sample of residential addresses)residential addresses)
Mode Sample Size
Field Dates Response Rate
Completion Time
Internet 1000 1/15/10-2/11/10
42.9% (RR1)
8.94 mins
Mail 1207 1/30/10-9/30/10
21.1% (RR3)
11.80 mins*
Phone 907 1/28/10-1/30/10
19.5% (RR3)
14.33 mins
Ansolabehere & Schaffner Ansolabehere & Schaffner (2011)(2011)
*mail recruits who took survey online
Ansolabehere & Schaffner Ansolabehere & Schaffner (2011)(2011)Item Response Internet Phone Mail Validating
Source
Home Ownership
Own .613 .632 .632 .669 (CPS)
Mobility Moved in past year
At address 5+ years
.152
.555
.155
.609
.162
.519
.154 (ACS)
.588 (ACS)
Smoked 100 Cigarettes
Yes .504 .471 .497 .430 (NHIS)
Smoke Cigarettes now
Every or some days
.259 .242 .241 .203 (NHIS)
Voted in ‘08 (if registered)
Yes .888 .876 .821 .896 (CPS)
Vote Choice in ‘08
Obama
McCain
.482
.469
.454
.505
.553
.431
.529
.456
Avg Diff. .036 .035 .043
Non-validatable Political Point Estimate Comparison, by ModeNon-validatable Political Point Estimate Comparison, by ModeIncludes State of Economy, Includes State of Economy, Approval of/Support for Obama, Congress, R’s Member, Abortion, Affirmative Action, Gay Marriage, Investing Social Security, Tax over $200k, Cut Approval of/Support for Obama, Congress, R’s Member, Abortion, Affirmative Action, Gay Marriage, Investing Social Security, Tax over $200k, Cut Spending, GovernmentSpending, GovernmentRight and WrongRight and WrongVoting MethodVoting MethodReligious and/or Political ContributionsReligious and/or Political ContributionsPolitical KnowledgePolitical KnowledgeNews SourceNews Source
Ansolabehere & Schaffner Ansolabehere & Schaffner (2011)(2011)
Internet vs. Phone
Phone vs. Mail
Internet vs. Mail
Avg Diff. (All measures)
.062 .042 .051
Avg Diff. (Attitudinal measures only)
.052 .042 .044
Weighted proportions of respondents in each category, excluding DK.
Small (“negligible”) differences across modesSmall (“negligible”) differences across modesExcept…Internet respondents more politically knowledgeable & Except…Internet respondents more politically knowledgeable & made more political contributionsmade more political contributionsMail costs 5 times more than Internet & twice as much as phoneMail costs 5 times more than Internet & twice as much as phoneInternet half as costly as phone and faster turnaroundInternet half as costly as phone and faster turnaroundDifferences from other studies that found Internet samples less Differences from other studies that found Internet samples less valid than phone samples attributed to (1) more Internet users than valid than phone samples attributed to (1) more Internet users than 5+ years ago when prior data samples collected, and (2) advances 5+ years ago when prior data samples collected, and (2) advances in “science of constructing, matching and weighting opt-in Internet in “science of constructing, matching and weighting opt-in Internet panels”panels”Conclusion: “...an opt-in Internet survey produced by a respected firm can produce results that are as accurate as those generated by a quality telephone poll and that these modes will produce few, if any, differences in the types of conclusions researchers and practitioners will draw in the realm of American public opinion.”
Ansolabehere & Schaffner Ansolabehere & Schaffner (2011)(2011)
U.S. Telephone OwnershipU.S. Telephone Ownership
Media Access,Media Access,Sweden vs. USASweden vs. USA
Sweden vs. USASweden vs. USA
Sweden vs. USASweden vs. USA
Sweden vs. USASweden vs. USA
Swedish Phone Use by Swedish Phone Use by AgeAge
Source: Axelsson (2010)
Why Persistent Mobile Why Persistent Mobile Phone & Internet Usage Phone & Internet Usage
Gaps?Gaps?
Income Inequality: USA vs. Sweden in 2011
Effects of Income on Effects of Income on Internet Usage in U.S.Internet Usage in U.S.
U.S. has more of these folks
What Do Data Tell Us?What Do Data Tell Us?Swedes more likely to be online in 2010 (by ~14 Swedes more likely to be online in 2010 (by ~14 percentage points), and make greater use of Internet percentage points), and make greater use of Internet
But, similar in fixed broadband and landline usageBut, similar in fixed broadband and landline usage
More likely to use mobile phonesMore likely to use mobile phones
But similar in volume of mobile calls sent and receivedBut similar in volume of mobile calls sent and received
Moral of story? Infrastructure looks, if anything, LESS Moral of story? Infrastructure looks, if anything, LESS hospitable to probability sampling in Sweden than in USAhospitable to probability sampling in Sweden than in USA
So, if RDD today works better in Sweden for generating So, if RDD today works better in Sweden for generating probability samples, reason seems likely to have more probability samples, reason seems likely to have more to do with attitudes toward surveys than infrastructureto do with attitudes toward surveys than infrastructure
ConclusionsConclusionsOpt-in Internet samples here to stayOpt-in Internet samples here to stay
Cheaper (by a lot!)Cheaper (by a lot!)
Faster (by a fair amount…)Faster (by a fair amount…)
Primary competitor (RDD phone surveys) increasingly difficult Primary competitor (RDD phone surveys) increasingly difficult 14% of adults “unreachable” in Sweden?14% of adults “unreachable” in Sweden?
Estimated 25% of U.S. households cell only in 2010.Estimated 25% of U.S. households cell only in 2010.
““Unreachable”: ~13% (AAPOR 2010); others say moreUnreachable”: ~13% (AAPOR 2010); others say more
Open up new possibilitiesOpen up new possibilitiesE.g., cross-national samples/panelsE.g., cross-national samples/panels
Most current evidence suggests that with current matching and Most current evidence suggests that with current matching and weighting techniques, Opt-in Internet samples weighting techniques, Opt-in Internet samples can can be be representative of target populationsrepresentative of target populations