Embed Size (px)
Citation preview
Results from Two Sampling Experiments
Michael J. Stern, PhD, Kirk Wolter, PhD,and Ipek Bilgen, PhD.
Can We Effectively Sample from SocialMedia Sites and Search Engines?
…"interactive platforms via which individuals ….. create and shareuser-generated content” (see Kietzmann, Hermkens, McCarthy,and Silvestre, 2011).
…use of search engines and ads.
• Why has there been a call for exploring this research?– Increasing difficulty reaching respondents; Expense of pre-testing
instruments; Interest and recent, tepid acceptance of non-probabilitymethods; Client expectations; Mobile devices lend themselves; andPanel construction.
What is Social Media?
Active versus Passive Social Media and Search Engine DataCollection
• Active (we’ll be focusing here today)• Using social media sites and search engines to collect sample,
collect survey responses, or both.
• Passive• Using social media sites and search engines to collect data
through scraping or other systematic techniques that allow foranalyzing aggregate user-generated data or paradata.
Types of Social Media and Search Engine Surveying
Selected Examples
• Active• In Bhutta’s (2012) study of baptized Catholics, she created a
Facebook group named “Please Help Me Find BaptizedCatholics!” and then contacted the administrators from otherCatholic-centered Facebook groups to recruit members for herstudy.
• Ramo and Prochaska (2012) used Facebook to recruitparticipants for a study of cigarette users between the ages of 18-25 years old. Within three months, they had obtained a sample of3,093 individuals who were eligible for the study with 1,548completing the survey at a cost of $4.28 per complete.
Examples of previous work
Example
• Passive• One example is the Google
Flu Trend data, whichoverestimated prevalencethis year.
Examples of previous work
Used By Permission--Butler, 2013Sources: Google Flu Trends (www.google.org/flutrends); CDC; FluNear You
Example
• Passive• Kosinski et al (2013)
analyzed “likes” onFacebook for 58Kvolunteers. Using thisdata they accuratelypredicted, to name afew:
Single/In Rel.: 67%Sex: 93%Race: 95%Political Affil.: 85%Sexual Orien.: 88%
Examples of previous work
Kosinski, Stillwell, and Graepel. (2013) “Private traits and attributesare predictable from digital records of human behavior.”www.pnas.org/cgi/doi/10.1073/pnas.1218772110
1. What is this data good for? Augment Supplement Primary All of the above
The question rests in how representative it is.
Questions that we must address
• To address this question, we must examine how it works
• Bids (suggested/who sees the ad/targeting)
• Measuring Response Rates
• Click Through Rates—The number of unique clicks you receiveddivided by the number of times your ad was shown.
• Clicks versus Visits versus Responses—examples fromFacebook, Google Adwords, and Google Analytics
How does active social media surveying work?
• Experiments• Design Facebook Ads (today n=134)• Google Ad (today n=384)• Both benchmarked by 2010 General Social Survey (2,041)
• Other• ABS and Email Blast (see Bilgen, Wolter, and Stern, 2013)• Purpose• Representativeness/weighting/effective recruitment/cost per case
• Incentives• $10• $5• $2
• Programming challenges• PINs and No PINs
• Three prong system using cookies, IP addresses, and email registration
Our Work in Active Data Collection
What’s a response rate based on?
Click throughby Reach
Is this theresponse rate?
Finding and recruiting respondents
We used avariety of adsas a way torecruitrespondents toour survey.Each adperforms at asomewhatdifferent leveland real timeanalytics foreach.
Finding and recruiting respondents
We included theincentive value($10 or $5depending ontreatment)
And, ouraffiliation.
Clicking on thead takesrespondents toour landing page
Finding and recruiting respondents through Google
We included the incentive value ($10or $5 depending on treatment), ouraffiliation. Clicking on the ad takesrespondents to our landing page
Google Adwords requires keywords
Differences by approach? Age
51.7
11.8
0.1 0.1 0.1
18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older
Age: Facebook Analytics-who clicked the ad?
Facebook Analytics (%)
Just over a third of cases in analytics were missing ageor sex
Differences by approach? Age
25.4
7.6
16.9
33.9
13.6
51.7
11.8
0.1 0.1 0.1
18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older
Age: Facebook Analytics and Survey Results
Facebook Survey Data (%) Facebook Analytics (%)
Obviously, the 0.1 folks don’t include their age, sex or both…andrespond at higher rates
Differences by approach? Age
8.9
18.5
26.9 26.6
19.3
25.4
7.6
16.9
33.9
13.6
10.4
20.8
39.0
21.7
8.2
18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older
Variation in Age Distribution: GSS, Facebook, and GoogleGSS Facebook Google FB vs. Google
Chi-Square= 46.52; p <0.001
Differences by approach? Sex
54.9
45.1
51.246.1
Male Female
Age: Facebook Analytics and Survey ResultsFacebook Survey Data (%) Facebook Analytics (%)
Differences by approach? Sex
43.6
56.454.9
45.142.9
57.1
Male Female
Variation in Sex Distribution: GSS, Facebook, and GoogleGSS Facebook GoogleFB vs. Google
Chi-Square= 5.02; p =0.03
Differences by approach? Education
63.9
25.5
10.7
47.2
39.2
13.6
62.2
25.9
11.5
Less than a College Degree College Degree Advanced Degree
Variation in Educational Attainment Distribution: GSS,Facebook, and GoogleGSS Facebook Google
FB vs. GoogleChi-Square= 49.23; p =0.01
Differences by approach? Work Status
44.9
11.5
35.8
4.6
32.5
15.4
48.0
4.1
37.6
16.9
31.4
14.1
Full Time Part Time Not employed In School
Variation in Work Status: GSS, Facebook, and GoogleGSS Facebook Google FB vs. Google
Chi-Square= 15.14; p =0.002
• Facebook and Google• Generalizability? Questionable but Google outperformed
Facebook.
• Appropriateness may hinge on sample requirements
Our Work in Active Data Collection
• Cost Per Case and Spending?
Our Work in Active Data Collection
FACEBOOK SPENT(DATA COLLECTION—does not include incentive )
4,215 total clicks
123 completes
34.27 clicks per complete
$0.87 spent per click (average)
$29.98 spent per complete
$3,687.04 TOTAL Spending
GOOGLE SPENT(DATA COLLECTION—does not include incentive )
5,127 total clicks
318 completes
16.12 clicks per complete
$0.79 spent per click (average)
$12.09 spent per complete
$3,846.00 TOTAL Spending
• Google looks more like GSS than Facebook.
• Google was faster and cheaper.
• Had one security blip with Google, though, withslickdeals.net
Ad Campaigns
• What have clients asked for
• Other uses
• What we’re doing
• IRB
Thinking Ahead
