Understanding mobility: Consent and capture of geoloca1on
data in web surveys
Sco4 D. Crawford [1] Colleen McClain [2] Robert H. Young [1] Toben F. Nelson [3]
[1] Survey Sciences Group, LLC; [2] Michigan Program in Survey Methodology; [3] University of Minnesota, Twin CiOes
Mobile Devices: A Disruptor Technology in Survey Research
• We must adjust – but will be rewarded with new opportuniOes! – “In the moment” surveys – Self-‐administered biomarker collecOon – Sound and image captures – “Internet of Things” integraOon (i.e. thermostat readings, electricity use, pedometers, etc.)
– Geoloca'on capture
2
What is GeolocaOon? Is it the same as GPS?
3
1
2 3
4
Easing in to GeolocaOon Capture Our Domains of Interest
A. Is it possible? • What would respondents think about it?
B. Does it work? • Can we actually capture the data? • Do respondents allow it?
C. What are the best pracOces? • How do we handle device prompts? • Is standard consent form appropriate and adequate? • Should we ask permission explicitly? • Do we capture more than once?
Quality: How well did it work? • What does the data look like? Is it complete?
• Can it cause any error? Can it help idenOfy error?
4
Study: Baseline QuesOonnaire (April 2013)
• Random sample of 8,000 U of Minnesota students • Baseline quesOonnaire (T12) – Alcohol, Drugs, Mental Health and related behaviors & experiences
– Web Survey Length • Mean=24.3 minutes; Median=22 minutes
– AAPOR RR#2: 28% – Other
• Prenote email requested users complete the survey on a desktop or laptop computer
• All Email (Invite, 3 reminders to NRs) • Sweepstakes incenOve for $500 cash
5
Study: Rapid Response QuesOonnaires • Follow-‐up Surveys of Responders to Baseline – Short version of key measures from baseline – No incenOve used, all email communicaOons, references made to survey being designed for a mobile device
6
Time Month in 2013 Length AAPOR RR#2
T14 June Mean=3.9 min; Median=3 67%
T16 August Mean=3.63 min; Median=3 66%
T18 October Mean=3.83 min; Median=3 61%
Study: Sample CharacterisOcs
48.6
72.9
8.6 15.6 17.9 25.5 32.4 38.3
77.6
7.5 14.2 15.4 23.4
39.4
Male White Class: 1st Year
Class: 2nd Year
Class: 3rd Year
Class: 4th Year
Class: Grad Student
Baseline Sample vs. Baseline Responders
Sample Baseline
7
All differences were significant in a Χ2 test p<0.01
Quality: How well did it work?
In Rapid Response surveys – no further differences. They were the same as Baseline responders.
More White
Less Male Older
Hypothesis 1a: Will respondents cooperate (hypotheOcally)
at T14 Rapid Response? • Hypothesis 1a: When asked, a majority of those responding will cooperate with a hypotheOcal request to collect geolocaOon data.
8 A. Is it possible?
Yes! 58% accepted when asked the hypotheOcal quesOon.
SUPPORTED …but, 42% did say NO…
The “HypotheOcals”… Were there any data quality concerns?
• No differences demographically • No differences on any substanOve measures captured, including: – Past 2 weeks binge drinking – NegaOve consequences from alcohol use – Past 4 hours drinking or drug use
9 Quality: How well did it work?
Will respondents cooperate with an ACTUAL geolocaOon request?
• Hypothesis 1b: When asked, a majority of those responding will cooperate with a request to collect actual geolocaOon data.
• So how do we get permission? – GeolocaOon data is above and beyond what a typical survey respondent would expect or even understand is being collected when they agree to complete a Web survey, thus, it should be described in the consen1ng process for the survey…
10 B. Does it work?
RequesOng GeolocaOon Data: Devices Already Do This – But Beware
• W3C GeolocaOon API SpecificaOons require permission1
• …While the technology currently requests permission prior to capturing this type of data, we do not have control over that and we cannot guarantee that it will be maintained. We believe that the automated request provided by the technology is not sufficient and it should be supplemented.
11
1) h4p://www.w3.org/TR/geolocaOon-‐API/#security
C. What are the best pracOces?
The SoluOons We Considered… Add Language to the Survey Consent Form
Add a Separate Geoloca'on Specific Consent Ques'on
12 C. What are the best pracOces?
In addi1on to the ques1ons in this brief survey we would like to collect data on the loca1on where you are comple1ng this survey using features available in desktop computers and mobile devices. You will be asked whether you will allow loca1on data to be collected and you may choose not to allow collec1on of loca1on data.
We would like to understand more about where respondents are when they par1cipate in surveys. We would like to collect informa1on made available by your computer/mobile device on your geographic loca1on. Do you accept or decline our request to collect your loca1on? o Yes, you may collect geographic
data o No you may not collect
geographic data
Two Hypotheses
• H2a: Respondents who are asked in a separate quesOon for permission to allow geolocaOon capture will be less likely to consent to capture than those who agree as part of the main consent form. – The “If you ask it, they will say no!” hypothesis
• H2b: Respondents who consent to capture will be more likely to actually provide geolocaOon data if they are consented with a separate quesOon. – The “Surprise, we are tracking you!” backfire hypothesis.
13 C. What are the best pracOces?
The Consent Experiment Treatments
14
Consent to Survey with GeolocaOon Text
Start Survey & GeolocaOon Capture
Consent to Survey with GeolocaOon Text Consent to GeolocaOon
Start Survey & GeolocaOon Capture
(if Consented)
Treatment A: Consent Form Only
Treatment B: Geoloca'on Consent Ques'on
Consent to Survey Start Survey
Treatment C: Control (T16 only)
C. What are the best pracOces?
B. Does it work?
Did respondents consent to the survey? (T16)
15
Consent to Survey 96% (n=136)
Start Survey & GeolocaOon Capture
Consent to Survey 96% (n=151) Consent to GeolocaOon
Start Survey & GeolocaOon Capture
(if Consented)
Treatment A: Consent Form Only (n=142)
Treatment B: GeolocaOon Consent QuesOon (n=157)
Consent to Survey 93% (n=135) Start Survey
Treatment C: Control
B. Does it work?
But what about the extra step with the Consent QuesOon? (T16)
16
Consent to Survey 95.8% (n=136)
Start Survey & GeolocaOon Capture
Treatment A: Consent Form Only (n=142)
Consent to Survey 96% (n=151)
Consent to GeolocaOon
60% Agreed (n=90) 40% said NO or blank
Start Survey & GeolocaOon Capture
(if Consented)
Consent to Survey 93.1% (n=135) Start Survey
Treatment C: Control
C. What are the best pracOces?
This replicated in T18 data collec'on with 67% agreeing to par'cipate.
Treatment B: Geoloca'on Consent Ques'on (n=157)
The “If you ask it, they will say no!” Result
• H2a: Respondents who are asked in a separate quesOon for permission to allow geolocaOon capture will be less likely to consent to capture than those who agree as part of the main consent form. – The “If you ask it, they will say no!” hypothesis
17 C. What are the best pracOces?
SUPPORTED
Did we capture geolocaOon data? Possible outcomes
18
• LaOtude / Longitude data and related data is received
1. Success: We Capture Data
• Permission Denied or Permission Unknown
2. Permission Error: Error Code Received
• And no informaOon as to why
3. No Data: We Get Nothing
Quality: How well did it work?
Consent to Survey • 96.9% (n=189)
Consent to GeolocaOon • 66.7% Agreed (n=130) • 33.3% said NO or leu blank
GeolocaOon Captured? • Success: 49.2% (n=64) • Permiss Denied: 14.6% (n=19) • No Data: 36.2% (n=47)
Did we capture data auer consent? (T18)
Consent to Survey (with GeolocaOon text) • 97% (n=212)
GeolocaOon Captured? • Success: 20% (n=43) • Permiss Denied: 29% (n=62) • No Data: 51% (n=107)
19
Treatment A: Consent Form Only (n=219)
Treatment B: Geoloca'on Consent Ques'on (n=195)
C. What are the best pracOces?
Quality: How well did it work?
Final Usable GeolocaOon Data • n=64 cases
• 32.8% of sample
Net: More Data With GeolocaOon Consent QuesOon
20
Final Usable GeolocaOon Data • n=43 cases
• 19.6% of sample
Treatment A: Consent Form Only (n=219)
Treatment B: Geoloca'on Consent Ques'on (n=195)
C. What are the best pracOces?
Quality: How well did it work?
Hypothesis 2b Result
• H2b: Respondents who consent to capture will be more likely to actually provide geolocaOon data if they are consented with a separate quesOon.
21 C. What are the best pracOces?
Quality: How well did it work?
SUPPORTED
Hypothesis 2c: Where geolocaOon providers demographically the same?
• H2c: Sample characterisOcs and substanOve measures collected from those who successfully provide geolocaOon data will be similar to those who did not.
Ø No differences found in gender, race/ethnicity and year in school.
22 Quality: How well did it work?
Hypothesis 2d: What did this do to survey break-‐off?
• H2d: Survey break-‐off rates among those who successfully provide geolocaOon data will be similar to the breakoff rates of those who did not.
23 Quality: How well did it work?
• Compared to the control, no significant impact (very small sample sizes) – T16: 2.3% Treat A / 1.8% Treat B / 0% Treat C (control) – T18: 5.5% Treat A (n=12) / 3.3% Treat B (n=6)
SUPPORTED
Hypothesis 2e: What did this do substanOve responses?
• H2e: SubstanOve responses provided by those who successfully provide geolocaOon data will be similar to responses provided by those who did not.
24 Quality: How well did it work?
H2e Results – So Far…
• Ongoing analysis here… some mixed early news: – No differences on most measures – Respondents more than twice as likely to have drank alcohol within the past four hours (10.5% vs. 4.2%) if they successfully provided geolocaOon data (p<0.01) • But not other drugs…
25 Quality: How well did it work?
Hypothesis 3: Mobile vs. Non-‐Mobile
• H3a: Mobile users (phones and tablets) will consent (when asked explicitly) to geolocaOon capture at a higher rate than non-‐mobile users.
• No difference found: – 66% of non-‐mobile users consented – 69% of mobile users consented
26 C. What are the best pracOces?
Quality: How well did it work?
NOT SUPPORTED
Hypothesis 3: Mobile vs. Non-‐Mobile
• H3b: Mobile users will end up providing a higher rate of successful geolocaOon captures than non-‐mobile users.
27 C. What are the best pracOces?
Quality: How well did it work?
H3b – Does mobile lead to a higher capture success rate?
28
27% 46%
17%
46% 56%
8%
0%
20%
40%
60%
80%
100%
Non-‐Mobile Respondents
Mobile Respondents
No Data
Error Capture
Successful GeolocaOon
NOTE: Difference was significant in a Χ2 test at p<0.01
Quality: How well did it work?
Yes SUPPORTED
And what did we get? LaOtude/Longitude
• Lat/Long Accuracy Data – 100% complete – Accuracy to within 5 to 95000 meters (16.4 feet to 59 miles)
• InteresOng arOfact – 5 cases with 22,000 meters and 1 case with 95,000 meters all with the SAME Lat/Long point
– Lat/Long point corresponds to a large 512 unit apartment building
– All Respondents with this point were using Windows
29 Quality: How well did it work?
And what did we get? AlOtude Data
• AlOtude – 25.6% with a non-‐NULL value; Min: 21 -‐ Max: 302 – Mean: 254 meters (833 feet) • Per Wikipedia: Eleva'on of Minneapolis is 830 feet
30 Quality: How well did it work?
So what does the data look like?
31 Quality: How well did it work?
Study LimitaOons
• Student populaOon at one University • Low response rate with some nonresponse bias • Short quesOonnaire • Small sample sizes
32
Next Steps
• Evaluate differences using baseline data • Build and test some models to be4er evaluate substanOve difference(s)
• How can we improve consent rate further? • How can we reduce geolocaOon failure to provide data?
• Explore what geolocaOon data itself can do to tell us about data quality (i.e. do those who stay sOll provide be4er data?)
• What else can we collect?
33
Thank You!
• QuesOons?
• Contact: Sco4 D. Crawford [email protected] 734-‐527-‐2150 (office) 734-‐395-‐8790 (cell)
34