23
Online Survey Participation via Mobile Devices AAPOR 2013, Boston, MA May 18, 2013 Michael Bosnjak, Teresio Poggio, Kai R. Becker, Frederik Funke, Alexandra Wachenfeld, Beat Fischer

Online Survey Participation via Mobile Devices - · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

  • Upload
    ngodat

  • View
    218

  • Download
    4

Embed Size (px)

Citation preview

Page 1: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Online Survey Participation via Mobile Devices

AAPOR 2013, Boston, MAMay 18, 2013

Michael Bosnjak, Teresio Poggio, Kai R. Becker, Frederik Funke, Alexandra Wachenfeld, Beat Fischer

Page 2: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

• Definitions of mobile surveys– Interviewer-administered surveys

• Interviews among mobile phone users• Interactive voice response surveys among mobile phone

users– Self-administered surveys

• SMS (text messaging) surveys• Browser-based surveys on mobile devices (e.g., mobile

phones having mobile Internet-access, Smartphones, etc.)

• Our focus: – Self-administered surveys AND– using a mobile device AND– touch-operated browser interface.

Mobile Surveys?Self-Administered Mobile Surveys?

Page 3: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Challenges for Survey Methodology

• Measurement equivalence?– Not treated here, but some experimental

comparisons do exist (e.g., Metzger & Bosnjak, 2009).

• (Non)Response Mobile vs Desktop/Laptop (selection):– (Non)Response rates?– Sample composition differences?– Differences in participation patterns/behavior?– ´Mode-Compliance´: Do participants actually stick

to a designated mode, i.e. the Desktop/Laptop mode for which a survey has been optimized?

Page 4: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Overall Research Questions

• Rate of mobile participants? Is the rate of mobile devices used when conducting online surveys substantial (or negligible)?

• Participation patterns? – Within online study comparisons (mobile versus other):

Amount of data collected, drop-out rates, amount of data available for drop-outs.

– Across online panel studies: Do mobile participants stick to the mobile mode across panel waves?

• Systematic differences between mobile/non-mobile participants?Can the propensity to use a mobile device to participate in online surveys be explained (using demographic variables)?

Page 5: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Studies: OverviewStudy 1:

LINK PanelSeven largely independent samples, market research

access panel

Study 2: GESIS Pilot Panel

Probability-based scientific access panel, German Internet users

Rate of mobile participants?

Weighted average % of mobile participants

(Meta-analytic synthesis)

Number of times a mobile device had been used across eight panel

waves(Descriptive analysis)

Participation patterns?

Mode comparisons for amount of data entered, drop-out

behavior

Participation patterns across 8 waves

Systematic differences

mobile/non-mobile?

Propensity to use mobile devices vs. other

(Logistic regression)

Predictors of (1) the likelihood (not) to use mobile devices at all and (2) incidence rate of using mobile devices in the 8 waves.

(Zero-inflated poisson regression model)

Page 6: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Method (Sketch)• Meta-analytic synthesis (RE model) of seven online panel

studies conducted by LINK Germany (March- May, 2012)• Parameters synthesized (selection): Completion rate,

dropout rate, mobile participation rate, length of open-ended entries

• Final sample sizes, topics, and Flash usage (k= 7):– Study 1 (DGE05419), n= 1839, 7 mins, Multitopic-Survey: Finance, Fonds, no

Flash– Study 2 (DGE05426), n= 1711, 58 mins, Political opinions, voting behavior,

Flash– Study 3 (DGE05427 ), n= 906, 54 mins, Political opinions, voting behavior, Flash– Study 4 (DGE05428), n= 867, 53 mins, Political opinions, voting behavior, Flash– Study 5 (DGE05429), n= 1380, 20 mins, Multitopic-Survey: digital liefstyle,

sweets/candy, no Flash– Study 6 (DGE05429_B), n= 274, 3 mins, Replication of study 5 with new sample

(only sweets/candy), no Flash– Study 7 (DGE05433), n= 1838, 19 mins, Advertising effectiveness (sweets),

Flash

Page 7: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Results: Mode Non-Compliance

Average Mobile OS participation rate: 5.8% (95 CI: 4.3%/7.3%),

homogeneous effect using RE model

Page 8: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Results: Participation Patterns

• No differences Mobile/Desktop OS for– # of pages with entries (20 on average)– # of open-ended question answered (4 on average)– # of characters entered (165 on average)

• Differences for drop-outs Mobile/Desktop OS:– Dropout rate comparison:

Mobile 12% (9/16), Desktop 6% (5/7)– # of pages completed before dropped out:

Mobile 5.9 (5.4/6.3), Desktop 4.9 (4.8/5.0)– # of open-ended questions answered:

Mobile 2.9 (2.5/3.3), Desktop 2.4 (2.2/2.5)– > Higher drop-out rates for Mobile OS, but also more data

for Mobile OS drop-outs compared to Desktop OS!

Page 9: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Results: Mobile Survey Propensity

Sketch of results from a micro-data analysis (logistic regression):

• Younger subjects are more likely to answer from Mobile OS devices

• Male subjects more than women• No significant effect of educational level

• Outlier Study 7: Being a participant in this study almost halves the likelihood of using Mobile OS devices (most likely because of the use of video content/stimuli)

Page 10: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Studies: OverviewStudy 1:

LINK PanelSeven largely independent samples, market research

access panel

Study 2: GESIS Pilot Panel

Probability-based scientific access panel, German Internet users

Rate of mobile participants?

Weighted average % of mobile participants

(Meta-analytic synthesis)

Number of times a mobile device had been used across eight panel

waves(Descriptive analysis)

Participation patterns?

Mode comparisons for amount of data entered, drop-out

behavior

Participation patterns across 8 waves

Systematic differences

mobile/non-mobile?

Propensity to use mobile devices vs. other

(Logistic regression)

Predictors of (1) the likelihood (not) to use mobile devices at all and (2) incidence rate of using mobile devices in the 8 waves.

(Zero-inflated poisson regression model)

Page 11: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Method (Sketch)

• Post-hoc analysis on the use of mobile devices by members of a probability-based on-line panel

• GESIS Pilot Panel, 8 survey waves (Dec 2010-Feb 2012)

• Analysis focused on the ones who had at least started all the 8 on-line questionnaires (n=587)

Page 12: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Results: Mode Non-Compliance

About 8% of respondents used a mobile device at least once. Distribution of mobile device usage across all eight survey waves (# of waves a mobile device was used):

Never: 91.7 %1: 2.9 %2: 1.7 %3: 0.7 %4: 0.3 %5: 0.3 %6: 0.3 %7: 1.4 %8: 0.7 %

Page 13: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Results: Participation Patterns

91.7 %: No use at all2.1 %: Almost only mobile users (in at least 7 out of 8 waves)2.2 %: Innovation-led users (used more in most recent waves)1.4 %: Tried mobiles and back (used less in most recent waves)2.6 %: Other (no clear pattern in the use of mobile devices)

Page 14: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Results: Mobile Survey Propensity

Two distinct processes likely to be in action here:1) using or not using smartphones (in everyday life)2) if using smartphones: number of participations via mobile

(propensity to use a mobile device)

Our dependent variable: Number of times a mobile device was used in the 8 waves

Type of analysis: Zero-inflated poisson regression model, which basically consists in estimating 2 simultaneous equations:

1) a logistic regression model for the probability of not using mobile devices at all (modeling ‘inflated’ zeros, due to no use of these devices)

2) a poisson regression model for counting the number of times these devices were used in the waves (modeling ‘true’ zeros and 1-8 counts)

Page 15: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Results: Mobile Survey Propensity

- For both equations (may have different predictors in principle) we tested the effect of several typical social stratification variables:

o Age and gendero Level of education, occupational class, living in

rural versus urban areaso Marital status and number of members in the

household- A variable ‘positive attitude towards technology was also

considered in the model’- Backward approach: Starting with the same set of

predictors for both equations, only significant variables included in the final model

Page 16: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

• Inflate model (logit)(likelihood not to use mobile devices at all)– The elder, the less likely to use mobile devices– Women also less likely (not fully significant)– No clear/significant effects of eduction– No clear/significant effects of urban/rural areas

• Count model (Poisson)(incidence rate of using mobile devices in the 8 waves)– Lower incidence rates for divorced/widowed/single compared to

participants with a partner– The larger the households, the lower the incidence rate– Blue collar workers, but also cadres, have lower incidence rate

than white collar workers– No significant effects of age, education, urban/rural areas

Study 2: Results: Mobile Survey Propensity

Page 17: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Summary of Findings

• Rate of mobile participants? Non ignorable rate of mobile devices used when conducting online surveys (about 6-8% in our two studies)

• Participation patterns? – Study 1: Higher drop-out rates for Mobile OS within

individual studies (but more data for drop-outs).– Study 2: Mainly inconsistent use of mobile devices

(participating /w mobile device does not guarantee later mobile participation).

• Systematic differences between mobile/non-mobile participants?Across both studies, associations with age (younger age groups more likely to use mobile) and gender (men more likely) found. Presumably many more predictors relevant (Study 2, but see limitations).

Page 18: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Discussion

• Major limitation: No data on everyday smartphone / tablet usage in both data sets available (therefore: zero-inflated count model). Replication within the GESIS Panel: Mobile usage assessed during recruitment interview and updated regularly.

• Actionable recommendations? Given these results, how to handle the issue of participants using Mobile OS devices to respond to online surveys?– Exclusion?– Adjustment of survey contents, measurement

options, and implementation procedures? How?

Page 20: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Appendix

Page 21: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Identification of ´Mobile´

Mobile OS Desktop/Laptop OS

AndroidBlackberry OS

Apple iOSSymbian

Symbian OSWindows Phone OS

Chrome OSLinux

Apple OS XWindows 2000

Windows 7Windows 98Windows NT

Windows Server 2003Windows VistaWindows XP

unknown

Page 22: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 1: Results: Flash & Drop-Out Rates

Mobile OS Desktop/Laptop OS

Without Flash(3 studies)

7.2%(5.3/9.7)

3.2%(1.9/4.4)

With Flash(4 studies)

27.7%(19.1/36.2)

14.1%(12.5/15.6)

Page 23: Online Survey Participation via Mobile Devices -  · PDF fileOnline Survey Participation via Mobile Devices ... • Interviews among mobile phone users ... questionnaires (n=587)

Study 2: Zero-inflated Poisson Regression Model