Migrating Surveys from Desktop to Mobile and Touch-based Interfaces

…where are we?

Dr. Bill MacElroyChairman, Socratic Technologies, Inc.

THE Market Research Technology Event

Las Vegas, NVMay, 2012

2

History of Survey Data Collection• Since the early days of survey work, the mode of data collection has had a tremendous impact on the

quality of data and the types of biases encountered

Creative Destruction (Schumpeter, 1942)Disruptive Innovation (Christensen, 2003)

3

Data Quality Effects• As technology becomes more embedded, it is creating a ubiquitous environment for higher degrees

of data quality and candor in data collection

• Lower the Perceived Human Presence the Greater Candor

4

Variables Influencing Dropout Rates in Web-based Surveys• Why are we interested in the effects of engagement and new interface opportunities?

• Design of surveys must take into account the interaction between burden and personal return variables

• The goal: to enhance salience

– Length of survey (both in terms of time to complete and number of questions)– Incentive (either total incentive offered as a prize package or the approximate value of the

incentive on an individual basis)

– Engagement Level lessens the perception of burden

• A combination of these factors influences the number and proportion of mid-survey abandoners (mid-terms)

5

Engagement Versus Other Survey Characteristics• Mid-terminates -- an indication of the point at which respondent fatigue, boredom or lack of perceived

value becomes "critical" – Note: Critical "pleasurable"

• As a rule of thumb, when surveys have a mid-terminate rate of more than 30% a post-hoc evaluation of factors leading to the problem is probably a prudent decision

• Findings from these studies indicated that no significant differences can be found across different geographies

Findings from 19 Web-based studies – All of the studies were with business-to-business, technology-related decision makers– Surveys included U.S., European and Asian respondents

(all surveys conducted in English)– The total number of respondents included in these surveys = 21,867

• Median sample size = 473

6

Number of Screens• The more screens/questions, the greater the number of the mid-terminates

– Surveys that exceed 30 screens/questions are predicted to exceed the maximum acceptable level of dropouts

R2 = 0.6953

0%

10%

20%

30%

40%

50%

60%

70%

80%

0 5 10 15 20 25 30 35 40 45

Number of Screens

Dro

po

ut

Ra

te

7

Total Average Time of Survey• Surveys that took more than 17.5 minutes led to predicted completion rates of less than 70%

R2 = 0.7968

0%

10%

20%

30%

40%

50%

0 5 10 15 20 25 30

Mean Time of Survey

Dro

pout

Rat

e

8

Time Tolerance for Online Surveys

0 5 10 15 20 25 30 35 40 45 500%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

2

4

6

8

10

12

14

Time in Minutes

% F

ind

ing

Acc

epta

ble

To

tal

Exp

ecte

d M

inu

tes

Acc

epta

ble

At 15 minutes, over 70% find a typical surveylength “acceptable.”

At 25 minutes,over 70% find surveylength “unacceptable.”

9

Known Value of the Incentive• A cash-equivalent value of only $5.00 would still leave a predicted 78% completion rate

– Once the value hit $22.00, the curve flattened noticeably– Increasing the individual incentive rate above a certain level does little to influence mid-terms

R2 = 0.6087

0%

5%

10%

15%

20%

25%

30%

$0.00 $20.00 $40.00 $60.00 $80.00 $100.00 $120.00

Known Incentive Amount

Dro

p O

ut

Ra

te

10

Incentive Rate Tolerances in Online Surveys

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 220%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0%

5%

10%

15%

20%

25%

30%

Incentive Rates in $

% F

ind

ing

Ra

te A

cc

ep

tab

le

Ma

rgin

al I

nc

rea

se

in A

cc

ep

tab

ility

Pe

r $

of

Inc

en

tiv

e

For a survey of 12-15 minutesa $5 incentive would beacceptable to 65%

Increasing the incentive to $10 would be acceptable to 85%

11

Engagement Factor• But engagement works to improve focus only up to a point…

• < 10 minutes, no sig. effect

• 17 to 42 minutes, improved completion rates

• > 45 minutes, no sig. effect

% Completes with and without Animation

0

10

20

30

40

50

60

100 90 80 70 60 50 40 30 20 10

% Completing (Once Started)

Du

rati

on

of

Su

rvey

(in

min

ute

s)

Zone of Effect

No Animation

With Animation

14% 20%

12

Engagement: Technology versus Engagement• New technologies tend to create a sense of excitement for survey takers due to the novelty of the

new medium– However, the novelty is wearing off more quickly as people become used to more-and-more

engaging technologies in their environment

• Sustaining engagementwill take more thantechnology…it willrequire content andinteractivity beyondthe current paradigmof just “asking questions”

The Next Generation of Data Collection Technology

14

Moving from the Keyboard/Mouse to Touchscreen• We are about to enter another period of rapid technology change

– The way people interact with surveys is about to shift radically as the Keyboard and Mouse paradigm (dominant since the inception of the personal computer revolution) is giving way to less manual input methods

15

Moving from the Keyboard/Mouse to Touchscreen

• Interface Challenges:– Size: Tablets versus SmartPhones– Speed: Click, Drag and Drop versus Activate, Define, Drag and Drop

16

Moving from the Keyboard/Mouse to Touchscreen

• Interface Challenges:– Accuracy: Mouse Pointer versus Stylus versus Finger

• Finger Structures– Non-structured Entry: Typing versus Virtual Keyboards/Writing Recognition

17

Moving from the Keyboard/Mouse to Touchscreen

• Interface Improvements:– Technical Engagement– Greater Flexibility to Customize Interface (Grow, Hide, Organize, Arrange)– Promise of Improved Voice-controlled Input

18

Moving from the Keyboard/Mouse to Touchscreen• Biggest Challenge: User Experience

• Bridging Mechanisms:– Demos– Instructions– Diagrams

19

Example: Scalar Ratings• Demo

20

Example: Sorting• Demo

21

Example Complex Interactions• Tools versus Direct Manipulation

– Grow– Rotate– Review/Swipe– Arrange

• Demo

22

Research-on-Research

23

Background• Since the late 1990's, "mobile technology" has been predicted to significantly overtake the PC as the

dominant data collection mode– Panels consisting of mobile users have been recruited and have been "survey ready" for years

(most of these have not been profitable)

• Software (and apps) for conducting research have been available since the first smartphones and tablets

• So why aren't more research agencies pushing the mobile platform for new, geo-locational and event-based research? Some observations:– Research companies, by nature, are not early adopters– Many companies have had poor to mixed results using mobile platforms; clients don’t want to

pay to experiment– Limited screen space lessens ability for complex questions– Mobile technology is still in "flux" meaning that surveys have to be programmed at the "lowest

common denominator" for cross-platform compatibility• Lack of Flash or other similar application can limit survey interactivity and engagement

– Researcher's doubt the "convenience" factor is really at play in driving surveys to mobile– Panelists have to pay for data transmission vs. "free" cost of internet connection

24

Hypotheses• Research Questions…

– Is mobile perceived to be more convenient? • Wouldn't people actually prefer to do survey via mobile?

– Is the mobile experience really more engaging, interesting, timely and user time friendly?– Does research "in the moment" produce higher quality data?

• For this conference, we looked at the results from a number of studies…– Dunkin’ Youth Panel – Mobile (previously presented in public)– IIR Mobile Evaluations (8 Studies)– TMRTE Mobile Research -- designed specially for discussion at this conference

25

Dunkin’ Youth Panel – Mobile • (Fielded June 29th to August 30th 2007)• Dunkin’ Youth Panel Recruitment Project

– Targeted 16-24 year olds in the Boston area• Part 1: Campaign to get folks to take a 1-question satisfaction survey on mobile device and provide email

address for follow-up survey OR go to a listed Website• Part 2: Participate in a follow-up survey

• Part 1:– Total Mobile Respondents958 (took more than a month)

• % Providing Name 73%• % Providing Age 72%• % Providing Email 69%• % Providing Rating 70%

• Part 2:– Socratic sent follow-up emails to all providing emails and got 34 responses in return– None opted to join panel

• Web-based Comparison– Those going to the Web site produced hundreds of survey takers– 85% of those responding via Web volunteered to join Dunkin' VIP panel– Cost of responses by mobile were 14.6X that of recruitment to Web

26

IIR Mobile Evaluations• July 2011 and continued through Dec 2011

– IIR interested in moving away from paper questionnaires toward real-time evaluations– 8 conferences– 325 completes from a total of 2,556 attendees (13%)

• Stats…– 85% on average completed following conference vs. completing over the course of the

conference– Respondents rated an average of 11 speakers (range was 26 to 181 possible speakers)

• In general, less than half of reasonable number of evaluations were completed…

– Twice as many done via PC• 221 of 325 completed on computer (8.8%)• 104 on mobile device (4%)

27

TMRTE Mobile Research• January 30, 2012 and continued through February 10, 2012

• Project to compare response rates, survey stats and satisfaction ratings between modes

• First step: Screener to identify owners of computers and smartphones/tablets with internet connectivity – n=1,458 completes– Total invites sent: 24,400

• Invites sent before 2/9 = 11,664• Invites sent after 2/9 or later = 12,736 (sent additional invites to target and reach quotas for

smartphone/tablet group – main survey)– Mean survey length = 1.6 minutes

28

TMRTE Mobile Research Main Survey • Recruitment

– Much easier to get people to do survey work on PCs versus Mobile Devices– 87% indicated preference for computer, 13% for tablet/smartphone

*98% completed on computer when instructed to use smartphone/tablet, 2% used smartphone/tablet instead of PC

Fielding Stats  Total Invites 779Total Completes 240Computer Completes 120Tablet/Smartphone Completes 120Total Invites - Computer 159Total Invites - Smartphone/tablet 620Mode violators* 177

29

Speed & Quality• PC users were able to complete up to 20% faster than Smartphone/Tablet users

• Both modes answered open-ended questions equally well

• However, a few Mobile users wrote a LOT more than average

Mean Survey LengthComputer 4.8 minsSmartphone/Tablet 5.3 mins

Open End Responses  Computer 119 of 120 Smartphone / Tablet 117 of 120

OE Character Length (includes spaces) Computer Smartphone / Tablet

Mean 57 61

Mode 42 22

Median 43 45

Max 289 382

Min 0 0

30

Demographics• Demographically, the samples were not significantly different for gender or age

Demos        Total Computer Smartphone/Tablet   240 120 120Males 55% 53% 58%Females 45% 48% 43%16-24 4% 3% 5%25-34 19% 22% 16%35-44 27% 28% 26%45-54 23% 24% 22%55-65 22% 18% 25%66+ 6% 6% 7%Mean 44 44 44

31

Demographics• However, education and income levels did vary significantly; lower income and education responded

via PC-based survey

  Total ComputerSmartphone/

Tablet   240 120 120

Completed some college or tech school 26% 30% 22%COLLEGE DEGREE AND HIGHER (NET) 64% 58% 70%

Graduated from college 33% 32% 33%Some graduate work 10% 14% 6%

Obtained post-graduate degree 21% 12% 31%Mean income (thousands) 92.91 85.57 100.86

32

Survey Experience• Computer users reported generally higher levels of satisfaction and enjoyment

MeansTotal

ComputerSmartphone/

Tablet This survey was easy to complete 4.71 4.80 4.63

I was able to complete this survey quickly 4.68 4.73 4.63

Overall, I am satisfied with this survey experience 4.50 4.70 4.29

This survey was enjoyable 4.20 4.48 3.92

33

Online Activities• Interestingly, PC users are more likely to share thoughts via social media

*Twitter, LinkedIn, Facebook, Other

 Total

ComputerSmartphone/

Tablet

 Social Media Sharer (shared thoughts re: Bin Laden, Tsunami, Casey Anthony or various celebrity women via social media*)

240 120 120

Yes 40% 45% 34%

No 60% 55% 66%

34

Summary

• Research technology is again at an inflection point where innovations may fundamentally change the way we collect data

• Mobile and Tablet interfaces have not yet stabilized…users still have to "learn" to use surveys in these new modes

• The general survey taker still far prefers the PC to mobile platforms

• Most surveys on the PC are perceived to be:– Faster– Easier– Less burdensome– More interesting– "Better use of my time"

• Both interfaces and public experience will have to mature in coming years before Mobile becomes a preferred/acceptable mode of standard research data collection

Questions and Discussion

Dr. Bill MacElroyChairman, Socratic Technologies, Inc.2505 Mariposa StreetSan Francisco, CA 94110

415-430-2200bill.macelroy@sotech.com