Embed Size (px)
DESCRIPTION
Citation preview
March 28, 2012
MOBILE RESEARCH: IT’S JUST THE SAME, RIGHT? OR IS IT? Dispatches from the cutting edge of research on mobile research
Agenda
2
Lessons learned about who wants to do mobile research, when and why
Do you get the same data with mobile and desktop?
Touch vs. non-touch: is BlackBerry different than an Android or iPhone?
Drawing on 3 RoR Studies
3
All studies conducted in US, UK and Canada
Study 1: 4 arms 500 respondents each, Short Mobile (11Q), Medium Mobile (16Q), Long Mobile (21Q), Desktop (21Q), Aug. 2011. All smartphone (iPhone, Android) owners
Study 2: 2025 smartphone owners, given the choice of mobile or desktop, Oct 2011
Study 3: 1116 BlackBerry users, 902 iPhone or Android users, completing on their mobile March 2012
March 28, 2012
Lessons learned about who wants to do mobile research, when and why
Response Rates amongst smartphone owners where mobile was required for 3 of the 4 arms
The response to the invite was basically the same across all arms but the actual completion rate was much lower for the mobile arms
22% 23% 23%
2%
16% 15% 16%
39%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Short Medium Long Desktop
Responded to Invite but Dropped Out Successfully completed survey
38% 38% 39% 41%
Mobile Mobile Mobile
Where they fell by the wayside
40% 41% 41%
2%
17% 17% 17%
3%
41% 40% 40%
96%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Short Medium Long Desktop
Completed survey
Dropped out when asked to go mobile
Dropped out in survey
Tried to do survey on wrong device and gave up
Only 4 in 10 of those who responded to the invite to do the study on a mobile device actually completed it. Many in the mobile arms tried to do it on a desktop, despite our instructions, and then gave up. Most of the others who were non compliant just gave up after being asked to go mobile.
Length and Dropping Out during the Survey
4% 5% 4% 0%
Short Medium Long Desktop
% dropped out in survey
Length had no effect on the % of people who dropped out mid-survey, at least not up to the 9 ½ minute mark
(4 min) (6 min) (9.5 min) (6 min)
Mobile
In a subsequent study of smartphone owners, we gave them a choice of mobile or desktop
8
• Within the invitation respondents were informed that they could use their smart phone to complete.
89%
11%
11% of respondents used their mobile phone
Desktop Mobile
Where they answered, given the choice
9
Amongst those smartphone owners who choose desktop, 97% answered the survey at home or at work.
Amongst those who choose mobile, 15% were answering either from transit or somewhere other than home or work.
When they answered, given the choice
10
Amongst those smartphone owners who choose desktop, 58% answered the survey the first day it launched
Amongst those who choose mobile, 88% answered it the first day.
Who answered mobile, when given the choice
11
45%
55%
48%
42%
10%
17%
35%
42%
29% 33% 32%
55%
45%
26%
47%
27%
16%
28%
48%
25%
35% 31%
Mobile Desktop
Those age 18-34, and women, were more likely to choose mobile
Who answered mobile, given the choice
12
Those smartphone owners who completed the study on a mobile device were more likely to be iPhone users and people who used their mobile for all sorts of things, like music, photos, surfing the internet, and games. In short, they are mobile aficionados.
It takes longer on mobile, but…
13
Study 2 took an average of 10 minutes to complete on a desktop and 15 minutes to complete on a mobile device. But it did not have any negative effect on drop-out rate or enjoyment of the survey.
14
“This survey was fun to complete”
50% 38%
Mobile Desktop
Agree strongly
Why mobile?
15
People chose to do research on their mobile when it is reaching them where they want, when they want and using the medium they want. Mobile should be a choice, not an either/or requirement. But for it to be a choice, the data you collect needs to be equivalent…
March 28, 2012
Do you get the same data with mobile and desktop?
Interface Matters
17
We know that the type of interface you use can make a difference. And the mobile interface is definitely different.
We have previously proven, in a number of studies, that our visual question types are more engaging. The result is better quality data, compared to traditional “radio button” questions. Depending upon the question type, we may also see better use of the entire scale, less flat lining and fewer “don’t know”.
Where they answered, given the choice
18
In Study 2, we gave people the option to choose mobile or desktop. And we asked a series of questions about the environment, poverty and other social issues.
We used a variety of different question types in the survey, to see if each question type was providing the same answers.
Because we know that those who choose mobile are different in some ways, in our analysis we controlled for differences in age, gender and type of mobile device owned between those who answered desktop vs. mobile.
Single choice Questions
19
The responses to these simple single choice questions were the same, despite a difference in orientation.
35%
29%
36%
38%
30%
32%
Increase
Remain the same
Decrease
Desktop Mobile
Multi-choice Questions
20
The responses to this multi-choice question were the same, despite the need for scrolling on the mobile version.
2%
19%
33%
37%
67%
70%
76%
75%
4%
18%
34%
37%
69%
74%
68%
75%
Bring your own cup when purchasing a beverage
Bring your own reusable bag when going shopping
Do not flush the toilet every time you use it
Put food scraps in the compost or green bin
Recycle bottles and cans, when you are not at home
Recycle paper and cardboard, when you are not at home
Reduce the amount of electricity you use
Take public transit, walk or bicycle, to reduce pollution
Desktop Mobile
Allocations
21
$25.60 $26.50
$17.00 $18.00
$26.20 $28.00
$26.90 $27.50
Mobile Desktop
Improving Public Education Reducing Poverty
Preserving the Environment Reducing Crime
Despite having quite different interfaces, there were no statistically significant differences in people’s allocations
Numeric Questions
22
15 16
21
28
15 16 20
26
Australia Canada UK USA
Mobile Desktop
There were no statistically significant differences between interfaces, in people’s estimates of the percent of people living in poverty
Visual Grid Single Choice Questions
23
There were no statistically significant differences between interfaces, in people’s attitudes toward poverty. This is notable because the interfaces are quite different, with one being a grid with a visual pattern reminder of previous answers and the other providing a written record.
Visual Grid Multi-Choice Questions
24
17% 9% 11%
25%
11% 8% 5% 8%
17% 9%
Greedy Resource Rich
Protect Environment
Wasteful Enviromental Innovators
Mobile Desktop
Descriptors for the UK
There were significant differences with the multi-choice grid question type. What tended to happen was that respondents choose more items with the mobile version. This possible difference needs to be explored further, as it could implications for use and data interpretation.
March 28, 2012
Touch vs. non-touch: is BlackBerry different than an iPhone?
BlackBerry vs. iPhone
26
There is something almost tribal about the BlackBerry vs. iPhone debate. We know that the profile of the users of each device differ.
But what about how they answer surveys? We have seen in Study 1 and 2 that, with our mobile interface, that touchscreen smart phones can deliver an enjoyable way to complete a survey. A survey that produces data that, with the probable exception of the multi-choice grid, is identical to that we gather from a computer-based survey. But what about a non-touch BlackBerry? Again, the interface is different…
When required in the survey invite to complete the study on their mobile, those using non-touch BlackBerries were less likely to go ahead and complete the study
27
57% 70%
43% 30%
0%
20%
40%
60%
80%
100%
iPhone/Android BlackBerry
Tried desktop/dropped out on mobile Completed on Mobile
Base is all those mobile device owners who opened the survey invite for Study 3
The 43% completion rate we see with the touch device owners is very similar to the 40% we saw in Study 1. Once they started the study, there were no differences in drop-out rate between touch and non-touch devices
Those who completed the survey on a non-touch BlackBerry tended to be less delighted than those who did it on a Touch device
“This survey was hard to complete.”
“This survey was fun to complete.”
“This survey was easy to complete.”
“This survey was more enjoyable than most.”
95% 89%
0%
50%
100%
Touch Non-‐touch
% agree
12% 16%
0%
50%
100%
Touch Non-‐touch
% agree
83% 72%
0%
50%
100%
Touch Non-‐touch
82% 74%
0%
50%
100%
Touch Non-‐touch
One reason for the slightly lower engagement and easy of use is likely that it took longer to complete the survey on a non-touch BlackBerry than on a Touch device
29
Median length of survey, in minutes
10.5
Touch Device
14.6
Non-‐Touch BlackBerry
Interface: Touch vs. Type
30
The fundamental interface difference that we are concerned with here is touch vs. typing.
There are touch BlackBerries, but our research has shown the interface behaves the same as other touch devices, be it iPhone or Android. So our focus is on touch vs. typing, because that’s where the interfaces differ. We know that there are significant differences in the age, gender and education of touch users vs. users of BlackBerries with a physical keyboard—differences that influence people’s answers. So we controlled for these age, gender and education differences in our analysis.
Touch vs. Type: For the most part, the interface looks fairly similar. It is how you navigate it that differs more
31
Single Choice Questions
Multi-choice Questions
Allocations
Numeric Questions
Visual Grid Single Choice Questions
Touch vs. Type: Our analysis revealed that there was no pattern of significant differences between touch and non-touch smartphones for…
32
There was, however, no clear pattern of difference. One confounding factor may be the fact these questions were about people’s image of different countries. There is variation in ownership of the types of touch and non-touch mobile devices by country. We attempted to control for that statistically but may not have been fully successful. Or there may be real differences by device. Further research is needed to confirm or deny these hypotheses.
Where we did see a difference between Touch and Non-Touch was the Multi-choice Grid
33
Conclusions
34
March 28, 2012
Lessons learned about who wants to do mobile research, when and why People chose to do research on their mobile when it is reaching them where they want, when they want and using the medium they want. Mobile should be a choice, not an either/or requirement. But for it to be a choice, the data you collect needs to be equivalent…
March 28, 2012
Do you get the same data with mobile and desktop? We have proven that our mobile touch-screen interface delivers the same data as our desktop computer interface, and is enjoyable and engaging. The one question type that may be different is the multi-choice grid. Further study is required, but until we clarify that, it should be kept in mind that the mobile interface for multi-choice questions may generate slightly more associations.
March 28, 2012
Touch vs. non-touch: is BlackBerry different than an iPhone? While the BlackBerry non-touch interface performed well, people were slightly less enamored with it, confirming that iPhones are simply more fun.* The data collected was equivalent to the Touch interface and the desktop interface, with the possible exception of the multi-choice grid. Further study is needed on this point. * Yes, I have an iPhone, and used to be a BlackBerry user
Mobile research is a new frontier and we need to explore it carefully before plunging ahead. Our research suggests that well designed interfaces can provide an engaging experience for respondents and allow them to do surveys where they want and when they want. We have seen that you cannot just assume equivalence of data when using different interfaces—you need proof. Further research will be done as we explore the frontiers of mobile research
Conclusion
38
Questions?
39
