Gamification of On-line Surveys for Public Health Data Collection

by

Corey P. J. Alexander

A Thesis

Presented to

The University of Guelph

In partial fulfilment of requirements

for the degree of

Masters of Science

in

Computer Science

Guelph, Ontario, Canada

c© Corey P. J. Alexander, January, 2017

ABSTRACT

Gamification of Online Surveys for Public Health Data

Collection

Corey P. J. Alexander Advisor:

University of Guelph, 2016 Dr. Daniel Gillis

Standard survey methods have many issues associated with them, especially when

they contain a large number of questions, or take a long time to complete. These

problems come in the form of user fatigue, a lack of interest in the survey, or inaccurate

and incomplete answers. We hypothesized that the use of badging - a self element

of gamification - would increase completion rates, increase time spent on the survey,

and have no effect on straight-lining behaviour. This research describes the specific

tools that have been developed to investigate the use of Gamification (specifically

badging) and answer these hypotheses. Results indicated that badging had no effect

on completion rates, although this could be an artifact of participant selection bias.

However, the study does suggest that gamification, specifically badging, increases the

time a user spends on the survey.

iii

DEDICATION

For CPJA, because you know.

iv

ACKNOWLEDGEMENTS

I would like to thank my advisor Dr. Daniel Gillis of the School of ComputerScience at the University of Guelph. I view Dr. Gillis as a mentor and a friend.I would like to thank him for the wisdom and guidance that he has provided mewhenever I needed it, not only in my research and academics but also life in general.I am grateful to have had the pleasure to work with and learn from Dr. Gillis. Thankyou.

I would also like to acknowledge Dr. Judi McCuaig of the School of ComputerScience at the University of Guelph as the second reader of this thesis. Dr. McCuaig,much like Dr. Gillis, was always there to support me and guide me in my researchand academics but also in my life. Thank you.

Without the support and guidance of both Dr. Daniel Gillis and Dr. Judi McCuaig,I would not have made it to where I am today. I would like to sincerely thank bothof them for all the support they have given me during my time with them.

I would like to thank Dr. Stacey Scott for being my external examiner and for thevaluable comments she has provided me.

I would like to thank Colin Howes and Dominic Gagne for their help with buildingparts of the experiment. It was a learning experience for all of us and I wouldn’t havemade it through as easily as I did without their help and dedication.

Finally, I would like to thank all of my friends and family who have supported meduring my degree.

v

Table of Contents

Dedication iii

Acknowledgements iv

List of Tables vii

List of Figures viii

1 Introduction 1

2 Literature Review 32.1 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.1 Self Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3.2 Social Elements . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3.3 Branching Elements . . . . . . . . . . . . . . . . . . . . . . . 142.3.4 Flow Elements . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3.5 Pros and Cons of Gamification . . . . . . . . . . . . . . . . . 15

2.4 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 Methods 193.1 Recruitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2 Standard Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3 Badge Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.1 Earning a Badge . . . . . . . . . . . . . . . . . . . . . . . . . 263.4 Building the Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4.1 Design of the System . . . . . . . . . . . . . . . . . . . . . . . 283.4.2 Design of the Database . . . . . . . . . . . . . . . . . . . . . . 283.4.3 Design of the Front End . . . . . . . . . . . . . . . . . . . . . 31

3.5 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

vi

3.6 Why are we collecting these data? . . . . . . . . . . . . . . . . . . . . 353.7 Delta Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.8 Relative Delta Times . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4 Results and Discussion 414.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.1.1 Results by Question Category . . . . . . . . . . . . . . . . . . 444.1.2 Results by Survey Question Type . . . . . . . . . . . . . . . . 464.1.3 Results by Survey Question Length . . . . . . . . . . . . . . . 494.1.4 Results by Survey Question . . . . . . . . . . . . . . . . . . . 50

4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5 Conclusion 585.1 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

A Ethics Documents 68A.1 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

A.1.1 Ethics Approval . . . . . . . . . . . . . . . . . . . . . . . . . . 68A.1.2 Ethics Letter Of Information & Consent Form . . . . . . . . . 70

A.2 Survey Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

B Code & Further Tables 81B.1 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81B.2 Category Code to Category Labels . . . . . . . . . . . . . . . . . . . 81B.3 Wilcoxon Rank Sum Results By Question . . . . . . . . . . . . . . . 83

vii

List of Tables

4.1 Badge Survey Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.2 Non-Badge Survey Metrics . . . . . . . . . . . . . . . . . . . . . . . . 444.3 Wilcoxon Rank Sum Results for Category Delta Timers . . . . . . . . 464.4 Wilcoxon Rank Sum Results for Category Relative Delta Timers . . . 474.5 Ordered Category & Median Question Times . . . . . . . . . . . . . . 484.6 Wilcoxon Rank Sum Results for Answer Type Delta Timers . . . . . 494.7 Wilcoxon Rank Sum Results for Answer Type Relative Delta Timers 504.8 Wilcoxon Rank Sum Results for Category Word Category Delta Timers 514.9 Wilcoxon Rank Sum Results for Category Word Category Relative

Delta Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

B.1 Lookup for the Category ID and the Name of the Category. . . . . . 82B.2 Wilcoxon Rank Sum Results for Question Delta Timers . . . . . . . . 85B.3 Wilcoxon Rank Sum Results for Question Relative Delta Timers . . . 88

viii

List of Figures

3.1 The standard survey. . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.2 An image of the badge survey, without badges. . . . . . . . . . . . . . 253.3 An image of the badge survey, with 1 earned badge. . . . . . . . . . . 263.4 Database visualization. . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1 Plot of time to complete survey. . . . . . . . . . . . . . . . . . . . . . 56

A.1 Ethics Approval Letter . . . . . . . . . . . . . . . . . . . . . . . . . . 69A.2 Letter of Information & Consent Form, Page 1 . . . . . . . . . . . . . 71A.3 Letter of Information & Consent Form, Page 2 . . . . . . . . . . . . . 72A.4 Letter of Information & Consent Form, Page 3 . . . . . . . . . . . . . 73A.5 Survey Questions & Answers, Page 1-2 . . . . . . . . . . . . . . . . . 75A.6 Survey Questions & Answers, Page 3-4 . . . . . . . . . . . . . . . . . 76A.7 Survey Questions & Answers, Page 5-6 . . . . . . . . . . . . . . . . . 77A.8 Survey Questions & Answers, Page 7-8 . . . . . . . . . . . . . . . . . 78A.9 Survey Questions & Answers, Page 9-10 . . . . . . . . . . . . . . . . 79A.10 Survey Questions & Answers, Page 11-12 . . . . . . . . . . . . . . . . 80

1

Chapter 1

Introduction

In 2013, the Public Health Agency of Canada (PHAC) began designing Foodbook

- a survey created by The Outbreak Management Division & Enteric Surveillance

and Population Studies Division in the Centre for Foodborne, Environmental, and

Zoonotic Infectious Disease - to collect data on Canadian food consumption habits

to better understand risks of food borne infections. To collect these data, the PHAC

opted to use a phone survey in place of other survey methods (e.g. pen and pa-

per or on-line data collection tools). However, recognizing the increasing effort and

growing costs of conducting a national survey, and the added challenges related to

data collection using pen and paper, and phone surveys, including for example lack

of user engagement (Harms et al., 2014; Puleston, 2014), premature survey termina-

tion (Downes-Le Guin et al., 2012) as well as the transition from traditional landline

phones to computers, tablets, hand held devices, and smart phones (Alfi, 2011; Dug-

2

gan, 2013), there is a need to develop alternative data collection tools to mitigate

these challenges. This thesis will explore the development and evaluation of one such

tool that combines elements of Gamification within the standard on-line survey frame-

work. Specifically, this research will show that on-line public health surveys will

have improved user engagement by incorporating self-elements of gamification

through badging compared to non-badged surveys.

This document is structured as follows. To understand the challenges associated

with data collection particularly in the domain of Public Health, we begin with a

review of the literature in Chapter 2. This chapter will also explore the use of gam-

ification to improve user engagement. It ends with a review of the thesis statement,

as well as a list of specific hypotheses that will be explored in subsequent chapters.

To evaluate the hypotheses described, Chapter 3 will describe the implementation

of an on-line Public Health survey tool. The chapter will include a list of relevant

data collected, as well as a description of the experimental design and statistical tests

to be used. Data collected are analyzed in Chapter 4, with further discussion and

conclusions provided in Chapter 5. The chapter concludes with a list of potential

future work

3

Chapter 2

Literature Review

This research will show that on-line public health surveys will have improved

user engagement by incorporating self-elements of gamification through badg-

ing compared to non-badged surveys. We begin by reviewing some of the standard

issues with traditional and on-line surveys from the point of view of user experience,

data quality, and data quantity in Section 2.1. This section will explore underlying

issues with traditional surveys as a data collection tool, specifically looking at user

engagement (described in Section 2.2). Following this, in Section 2.3 we are going to

introduce the concept of Gamification as a tool to increase user engagement. Specif-

ically, we will define what Gamification means, describe the major design elements

related to it, the pros and cons of implementing it, and explain how it has been used

in various domains with a specific focus on the Public Health domain.

4

2.1 Surveys

In this section, we will discuss the current state of surveys as a data collection tool,

including the various issues associated with their use, and with a specific look at user

engagement and how it affects the collected data.

Currently, standard survey methods (including traditional methods using paper and

pencil or phone surveys such as those used to collect data for the PHAC’s Foodbook

project) face various issues. These issues include, but are not limited to, surveys being

uninteresting to participants, or time consuming to complete. In both situations,

this can lead to missing, unrepresentative, or incomplete data. Surveys viewed as

taking excessively long to complete or having a large number of questions, can also

affect the completion rates, defined as the number of people who finish the survey

over the number of people who started the survey, and response rates, defined as

the number of people who completed the survey over the number of people who

were contacted to take part in the survey (Cook, 2000; FluidSurveysTM, 2013, 2014;

Kviz, 1977; Mavletova, 2013). This happens as the participant begins to lose their

motivation or interest with the survey, which in turn also increases data missingness

as a result of a lack of engagement with participants, or premature termination of

the survey. That is, a lack of user engagement due to user fatigue and drop-outs

or participants straight-lining the survey, begins to affect the quality of the data

collected. Straight-lining is when a participant picks either the first answer or the

5

first answer they read, without consuming the information being presented to them

(Herzog and Bachman, 1981). Due to these issues, a lot of time and money are

required to gather a representative sample of the population (Greene et al., 2008;

Harms et al., 2014). This issue is especially significant because of the observed drop

in completion rates noted by Downes-Le Guin et al. (2012). Their findings suggest

that as completion rates continue to fall, the issues outlined above (particularly related

to data missingness), are growing.

To provide context, we consider two separate studies which suffered from very

low completion and response rates. Our first example, a study done on e-cigarette

cessation by Siegel et al. (2016) describes a completion rate for their survey of a mere

4.5% of the 222 respondents. While the paper does not explain why the authors saw

such low completion rates, it does point out that the low completion rate means that

the sample is not representative of all smokers who have tried e-cigarettes, limiting

the authors’ ability to generalize their findings. Whatever the cause, users were not

engaged sufficiently to complete the survey. A second example is a study by Downes-

Le Guin et al. (2012) which describes a response rate of 8% that was attributed in

part to the length of time it took to load the survey (or game) before being able

to begin. That is, respondents disengaged before the survey even loaded. Such low

rates for these (and many more) studies introduce many issues for statistical analyses

due to unrepresentative or missing data sets, making it harder to make accurate

observations, generalizations, and predictions about the topics being studied.

6

2.2 Engagement

While useful, traditional pen and paper or phone surveys are often lengthy, requir-

ing a lot of time to complete, or are simply uninteresting to the intended respondents

(Downes-Le Guin et al., 2012; Harms et al., 2014; Puleston, 2014). When implement-

ing a survey, designers are often faced with several key questions, including:

1. How does one get users interested in the survey? (Downes-Le Guin et al., 2012)

2. How does one keep the attention of survey participants? (Downes-Le Guin

et al., 2012; Galesic and Bosnjak, 2009; Kruat et al., 1975)

3. How can one design a survey to increase engagement and completion rates?

(Wright, 2005)

Ignoring these key questions, or not designing with them in mind can lead to

problems with the data that are collected, including increased costs, longer times to

implement the survey, incomplete data sets, issues with representation, and biases

(Greene et al., 2008; Harms et al., 2014; Siegel et al., 2016).

User engagement is defined as “a desirable – even essential – human response to

computer mediated activities” (O’Brien and Toms, 2008) and with regards to surveys,

it is how interested the participant is with the content of the survey (Puleston, 2014).

As O’Brien and Toms (2008) explain, “engaging experiences have the attention of

7

the user.” and point out that “[engagement] manifests itself in the form of attention,

intrinsic interest, curiosity, and motivation.”

On-line surveys are being used to help reduce some of the issues present in tra-

ditional surveys. However, on-line surveys still suffer from some of the same issues

present in traditional surveys, such as keeping participants motivated and interested

in the content of the survey until completion (Cook, 2000; FluidSurveysTM, 2013;

Kviz, 1977; Mavletova, 2013; Puleston, 2014).

As survey completion and response rates continue to drop, there needs to be explo-

ration of alternative data collection methods to increase user engagement, improve

completion rates, and thereby increasing the quality of data being collected. For the

purposes of this thesis, we define user engagement as how interested the participants

are with the content of the survey (Puleston, 2014).

To make on-line surveys more interesting and enjoyable, researchers have incor-

porated additional visual and interactive features within their on-line survey design,

including visually appealing surveys, animated survey components, and gamified sur-

veys (Downes-Le Guin et al., 2012; Leaman, 2014).

One study points out that making on-line surveys more enjoyable is an important

goal, since it helps to avoid negative respondent behaviours, such as premature ter-

mination (Cechanowicz et al., 2013; Harms et al., 2014). It also keeps users engaged

8

and increases completion rates (Harms et al., 2014). Recently, studies investigating

different methods to make on-line surveys more appealing have shown that Gami-

fication tools show promise in dealing with both user engagement and completion

rates of on-line surveys (Cechanowicz et al., 2013; Goehle, 2013; Hamari et al., 2014;

Leaman, 2014).

Researchers have used Gamification of on-line surveys/activities to increase user

engagement [and completion rates] in several domains. It has been successful in

the educational domain, where on-line homework was Gamified to increase user en-

gagement (Goehle, 2013). Specifically, on-line homework was gamified by adding

achievements that students could unlock throughout the course.

Gamification has also been used “in the real world of corporate learning” (Leaman,

2014), by adding small 15 minute learning games to be played at work, users become

more engaged, and there is an increase in user performance. Another study, used on-

line surveys for market research. Participants were asked to complete various tasks

related to market research, such as image identification, slogan matching, and five-

second memory quizzes (Cechanowicz et al., 2013). On-line surveys have also been

actively used in the public health domain, for investigations of infectious diseases,

like the H1N1 virus (Bults et al., 2011), e-cigarette satisfaction (Siegel et al., 2016),

and substance abuse among young adults (Bachman et al., 2006).

9

This section has shown how researchers have explored user engagement to improve

data collection activities. In many cases, researchers have used Gamification as a

means to improve engagement. The following section will explore Gamification in

further detail.

2.3 Gamification

Gamification is a tool that has been used in numerous disciplines and across mul-

tiple applications to improve user engagement. As such, this section will introduce

the concept of Gamification; explaining the major design elements of it, as well as

the pros and cons of implementing it, and showing how it has been used in various

domains including Public Health.

One definition of Gamification includes the act of taking a task that is to be

completed and making it into a game (Goehle, 2013; Leaman, 2014). As proposed

by Deterding et al. (2011), “Gamification is the use of game design elements in non-

game contexts.” However, Goehle (2013) is more specific, stating that “Gamification

refers to the use of video game mechanics and techniques to increase engagement

and interest in an activity which is, usually, unrelated to video games.” The idea

behind adding various video game mechanics to on-line surveys is that it will make

the surveys more enjoyable or exciting for the respondents (Seaborn and Fels, 2015).

This idea has come about due to the popularity and success of video games, and their

ability to hold the attention of their users (Seaborn and Fels, 2015). For the purposes

10

of this review, the more specific definition of Gamification by Goehle (2013) will be

used.

Gamification has been used in various domains, including Education, Health and

Wellness, on-line Communities and Social Networks, as well as Crowdsourcing (Seaborn

and Fels, 2015). The literature survey done by Seaborn and Fels (2015) shows a broad

range of studies that have implemented various elements of Gamification, ranging

from points and badges, to leaderboards and in-game avatar progression. Looking

specifically at the studies related to Public Health and Wellness, 19 papers were iden-

tified by Seaborn and Fels (2015), with 59% of the papers indicating positive effects

for improving health and well-being while the other 41% were mixed. The papers fo-

cussed on 1) active video games, 2) behavioural modification, 3) rehabilitation games,

and 4) educational games. Although it has been noted that Gamification has been

used in the Public Health domain (such as the study by Cafazzo et al. (2012) which

looked at creating a gamified mobile application to help monitor blood glucose levels

in diabetics) and that surveys are being used to collect public health data or to assess

the effectiveness of gamified tools for public health, there is little evidence to suggest

that the combination of gaming elements within survey design has been investigated

(Johnson et al., 2016; Seaborn and Fels, 2015).

Given the challenges identified with a lack of user engagement within on-line sur-

veys, leading to low completion and response rates, and the effect these have on data

11

quality, and considering that Gamification has been used to improve engagement, it’s

worth exploring this as a potential tool to improve survey design.

To successfully employ Gamification within survey design, it’s important to consider

the four elements of game design. These elements are: self elements, social elements,

branching elements, and flow elements (Csikzentmihalyi, 1975; Hemley, 2012; Hsin-

Yuan Huang and Soman, 2013; Shin, 2006). Each are described in sections 2.3.1

- 2.3.4. For the purposes of this thesis, the focus will be on the self elements of

Gamification.

2.3.1 Self Elements

Self elements are design elements that get a user to compete with themselves (Hsin-

Yuan Huang and Soman, 2013). These elements include but are not limited to points,

badges, or time trials. An example of self elements for surveys would be earning points

or badges when completing a question or a category of questions (Hsin-Yuan Huang

and Soman, 2013).

The use of self elements through addition of points and badges has been successfully

implemented for both on-line based homework as well as market research surveys

(Cechanowicz et al., 2013; Goehle, 2013). The on-line homework study found that

most of the participants in the study actually kept track of the badges that they had

earned during the course of the experiment and actively tried to earn badges. Goehle

12

(2013) concludes that adding elements of Gamification to the on-line homework was

a success and increased the engagement of the users.

In comparison, Cechanowicz et al. (2013) used a bunch of different characteristics

of video games, including a point based system. They also looked at implementing

different levels of Gamification by using one or more gaming elements in their exper-

iments. A standard baseline survey was presented as well as a fully gamified version,

including self elements of Gamification. However, their experiments did not consider

the individual effect of the gamified elements, instead they were all presented at one

time. This poses an issue when trying to determine the effect that the self elements

used had on the participant. Though it should be pointed out that Cechanowicz

et al. (2013) concluded, “Our evaluation demonstrates that the motivational bene-

fits of games increase with the level of gamification, and that these benefits apply

regardless of age, gender, length of a panelist’s tenure, and game experience.” This

provides the impetus to explore the utility of self elements as a method to improve

the engagement of participants.

Various social media sites also use some type of a “scoring” system such as likes,

shares, and follows in order to get users more engaged on the sites (Birman, 2014).

This type of system can be seen as both a self element and social element of Gamifica-

tion because users compete with themselves and other users to earn more of each type

of score (e.g. likes, shares, follows). Currently, there is little evidence to suggest that

13

self elements have been used specifically in on-line Public Health surveys (Johnson

et al., 2016; Seaborn and Fels, 2015).

2.3.2 Social Elements

Social elements include methods of creating competition/interaction between users

(Hsin-Yuan Huang and Soman, 2013). Examples of social elements include leader-

boards, virtual goods, or cooperative goals. Social elements can be included in surveys

simply by giving the participant feedback on how they compare to others who have

completed the survey (Hsin-Yuan Huang and Soman, 2013). Leaderboards and other

on-line social elements have been included in on-line homework to show students how

they compare to others in a course (Goehle, 2013). Social elements have also been

used in a study that attempted to use crowd sourcing as a method to get participants

involved in pro-environmental activities. Virtual and Financial leaderboards were

used to create competition between the various participants and as Massung et al.

(2013) noted, “Quantitative results suggested that the participants near the top of

the Virtual and Financial leaderboards were actively competing for the top position,

and that this competitive aspect can provide extrinsic motivation to maintain en-

gagement with the app.” This shows that social elements of Gamification do have

the potential to increase user engagement. Social elements have been used in various

on-line surveys but not specifically in the domain of Public Health.

14

2.3.3 Branching Elements

Branching elements provide users with the ability to reach a goal along multiple

different pathways; they feel unique to the user, but all lead to the same result (Hsin-

Yuan Huang and Soman, 2013). To include branching elements in on-line surveys,

the user could be given the choice of different sections to complete, allowing them

to take different ‘paths’ to complete the survey, similar to interactive story telling

such as a choose your own adventure type of activity. To our knowledge, branching

elements have not been used in Public Health surveys.

2.3.4 Flow Elements

Flow elements immerse the users within the game (Hemley, 2012). In this way,

users lose track of time and are highly focused on their task (Hemley, 2012; Hsin-

Yuan Huang and Soman, 2013). Flow elements give the user clear goals, involving

a high degree of concentration for a task, or giving the user a sense of personal

control over the situation/activity (Hemley, 2012). Flow elements also give users new

motivations as the task progresses to help avoid them becoming bored or frustrated

by doing the same task at a specific level for a long time (Sy, 2010). The study

completed by Sussane et al. (2011) shows the use of Gamification in on-line idea

competitions. The elements added included the use of “game points, social points,

and leaderboards.” Although these elements fall into the other categories such as

self elements or social elements, the use of them can increase the user’s immersion

15

in the task at hand. Points were earned by the participants for interacting in the

system in various ways, like contributing ideas, writing comments, evaluating ideas,

or leaving other participants a message. After the competition, a survey was used

to collect information about the participants (demographics, general experience with

these types of competitions, and an evaluation of this competition.), this survey was

not gamified but was used to gather information about the participants feeling of

immersion in the activity. The data collected showed that the implementation of

these elements did in fact increase the participants enjoyment of the competition,

and they reported the feeling of time passing faster (Sussane et al., 2011). Flow

elements have not been used in Public Health surveys.

After having defined Gamification, and looking at some of the places that Gami-

fication has been successfully implemented, we will take a look at some of the pros

and cons for implementing elements of Gamification in a system, with a specific look

at on-line Public Health surveys.

2.3.5 Pros and Cons of Gamification

While researchers have begun to explore the use of Gamification in various domains

and across numerous applications, it is important to review some of the challenges and

opportunities that come with implementing Gamfication. In this section we explore

the pros and cons associated with Gamification, with a focus on survey design.

16

From the literature reviewed, it can be seen that the pros of applying Gamification

to on-line surveys are that it may increase the engagement of users, by giving users

different ways to measure their progress throughout the survey, as well as short term

goals to achieve while participating in the survey, leading to users having a feeling of

accomplishment throughout the survey (Downes-Le Guin et al., 2012; Goehle, 2013).

As well, Gamified surveys minimize any negative emotions or behaviours, such as

straight-lining or inaccurate data sets that may be encountered when completing

the survey (Bhaskaran, 2013; Hsin-Yuan Huang and Soman, 2013). Another pro

of Gamification of surveys, as Cechanowicz et al. (2013) points out is that, “The

motivational benefits of games increase with the level of gamification.” This suggests

that the more game-like a survey is, the more likely it is to have the participant

engaged and immersed in the survey.

Of course, Gamification also has a unique set of challenges and issues that must

be considered when trying to implement it within a system. Some of the cons to

be considered when gamifiying a survey are: structuring the survey in such a way

that it is a game can be difficult as some types of surveys lend themselves more

freely to be gamified than others (Goehle, 2013). Another con includes the increased

time and cost of setting up the survey. For example, Downes-Le Guin et al. (2012)

indicated that “The Gamified presentation consumed more than twice as many hours

(as well as significant subcontracted resources to create original artwork) to conceive

and design the game structure, narrative and artwork, and to program and iteratively

17

test” which greatly increases the cost and time of the survey.

2.4 Thesis Statement

Sections 2.1 and 2.2 explained how standard surveys are suffering, specifically

through a lack of user engagement which affects the quality of the data being col-

lected. From there, Section 2.3 introduced the concept of Gamification and showed

how it has been used in various domains to increase user engagement, leading to

better quality data being collected. With these sections in mind, this section will

propose that Gamification can be used to increase the user engagement of on-line

Public Health surveys and explain the steps needed to achieve this goal. It will also

outline a set of hypotheses that we seek to address in this thesis.

Public health surveys are not engaging which has led to reduced completion rates

and response rates, causing a decrease in data quality. Gamification at a broad level

has been applied to many other domains, such as marketing research, education,

and public health (in general) (Batterham, 2014; Cechanowicz et al., 2013; Hsin-

Yuan Huang and Soman, 2013). And with the show of promise for Gamification

tools in other domains for increasing user engagement, it becomes clear that there is

motivation to explore Gamification of on-line Public Health surveys to address some

or all of the issues identified in the previous sections. This research will show that

on-line public health surveys will have improved user engagement by incorporating

18

self-elements of gamification through badging compared to non-badged surveys.

To demonstrate this, this thesis will investigate the following hypotheses:

1. Gamification can be used to increase participant completion rates. More specif-

ically, the use of self elements (such as badges) will increase participants com-

pleting the survey.

2. The participant will spend more time on the gamified survey as opposed to the

non-gamified survey. More specifically, the use of self elements will engage the

participant and they will take more time to answer the survey.

3. There will be no difference between a gamified survey and a non-gamified survey

in terms of participants straight-lining the survey. More specifically, the partici-

pant is just as likely to straight-line a gamified survey as they are a non-gamified

survey.

By answering these hypotheses, we will show that the use of badges have an impact

on the participants’ engagement.

19

Chapter 3

Methods

The goal of this research is to investigate the use of Gamification’s self elements

(e.g. badges or points) by applying them to on-line Public Health surveys, thereby

leading to increased participant completion rates and increased time on survey, while

attempting to reduce participant straight-lining throughout the survey. It is hoped

that increasing engagement in this way will lead to the goal of increasing the quality

of data being collected. To achieve the goal of this research, it is necessary to satisfy

the following objectives:

• develop a standard survey, without any of the elements of Gamification included,

to act as the baseline for the experiment.

• extend the standard survey to incorporate self elements. This will be an exten-

sion to add badges to the survey.

20

• collect data using the various surveys.

• analyze the data, comparing the two different surveys.

The reason for building the platform was to allow access to gather the data needed

to measure engagement, such as the timers per question and the badges. It was

important to build the platform so that we could ensure that the surveys were the

exact same with the only difference being the badges. This allowed for more control

over the experiment and the data that was being gathered.

Each survey (badged, and non-badged, as described in detail in sections 3.2-3.3)

will be on the topic of food and food consumption habits, asking similar questions to

the food consumption habits survey that the PHAC conducted. This allows for the

data collected to potentially be compared in the future to those data collected by the

PHAC to determine if the quality and quantity of data collected through gamified

surveys differ from those collected through non-gamified surveys. Both the badged

and non-badged surveys will contain the same number of categories and the same

number of questions per category. The only difference between the surveys will be

that the badge survey will give out badges to the participant at random points during

the survey. This will be described in detail in section 3.3.1. For a complete list of

the survey categories, questions and answers presented to the respondents, refer to

Appendix B.1 and A.2.

21

The survey was designed to support three different question formats: a radio but-

ton, checkbox, or textfield. Radio button questions will typically be structured in the

following manner:

Q: Did you consume <food item> in the past 7 days?

a) Yes.

b) No.

c) Prefer not to answer.

d) I don’t know.

Checkbox questions will typically be structured in the following manner:

Q: Did you consume any of the following fresh or frozen <food item>?

� Strawberries

� Blueberries

� Raspberries

� Blackberries

� Other berries

� I didn’t eat any of these foods

� I don’t know

� I prefer not to answer

22

Textfield questions will typically be used in a situation where a radio button or

checkbox does not make sense, such as asking for the postal code. Due to the structure

of most questions, textfields have only been used in the Demographics category of

the surveys.

3.1 Recruitment

To recruit participants to the survey, we used a social media and e-mail campaign.

For social media we used Twitter, Facebook, and personal blogs to reach out to

potential participants to inform them that a survey was being conducted. E-mails

were also sent to potential participants, and shared on various list-serves.

The experiment was presented to potential participants as the “Alternative Data

Collection to Understand Food Behaviours” survey. Before beginning the survey,

participants were presented with a Letter of Information and Consent Form (see

Appendix A.1, Figures A.2-A.4). As described in the Letter of Information, users

were informed that there were two types of surveys (standard or badged) but they

would not know which survey type they would get until they began. That is, users

were aware that they could earn badges while completing the survey. It should also

be noted that there was a chance that a participant with the badged survey might

never earn a badge.

23

Several comments need to be outlined regarding the social media campaign and

selection bias. First, there could be the potential for bias in the selection of partic-

ipants since many of the respondents targeted through Facebook were largely made

up of other university students. Further, due to the nature of the e-mails being sent

to various university faculty members (from different universities as well) and list-

serves, there was a potential bias present as some of these participants might also

be researchers and thus more likely to complete the full survey. This could present

an issue when it comes to representativity within the experiment, and our ability to

generalize the findings.

3.2 Standard Survey

The Standard Survey (SS) will be designed to have 1 category per page, with 9

categories in total for the survey. The survey pages will make use of colour (i.e.

they will not be simply black and white), and will include navigation controls (e.g.

next or previous section), and the ability to leave at any time. Each section will

include a short description for the participant. There will be no badges included.

This survey is expected to have the lowest amount of questions or sections completed

and the lowest completion rate compared to the surveys incorporating self elements

of Gamification. It is expected that this type of survey will have a higher amount of

participants straight-lining the survey, due to a lower motivation to complete it since

there are no elements of Gamification present. Figure 3.1 shows an example of the

24

standard survey.

Figure 3.1: An image of the standard survey.

3.3 Badge Survey

The Badge Survey (BS) will be an extension of the SS that will incorporate a

badge system. The respondent will be rewarded with badges at random, this will be

discussed in section 3.3.1. By including badges into the survey, it is hypothesized that

participants will be more likely to complete the entire survey, since the participant

25

will want to continue earning badges (Birman, 2014). This version of the survey is

expected to have a higher number of questions answered, and a higher completion

rate compared to the SS. It is also expected that this type of survey will have a lower

amount of participants straight-lining the survey, due to being more motivated to

continue earning badges (Birman, 2014; Hamari, 2015). Figure 3.2 shows an example

of the badged survey with no badges added while Figure 3.3 shows an example of the

badged survey, with one badge earned.

Figure 3.2: An image of the badge survey, without badges.

26

Figure 3.3: An image of the badge survey, with 1 earned badge.

3.3.1 Earning a Badge

To determine if a participant should earn a badge while they are completing the

survey, a test similar to the FizzBuzz test (Ghory, 2014) is used. When a participant

completes a question, including changing their answer, a time stamp is taken. If

the time stamp is divisible by 20, which was chosen arbitrarily, the participant will

earn a badge. This means that a participant will now have to ability to earn at

least one badge per question if every time stamp taken happens to be divisible by

20. This also means that if the participant chooses to change an answer, they will

27

also have the chance of earning another badge, allowing the badges to be seen as

random. The decision to randomize the badge allocation was to try and discourage

participants from only completing the requirements to earn the next badge instead

of being focussed on the survey.

When a badge is earned, a modal window pops up that the participant can close,

or wait for a maximum of 3 seconds for the window to close itself. This becomes

important when looking at Hypothesis 2.

Due to the nature of every question having the ability to have the participant earn

a badge, badges will look the same for each individual category.

3.4 Building the Experiment

This section will describe how the system was designed and implemented, along

with the design of the database and the technologies that went into the project.

For this experiment, a survey system was designed and implemented. This system

was designed to be easily extensible and scalable, allowing for multiple surveys to be

hosted, requiring an extensive amount of time designing and planning the system.

The database has been designed to allow for multiple different questions, categories,

and surveys to be stored. Figure 3.4 shows the relationships between the various

tables that make up the database and will be discussed more in section 3.4.2.

28

3.4.1 Design of the System

The front end was designed using HTML5, AngularJS (JavaScript), and Bootstrap,

(Cascading Style Sheet). This allowed for the design to be easily extensible with

limited modifications needed to implement new things. The back end of the system

was designed using PHP and MySQLi (Changed to MySQLi because MySQL was

depreciated) and the PHPSlim Framework in order to create a simple yet scalable

API and database, allowing for the back end to be easily extended if there is new

data to be stored or new features are added to the system.

3.4.2 Design of the Database

As for storing the different pieces of information, everything has been split into

separate tables according to how the data is related, with foreign keys being used to

link the data together. For example, there is a table to store the information about a

question. This table does not store the information about the answer set or the link

set, instead storing only foreign keys to the respective tables.

Here, we describe how it all fits together, and how the surveys, categories, questions,

and answers are related.

A survey knows what categories are in it and what type of survey it is (Standard

or Badged). This allows for easy extension in the front end to include different types

29

Figure 3.4: An image of the database with relationships between the various tables.

of surveys if more elements of Gamification are added in future work.

A category is made up of questions, each with their own set of answers, that are

going to be displayed in the survey. The survey will display one category per page,

with all of the questions in the category being displayed at once.

A question is made up of multiple parts, including the question’s text, the answer

type (radio button, checkbox, textfield), the set of answers (answers the participant

30

can choose from as long as answer type is not a textfield), and a set of links (links

to other questions to help control the flow of the survey). The set of links has been

included to try to mimic the PHAC survey; when a participant answers a question,

depending on the answer other questions may or may not appear. For example, if a

participant answers that they have not eaten apples or apple products, the following

questions should not be related to apples or apple products. Currently, links are not

implemented but have been included in such a way that the surveys can easily be

extended to accommodate them.

There are also various pieces of information about the participant that are being

stored, such as their answers to the questions (only if they want this data to be

recorded, by completing the survey) and their measures (outlined in section 3.5).

Since there are no participant accounts, the participants’ measures can be stored

without their consent, due to the data being completely anonymous, with no way to

trace a set of measures back to a specific participant. For example, if information

about a participant’s measures were to be leaked, knowing that it took a participant,

with a numeric ID of x, t amount of time to complete question y, does not give

information about who that participant was.

The various measures that are being collected in relation to a participant are also

stored in their own respective tables, such as a table for timers and badges. By having

these all separated into their own tables, it is very easy to extend the measures that

31

are being stored, by having a table that stores foreign keys to each of the other

measures tables and a foreign key to link to a participant.

With the consent of the participant (by completing the survey and pressing ‘Submit

Survey’), their answers are being stored in the database as well. The format for the

participants’ answers depends on the given answer type, since there are currently

three different types of answers (radio buttons, checkboxes, and textfields). If the

answer type is a radio button, a string that consists of multiple pairs of values, the

question id and the index in the answer set array for the selected answer will be

stored. For a checkbox, an array of true/false, that indicate which answers were

selected, will be stored in place of the index value listed above. And for a textfield,

the text submitted will be stored. Since the questions know which set of answers are

linked to them, it will be easy to look up the participants’ answer.

3.4.3 Design of the Front End

A participant is brought to the landing page, on which the information about con-

sent for the survey is displayed and is presented with the option to accept these terms

and begin the survey or to deny them/leave the survey. Once the participant begins

the survey, multiple timers will be started to record the time it takes to complete a

question, category, and the survey. When a question is answered, a time is recorded.

If the participant goes back to change an answer, the timer will be updated with a

32

new time, with the same holding true for the categories. After the participant sub-

mits a survey, the time is recorded and there will be no way for them to go back and

change their answers.

While going through the survey, the participant is not allowed to move onto another

category without answering all questions in the section. However, the questions can

be answered in any order within a category. The order can then be determined simply

by ordering the times observed. The surveys will also include a button allowing the

participant to leave at any time throughout the survey. This allows the participant

to stop doing the survey at any time if they decide to no longer participate. If the

participant closes their browser, the measures will be stored. That is, their timing

data, or number of badges if applicable, will be stored. However, their specific answers

will not be stored. If the participant loses their internet connection during the survey,

the times for the completed categories will be stored and times for the current category

will be lost.

At any point when the participant attempts to leave, the participant’s measures

will be saved. The only time the participant’s answers will be saved is if they complete

the survey and press the “Submit Survey” button. Upon completion, the participant

will be shown a thank you page for each survey type.

Other pages of the website include:

33

• A page thanking the participant for participating, and

• A page informing participants that they cannot participate without agreeing to

the consent form.

3.5 Data Collection

This section will outline the data that are necessary to investigate the hypotheses

outlined in this thesis, as well as other data that are being collected that could be

useful in answering potential questions of interest. The data that will be collected

to help answer the questions outlined in Section 2.4 are the following, and will be

discussed in full detail in Section 3.6.

• Time spent on a question (in milliseconds),

• Time spent on a category (in milliseconds),

• Time spent on the survey (in milliseconds),

• The deltas of the timers collected,

• The relative delta of the timers collected,

• Number of questions completed,

• Number of questions per category completed,

• Number of categories completed,

34

• The number of people that started the survey,

• The number of people that came to the site,

• The number of people that did not take part in the survey, and

• The number of people that completed the survey.

In addition to these data, the following information will be collected to help answer

other potential questions of interest:

• Basic demographics (e.g. age, gender, education, and income levels), and

• The answers provided by the respondent.

The respondents were required to answer the questions in each of the categories

in the following order: Demographics, Food Habits, Food Insecurity, Fruits, Vegeta-

bles, Meat and Alternatives, Dairy and Alternatives, Grains and Alternatives, Other

Foods. Participants were asked demographic questions at the beginning of the survey

to try to eliminate user fatigue that may have occurred if demographics were located

at the end of the survey. While this does not have an impact on the research question

describe by this thesis, it becomes important for future work where it is necessary to

compare the survey results (i.e. the participant responses to the food survey) with

the survey results collected by the PHAC.

35

3.6 Why are we collecting these data?

It should be noted that although basic demographics are being collected, and they

could be used to separate the data collected by different user groups, like age or

gender, this thesis is looking specifically at the use of the self elements at increasing

overall user engagement. As such, breaking the data up by user groups (although

interesting) is outside the scope of this thesis. This is because we are looking at how

the self elements affect all participants and not individual sub-groups. However, the

data is being collected to allow for further investigation to be done in the future.

Various timers will be collected from the participants and used in multiple ways.

The timers will be used to determine if a participant has straight-lined one or more

categories, or the entire survey, and the order that a participant has answered the

questions. The timers being recorded include: the time spent on a question (tquestion),

the time spent on a category (tcategory), and the time spent on a survey (tsurvey). These

timers are easily scalable for multiple surveys. From here, each of the types of timers

(question, category, survey) had the delta and relative deltas calculated. For the

deltas, the equation used was tx+1 - tx where x was either question, category, or

survey, and for the relative deltas the equation used was (tx+1 - tx)/tx+1, again where

x was either a question, category, or survey. A Wilcoxon Rank Sum test can be used

to determine if there is a difference between the surveys when it comes to the delta

timers and relative deltas. To determine if straight-lining has occurred, it would be

36

expected that the timers would be relatively small and that there may be multiple

timer groups within a survey; a group of very small timers, showing participants

completing the survey quickly, and a group of big timers, showing participants are

taking more time to complete the survey. It is currently assumed that participants

completing the survey rather quickly may not be as engaged in the content of the

survey as those taking more time are. It is assumed that if participants are taking

longer to complete questions or categories this could be an indicator of the participant

being more engaged during the survey.

The number of people that came to the website will be a non-negative value, that

does not represent the actual amount of people that have come to the website but

instead represents the maximum people that could have come to the website. This

value could be inflated due to people refreshing the website, or accidentally closing

the page and having to come back to the site. The first problem is more difficult to

deal with since it is hard to figure out if a participant is refreshing the page but the

website has been equipped with the ability to stop a participant from accidentally

navigating away.

The number of people that have clicked the deny button, denying/ refusing to

accept the terms of consent, will also be a non-negative value. it does not represent

the actual amount of people that have denied/ refused to accept the terms of consent

and left the survey but will be a lower bound on how many participants have not

37

participated. This is because people do not have to click on the deny button for them

to leave the survey. From here an approximation of the amount of people who did not

participate in the survey can be calculated by removing the people who started the

survey from the amount of people who came to the site. This will give the equation

pnotParticipate = pcame - pstarted, where pnotParticipate is the number of people who did

not participate in the survey, pcame is the number of people who came to the website,

and pstarted is the number of people who started the survey. This will only be an

approximation because the number of people who came to the site is not a perfect

representation of the actual number of people who came to the site; the same holds

true for the people who started the survey.

The number of participants that have completed they survey will be the number of

answer sets that are stored, since the only way to store an answer set is to complete

the survey. This piece of data will be used to help determine if there is statistical

evidence to prove that the completion rates of the surveys that have been implemented

are different from one another and if the difference is due to the use of Gamification.

The timers will help to determine how far into the surveys the participants are

getting before quitting or before a noticeable amount of straight-lining or termination

has occurred. It is expected that if straight-lining has occurred, the timing data that

was collected will begin to speed up, while if termination has occurred, the timing

data will stop at a specific question.

38

The participant has multiple methods of leaving the survey; clicking on the exit the

survey button, or by completing the survey. If a participant selects the first option,

the only data that is stored will be the measures of the participant. From here it

is possible to get information on how many questions the participant completed, the

order of the questions and how long it took to complete them. If the participant uses

the second option (completing the survey) to leave, they will also be given the option

to save their answers or not.

Each of the questions will be split into one of three arbitrary categories, based on

the length of both the question and answer text, or the number of words contained

in both. Here, answer text refers to the length of prescribed answers such as those

provided in radio button or checkbox style questions. This excludes answers where the

participant would input using a textfield which which is only used in one question in

the demographics section (i.e. Enter your postal code). The categories are as follows,

short word length; 0 words inclusive to 25 words exclusive, medium word length; 25

words inclusive to 35 words exclusive, and long word length; 35 words inclusive to 50

words inclusive (since the longest question and answer text was 49 words).

The word length categories can be used to determine if participants are more en-

gaged during a specific word length category, since both surveys have the same ques-

tions and therefore the same word length categories, it would be a good indicator of

participants being more engaged for the badged survey or the non-badged survey.

39

3.7 Delta Times

Delta times are calculated using the formula: tx+1 - tx where x represents a question.

Delta times are used to determine the total length of time it takes a participant to

answer a particular question. For example, if the participant answered the first 4

questions of the survey in 10s, 22s, 35s, 50s, respectively, the delta times would be

10s, 12s, 13s, 15s. The delta times were calculated for every question and these were

used to determine delta times for answer type, question word length, and category.

For example, to determine the delta time for the radio button answer type, we simply

summed up the delta times for each question that used this answer type. That is, for

each participant we calculated the delta time per question, their delta time for each

of the 3 answer types, the delta time for each of the 3 question word lengths, and the

delta time for each of the 9 categories. We did this for each participant who answered

the badged survey and the non-badged survey providing us with a distribution of

results that could be compared between the two survey types.

3.8 Relative Delta Times

The relative delta times show the relative difference in time taken to complete two

questions. The formula for the relative delta times is (tx+1 - tx)/tx+1, again where x

represents a question. The relative delta times were calculated for each question and

summed in the same way as described in the previous section for each of the answer

40

types, question word lengths, and categories. The relative delta times are useful

because they allow a comparison between participants to see if there is a difference,

relatively, between the participants being compared. That is to say that if there is

a difference between two participants, the difference will be apparent when viewing

the relative delta times because there will be a huge difference in one or more of the

relative delta time values.

41

Chapter 4

Results and Discussion

Data collection occurred over the course of 4 weeks in March, 2016. Participants

were recruited through an email campaign, as well as through a social media cam-

paign that included Twitter, Facebook, and blog posts (e.g. https://danielgillis.

wordpress.com/2016/03/03/understanding-food-behaviours-survey/). In Sec-

tion 4.1 we provide summary and analyses of data collected during this period, fol-

lowed by a discussion of the findings in Section 4.2.

4.1 Results

During the course of the data collection period, the total number of visits to the

website was 457. This may represent 457 unique individuals but there was no way to

track this due to the use of social media as a method for recruitment. Of these, 111

people (roughly 24%) decided to take part in the survey, while 346 did not. Of the

42

Survey Started Completed Unfinished

Badge 54 45 9

NonBadge 57 46 11

Total 111 91 20

Table 4.1: Completion data collected for each type of survey given.

people who started a survey, 91 (82%) completed the survey, and 20 left the website

before finishing their survey. Table 4.1 breaks down further this information based on

the type of survey a user was randomly assigned; badged or non-badged. Specifically,

54 of the 111 people who started a survey were given a badge survey to complete,

while 57 were given a non-badged survey. Of those who started a badge survey, 45 of

them were able to finish the given survey leaving only 9 unfinished. Similarly, 46 of

57 people were able to complete a non-badged survey. That is, approximately 83%

and 81% of respondents completed the respective surveys.

Respondents who answered the non-badged survey spent an average of 553.53 sec-

onds on the survey, while badged respondents spent an average of 635.34 seconds.

That is, badged survey respondents spent on average 81.81 seconds longer answering

the survey. Median time spent on the surveys was 519.81 seconds and 553.53 seconds

for non-badged and badged surveys, respectively. The difference in medians is 33.72

seconds.

43

By design, respondents who were randomly assigned a badged survey received a

random number of badges. Table 4.2 outlines some basic statistics describing the dis-

tribution of badges received. The maximum number of badges given to a participant

during the experiment was 13, the minimum number of badges given was 0, and the

median number of badges that participants earned was 5. Table 4.2 also shows the

maximum estimated time that earning a badge could add to a participant’s survey

completion time if they were to allow the badge pop up to stay on screen for the full 3

seconds. Recall that the badge pop ups were designed to appear for at most 3 seconds.

Based on this, the longest time added to the total survey completion time (assuming

the extra time is only related to the badge pop ups) was an extra 39 seconds. That is,

we’d expect that the badged survey would be on average 14.7 seconds longer, and no

more than 39 seconds longer than the non-badged survey, if the badges had no effect

on the time a user spends on the survey. Further, we’d expect the median difference

between survey completion times to be approximately 15 seconds.

A Wilcoxon Rank Sum test was preformed on each of the survey types to determine

if there was a difference between the time to complete each of the surveys. The results

of the test showed a p-value of 0.057. Although this p-value was not statistically

significant, further investigation was explored to determine if there were differences

based on different factors. Specifically, we analyzed the time to complete the survey

based on the survey categories, the different answer types, the length of text in each

44

Statistic Number of badges collected Estimated extra time (s)

Mean 4.9 14.7

Median 5.0 15.0

Maximum 13.0 39.0

Minimum 0.0 0.0

Table 4.2: Summary of badges collected by participants in the badged survey, andthe expected extra time these would add to the total survey if the user allowed thebadge pop up to remain on screen for 3 seconds per badge as designed.

question, and by question, to see if there were any differences between the time to

complete the two surveys for any of these factors. A Wilcoxon Rank Sum test was

performed for each of the factors. Wilcoxon Rank Sum tests were applied to both the

delta times, and the relative delta times for each of the factors. Results are provided

in Sections 4.1.1-4.1.4.

4.1.1 Results by Question Category

The first factor examined was the question categories. Wilcoxon Rank Sum tests

were performed to see if there was a significant difference between the time required

to complete each question category. The null hypothesis of each test was that there

was no significant difference between the non-badged and badged category times. A

full list of categories (including the number of questions per category) can be found

in Appendix B.1.

45

Tables 4.3 and 4.4 both show the results from the Wilcoxon Rank Sum tests that

were performed on the aggregated delta times and the relative delta times per cate-

gory, as well as the medians of both sets of times for both surveys. The tables also

include the p-values associated with each of the test statistics, where p-values that

are less than 0.05 represent statistically significant results and are indicated with bold

face font. It should be noted that in the delta times result set, the demographics and

food habits categories have p-values less than 0.05, where in the relative delta timers,

the demographics and food insecurity categories are both close to 0.05, with their

values being 0.065 and 0.078 respectively. Further, it should be noted that the me-

dian times for the badged survey are greater for every category than the non badged

survey, except for food insecurity category.

In the case of the significant results for the tests performed on the delta timers, the

badged survey took longer than the non-badged survey. In particular, badged survey

respondents spent approximately 20 seconds longer on the demographics category,

and 7 seconds longer on the food habits category questions than non-badged survey

respondents. That is, the first two categories presented to the respondents had median

timers that were significantly different between badged and non-badged surveys. In

both cases, the badged survey had a higher median timer.

Table 4.5 provides another view of the data. Here we divide the median time per

category for both badged and non-badged surveys by the number of questions in each

46

category. For both badged and non-badged surveys, the proportion of median time

spent per question seems to increase towards the end of the survey.

Category p-value Median B Median NB

Demographics 0.007 104.165 84.708

Food Habits 0.020 33.784 26.344

Food Insecurity 0.633 45.328 45.442

Fruits 0.145 58.378 56.043

Vegetables 0.086 62.625 59.543

Meat and Alternatives 0.336 78.868 77.833

Dairy and Alternatives 0.377 48.050 45.221

Grains and Alternatives 0.586 36.057 33.530

Other Foods 0.158 48.106 41.795

Table 4.3: Wilcoxon Rank Sum results for both surveys, of the delta times percategory. Where Median Badged and Median Non-Badged are the medians of eachcategory for the badge survey and non-badge survey respectively.

4.1.2 Results by Survey Question Type

The questions were then categorized by answer type: radio button, checkbox, or

textfield. In both surveys, the only textfield was for the postal code. All other

questions were either a radio button, or a checkbox, with 26 answers being radio

47

Category p-value Median B Median NB

Demographics 0.065 2.684 2.558

Food Habits 0.921 0.256 0.256

Food Insecurity 0.078 0.263 0.289

Fruits 0.853 0.265 0.268

Vegetables 0.805 0.214 0.221

Meat and Alternatives 0.412 0.218 0.230

Dairy and Alternatives 0.312 0.106 0.112

Grains and Alternatives 0.249 0.072 0.081

Other Foods 0.780 0.087 0.091

Table 4.4: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper category. Median Badged and Median Non-Badged are the medians of eachcategory for the badge survey and non-badge survey respectively.

48

Category A = Number Questions MedianBA

MedianNBA

Demographics 15 6.94 5.65

Food Habits 6 5.63 4.39

Food Insecurity 8 5.67 5.68

Fruits 5 11.68 11.21

Vegetables 7 8.95 8.51

Meat and Alternatives 9 8.76 8.65

Dairy and Alternatives 6 8.01 7.54

Grains and Alternatives 4 9.01 8.38

Other Foods 5 9.62 8.36

Table 4.5: Total number of questions per category (A), with median badge time (Me-dian B) and median non-badged time (Median NB) divided by number of questionsper category.

49

buttons and 38 answers being checkboxes. Again, we explored whether or not there

was a difference in the timers for each question type between the different surveys

using the Wilcoxon Rank Sum test.

Tables 4.6 and 4.7 both summarize the results from the Wilcoxon Rank Sum tests

that were performed on the various answer types that are used in the two surveys.

In the delta times, radio buttons (answer type 0) have a p-value less than 0.05, and

checkboxes (answer type 1) have a p-value that close the 0.05, at 0.075, while in the

relative deltas set, the radio buttons p-value is close to 0.05 (at 0.062).

AnswerType p-value Median B Median NB

0 0.034 142.722 132.446

1 0.075 395.225 347.945

2 0.335 8.627 8.488

Table 4.6: Wilcoxon Rank Sum results for both surveys, of the delta times per answertype (0 = Radio Button, 1 = Checkbox, 2 = Textfield). Where Median Badged andMedian Non-Badged are the medians of each answer type for the badge survey andnon-badge survey respectively.

4.1.3 Results by Survey Question Length

It should be noted that each of the questions were broken into 3 categories for word

length. The categories were short, [0 - 25), medium, [25 - 35), and long, [35, 50). The

question with the shortest text contained 11 words in both the question text and the

50

AnswerType p-value Median B Median NB

0 0.062 2.746 2.543

1 0.696 1.220 1.282

2 0.789 0.264 0.265

Table 4.7: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper answer type (0 = Radio Button, 1 = Checkbox, 2 = Textfield). Where MedianBadged and Median Non-Badged are the medians of each answer type for the badgesurvey and non-badge survey respectively.

answer text, while the question with the longest text contained 49 words in both the

question text and the answer text. The median word length for both question and

answer of the both surveys was 33 words.

Tables 4.8 and 4.9 sum up the results of the Wilcoxon Rank Sum tests of each

survey for the word length category of each question. For the delta times of the word

length categories, there is one result that is of interest, the questions that fell into the

short category ([0 - 25) total word length) have a p-value of 0.013, while the p-values

for the other categories are greater than 0.05. The relative deltas do not have any

p-values that are less than 0.05.

4.1.4 Results by Survey Question

We further investigated to determine if there was a significant difference in times

spent per question between the two surveys. Wilcoxon Rank Sum tests were per-

51

Word Category p-value Median B Median NB

1 0.013 75.146 59.389

2 0.121 247.014 235.212

3 0.103 225.088 212.318

Table 4.8: Wilcoxon Rank Sum results for both surveys, of the delta times per wordlength category (1 = [0 - 25) total words, 2 = [25 - 35) total words, 23= [35 - 50)total words). Where Median Badged and Median Non-Badged are the medians ofeach word length category for the badge survey and non-badge survey respectively.

Word Category p-value Median B Median NB

1 0.100 1.660 1.514

2 0.216 1.486 1.367

3 0.532 1.205 1.175

Table 4.9: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper word length category (1 = [0 - 25) total words, 2 = [25 - 35) total words, 23= [35- 50) total words). Where Median Badged and Median Non-Badged are the mediansof each word length category for the badge survey and non-badge survey respectively.

52

formed for each of the questions. A full summary of results can be found in Appendix

B.3.

Tables B.2 and B.3 show the results for both surveys, for the delta times and the

relative deltas per question. These results show that there are questions with p-values

less than 0.05 for the delta times per question, specifically questions 2, 11, 22, 24,

29, 54, 57, 60. And that the relative delta times also have a couple of questions with

p-values less than 0.05, specifically questions 9, 16, 22, 54, 63. It should be noted

that questions 22 and 54 show up in with p-values less than 0.05 for both the delta

times and relative deltas.

53

4.2 Discussion

4.1 outlines the various results that have been gathered throughout the experiment.

This section will discuss what the results mean with respect to the experiment [and

hypotheses] outlined in Section 2.4. As Section 4.1 points out, the overall results

from the experiment show that of the 457 people who came to the website, only 111

actually took part in the experiment, which is approximately 24% of the people who

came to the site actually participating in the experiment, with approximately 48.6%

taking part in a badged survey and 51.3% taking part in a non-badged survey. From

here, it can also be seen that 91 people who took part in the experiment were able

to complete it, leaving only 20 unfinished surveys between the two types of surveys.

That is, approximately 81% of the people who took part in the experiment were able

to finish.

Breaking this down by survey type, approximately 83.3% of people who started a

badged survey completed it, and 80.7% of people who started a non-badged survey

completed it. A simple test of proportions indicates that there is no significant dif-

ference (p-value=0.91) in the completion rates for those who took the badged versus

the non-badged survey. Interestingly, this contradicts Hypothesis 1 in Section 2.4.

That is, badging does not appear to have influenced the completion rate.

While the Wilcoxon Rank Sum tests indicate that there was no statistically signif-

icant difference between the time it took to complete either types of surveys, some

54

important observations need to be noted. Upon taking a deeper look at the survey

results, we can see that for the badge survey, the median number of badges that were

given during the experiment was 5, the average number 4.7, and the maximum given

out being 13. Since each badge would produce a pop up window that would appear

for a maximum of 3 seconds (unless the participant closed it), this suggests that the

badged survey should take around 15 seconds longer (using either the median or av-

erage number of badges received) for a user to complete, while at most it should take

a participant around 39 seconds longer. In reality, we observed the following:

• The average time for a non-badged survey was 553.53 seconds.

• The average time for a badged survey was 635.34 seconds.

• The difference in the averages is 76.39 seconds.

• The median time for a non-badged survey was 519.81 seconds.

• The median time for a badged survey was 553.53 seconds.

• The difference in medians was 33.72 seconds.

Referring to Figure 4.1, the difference in the time it takes to finish a badged survey,

on average, is much higher than the time taken to finish a non-badged survey. That

is, the average difference of 76.39 seconds is 5.19 times as much as what would be

expected if we only considered the additional time required to open and close the

badge pop up windows. This hints that participants may be more engaged during a

55

badged survey than with their non-badged counterparts. Further, the median times

between the two survey types also shows that participants appear to be taking long to

complete the badged survey than what would be expected. Specifically, the difference

in median times to complete the survey is 2.25 times longer than what would be

expected if the difference were only related to the badge pop ups. While referring to

Figure 4.1, the orange bar represents the non-badged survey results with the addition

of the time required to receive the average number of badges that were assigned. This

shows that even by adding this extra time to the non-badged survey, there is still a

difference between the two surveys that is larger than might be expected. Again,

this trend suggests that the badged survey respondents are spending more time on

the surveys, and are possibly more engaged. This provides support for Hypothesis 2

outlined in Section 2.4, although it is not statistically significant.

As described in Section 4.1.1, the demographics and food habits categories showed

a significant difference in the delta time taken to complete them, while only the

demographics category showed a significant difference in the relative time taken to

complete it. This suggests that the distribution of times for the demographics cat-

egory is shifted more to the right than the distribution of times for the non-badged

survey. When the categories are ordered, these two categories become the first that

are presented to the participant. The differences observed might be associated with

the participant becoming familiarized with the survey itself.

56

Figure 4.1: A plot of the average and median times to complete the survey for thebadged (in light blue) and non-badged (in dark blue) surveys and the non-badgedsurvey with the addition of the time to earn the average number of badges given (inorange).

57

Other significant results between the surveys were limited to delta timers for radio

button type questions, and questions with fewer than 25 words. No other significant

differences were identified. That is, if straight-lining were occurring, it’s occurring in

the same way for both surveys. However, it should be noted that the median results

for delta timers were always larger for the badged survey than the non badged survey.

That is, despite a lack of statistical significance, it appears that the respondents were

taking longer to complete badged survey questions.

However, when considering the proportion of median time spent per question for

each category, it appears as if the respondents for both badged and non-badged

surveys spent more time answering questions closer to the end of the survey. That is,

it does not appear as if straight-lining occurred, or if it did, it occurred in the same

way. Since none of the categories near the end of the survey were found to have times

that were significantly different between the survey types, this neither supports nor

contradicts Hypothesis 3 in Section 2.4.

58

Chapter 5

Conclusion

As a consequence of issues with pen and paper (or phone) surveys, such as a lack of

user engagement, premature survey termination, and transitions from from traditional

landline phones to computers and hand held devices such as tablets and phones,

investigation into new methods of data collection becomes necessary. Alternative data

collection methods, e.g. on-line surveys, provide a method to collect data without the

use of standard pen and paper, or telephone surveys. Self elements of Gamification

were used to increase user engagement and completion rates during the online survey,

by adding badges that participants can earn during the survey.

5.1 Findings

We now review the hypotheses that this thesis was attempting to answer.

59

1. Gamification can be used to increase participant completion rates.

More specifically, the use of self elements (such as badges) will in-

crease participants completing the survey.

In regards to hypothesis 1, it was found that both types of surveys had very

similar completion rates, with the badge survey having 83.3% of participants

finish it, while 80.7% of non-badge survey participants were able to finish. As

Table 4.1 shows, the number of user’s that started for each survey is close,

with the badged survey having 54 participants start and the non-badged survey

having 57. Table 4.1 also shows that the number of participants that com-

pleted were close as well, with the badge survey having 45 participants finishing

and the non-badged survey having 46, with 9 and 11 participants not finishing

the badged or non-badged survey respectively. That is, we can not conclude

that Gamification was able to increase completion rates, as both the badged

and non-badged surveys have similar completion rates. Evidence suggests that

hypothesis 1 is FALSE.

2. The participant will spend more time on the gamified survey as op-

posed to the non-gamified survey. More specifically, the use of self

elements will engage the participant and they will take more time to

answer the survey.

The average and median times to complete a survey was greater for the badged

60

survey than the non badged survey. The difference, although not statistically

significant, was even larger than what would have been expected when taking

into account the time a badge stays on screen when it is earned. This trend

seems to suggest that hypothesis 2 is TRUE, and perhaps with further investi-

gation and a larger sample size, the true results will be known. Despite the fact

that statistical significance was not observed, the trend suggests that Gamifi-

cation may increase the amount of time that participants were spending on the

questions.

3. There will be no difference between a gamified survey and a non-

gamified survey in terms of participants straight-lining the survey.

More specifically, the participant is just as likely to straight-line a

gamified survey as they are a non-gamified survey.

The results indicate that straight-lining likely did not occur due to the amount

of time that participants of the badged survey spent on each question. However,

since there were mostly non significant results between the two surveys when

compared by delta timers and relative delta timers for category, question type,

question length, question, and overall survey, there is insufficient evidence to

suggest the time spent on each survey differed, or that participants were behav-

ing differently for each survey. That is, the evidence suggests that hypothesis 3

is INCONCLUSIVE.

61

To summarize, hypothesis 1 was found to be true, hypothesis 2 was found to be

false, and hypothesis 3 was found to be inconclusive. Regardless, it appears as if

badging has at least in some way contributed to improving participant engagement.

As such, it is recommended that badging be considered for future online public health

surveys, and that further study be done.

5.2 Future Work

In the future, further investigation of other elements of Gamification, and other

types of Self Elements should be looked at to see if they have an effect on user

engagement. More specifically, Social Elements of Gamification should be investigated

such as leaderboards or being able to share results from the survey if the participant

chooses. While other types of Self Elements that should be further looked at are

Pointsification, the use of numeric scores or points, or immediate feedback as a user

finishes a question, section/category, or survey, to see if these have an impact on the

engagement of users.

Another element of Gamification to look into would be Branching Elements, by

sending the user to different questions or sections of the survey depending on the

answer that was given for a specific question. This would help avoid premature

termination or fatigue for the user by having users skip specific questions that may

be unnecessary for the user to answer based on a previous answer. An example of

62

this would be asking the user if they have eaten a category of food such as fruits; if

the user answers ‘no’ then there is no reason to ask more questions about the type

of fruits a user has eaten just to force them to select an answer such as “I have not

eaten this.”

In regards to the data that was collected, further future work would include taking

a look at the subgroups, such as gender, age, location, within the demographics of

the data and seeing if the same conclusions can be drawn. It should be noted that

there will be biases present in this data due to the biases outlined in section 3.1

As well, since the contents of the survey were modelled to be similar to that of the

survey that the PHAC was running, other future work would include comparing the

data they collected to the data that was collected by this experiment. This would

help to further determine the impact of Gamification by comparing the number of

user’s that completed the survey, timing data if available, or if data collected are of

the same quality.

63

References

C. Alfi. The world in 2011: Ict facts and figures, 2011. URL http://www.slideshare.net/

cokyfauzialfi/ict-facts-figures2011.

Jerald G. Bachman, Lloyd D. Johnston, Patrick M. O’Malley, and John E. Schulenberg. Monitoring

the future. Technical report, Ann Arbor, MI: Institute for Social Research, 2006.

Philip Batterham. Recruitment of mental health survey participants using internet advertising:

content, characteristics and cost effectiveness. International Journal of Methods in Psychiatric

Research, doi: 10.1002/mpr.1421, 2014.

V. Bhaskaran. Increasing confidence in responses to electronic surveys, January 3 2013. URL

https://www.google.com/patents/US20130004933. US Patent App. 13/539,180.

Dan Birman. Ux flows: How to drive deep user engagement, 2014.

Marloes Bults, Desiree JMA Beaujean, Onno Zwart, Gerjo Kok, Pepijn Empelen, Jim E. Steenber-

gen, Jan Hendrik Richardus, and Helene ACM Voeten. Perceived risk, anxiety, and behavioural

responses of the general public during the early phase of the influenza a (h1n1) pandemic in

the netherlands: results of three consecutive online surveys. BMC Public Health, 11(1):1–13,

2011. ISSN 1471-2458. doi: 10.1186/1471-2458-11-2. URL http://dx.doi.org/10.1186/

1471-2458-11-2.

64

A. Joseph Cafazzo, Mark Casselman, Nathaniel Hamming, K. Debra Katzman, and R. Mark

Palmert. Design of an mhealth app for the self-management of adolescent type 1 diabetes:

A pilot study. J Med Internet Res, 14(3):e70, May 2012. doi: 10.2196/jmir.2058. URL

http://www.ncbi.nlm.nih.gov/pubmed/22564332.

Jared Cechanowicz, Carl Gutwin, Briana Brownell, and Larry Goodfellow. Effects of gamification on

participation and data quality in a real-world market research domain. In Proceedings of the First

International Conference on Gameful Design, Research, and Applications, Gamification ’13, pages

58–65, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2815-9. doi: 10.1145/2583008.2583016.

URL http://doi.acm.org/10.1145/2583008.2583016.

Colleen Cook. A meta-analysis of response rates in web- or internet-based surveys. Educational and

Psychological Measurement, 60(6):821–836, 2000.

Mihaly Csikzentmihalyi. Beyond Boredom and Anxiety. Jossey-Bass, 1975.

Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. From game design elements

to gamefulness: Defining ”gamification”. In Proceedings of the 15th International Academic

MindTrek Conference: Envisioning Future Media Environments, MindTrek ’11, pages 9–15, New

York, NY, USA, 2011. ACM. ISBN 978-1-4503-0816-8. doi: 10.1145/2181037.2181040. URL

http://doi.acm.org/10.1145/2181037.2181040.

Theo Downes-Le Guin, Reg Baker, Joanne Mechling, and Erica Ruyle. Myths and realities of

respondent engagement in online surveys. International Journal of Market Research, 54(5):1–21,

2012.

Maeve Duggan. Cell phone activities 2013. Technical report, Pew Research Center, 2013.

FluidSurveysTM. Finding the correct survey length, August 2013. URL http://fluidsurveys.

com/university/finding-the-correct-survey-length/.

65

FluidSurveysTM. What is the difference between a response rate and a completion rate?, 2014. URL

http://fluidsurveys.com/university/difference-response-rate-completion-rate/.

Mirta Galesic and Michael Bosnjak. Effects of questionnaire length on participation and indicators

of response quality in a web survey. Public Opinion Quarterly, 73(2):349–360, 2009.

Imran Ghory. Using fizzbuzz to find developers who grok coding, 2014. URL http://imranontech.

com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/.

Geoff Goehle. Gamification and web-based homework. PRIMUS, 23(3):234–246, 2013. doi: 10.1080/

10511970.2012.736451. URL http://dx.doi.org/10.1080/10511970.2012.736451.

Jessica Greene, Howard Speizer, and Wyndy Wiitala. Telephone and web: mixed-mode challenge.

Health Serv Res, 43(1 Pt 1):230–48, Feb 2008. doi: 10.1111/j.1475-6773.2007.00747.x.

J. Hamari, J. Koivisto, and H. Sarsa. Does gamification work? – a literature review of empirical

studies on gamification. In System Sciences (HICSS), 2014 47th Hawaii International Conference

on, pages 3025–3034, 2014. doi: 10.1109/HICSS.2014.377.

Juho Hamari. Do badges increase user activity? a field experiment on the effects of gamifica-

tion. Computers in Human Behavior, pages –, 2015. ISSN 0747-5632. doi: http://dx.doi.

org/10.1016/j.chb.2015.03.036. URL http://www.sciencedirect.com/science/article/pii/

S0747563215002265.

Johannes Harms, Christoph Wimmer, Karin Kappel, and Thomas Grechenig. Gamification of online

surveys: conceptual foundations and a design process based on the mda framework. In Proceedings

of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, pages

565–568. ACM, 2014.

Debbie Hemley. 26 elements of a gamification marketing strategy., 2012. URL http://www.

socialmediaexaminer.com/26-elements-of-a-gamification-marketing-strategy/.

66

A. Regula Herzog and Jerald G. Bachman. Effects of questionnaire length on response quality, 1981.

W Hsin-Yuan Huang and D Soman. A practitioner’s guide to gamification of education, 2013.

Daniel Johnson, Sebastian Deterding, Kerri-Ann Kuhn, Aleksandra Staneva, Stoyan Stoyanov, and

Leanne Hides. Gamification for health and wellbing: A systematic review of the literature.

Internet Interventions, doi: 10.1016/j.invent.2016.10.002, July 2016.

Allen I. Kruat, Alan D. Wolfson, and Alan Rothenberg. Some effects of position on opinion survey

items. Journal of Applied Psychology, 60(6):774–776, 1975.

Frederick J. Kviz. Toward a standard definition of response rate. The Public Opinion Quarterly, 41

(2):265–267, 1977. ISSN 0033362X, 15375331. URL http://www.jstor.org/stable/2748344.

Carol Leaman. Report card: Gamification in learning (what works?), 2014.

Elaine Massung, David Coyle, Kirsten F. Cater, Marc Jay, and Chris Preist. Using crowdsourcing

to support pro-environmental community activism. In Proceedings of the SIGCHI Conference

on Human Factors in Computing Systems, CHI ’13, pages 371–380, New York, NY, USA, 2013.

ACM. ISBN 978-1-4503-1899-0. doi: 10.1145/2470654.2470708. URL http://doi.acm.org/10.

1145/2470654.2470708.

Aigul Mavletova. Data quality in pc and mobile web surveys. Social Science Computer Review, 31

(6):725–743, December 2013.

Heather L. O’Brien and Elaine G. Toms. What is user engagement? a conceptual framework for

defining user engagement with technology. Journal of the American Society for Information

Science and Technology, 59(6):938–955, 2008. ISSN 1532-2890. doi: 10.1002/asi.20801. URL

http://dx.doi.org/10.1002/asi.20801.

Jon Puleston. Trimming mobile surveys. Research World, pages 40–42, October 2014.

67

K. Seaborn and D. I. Fels. Gamification in theory and action: A survey. International Journal of

Human-Computer Studies, 74:14–31, 2015.

Namin Shin. Online learner’s ‘flow’ experience: an empirical study. British Journal of Educational

Technology, 37(5):705–720, 2006. ISSN 1467-8535. doi: 10.1111/j.1467-8535.2006.00641.x. URL

http://dx.doi.org/10.1111/j.1467-8535.2006.00641.x.

Michael B. Siegel, Kerry L. Tanwar, and Kathleen S. Wood. Electronic cigarettes as a smoking-

cessation tool. American Journal of Preventive Medicine, 40(4):472–475, 2016/02/01 2016. ISSN

0749-3797. doi: 10.1016/j.amepre.2010.12.006. URL http://dx.doi.org/10.1016/j.amepre.

2010.12.006.

Robra-Bissantz Sussane, Scheiner Christian, and Witt Maximilian. Gamification of online idea

competitions: Insights from an explorative case. Informatik schafft Communities, 2011.

Sharleen Sy. Engagement flow in gamification, 2010. URL https://stratsynergy.wordpress.

com/2010/11/02/engagement-flow-in-gamification/.

Kevin B. Wright. Researching internet-based populations: Advantages and disadvantages of online

survey research, online questionnaire authoring software packages, and web survey services. Jour-

nal of Computer-Mediated Communication, 10(3):00–00, 2005. ISSN 1083-6101. doi: 10.1111/j.

1083-6101.2005.tb00259.x. URL http://dx.doi.org/10.1111/j.1083-6101.2005.tb00259.x.

68

Appendix A

Ethics Documents

A.1 Ethics

A.1.1 Ethics Approval

Since this thesis required experiments involving human participants, it was necessary to review

the experimental design to ensure that risks and harm were to any participant was minimized.

As indicated in Figure A.1, the Research Ethics Boards of the University of Guelph reviewed the

proposed study, and provided the Certification of Ethical Acceptability of Research Involving Human

Participants, and REB Number 15AU004.

69

Page 1 of 1

APPROVAL PERIOD: February 10, 2016 EXPIRY DATE: February 10, 2017 REB: NPES REB NUMBER: 15AU004 TYPE OF REVIEW: Delegated Type 1 PRINCIPAL INVESTIGATOR: Gillis, Daniel (dgillis@uoguelph.ca) DEPARTMENT: School of Computer Science SPONSOR(S): None TITLE OF PROJECT: Alternative Data Collection to Understand Food

Behaviours

The members of the University of Guelph Research Ethics Board have examined the protocol which describes the participation of the human participants in the above-named research project and considers the procedures, as described by the applicant, to conform to the University's ethical standards and the Tri-Council Policy Statement, 2nd Edition. The REB requires that researchers:

Adhere to the protocol as last reviewed and approved by the REB.

Receive approval from the REB for any modifications before they can be implemented.

Report any change in the source of funding.

Report unexpected events or incidental findings to the REB as soon as possible with an indication of how these events affect, in the view of the Principal Investigator, the safety of the participants, and the continuation of the protocol.

Are responsible for ascertaining and complying with all applicable legal and regulatory requirements with respect to consent and the protection of privacy of participants in the jurisdiction of the research project.

The Principal Investigator must:

Ensure that the ethical guidelines and approvals of facilities or institutions involved in the research are obtained and filed with the REB prior to the initiation of any research protocols.

Submit a Status Report to the REB upon completion of the project. If the research is a multi-year project, a status report must be submitted annually prior to the expiry date. Failure to submit an annual status report will lead to your study being suspended and potentially terminated.

The approval for this protocol terminates on the EXPIRY DATE, or the term of your appointment or employment at the University of Guelph whichever comes first. Signature: Date: February 10, 2016

RESEARCH ETHICS BOARDS Certification of Ethical Acceptability of Research Involving Human Participants

A. Papdopoulos Chair, Research Ethics Board-NPES

Figure A.1: Ethics approval from the University of Guelph Research Ethics Board,February 10, 2016

70

A.1.2 Ethics Letter Of Information & Consent Form

The following Letter of Information and Consent Form (Figures A.2-A.4) was presented to the

participants of the survey prior to beginning the survey.

71

Alternative  Data  Collection    to  Understand    Food  Behaviours    

Letter of Information If you have access to the internet and are 18 years of age or older, we’re inviting you to take part in a study being conducted by Dr. Daniel Gillis (an Assistant Professor and Statistician in the School of Computer Science at the University of Guelph). The project has two major goals:

1.   We want to know how people across the country consume food (including how often they typically eat, and the kinds of food they eat), and what challenges they might face that could get in the way of them eating the types of foods that they want to eat.

2.   We want to figure out how to make surveys more enjoyable for the person answering questions, and in such a way as to provide researchers with better, more complete information. To do this we’re asking some people to answer regular surveys, and others to answer surveys that give them feedback on the information they’ve provided in the form of badges that they can share on Facebook or Twitter (if they choose to).

As a participant, you’re invited to answer a series of questions about yourself, about your food habits, your understanding and level of food insecurity (that is, not being able to eat the way you’d like to eat because of a lack of money), and food consumption. The questionnaire consists of 9 sections: demographics, food habits, food insecurity, fruits, vegetables, meats and alternatives, dairy and alternatives, grains and alternatives, and other foods. To proceed from one section to the next, you’ll need to provide an answer to every question within a section. However, you are not obliged to answer every question. Instead, you have the option of answering “I don’t know”, or “I’d prefer not to answer” on any question. Every question is multiple choice: some requiring a single answer, others that allow for multiple answers. The research program also investigates the use of new technologies (mobile phones, tablets, gesture control devices, etc.) for collecting data for research. While the results are not directly commercializable, they could be used to in the development of tools for marketing or business data collection, and thus might be commercializable in the future. The first question of the survey will ask for your consent for our team to analyze the data provided, as well as share and publish results based on the aggregate answers collected. All of your data are completely anonymous. We won’t (and don’t) ask for your name, address (other than the first three digits of your postal code), or email. To ensure anonymity, please do not enter your name anywhere in the survey.

Figure A.2: Page 1 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.

72

Alternative  Data  Collection    to  Understand    Food  Behaviours    

Letter of Information If you have access to the internet and are 18 years of age or older, we’re inviting you to take part in a study being conducted by Dr. Daniel Gillis (an Assistant Professor and Statistician in the School of Computer Science at the University of Guelph). The project has two major goals:

1.   We want to know how people across the country consume food (including how often they typically eat, and the kinds of food they eat), and what challenges they might face that could get in the way of them eating the types of foods that they want to eat.

2.   We want to figure out how to make surveys more enjoyable for the person answering questions, and in such a way as to provide researchers with better, more complete information. To do this we’re asking some people to answer regular surveys, and others to answer surveys that give them feedback on the information they’ve provided in the form of badges that they can share on Facebook or Twitter (if they choose to).

As a participant, you’re invited to answer a series of questions about yourself, about your food habits, your understanding and level of food insecurity (that is, not being able to eat the way you’d like to eat because of a lack of money), and food consumption. The questionnaire consists of 9 sections: demographics, food habits, food insecurity, fruits, vegetables, meats and alternatives, dairy and alternatives, grains and alternatives, and other foods. To proceed from one section to the next, you’ll need to provide an answer to every question within a section. However, you are not obliged to answer every question. Instead, you have the option of answering “I don’t know”, or “I’d prefer not to answer” on any question. Every question is multiple choice: some requiring a single answer, others that allow for multiple answers. The research program also investigates the use of new technologies (mobile phones, tablets, gesture control devices, etc.) for collecting data for research. While the results are not directly commercializable, they could be used to in the development of tools for marketing or business data collection, and thus might be commercializable in the future. The first question of the survey will ask for your consent for our team to analyze the data provided, as well as share and publish results based on the aggregate answers collected. All of your data are completely anonymous. We won’t (and don’t) ask for your name, address (other than the first three digits of your postal code), or email. To ensure anonymity, please do not enter your name anywhere in the survey.

Figure A.3: Page 2 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.

73

2.   Understand that your participation is voluntary and you are free to withdraw at any time 3.   Understand the provisions for confidentiality

§   I agree to participate.

§   I do not agree to participate.  

Figure A.4: Page 3 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.

74

A.2 Survey Questions

Survey questions and answers presented to the participants of the survey are outlined in Figures

A.5-A.10.

75

Questionnaire Category Category

Description/Instructions Question Answer Set

Demographics The following questions allow us to get to know you a little better. The information you provide here will be used to identify patterns in food consumption, food habits, and food insecurity. If you are unsure of your answer, please select "don't know" or follow the specific instructions of the question. If you prefer not to answer, please select that option, or follow the specific instructions of the question. You can leave the survey at any time by simply closing your browser.

Please enter your gender:

{"0":"Male", "1":"Female", "2":"Other", "3":"prefer not to answer"}

Please enter your age:

{"0":"<18", "1":"18 to 29","2":"30 to 39", "3":"40 to 59", "4":"60+", "5":"prefer not to answer"}

In the past 12 months, were you a part time or full time university or college student?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Which of the following best describes your current

{"0":"Single", "1":"Married/Partnered", "2":"Separated/Divorced/Widowed", "3":"Other", "4":"prefer not to answer"}

relationship status?

Do you have any dependents that live with you?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Please enter the first 3 digits of your postal/zip code. Enter XXX if you prefer not to answer. Enter YYY if you do not have a postal code. Enter ZZZ if you do not know your postal code.

{"0":"Postal Code"}

What is your annual income?

{"0":"<$30000","1":"$30000-$59999","2":"$60000-$79999","3":"$80000-$99999","4":"$100000+","5":"don't know","6":"prefer not to answer"}

What is your highest level of education?

{"0":"Primary","1":"Secondary","2":"University/College Degree/Diploma","3":"Graduate Degree/Diploma","4":"prefer not to answer"}

Which of the following best describes your current university student status?

{"0":"I am not currently a university student","1":"Undergraduate","2":"Master's","3":"PhD","4":"Post Doc","5":"don't know","6":"prefer not to answer"}

If you are a student in university or college, are you an international student?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Do you identify as any of the following?

{"0":"First Nations Status","1":"First Nations Non-Status","2":"Metis","3":"Inuit","4":"Senior Citizen (65+)","5":"Newcomer to Canada (<5yrs in Canada)","6":"I do not identify as any of the above","7":"don't know","8":"prefer not to answer"}

Figure A.5: Page 1 and 2 of the survey questions and answers provided to the par-ticipants

76

In general, how would you describe your physical health?

{"0":"Excellent","1":"Very good","2":"Good","3":"Fair","4":"Poor","5":"Very poor","6":"don't know","7":"prefer not to answer"}

In general, how would you describe your mental health?

{"0":"Excellent","1":"Very good","2":"Good","3":"Fair","4":"Poor","5":"Very poor","6":"don't know","7":"prefer not to answer"}

Thinking about the amount of stress in your life, would you say that most days are:

{"0":"Not at all stressful","1":"Not very stressful","2":"A bit stressful","3":"Quite a bit stressful","4":"Extremely stressful","5":"don't know","6":"prefer not to answer"}

Identify all sources of income for the previous year.

{"0":"Government student loan","1":"Scholarship/Bursary","2":"Bank loan","3":"Research/Teaching Assistantship","4":"Personal savings","5":"Family","6":"Part time employment","7":"Full time employment","8":"Child tax benefits","9":"Social assistance","10":"don't know","11":"prefer not to answer"}

Fruits The following questions are related to different types of fruit you've eaten in the last 7 days. Do not include fruit that was consumed as part of a prepared dessert (such as a pie or a tart), unless it was, for example, a bowl of cherries with whipped cream. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can

Did you eat any of the following fresh or frozen tropical fruits?

{"0":"Bananas","1":"Mangos","2":"Papayas","3":"Kiwis","4":"Pomegranates","5":"Avocado","6":"Other tropical fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

leave the survey at any time by simply closing your browser.

Did you eat any of the following fresh or frozen tree fruits?

{"0":"Apples","1":"Pears","2":"Plums","3":"Peaches","4":"Nectarines","5":"Apricots","6":"Other tree fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Did you eat any of the following citrus fruits?

{"0":"Lemons","1":"Limes","2":"Grapefruits","3":"Oranges","4":"Tangerines","5":"Clementines","6":"Other citrus fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Did you eat any of the following fresh or frozen melons?

{"0":"Cantaloupe","1":"Watermelon","2":"Honey Dew","3":"Other melons","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Did you eat any of the following fresh or frozen berries?

{"0":"Strawberries","1":"Blueberries","2":"Raspberries","3":"Blackberries","4":"Cherries","5":"Other berries","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Vegetables The following questions are related to different types of vegetables you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

Did you eat any of the following fresh or frozen leafy vegetables?

{"0":"Lettuce","1":"Cabbage","2":"Spinach","3":"Swiss Chard","4":"Kale","5":"Other leafy vegetables","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following fresh or frozen

{"0":"Carrots","1":"Beets","2":"Potatoes","3":"Turnips","4":"Parsnips","5":"Radish", "6":"Other root vegetables/tubers","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Figure A.6: Page 3 and 4 of the survey questions and answers provided to the par-ticipants

77

root vegetables/tubers?

Did you eat any of the following fresh or frozen strong vegetables?

{"0":"Onions","1":"Leeks","2":"Garlic","3":"Shallots","4":"Scallions","5":"Other strong vegetables","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following fresh or frozen stalk vegetables?

{"0":"Celery","1":"Broccoli","2":"Cauliflower","3":"Bok choy","4":"Other stalk vegetables","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you eat any of the following fresh or frozen pod vegetables?

{"0":"Beans","1":"Peas","3":"Other pod vegetables","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Did you eat any of the following fresh or frozen squash vegetables?

{"0":"Acorn","1":"Butternut","2":"Spaghetti","3":"Pumpkin","4":"Other squash","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you eat any of the following fresh or frozen vegetables?

{"0":"Cucumbers","1":"Tomatoes","2":"Mushrooms","3":"Zucchini","4":"Other vegetables","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Meat and Alternatives

The following questions are related to different types of meat (or meat alternatives) that you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can

Did you eat any of the following fresh or frozen unprocessed red meats (e.g. steaks, roasts, chops, etc.)? Do not include ground meats or deli meats here.

{"0":"Beef","1":"Deer","2":"Elk","3":"Bison","4":"Lamb","5":"Goat","6":"Other red meat","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

leave the survey at any time by simply closing your browser.

Did you eat any of the following fresh or frozen ground meats?

{"0":"Ground beef","1":"Ground pork","2":"Ground chicken","3":"Ground lamb","4":"Ground turkey","5":"Other ground meat","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following fresh or frozen white meats (e.g. breasts, legs, chops, etc.)?

{"0":"Chicken","1":"Pork","2":"Turkey","3":"Other white meat","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Did you eat any of the following fresh or frozen unprocessed fish or seafood?

{"0":"Fish","1":"Mussels","2":"Shrimp","3":"Scallops","4":"Clams","5":"Lobster","6":"Crab","7":"Other seafood or fish","8":"I didn't eat any of these foods","9":"don't know","10":"prefer not to answer"}

Did you eat any of the following nuts or seeds?

{"0":"Pecans","1":"Walnuts","2":"Almonds","3":"Cashews","4":"Hazelnuts","5":"Sunflower seeds","6":"Other nuts or seeds","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Did you eat any of the following raw or cooked bean/protein replacements?

{"0":"Lentils","1":"Chickpeas","2":"Kidney beans","3":"Black eyed peas","4":"Soy or soy products (e.g. tofu)","5":"Other meat replacement proteins","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following processed/deli meats?

{"0":"Sliced ham","1":"Sliced beef","2":"Sliced turkey","3":"Sliced chicken","4":"Salami","5":"Bacon","6":"Other processed meats","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Did you eat any of the following veggie/vegan protein replacements?

{"0":"Veggie/vegan burger","1":"Veggie/vegan hot dog","2":"Ground veggie meat substitute","3":"Other veggie/vegan meat replacements","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Figure A.7: Page 5 and 6 of the survey questions and answers provided to the par-ticipants

78

Did you eat any of the following nut butters?

{"0":"Peanut butter","1":"Cashew butter","2":"Almond butter","3":"Hazelnut butter","4":"Other nut butters","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Dairy and Alternatives

The following questions are related to different types of dairy (or dairy alternatives) that you've consumed in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

Did you drink any of the following milks?

{"0":"Milk (cow, goat, etc.)","1":"Almond milk","2":"Soy milk","3":"Cashew milk","4":"Rice milk","5":"Coconut milk","6":"Other milks","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}

Did you eat any of the following medium to hard cheeses?

{"0":"Cheddar","1":"Parmesan","2":"Romano","3":"Colby","4":"Monterey Jack","5":"Emmental","6":"Gouda","7":"Edam","8":"Other medium to hard cheeses","9":"I didn't eat any of these foods","10":"don't know","11":"prefer not to answer"}

Did you eat any of the following soft or semi-soft cheeses?

{"0":"Cream cheese","1":"Brie","2":"Havarti","3":"Munster","4":"Other soft or semi-soft cheeses","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you eat any of the following frozen dairy products? Include frozen ice cream bars here.

{"0":"Ice cream (dairy based)","1":"Rice milk ice cream","2":"Soy milk ice cream","3":"Coconut milk ice cream","4":"Other milk ice cream","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you eat any of the following yogurts?

{"0":"Yogurt (dairy based)","1":"Rice milk yogurt","2":"Soy yogurt","3":"Cashew yogurt","4":"Coconut yogurt","5":"Other milk yogurts","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following processed dairy based products?

{"0":"Cheese slices","1":"Cheese strings","2":"Pre-made milk based beverage","3":"Other processed dairy","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Grains and Alternatives

The following questions are related to different types of grains (or grain alternatives) that you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

Did you consume any of the following processed grain items?

{"0":"White bread","1":"Hot dog/hamburger buns","2":"White pasta","3":"White rice","4":"Other similar processed carbohydrates","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you consume any of the following whole grain products?

{"0":"Rye bread","1":"Whole wheat bread","2":"Whole wheat pasta","3":"Brown or wild rice","4":"Other complex carbohydrates","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you eat any of the following cold cereals?

{"0":"Bran flakes or other high fibre cereal","1":"Cheerios or other grain based cereals","2":"Granola based cereals","3":"Rice puff cereal","4":"Sugary cereal","5":"Other processed cereals","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}

Did you eat any of the following hot cereals?

{"0":"Oatmeal", "1":"Cream of wheat","2":"Multigrain cereal","3":"Other hot high fibre low sugar unprocessed cereals","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Food Insecurity

The following questions allow us to understand your current food security situation. Your answers will help us understand any issues you might have faced

In the past 12 months, have you NOT HAD enough food to eat because of a lack of money?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Figure A.8: Page 7 and 8 of the survey questions and answers provided to the par-ticipants

79

trying to obtain and eat the types of food you want to eat. If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

In the past 12 months, have you NOT eaten the quality or variety of foods that you wanted to eat because of a lack of money?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 12 months, have you worried that there might not be enough to eat because of a lack of money?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 12 months, have you sought assistance from a food bank or other food provider?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

If you needed to, do you know where you could go to find food in an emergency?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 12 months, have you ever not eaten balanced meals because you couldn't afford to?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 12 months, have you ever reduced your food consumption because there wasn't enough to eat because you couldn't afford it?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 12 months, did you experience weight loss because you couldn't afford to buy food?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Other Foods The following questions are related to other types of food or drinks that you've consumed in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

Did you eat any of the following items?

{"0":"Cake/Cupcakes/Muffins/Dessert Bread","1":"Pie/Tarts","2":"Cookies","3":"Pastries/Croissants","4":"Other desserts or pastries","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Figure A.9: Page 9 and 10 of the survey questions and answers provided to theparticipants

80

Did you consume any of the following beverages?

{"0":"Latte","1":"Capuccino","2":"Milkshake","3":"Other tea/coffee milk based beverage","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}

Did you consume any of the following beverages?

{"0":"Soda/Pop","1":"Energy drink","2":"Vitamin water","3":"Sports drink","4":"Other energy drinks","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you consume any of the following alcoholic beverages?

{"0":"Beer/cider","1":"Wine","2":"Clear liquor (e.g. vodka, gin, etc.)","3":"Other liquor (e.g. rye, scotch, etc.)","4":"Other alcoholic beverages","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}

Did you consume any of the following items?

{"0":"Chocolate bar","1":"Popcorn","2":"Granola bars","3":"Energy bars","4":"Potato chips","5":"Corn chips","6":"Crackers","7":"Other snack foods","8":"I didn't eat any of these foods","9":"don't know","10":"prefer not to answer"}

Food Habits The following questions give us a better idea of your current eating habits. These will be used to better understand your food consumption habits, which you'll be asked about in following sections. If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.

In the past week, have you eaten breakfast every day?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past week, have you eaten lunch every day?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past week, have you eaten dinner every day?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Are you a vegetarian or vegan?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 7 days, were you fasting for religious, medical, or other purposes?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

In the past 7 days, have your eating habits changed for any reason?

{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}

Figure A.10: Page 11 and 12 of the survey questions and answers provided to theparticipants

81

Appendix B

Code & Further Tables

B.1 Code

The code to create and implement the surveys used in this thesis can be found in the following

repository: https://bitbucket.org/altdatacollection/alt-data-collection.

B.2 Category Code to Category Labels

Table B.1 outlines the categories used in the survey, and the number of questions in each category.

The categories were presented to the respondent in the following order: 0, 8, 6, 1, 2, 3, 4, 5, 7.

82

Category ID Category Name Number of Questions in Category

0 Demographics 15

1 Fruits 5

2 Vegetables 7

3 Meat and Alternatives 9

4 Dairy and Alternatives 6

5 Grains and Alternatives 4

6 Food Insecurity 8

7 Other Foods 5

8 Food Habits 6

Table B.1: The category IDs for the surveys and the corresponding name for thecategory.

83

B.3 Wilcoxon Rank Sum Results By Question

Question Test Statistic p.value Median B Median NB

1 1150.500 0.151 22.046 18.558

2 1022.000 0.034 13.828 10.321

3 1510.500 0.313 0.000 0.000

4 1463.000 0.656 2.683 2.760

5 1142.000 0.234 7.472 6.840

6 1063.000 0.063 8.640 7.731

7 1334.000 0.956 6.860 7.172

8 1103.000 0.109 6.638 6.036

9 1455.500 0.495 6.398 7.294

10 1275.000 0.524 10.650 9.368

11 1038.000 0.031 9.649 7.669

12 1241.000 0.481 6.947 6.877

13 1146.000 0.143 6.234 5.968

14 1460.000 0.588 7.890 8.814

15 1114.000 0.095 5.597 4.831

16 1486.000 0.479 4.231 4.376

17 1145.000 0.141 13.102 9.607

18 1134.000 0.123 3.508 2.678

19 1180.000 0.213 3.090 2.455

20 1070.000 0.051 3.809 3.273

21 1200.000 0.263 5.766 4.969

22 1136.500 0.018 5.606 4.491

84

23 1245.500 0.084 5.039 4.607

24 1200.500 0.047 3.507 3.089

25 1375.000 0.335 8.627 8.488

26 1341.500 0.314 5.434 5.373

27 1383.000 0.448 5.151 4.877

28 1263.000 0.475 2.587 2.430

29 1038.000 0.031 4.587 3.651

30 1246.500 0.412 3.899 3.719

31 1269.000 0.601 7.539 6.563

32 1097.000 0.231 14.294 11.982

33 1186.000 0.553 7.561 6.453

34 1258.000 0.917 5.944 6.832

35 1116.000 0.448 7.635 6.384

36 1273.000 0.998 8.348 9.202

37 1198.000 0.608 9.215 8.655

38 1292.000 0.773 7.558 7.264

39 1251.000 0.987 6.035 6.202

40 1217.000 0.702 3.740 3.934

41 1247.000 0.742 9.138 9.493

42 1096.000 0.463 8.926 7.101

43 999.000 0.155 6.966 6.667

44 1107.000 0.511 6.126 5.427

45 1024.000 0.213 4.600 4.211

46 1150.000 0.726 6.451 6.479

85

47 1048.000 0.804 10.901 9.764

48 885.000 0.099 7.504 7.299

49 1083.000 0.742 6.445 6.195

50 911.000 0.146 7.400 5.919

51 923.000 0.129 9.671 8.537

52 1155.500 0.886 12.012 11.696

53 1156.000 0.889 7.243 6.705

54 762.500 0.005 8.669 7.119

55 1293.000 0.305 4.449 4.540

56 1301.000 0.274 6.660 6.640

57 1136.000 0.026 6.709 5.282

58 1434.000 0.650 7.228 7.846

59 1333.000 0.290 5.672 5.018

60 1049.000 0.006 4.557 3.686

61 1350.000 0.339 6.852 6.188

62 1360.000 0.455 11.990 10.691

63 1440.000 0.679 4.890 5.091

64 1114.000 0.095 5.100 4.887

65 1309.000 0.675 3.783 3.566

Table B.2: Wilcoxon Rank Sum results for both surveys, of the

delta times per question. Where Median Badged and Median Non-

Badged are the medians of each question for the badge survey and

non-badge survey respectively.

86

Question Test Statistic p.value Median B Median NB

1 1213.000 0.301 0.111 0.097

2 1158.000 0.213 0.049 0.044

3 1510.500 0.313 0.000 0.000

4 1377.000 0.341 0.389 0.344

5 1328.000 0.977 0.030 0.029

6 1189.000 0.297 0.029 0.028

7 1610.000 0.061 0.021 0.025

8 1314.000 0.818 0.023 0.021

9 1652.000 0.050 0.021 0.024

10 1420.000 0.776 0.052 0.052

11 1196.000 0.253 0.042 0.039

12 1371.000 0.894 0.029 0.028

13 1275.000 0.524 0.028 0.027

14 1632.000 0.100 0.054 0.065

15 1373.000 0.993 0.032 0.035

16 1691.000 0.043 0.026 0.031

17 1278.000 0.536 0.096 0.093

18 1311.000 0.684 0.026 0.024

19 1401.000 0.871 0.024 0.021

20 1279.000 0.541 0.023 0.022

21 1365.000 0.952 0.032 0.033

22 1163.000 0.027 0.417 0.350

23 1324.000 0.206 0.283 0.247

87

24 1457.000 0.631 0.153 0.150

25 1585.000 0.789 0.264 0.265

26 1504.000 0.972 0.143 0.140

27 1463.000 0.779 0.117 0.107

28 1539.000 0.295 0.021 0.022

29 1331.000 0.781 0.030 0.031

30 1421.000 0.771 0.030 0.030

31 1510.000 0.300 0.024 0.024

32 1233.000 0.784 0.038 0.039

33 1444.000 0.250 0.018 0.023

34 1403.000 0.383 0.017 0.020

35 1329.000 0.469 0.018 0.019

36 1417.000 0.333 0.020 0.024

37 1417.000 0.333 0.024 0.026

38 1434.000 0.205 0.018 0.021

39 1409.000 0.269 0.016 0.017

40 1405.000 0.376 0.010 0.011

41 1395.000 0.167 0.022 0.023

42 1253.000 0.710 0.019 0.019

43 1203.000 0.986 0.017 0.018

44 1232.000 0.823 0.014 0.014

45 1204.000 0.981 0.012 0.011

46 1411.000 0.135 0.014 0.016

47 1213.000 0.314 0.021 0.023

88

48 1097.000 0.961 0.016 0.018

49 1292.000 0.225 0.012 0.014

50 1050.000 0.685 0.013 0.013

51 1090.000 0.782 0.018 0.018

52 1284.000 0.440 0.025 0.027

53 1282.000 0.449 0.015 0.016

54 862.000 0.034 0.019 0.016

55 1414.000 0.056 0.010 0.011

56 1386.000 0.563 0.142 0.126

57 1331.000 0.285 0.107 0.095

58 1701.000 0.256 0.099 0.111

59 1489.000 0.901 0.072 0.068

60 1349.000 0.336 0.055 0.053

61 1644.000 0.427 0.075 0.076

62 1560.000 0.648 0.116 0.112

63 1724.000 0.026 0.027 0.033

64 1398.000 0.886 0.027 0.030

65 1528.000 0.328 0.019 0.021

Table B.3: Wilcoxon Rank Sum results for both surveys, of the rel-

ative delta times per question. Where Median Badged and Median

Non-Badged are the medians of each question for the badge survey

and non-badge survey respectively.