This is an electronic reprint of the original article.This reprint may differ from the original in pagination and typographic detail.

Powered by TCPDF (www.tcpdf.org)

This material is protected by copyright and other intellectual property rights, and duplication or sale of all or part of any of the repository collections is not permitted, except that material may be duplicated by you for your research use or educational purposes in electronic or print form. You must obtain permission for any other use. Electronic or print copies may not be offered, whether for sale or otherwise to anyone who is not an authorised user.

Savela, Nina; Oksanen, Atte; Kaakinen, Markus; Noreikis, Marius; Xiao, YuDoes Augmented Reality Affect Sociability, Entertainment, and Learning? A Field Experiment

Published in:APPLIED SCIENCES

DOI:10.3390/app10041392

Published: 01/02/2020

Document VersionPublisher's PDF, also known as Version of record

Please cite the original version:Savela, N., Oksanen, A., Kaakinen, M., Noreikis, M., & Xiao, Y. (2020). Does Augmented Reality AffectSociability, Entertainment, and Learning? A Field Experiment. APPLIED SCIENCES, 10(4).https://doi.org/10.3390/app10041392

applied sciences

Article

Does Augmented Reality Affect Sociability,Entertainment, and Learning? A Field Experiment

Nina Savela 1,* , Atte Oksanen 1 , Markus Kaakinen 2, Marius Noreikis 3 and Yu Xiao 3

1 Faculty of Social Sciences, Tampere University, 33100 Tampere, Finland; atte.oksanen@tuni.fi2 Institute of Criminology and Legal Policy, University of Helsinki, 00100 Helsinki, Finland;

markus.kaakinen@helsinki.fi3 Department of Communications and Networking, Aalto University, 02150 Espoo, Finland;

mnoreikis@yahoo.com (M.N.); yu.xiao@aalto.fi (Y.X.)* Correspondence: nina.savela@tuni.fi

Received: 14 January 2020; Accepted: 15 February 2020; Published: 19 February 2020�����������������

Abstract: Augmented reality (AR) applications have recently emerged for entertainment andeducational purposes and have been proposed to have positive effects on social interaction. In thisstudy, we investigated the impact of a mobile, indoor AR feature on sociability, entertainment, andlearning. We conducted a field experiment using a quiz game in a Finnish science center exhibition.We divided participants (N = 372) into an experimental group (AR app users) and two control groups(non-AR app users; pen-and-paper participants), including 28 AR users of follow-up interviews.We used Kruskal–Wallis rank test to compare the experimental groups and the content analysismethod to explore AR users’ experiences. Although interviewed AR participants recognized theentertainment value and learning opportunities for AR, we did not detect an increase in perceivedsociability, social behavior, positive affect, or learning performance when comparing the experimentalgroups. Instead, AR interviewees experienced a strong conflict between the two different realities.Despite the engaging novelty value of new technology, performance and other improvements do notautomatically emerge. We also discuss potential conditional factors. Future research and developmentof AR and related technologies should note the possible negative effects of dividing attention toboth realities.

Keywords: augmented reality; sociability; social interaction; entertainment; learning

1. Introduction

A new software technology combining physical and virtual realities by adding layers of digitalinformation onto the physical world using suitable devices, such as wearables, has enabled newapplications (e.g., Pokémon Go) and has introduced a novel way to communicate with others [1,2].Changing the way people communicate and interact with each other, social media has been the mostrecent technological revolution that has impacted social interaction, entertainment, and learning [3].These aspects are also apparent in perceived benefits that motivate people to use social media [4–6].As new technology has the potential to permanently change the way we behave, besides user studiesand product development, cutting-edge technology should be examined with regard to its implicationson social interaction, sociability, entertainment, and learning.

Augmented reality (AR) applications enable people to interact with virtual objects overlaid in thephysical environment and visible through digital screens. AR or mixed reality (MR) can be located in acontinuum between real and virtual reality or seen as a combination of the two different elements [7,8].As a result of the incoherent use of the terminology and commercial success of AR applications,previous researchers and other stakeholders have often referred to MR as AR, even though only some

Appl. Sci. 2020, 10, 1392; doi:10.3390/app10041392 www.mdpi.com/journal/applsci

Appl. Sci. 2020, 10, 1392 2 of 15

of the experts in the field use them as synonyms [8]. The suggested definitions of MR summarized bySpeicher and colleagues [8] indicate that collaboration is one of the key aspects used in defining theconcept. Piumsomboon and colleagues [9,10], for example, studied the collaboration between the localAR user and remote virtual reality (VR) user.

Increasing amounts of AR and MR applications are being designed and introduced each year tothe public and to various industry fields, such as education, health care, and maintenance, to namea few [11–13]. As an entertainment technology combined with a global positioning system, AR isbest known to the public thanks to Pokémon GO, but new games and applications emerge, includingMinecraft Earth and Harry Potter: Wizards Unite. Based on previous research, AR applications suchas AR games could potentially enhance social communication and social interaction between people.Researchers have focused on the case of Pokémon Go and concluded that AR games have the potentialfor positive behavioral effects such as socialization [14]. Some games, not just AR games, incorporatemore social and collaborative features to the gameplay than others, but even single-player games havebeen argued to have inherent social aspects [15]. Thus, augmented reality games could potentiallyfoster sociability by gathering users together in a shared space and offering affordances for interactingwith other users via options to comment, read, and reply to others. Mäyrä and colleagues [16] describethe single-player games in general, “knowledge of others playing the same game makes the gamemore social”, which presumably concerns AR technology as well. Against this, it is interesting that asense of social presence in MR technology has been found to positively affect visitor experiences inpublic places [17].

Entertainment value is also an important factor in itself and becomes even more relevant wheninvestigating AR game outcomes in the context of recreational activities. More generally, gamificationhas combined research of technology and game design, motivation, and human–computer interaction,among others [18]. In a review of engagement in digital entertainment games, researchers categorized,for example, subjective experience, physiological responses, motives for playing, and impact of gameon life satisfaction as aspects of engagement [19]. Recently, researchers studying AR identified hedonicand utilitarian as important aspects in AR adoption and found that consumers regard AR to be useful intourism and education [20], and based on a literature review, AR had positive effects on engagement (e.g.,in the museum context) [21]. Pervasive MR games are not without their challenges from hedonic andemotional perspectives. Enjoyment and engagement could be disrupted by, for example, awkwardness ofplayful behavior in public places [22]. Immersive games also impact the sense of presence [23], whichcould result in conflict between realities in the case of MR, but research on this remains scarce.

Based on a systematic literature review on the effects of computer games and serious games,there exist evidence about the effectiveness of game-based learning [24]. The authors concludedthat “playing computer games is linked to a range of perceptual, cognitive, behavioral, affective andmotivational impacts and outcomes” [24] (p. 661). For example, they found evidence on knowledgeacquisition, motivational, and affective outcomes, but they noted the difficulties in defining learningoutcomes [24]. In previous studies on AR and similar technologies, researchers have indeed focusedheavily on their learning potential and have found similar findings than the aforementioned literatureon game-based learning e.g., [25,26].

Wu and colleagues showed in their literature review on AR in education that research hasemphasized the potential of AR to improve learning and aid in learning difficulties, but somechallenges have also been noted regarding cognitive overload, lack of complex skills, and confusionbetween reality and fantasy [27]. In another systematic review, Bacca and colleagues demonstrate thatthe target group of most of the research on AR and learning has been students of bachelor level orlower, with only a couple targeting informal learners [28]. According to the authors, learning gains,motivation, and facilitated interaction and collaboration were the most reported advantages of AR [28].Finally, in a more recent systematic literature review, researchers found mixed evidence about cognitiveoverload and called for future research on the effects of informal and ubiquitous learning, motivation,engagement, satisfaction, and interactions [25].

Appl. Sci. 2020, 10, 1392 3 of 15

Researchers have highlighted social aspects such as social presence, social space, and sociabilityin learning in other technology contexts as well [29]. Kreijns and Kirschner [30] coupled educationalaffordance with social and entertaining affordances in a paper calling for consideration of these threeperspectives on online collaborative learning. These aspects are also part of the uses and gratificationsapproach that seeks to explain the reasons and motivations behind media use with social, affective,and cognitive needs, among others [31]. However, our interest lies not in the end results of thesemotivations, such as perceived benefits and motivations to use, but, in the spirit of mass communicationtheory [32], in how the new AR technology is actually able to affect sociability, entertainment, andlearning. This kind of research has not been done on AR technology before.

This Study

In this experimental study, we investigated how indoor AR technology influences users’ sociability,entertainment, and learning in shared space. To test whether AR technology has an independenteffect on the three aforementioned perspectives, we conducted a field experiment in a science centerexhibition and designed an experiment comparing AR application participants to non-AR (a mobileapplication without AR features) and pen-and-paper participants. Based on previous research, ARtechnology could potentially have effects on sociability and communication with other players [14–16],as well as entertainment and learning-related consequences [20,21,25–28]. To investigate this further,this study has the following research questions:

1. Does augmented reality have an effect from the social perspective on (a) perceived sociability orsocial behavior of (b) leaving messages or (c) talking with others face-to-face?

2. Does augmented reality have an effect from the entertainment perspective on (a) excitement, (b)nervousness, (c) tiredness afterwards, (d) app satisfaction, or (e) exhibition enjoyment?

3. Does augmented reality have an effect from the educational perspective on learning performance?

More specifically, from the social perspective, we include aspects of perceived sociability andactual social behavior with other people. From the entertainment point of view, we examine perceivedsatisfaction with the exhibition and application, reported affects of excitement and nervousness duringthe experiment, and tiredness after the exhibition. Lastly, the educational perspective encompasseslearning performance during the experiment. We compared all these aspects among three experimentalgroups that provided three different stimuli: AR application, an application without AR technology,and without smartphone application using only pen and paper. The experiment environment providesstimuli and opportunities for learning, entertainment, and social behavior through shared social space.Figure 1 shows the detailed study design.

Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 18

affordance with social and entertaining affordances in a paper calling for consideration of these three

perspectives on online collaborative learning. These aspects are also part of the uses and gratifications

approach that seeks to explain the reasons and motivations behind media use with social, affective,

and cognitive needs, among others [31]. However, our interest lies not in the end results of these

motivations, such as perceived benefits and motivations to use, but, in the spirit of mass

communication theory [32], in how the new AR technology is actually able to affect sociability,

entertainment, and learning. This kind of research has not been done on AR technology before.

This Study

In this experimental study, we investigated how indoor AR technology influences users’

sociability, entertainment, and learning in shared space. To test whether AR technology has an

independent effect on the three aforementioned perspectives, we conducted a field experiment in a

science center exhibition and designed an experiment comparing AR application participants to non-

AR (a mobile application without AR features) and pen-and-paper participants. Based on previous

research, AR technology could potentially have effects on sociability and communication with other

players [14–16], as well as entertainment and learning-related consequences [20,21,25–28]. To

investigate this further, this study has the following research questions:

1. Does augmented reality have an effect from the social perspective on (a) perceived sociability or

social behavior of (b) leaving messages or (c) talking with others face-to-face?

2. Does augmented reality have an effect from the entertainment perspective on (a) excitement, (b)

nervousness, (c) tiredness afterwards, (d) app satisfaction, or (e) exhibition enjoyment?

3. Does augmented reality have an effect from the educational perspective on learning

performance?

More specifically, from the social perspective, we include aspects of perceived sociability and

actual social behavior with other people. From the entertainment point of view, we examine

perceived satisfaction with the exhibition and application, reported affects of excitement and

nervousness during the experiment, and tiredness after the exhibition. Lastly, the educational

perspective encompasses learning performance during the experiment. We compared all these

aspects among three experimental groups that provided three different stimuli: AR application, an

application without AR technology, and without smartphone application using only pen and paper.

The experiment environment provides stimuli and opportunities for learning, entertainment, and

social behavior through shared social space. Figure 1 shows the detailed study design.

Figure 1. The study design. Figure 1. The study design.

Appl. Sci. 2020, 10, 1392 4 of 15

2. Materials and Methods

2.1. Procedure

We conducted a field experiment in the Finnish Heureka Science Centre to investigate the effectsof AR technology on visitor experiences and behavior. Using the science center’s own quiz related tobiotechnology, we developed a quiz game for the AR app, app without AR feature (non-AR app), andpen-and-paper questionnaire. We invited the participants to participate in our study, and then werandomly assigned them to one of the three different conditions in the science center’s main exhibitionroom, which presents multiple themes included in the quiz. All three conditions (AR, non-AR, andpen-and-paper) were carried out simultaneously and there was some variation depending on theday and number of visitors at the exhibition space. Participants received a small candy reward afterthe experiment.

The key feature of the AR app was for the visitors to interact with the quiz using AR. As they start,visitors look around the exhibition through their phone screens and discover the hidden quiz questions,represented as virtual floating question marks, scattered around the exhibition (see Appendix A).When a user approaches a question mark, the quiz questions are shown. Questions are placed nearexhibits that are related to the question contents. Altogether, 15 questions were located around theroom in eight different places. Therefore, one must tour the entire exhibition room to discover andanswer all the quiz questions. Questions can be answered by tapping on one of the four possibleanswers. Once answered, the questions disappear, as well as the question mark hints. Users can keeptrack of the number of remaining questions. After all the questions are answered or if the users decideto quit the quiz, they are presented with a quiz score. Within the exhibition, users may also leavecomments that will appear as an augmented 3-D object, placed on the museum floor. Other users canimmediately see the comments left by others, read them, and post replies.

The non-AR app contains the same functionality as the AR app; however, a user solves a quiz onthe phone’s screen without AR interaction (see Appendix B). In contrast to the AR app, a non-AR appuser can immediately browse through and answer all the quiz questions without the need to explorethe exhibition. The quiz questions and scoring system are the same for both applications. Non-AR appusers can also write comments next to a selected question and reply to comments left by other non-ARapp users. They can also read the comments, but because opening the comments section immediatelyshows all the comments, we did not have a reliable way of determining which comments were actuallyread by a user and thus we have missing information about the read comments of non-AR app users.

Both applications were developed for the Android platform. To implement the AR feature inthe AR app, we utilized image-base localization, the ARCore library by Google, and the Unity gameengine by Unity Technologies. The non-AR app was developed using Google’s Android SoftwareDevelopment Kit. The applications obtain questions and comments from a server and continuouslysend users’ activity information to the server. The activity information indicates times when the quizwas started and finished, quiz scores, and read (for AR app only) and written comments. Furthermore,before and after the quiz, both applications ask users a series of questions that are also sent to theserver. We stored all collected data in the database for later analysis.

We gave the other control group a pen-and-paper questionnaire with the quiz in the middle andquestionnaire items before and after the quiz. We provided additional papers on a side desk next tothe starting point and informed them that they could leave comments for other visitors to read andreply to. Similar to non-AR condition, we did not have a reliable way to detect if participants readany comments and thus, we have missing information about the read comments of pen-and-paperparticipants as well. All the experimental groups were offered two comments as a starting point: “Youcan leave comments and reply to comments left by others!” and “When you stand in a certain point ofthe room, you can see the Big Dipper.”

Appl. Sci. 2020, 10, 1392 5 of 15

2.2. Participants

We collected the data in autumn 2018 from science center visitors (N = 372) who were aged 15–73(M = 36.49, SD = 11.71), and 55.95% of them were female. Of all the participants, 231 were given ARapplications, 71 participated with an application without AR technology, and 70 were handed pen andpaper. Because our main interest was in the impact of AR technology, we recruited a larger number ofAR participants to ensure enough information about the key phenomena. To assess the success of ourrandomization, we tested whether there were significant differences between our experimental groupsin terms of background factors. We found that there were no differences between experimental groupsin age, gender, extraversion trait, smartphone application self-efficacy, or previous experience with theexhibition. This means that the randomization was successful and found differences cannot be due tothese background variables. In addition to the experiment, 28 users of AR application participated in afollow-up interview. All subjects gave their informed consent for inclusion before they participated inthe study.

2.3. Measures

We measured the dependent variable of sociability with a same four-item instrument [33] beforeand after the game. This instrument was chosen because of its validated design of before and aftermeasurement points. Participants answered statements on a scale from 1 (strongly disagree) to 7(strongly agree): “Right now I feel social,” “Right now I feel unsocial,” “Right now I feel talkative,”and “Right now I feel quiet.” The sum variable of the items had a Cronbach’s alpha of 0.86 before and0.90 after the quiz. Average sum variable was used for the remainder after subtracting before fromafter values of perceived sociability. We measured the dependent variable related to social behavior byasking participants in the questionnaire whether they talked with other visitors and, if yes, whetherthey were unknown to them from before. We used these measures as dummy variables. In additionto the questionnaire items, we collected information about written comments and replies that werecollected for every AR and non-AR app participant via the applications. The AR app also collecteddata about how many comments or replies were read by the users. After the field days, we collectedpapers left for pen-and-paper participants on a side desk for commenting.

The second set of dependent variables is related to the entertainment value and affects. Consideringthe reliable results of previous research in measuring immediate moment positive and negativeaffects [34,35], we asked all participants two questions in the middle of the quiz—that is, how excitedor nervous they felt at the moment of filling the quiz. At the end of the questionnaire, participantsalso scored one statement about feeling tired after the exhibition. In addition to measuring perceivedaffects, we measured entertainment in the light of application and museum context. After the quiz, weasked them about their app satisfaction using three items (e.g., “the application I tested was fun touse”) with a Cronbach’s alpha of 0.83. They also answered to two items of exhibition satisfaction (e.g.,“the exhibition was interesting”) with a Cronbach’s alpha of 0.80. We measured all items regardingentertainment value and affects on a scale from 1 to 7. Average sum variable was used for the 3- and2-item measures of app and exhibition satisfaction.

Thirdly, we measured participants’ performance in the quiz. Scoring was based on how thescience center scored its quizzes. If the correct answer to the question was “All of the above,” a userreceived 3 points, and if only a single choice was selected when “All of the above” was a correct answer,a user received 1 point. After the first question, a user received 1 point for a correct answer and 0for incorrect ones. In the analysis, we excluded those participants who did not finish the quiz. Thisinformation was collected with applications and calculated for pen-and-paper participants based onwhether they answered the last question of the quiz.

We used a semi-structured interview method to collect a more in-depth account of user perceptions.The interview form contained 10 questions related to the experiences of the previous experiment (seeAppendix C). The answers to the questions allowed us to examine the underlying reasons for the

Appl. Sci. 2020, 10, 1392 6 of 15

results of the experiment with the AR quiz application. Table 1 shows the descriptive statistics of allmeasures used in this study.

Table 1. Summary of Descriptive Statistics of the Study Variables (N = 372): Frequency (n), Percent (%),Mean (M), Standard Deviation (SD), Range, Number of Items in a Sum Variable, and Alpha (α).

Measure n % M SD Range n ofItems α

Experimental groups 372Experimental group (augmented reality [AR]

app participant) 231 62.10

Control group (non-AR app participant) 71 19.09Control group (pen and paper participant) 70 18.82

Sociability before the experiment 370 4.76 1.19 1–7 4 0.86Sociability after the experiment 370 4.44 1.30 1–7 4 0.90

Sociability after–before 368 −0.32 1.18 −6–3.25 8

Talked with other visitors 357 0–1 1No 201 56.30Yes 156 43.70

Talked with unfamiliar visitors 156 0–1 1No 112 71.79Yes 44 28.21

Left comments 372 0–1 1No 346 93.01Yes 26 6.99

Excitement during experiment 208 4.34 1.24 1–7 1Nervousness during experiment 188 3.30 1.66 1–7 1

Tiredness after the exhibition 371 3.61 1.49 1–7 1

App satisfaction 302 3.88 1.42 1–7 3 0.83Exhibition satisfaction 371 5.37 1.13 2–7 2 0.80

Quiz score (from those who finished) 155 10.54 2.67 3–16 1

2.4. Analysis

We used descriptive statistics, cross tabulation, χ2 test, Kruskal–Wallis equality of populationsrank test, and Dunn’s pairwise multiple comparison with Bonferroni corrections post hoc test tocompare the experimental groups in measures of social behavior, perceived sociability, entertainment,and quiz performance. We chose non-parametric Kruskal–Wallis test by ranks over the ANOVAanalysis of variance, because of the unequal sizes of the experimental groups and unequal variancesbased on Barlett’s test for equal variances. For additional analysis we used the ordinary least squares(OLS) regression analysis and binary logistic regression methods, for which we report unstandardizedregression coefficient (B), odds ratio (OR), and p values. We performed all statistical analyses with Stata12 and Stata 16. For Dunn’s pairwise multiple comparisons, we used a Stata package programmed byAlexis Dinno [36].

We used the directed content analysis method [37] to explore AR application users’ experiences oflearning, entertainment, and social interaction, and also reasons for, for example, communicating (ornot) with others. With this approach, we could use the research findings from the quantitative analysisto find relevant information related to the rationale of participants’ behaviors.

3. Results

3.1. Sociability and Social Behavior

The results of the change in sociability before and after the quiz show that participants in allexperimental groups felt, on average, less sociable after than before the quiz (M = −0.32, SD = 1.18; seeTable 2). Compared to AR users (M = −0.32, SD = 1.25), non-AR users felt even less sociable (M = −0.52,SD = 1.09), but the negative effect was smaller for pen-and-paper participants (M = −0.08, SD = 0.95).However, based on the Kruskal–Wallis rank test, the difference among the experimental groups was

Appl. Sci. 2020, 10, 1392 7 of 15

not statistically significant (χ2 [2, N = 368] = 5.64, p = 0.060, with ties), except between the two controlgroups (χ2 [1, N = 137] = 5.99, p = 0.014, with ties).

Table 2. Means (M), Standard Deviations (SD), Frequencies (n), Rank Sum, and Dunn’s MultipleNonparametric Pairwise Post Hoc Test with Bonferroni Corrections for Differences in DependentVariable of Sociability (N = 372).

Sociability M SD n Rank Sum 1. AR App 2. Non-AR App

Sociability after–before −0.32 1.18 3681. AR app −0.32 1.25 231 42,473.00

2. Non-AR app −0.52 1.09 71 11,707.00 1.323. Pen and paper −0.08 0.95 66 13,716.00 −1.62 −2.37 *

* p < 0.05. ** p < 0.01. *** p < 0.001. Note. The dependent variable: Perceived Sociability (After—Before the Quiz).

Table 3 shows the cross tabulation of talking with other visitors. We did not find statisticallysignificant differences in talking to other visitors between the experimental groups, but the χ2 testresult suggests that AR app users were slightly less likely to talk with others than the two controlgroups put together (χ2 [1] = 5.97, p = 0.015). In addition, from those participants who talked withsomeone, pen-and-paper participants were more likely to engage in a discussion with someone whowas unknown to them from before compared to AR app users (χ2 [1] = 14.41, p < 0.001) and, to someextent, to non-AR app users (χ2 [1] = 4.35, p = 0.037).

Table 3. Cross Tabulation of Talking with Other Visitors by Experimental Groups (N = 372).

Measure All Participants AR App Non-AR App Pen and Paper

Talked with othervisitors 357 (100%) 231 (100%) 71 (100%) 55 (100%)

No 201 (56.3%) 141 (61.0%) 34 (47.9%) 26 (47.3%)Yes 156 (43.7%) 90 (39.0%) 37 (52.1%) 29 (52.7%)

Talked withunfamiliar visitors 156 (100%) 90 (100%) 37 (100%) 29 (100%)

No 112 (71.8%) 73 (81.1%) 26 (70.3%) 13 (44.8%)Yes 44 (28.2%) 17 (18.9%) 11 (29.7%) 16 (55.2%)

Most participants did not leave any comments (see Appendix D). We provided pen-and-paperparticipants additional papers on a side desk next to the starting point for leaving comments, but theydid not write anything down or reply to comments made by others. Non-AR app participants did notreply to any comments, and only four of them (4/71, 5.6%) left comments with the non-AR application.From the AR app participants, nearly every tenth left at least one comment (22/231, 9.5%) or reply(27/231, 11.7%), and every other AR participant read at least one comment (114/231, 49.4%). Within thelimitations of missing information of read comments by pen-and-paper and non-AR participants, theresults show that AR users left more comments and replied more to other visitors’ comments than thecontrol groups.

In the follow-up interviews of AR users, some participants commented that AR could be suitablefor social interaction among introverted people. However, they also reported conflict and disorientationin trying to pay attention to their surroundings and to other people while navigating the room andadvancing in the quiz. Some reported not paying any attention to other people, and some perceivedthat other visitors were more of an annoyance because it was difficult to get close enough to the questionspots and to move around the room, as presented in this comment: “I feel like I don’t necessarily payattention to other people, that I could even walk over someone. I was in there at the phone, but still itfelt as if I wasn’t here in this space” (Interviewee 11). One interviewee explicitly expressed not feelingsociable while participating in our experiment: “I definitely didn’t feel myself sociable. I would have

Appl. Sci. 2020, 10, 1392 8 of 15

been more social without [the phone or app]” (Interviewee 5). Another interviewee did not thinkthis app increases talking to other people because it was more of an individual task: “I feel that thisapp does not help interacting with others, at least not the one I used now. This was private activity”(Interviewee 9). Interviewees who reported having found comments and who talked about beingengaged with the game were excited about finding the comments and question marks. Some reportednot quite catching the point of them, and other times, people discussed the user interface of commentsand questions.

3.2. Entertainment

Table 4 shows the results for excitement and nervousness during the quiz. Excitement duringthe quiz was moderately high for all participants (M = 4.28, SD = 1.17). However, AR app userswere less excited (M = 4.09, SD = 1.36) than pen-and-paper participants (M = 4.68, SD = 1.10), withnon-AR participants being somewhere in the middle (M = 4.28, SD = 1.17). The difference amongexperimental groups was statistically significant (χ2 [2, N = 208] = 9.73, p = 0.008, with ties) basedon the Kruskal–Wallis rank test. We found significant pairwise difference between AR users andpen-and-paper participants but not when comparing non-AR participants to other experimentalgroups. During the quiz, participants using phone applications were more nervous than pen-and-paperparticipants (M = 2.46, SD = 1.43), with non-AR app users feeling most nervous (M = 4.35, SD = 1.54)and AR app users somewhere between (M = 3.32, SD = 1.48). The difference between the experimentalgroups and among all groups was statistically significant (χ2 [2, N = 188] = 39.11, p < 0.001, with ties)based on the Kruskal–Wallis rank test and Dunn’s pairwise comparison with Bonferroni correctionspost hoc test.

Table 4. Means (M), Standard Deviations (SD), Frequencies (n), Rank Sum, and Dunn’s MultipleNonparametric Pairwise Post Hoc Test with Bonferroni Corrections for Differences in DependentVariables of Entertainment (N = 372).

Entertainment M SD n Rank Sum 1. AR App 2. Non-AR App

Excitement during the quiz 4.34 1.24 2081. AR app 4.09 1.36 80 7341.00

2. Non-AR app 4.28 1.17 60 6134.50 −1.063. Pen and paper 4.68 1.10 68 8260.50 −3.10 ** −1.87

Nervousness during the quiz 3.30 1.66 1881. AR app 3.32 1.48 66 6307.00

2. Non-AR app 4.35 1.54 54 6905.00 −3.29 **3. Pen and paper 2.46 1.43 68 4554.00 3.10 ** 6.25 ***

Tiredness after the exhibition 3.61 1.49 3711. AR app 3.54 1.42 231 41,995.50

2. Non-AR app 3.34 1.52 71 11,905.50 0.993. Pen and paper 4.12 1.62 69 15,105.00 −2.58 * −2.89 **

App satisfaction 3.88 1.42 3021. AR app 3.78 1.37 231 33,401.50

2. Non-AR app 4.20 1.56 71 12,351.50 −2.49 **

Exhibition satisfaction 5.37 1.13 3711. AR app 5.33 1.11 231 41,776.50

2. Non-AR app 5.36 1.24 71 13,207.00 −0.363. Pen and paper 5.54 1.08 69 14,022.50 −1.53 −0.96

* p < 0.05. ** p < 0.01. *** p < 0.001. Note. The dependent variables: excitement and nervousness during the quiz,app satisfaction, exhibition enjoyment, and tiredness after the exhibition.

Table 4 shows results also for tiredness after the exhibition, app satisfaction, and exhibitionsatisfaction. The reported tiredness after the exhibition was close to neutral for all participants(M = 3.61, SD = 1.49). Participants using the pen-and-paper questionnaire felt more tired (M = 4.12,SD = 1.62) compared to participants using phone applications, namely AR app users (M = 3.54,

Appl. Sci. 2020, 10, 1392 9 of 15

SD = 1.42) and non-AR app users (M = 3.34, SD = 1.52). The difference among the experimental groupswas statistically significant (χ2 [2, N = 371] = 9.34, p = 0.009, with ties) based on the Kruskal–Wallis ranktest. Both app participants’ satisfaction with the application was little above the middle of the scale. ARapp users were slightly less satisfied with the application (M = 3.78, SD = 1.37) than non-AR app users(M = 4.20, SD = 1.56), the difference being statistically significant (χ2 [1, N = 302] = 6.19, p = 0.013, withties) based on the Kruskal–Wallis rank test. All participants were highly satisfied with the exhibition(M = 5.37, SD = 1.13). Participants using pen and paper were most satisfied (M = 5.54, SD = 1.08),followed by non-AR participants (M = 5.36, SD = 1.24) and AR app users (M = 5.33, SD = 1.11). Thedifferences between the experimental groups and among all groups based on the Kruskal–Wallis ranktest were not statistically significant (χ2 [2, N = 371] = 2.36, p = 0.308, with ties).

AR interviewees reported high excitement in testing out new cutting-edge technology. Participantsregarded the AR technology for the science center to be appropriate and complementary for theexhibition: “It felt quite natural and of course supports and expands this kind of exhibition environmentwell. I felt that the app was naturally well-suited to the exhibition” (Interviewee 13). Dissatisfiedcomments about app satisfaction were related to difficulties with some features of the app and learninghow to use it: “At first it was a little confusing, but you get used to it” (Interviewee 12). One majortheme in the interview data was the competitive side of the game and getting hooked in finding all theelements and finishing the game. This can be noted when one person mentioned being attracted to thecomments and questions: “I noticed that the app attracted my attention and was searching for thoseothers’ comments and questions. I did like [the app]” (Interviewee 9).

3.3. Learning Performance

Table 5 shows the results regarding quiz performance. We excluded those who did not finish thequiz from this analysis. Based on the results, AR users received lower scores from the quiz (M = 8.59,SD = 2.14) than the two control groups: pen-and-paper participants (M = 12.11, SD = 2.05) and non-ARusers (M = 10.75, SD = 2.56). The difference among the experimental groups was statistically significant(χ2 [2, N = 155] = 47.36, p < 0.001, with ties). Dunn’s pairwise comparison with Bonferroni correctionspost hoc test shows a statistically significant result for all pairwise comparisons. We did additionalregression analysis and found that AR users who reported higher satisfaction on the app were morelikely to finish the quiz (OR = 1.65, p < 0.001). In turn, participants who finished the quiz had higherquiz scores (B = 7.41, p < 0.001).

Table 5. Means (M), Standard Deviations (SD), Frequencies (n), Rank Sum, and Dunn’s MultipleNonparametric Pairwise Post Hoc Test with Bonferroni Corrections for Differences in DependentVariable of Learning Performance (N = 372).

LearningPerformance M SD n Rank Sum 1. AR App 2. Non-AR App

Quiz score 10.54 2.67 1551. AR app 8.59 2.14 49 2170.50

2. Non-AR app 10.75 2.56 53 4378.50 −4.34 ***3. Pen & paper 12.11 2.05 53 5541.00 −6.82 *** −2.53 *

* p < 0.05. ** p < 0.01. *** p < 0.001. Note. The dependent variable: Learning performance based on quiz score.

Interviews with AR users after the experiment revealed that AR participants assumed a highpotential of AR technology in teaching and providing engaging learning tools, as was apparent inone comment:

Definitely, I can see it could support learning, mine and others as well. It was especially nice that itwas a part of the exhibition space. I also liked the scoring. If those were utilized more, I think it wouldhave supported the learning more. (Interviewee 16).

Appl. Sci. 2020, 10, 1392 10 of 15

High engagement in the competitive features of finding and answering all the questions was a commontheme. However, some interviewees had doubts that AR technology could actually improve learningand performance outcomes: “I believe using augmented reality was more meaningful, but I don’tbelieve it increased learning” (Interviewee 10). The previously reported results of comparing AR users’quiz performance to the two control groups provide support for this suspicion, but the interviews stillhighlight the potential entertainment value in learning, which cannot be measured by performance.

4. Discussion

In this experimental field study in the science center environment, we investigated the social,entertaining, and educational aspects of indoor AR technology. Our findings from comparing theAR app to the app without the AR feature and pen-and-paper condition did not confirm a positiveimpact of AR technology toward perceived sociability, social communication, entertainment, orlearning performance, although we detected some evidence of engaging novelty value and increasedcommenting behavior. In addition, we found that AR users experienced some discomfort in dividingtheir attention between the virtual and physical world.

We did not find evidence on the positive impact of AR technology on perceived sociability. Ourfindings suggest that AR app users are slightly less likely to talk to other people outside the applicationcompared to participants using non-AR app or pen-and-paper, and less likely to talk with unfamiliarpeople compared to the latter. However, based on the descriptive analysis of comments left, read, andreplied to, AR users’ commenting activity was higher than that of the control groups. One reason for thiscould be that the AR application provided easily locatable and accessible opportunities to communicatewith others through technology, which increased the amount of comments compared to non-AR app usersand pen-and-paper participants. It should be noted, however, that the commenting options betweenthe three groups were fundamentally different in regard to user interface and realistic commentingopportunities with pen and paper. Considering these as qualities of the three features used in theexperiment, we found some tentative evidence that the AR feature has the potential to increase socialbehavior inside the app. To conclude, we found some potential for AR to increase social behavior withinthe app but did not find positive effects on social behavior outside the app or on perceived sociability.Our results were somewhat different compared to what other researchers have argued regarding socialaspects of single-player games [15,16] and suggesting that AR has a positive impact on sociability [14].

From the entertainment perspective, applications seem to arouse less excitement and morenervousness than the traditional pen-and-paper survey method, yet users of applications were lesstired after the exhibition. One possible explanation is the learning curve of new and unfamiliartechnology, as well as the feeling of uncertainty and not being in control. Filling out a survey and aquiz via a smartphone application can still perhaps be quicker and less arduous than circling answerson a paper form. Participants of the non-AR app were more satisfied with their app than AR app users.The unfamiliarity of AR applications could contribute to app satisfaction. We found no differencesin exhibition satisfaction among the experimental groups; thus, none of the methods increased ordecreased exhibition satisfaction. In comparing AR to other conditions, we did not find the positiveeffects of AR on the aspects of entertainment suggested in the literature [20,21], although a lower levelof tiredness in AR participants could imply higher engagement or ease of using the AR application.

From the perspective of learning, we considered learning performance via measuring theparticipants’ quiz scores. AR users scored significantly lower compared to the control groups, despitemany AR interviewees reporting being engaged and ambitious toward the gamified quiz. Based onthe interviews, some AR users also thought it could stimulate learning. The AR app was thought tooffer novelty and meaningfulness to the learning experience, and this does not necessarily increaselearning or performance compared to other means of learning. As an additional analysis, however, wefound that those AR participants that were satisfied with the application were more likely to finish thequiz and, in turn, performed better in the quiz. The findings show that implementing new technologyor gamification does not always enhance performance, although their effectiveness is supported by

Appl. Sci. 2020, 10, 1392 11 of 15

previous research on game-based learning [24] and learning with AR or similar technologies [25–28].Our study contributes to the research on educational uses of AR by offering valuable information froman empirical experiment on informal learners in the context of recreation [28].

The interviews revealed that AR users saw a lot of potential for AR technology as a learning toolfor education and as a complementary feature in the science center and other attractions. However, theinterviewees saw less potential in AR for social behavior. Interview results show that AR technologycauses a strong conflict between the two different environments: the reality seen through the technologyenhanced with virtual elements and the reality shared with people around the same room. For theseAR users, the latter social environment was seen as a disruption while trying to focus on the task andthe reality seen through the AR application. Thus, the pervasive AR games pose some hedonic andemotional challenges for the users. The findings were in line with previous research on the perceivedawkwardness of playful behavior in public places [22] and the immersive nature of games that impactsthe sense of presence [23]. Although mixed results regarding cognitive overload exist [25,27], perceivedconflict between realities using AR or similar technology has not been addressed in previous researchand should be investigated further.

4.1. Strengths and Limitations

Our study was limited to certain applications and a science center context, which should be notedwhen interpreting the results. Our application was fairly simple, and some participants may havehad high expectations based on commercial applications with advanced design processes such asPokémon Go that has been developed by Niantic, Nintendo, and The Pokémon Company. On the otherhand, we consider that the simple design of our application was the strength of this study. It madethe AR condition more similar to other experimental conditions (i.e., app without AR feature andpen-and-paper), which were also simple by design, mainly based on the exhibition quiz. In addition,we were able to show the difficulties that users face when using an app for the first time. However, thisalso means that our experiment was limited to one user experience at one point in time. The negativeeffects could arguably disappear after people become more familiar with AR technology. For thisreason, future researchers should aim to understand how the use of AR apps develops over time andwhether AR technology has a social, entertaining, or educational impact when using the technologyfor a longer period of time.

Lastly, the possible impact of the lack of something in common with others and the self-awarenesscaused by the experimental nature of the situation should also be considered as conditional factors.A science exhibition itself is not always considered a social space, and participants may have reservationsin commenting, as they are not used to this during exhibitions. It might be more natural and relevantfor users to participate outside the experimental environment, where they can leave and read commentsfrom people of their own social network or from people with common interests and goals, comparedto a wide and unfamiliar group of science center visitors. In that case, the social features could perhapsbe more effective in generating social interaction. Yet, by not explicitly encouraging participantsto interact but informing them on the communication facilities available, our experimental settingallowed us to analyze whether the mere communicative facilities of AR technology (not the usedencouragement or incentive) can foster social interaction and ensured the comparability between thedifferent experimental conditions. The effect of different encouragement strategies would still beworthy of future investigations.

4.2. Conclusions

Based on the field experiment, our conclusion is that an AR game does not necessarily provideimprovements to learning performance or social interaction, at least in communicating outside thetechnology. Phone applications seem to cause more nervousness and less excitement but also lesstiredness than using the traditional survey method of pen and paper. Our results indicate that ARdoes have entertainment value as a novel technology, but it poses some challenges for people to be

Appl. Sci. 2020, 10, 1392 12 of 15

present and devoted to both realities at the same time. Thus, the results suggest that the positiveeffects of augmented reality are not self-evident. On the contrary, utilizing AR technology in naturalenvironments can produce unexpected and even negative results. Given our findings, more fieldexperiments on AR technology is needed. Future research and the development of AR technologyshould consider the negative emotions related to choosing between the two realities and the learningcurve of using AR technology when high performance or learning outcomes are desired.

Author Contributions: Conceptualization, N.S., A.O., M.K., M.N. and Y.X.; methodology, N.S., A.O. and M.K.;software, M.N. and Y.X.; validation, N.S., A.O., M.K., M.N. and Y.X.; formal analysis, N.S. and A.O.; investigation,N.S. and M.N.; resources, A.O. and Y.X.; data curation, N.S. and M.N.; writing—original draft preparation, N.S.;writing—review and editing, N.S., A.O., M.K., M.N. and Y.X.; visualization, N.S. and M.N.; supervision, A.O.and Y.X.; project administration, A.O. and Y.X. All authors have read and agreed to the published version ofthe manuscript.

Funding: This research received no external funding.

Acknowledgments: We acknowledge the resources and support to this research provided by the Finnish HeurekaScience Centre, Aalto University, University of Helsinki, and Tampere University.

Conflicts of Interest: The authors declare no conflicts of interest.

Appendix AAppl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 18

Appendix A

Figure A1. A Screenshot of the AR App Game Mode.

Figure A1. A Screenshot of the AR App Game Mode.

Appl. Sci. 2020, 10, 1392 13 of 15

Appendix B

Appl. Sci. 2020, 10, x FOR PEER REVIEW 14 of 18

Appendix B

Figure B1. A Screenshot of the Non-AR App Game Mode.

Figure A2. A Screenshot of the Non-AR App Game Mode.

Appendix C

The Interview Form

(1) What was your experience of the application?(2) Did the experience here differ from other museum exhibition experiences you have? How?(3) How did the application influence your behavior in the exhibition space?(4) What kind of cell phone application would make the exhibition more interesting?(5) Do you think you are outgoing?(6) Do augmented reality applications suit you?(7) Have you used mobile applications with augmented reality before?(8) What kind of learning experience was it for you to answer the quiz?

Appl. Sci. 2020, 10, 1392 14 of 15

(9) Do you think augmented reality technology had anything to do with your learning?(10) Did you finish the quiz? Do you think augmented reality technology had anything to do with

your motivation to finish the quiz?

Appendix D

Table A1. Commenting Activity by AR Users and Non-AR Users: Frequencies of Comments Left,Replied, and Read (N = 372).

Amount of comments left: 0 1 2 3 4 5 6 Total

1. AR app 209 15 3 2 1 1 2312. Non-AR app 67 2 2 71

3. Pen and paper 70 70

Amount of replies made: 0 1 2 3 4 5 6< Total

1. AR app 204 17 6 2 2 2312. Non-AR app 71 71

3. Pen and paper 70 70

Amount of comments read: 0 1 2 3 4 5 6< Total

1. AR app 117 30 30 18 17 10 9 231

References

1. Bastug, E.; Bennis, M.; Médard, M.; Debbah, M. Toward interconnected virtual reality: Opportunities,challenges, and enablers. IEEE Commun. Mag. 2017, 55, 110–117. [CrossRef]

2. Sherman, W.R.; Craig, A.B. Understanding Virtual Reality: Interface, Application, and Design, 2nd ed.; MorganKaufmann: Cambridge, MA, USA, 2018.

3. Van Dijck, J.; Poell, T. Understanding social media logic. Media Commun. 2018, 1, 2–14. [CrossRef]4. Kim, Y.; Sohn, D.; Choi, S.M. Cultural difference in motivations for using social network sites: A comparative

study of American and Korean college students. Comput. Hum. Behav. 2011, 27, 365–372. [CrossRef]5. Lin, K.Y.; Lu, H.P. Why people use social networking sites: An empirical study integrating network

externalities and motivation theory. Comput. Hum. Behav. 2011, 27, 1152–1161. [CrossRef]6. Whiting, A.; Williams, D. Why people use social media: A uses and gratifications approach. Qual. Mark. Res.

Int. J. 2013, 16, 362–369. [CrossRef]7. Ohta, Y.; Tamura, H. Mixed Reality: Merging Real and Virtual Worlds; Springer Publishing: New York, NY,

USA, 2014.8. Speicher, M.; Hall, B.D.; Nebeling, M. What is mixed reality? In Proceedings of the 2019 CHI Conference on

Human Factors in Computing Systems, CHI ’19, Glasgow, UK, 4–9 May 2019. [CrossRef]9. Piumsomboon, T.; Day, A.; Ens, B.; Lee, Y.; Lee, G.; Billinghurst, M. Exploring enhancements for remote mixed

reality collaboration. In Proceedings of the SA ’17 SIGGRAPH Asia 2017 Mobile Graphics & InteractiveApplications, Bangkok, Thailand, 27–30 November 2017. [CrossRef]

10. Piumsomboon, T.; Lee, G.A.; Hart, J.D.; Ens, B.; Lindeman, R.W.; Thomas, B.H.; Billinghurst, M. Mini-me:An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI Conference onHuman Factors in Computing Systems, CHI ’18, Montréal, QC, Canada, 21–26 April 2018. [CrossRef]

11. Gavish, N.; Gutiérrez, T.; Webel, S.; Rodríguez, J.; Peveri, M.; Bockholt, U.; Tecchia, F. Evaluating virtualreality and augmented reality training for industrial maintenance and assembly tasks. Interact. Learn.Environ. 2015, 23, 778–798. [CrossRef]

12. Kurilovas, E. Evaluation of quality and personalisation of VR/AR/MR learning systems. Behav. Inf. Technol.2016, 35, 998–1007. [CrossRef]

13. Rauschnabel, P.A.; Rossmann, A.; tom Dieck, M.C. An adoption framework for mobile augmented realitygames: The case of Pokémon Go. Comput. Hum. Behav. 2017, 76, 276–286. [CrossRef]

14. Serino, M.; Cordrey, K.; McLaughlin, L.; Milanaik, R.L. Pokémon Go and augmented virtual reality games: Acautionary commentary for parents and pediatricians. Curr. Opin. Pediatr. 2016, 28, 673–677. [CrossRef]

Appl. Sci. 2020, 10, 1392 15 of 15

15. Stenros, J.; Paavilainen, J.; Mäyrä, F. Social interaction in games. Int. J. Arts Technol. 2011, 4, 342–358.[CrossRef]

16. Mäyrä, F.; Stenros, J.; Paavilainen, J.; Kultima, A. From social play to social games and back: The emergenceand development of social network games. In New Perspectives on the Social Aspects of Digital Gaming:Multiplayer 2; Kowert, R., Quandt, T., Eds.; Routledge: Abingdon, UK, 2017; pp. 11–32.

17. Jung, T.; tom Dieck, M.C.; Lee, H.; Chung, N. Effects of virtual reality and augmented reality on visitorexperiences in museum. In Information and Communication Technologies in Tourism; Inversini, A., Schegg, R.,Eds.; Springer: Cham, Switzerland, 2016; pp. 621–635. [CrossRef]

18. Deterding, S. The lens of intrinsic skill atoms: A method for gameful design. Hum. Comput. Interact. 2015,30, 294–335. [CrossRef]

19. Boyle, E.A.; Connolly, T.M.; Hainey, T.; Boyle, J.M. Engagement in digital entertainment games: A systematicreview. Comput. Hum. Behav. 2012, 28, 771–780. [CrossRef]

20. Grzegorczyk, T.; Sliwinski, R.; Kaczmarek, J. Attractiveness of augmented reality to consumers. Technol.Anal. Strateg. 2019, 31, 1257–1269. [CrossRef]

21. Berryman, D.R. Augmented reality: A review. Med. Ref. Serv. Q. 2012, 31, 212–218. [CrossRef] [PubMed]22. Montola, M. A ludological view on the pervasive mixed-reality game research paradigm. Pers. Ubiquitous

Comput. 2011, 15, 3–12. [CrossRef]23. Cummings, J.J.; Bailenson, J.N. How immersive is enough? A meta-analysis of the effect of immersive

technology on user presence. Media Psychol. 2016, 19, 272–309. [CrossRef]24. Connolly, T.M.; Boyle, E.A.; MacArthur, E.; Hainey, T.; Boyle, J.M. A systematic literature review of empirical

evidence on computer games and serious games. Comput. Educ. 2012, 59, 661–686. [CrossRef]25. Akçayır, M.; Akçayır, G. Advantages and challenges associated with augmented reality for education: A

systematic review of the literature. Educ. Res. Rev. 2017, 20, 1–11. [CrossRef]26. Garzón, J.; Pavón, J.; Baldiris, S. Systematic review and meta-analysis of augmented reality in educational

settings. Virtual Real. 2019, 23, 447–459. [CrossRef]27. Wu, H.K.; Lee, S.W.Y.; Chang, H.Y.; Liang, J.C. Current status, opportunities and challenges of augmented

reality in education. Comput. Educ. 2013, 62, 41–49. [CrossRef]28. Bacca, J.; Baldiris, S.; Fabregat, R.; Graf, S. Augmented reality trends in education: A systematic review of

research and applications. Educ. Technol. Soc. 2014, 17, 133–149.29. Kreijns, K.; Kirschner, P.A.; Jochems, W.; Van Buuren, H. Determining sociability, social space, and social

presence in (a) synchronous collaborative groups. CyberPsychol. Behav. 2004, 7, 155–172. [CrossRef] [PubMed]30. Kreijns, K.; Kirschner, P.A. Extending the SIPS-Model: A research framework for online collaborative

learning. In European Conference on Technology Enhanced Learning; Pammer-Schindler, V., Pérez-Sanagustín, M.,Drachsler, H., Elferink, R., Scheffel, M., Eds.; Springer: Cham, Switzerland, 2018; pp. 277–290. [CrossRef]

31. Katz, E.; Blumler, J.G.; Gurevitch, M. Uses and gratifications research. Public Opin. Q. 1973, 37, 509–523.[CrossRef]

32. McQuail, D. Mass Communication Theory, 5th ed.; Sage: London, UK, 2005.33. Whelan, D.C.; Zelenski, J.M. Experimental evidence that positive moods cause sociability. Soc. Psychol. Pers.

Sci. 2012, 3, 430–437. [CrossRef]34. Laurent, J.; Catanzaro, S.J.; Joiner, T.E., Jr.; Rudolph, K.D.; Potter, K.I.; Lambert, S.; Osborne, L.; Gathright, T. A

measure of positive and negative affect for children: Scale development and preliminary validation. Psychol.Assess. 1999, 11, 326–338. [CrossRef]

35. Watson, D.; Clark, L. The PANAS-X: Manual for the Positive and Negative Affect Schedule-Expanded Form; TheUniversity of Iowa: Iowa City, IA, USA, 1999. [CrossRef]

36. Dinno, A. Nonparametric pairwise multiple comparisons in independent groups using Dunn’s test. Stata J.2015, 15, 292–300. [CrossRef]

37. Hsieh, H.F.; Shannon, S.E. Three approaches to qualitative content analysis. Qual. Health Res. 2005, 15,1277–1288. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).