Teaching Language and Culture with a Virtual Reality Game eland/papers/chi2017_vr.pdf‚ ‚ Teaching Language and Culture with a Virtual Reality ... Learning a second language

  • View

  • Download

Embed Size (px)

Text of Teaching Language and Culture with a Virtual Reality Game...

  • Teaching Language and Culture with a Virtual Reality GameAlan Cheng1, Lei Yang2, and Erik Andersen1

    1Department of Computer Science, 2Department of City and Regional Planning, Cornell University{ayc48, ly292, ela63


    ABSTRACTMany people want to learn a language but find it difficult tostay engaged. Ideally, we would have language learning toolsthat can make language learning more enjoyable by simulatingimmersion in a foreign language environment. Therefore, weadapted Crystallize, a 3D video game for learning Japanese,so that it can be played in virtual reality with the Oculus Rift.Specifically, we explored whether we could leverage virtualreality technology to teach embodied cultural interaction, suchas bowing in Japanese greetings. To evaluate the impact ofour virtual reality game designs, we conducted a formativeuser study with 68 participants. We present results showingthat the virtual reality design trained players how and when tobow, and that it increased participants sense of involvementin Japanese culture. Our results suggest that virtual realitytechnology provides an opportunity to leverage culturally-relevant physical interaction, which can enhance the design oflanguage learning technology and virtual reality games.

    Author Keywordslanguage learning, video games, virtual reality

    ACM Classification KeywordsH.5.0. Information Interfaces and Presentation: General

    INTRODUCTIONLearning a second language is a goal shared by many, fromchildren in bilingual environments to adult immigrants seek-ing employment to people wishing to travel abroad. How-ever, many people find it difficult to stay engaged and also tolearn vocabulary and grammar in context. Although manypeople learn languages through popular online tools likeDuoLingo [36] and Rosetta Stone [26], much of the vibrancyof the real-world experiences that can make language learningfun and relevant do not translate to these systems.

    On the other end of the spectrum, studying abroad providesample opportunities to be exposed to a foreign language, as-similate into another culture, and apply language in context.Indeed, many studies have analyzed the positive effects ofstudying abroad on language acquisition [31, 15]. However,studying abroad is not an option for everyone due to finan-cial cost, lack of opportunity, or insufficient time. Ideally, wePermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

    CHI 2017, May 06 - 11, 2017, Denver, CO, USA 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM.ISBN 978-1-4503-4655-9/17/05. . . $15.00

    DOI: http://dx.doi.org/10.1145/3025453.3025857

    would have immersive language tools that simulate the experi-ence of being in a foreign language environment as deeply aspossible, so that an aspiring learner can learn both languageand culture from observation, as well as harness their potentialfor motivation.

    The emergence of high-quality, low-cost virtual reality head-sets in recent years has sparked a surge in interest in the tech-nology, including for educational purposes [17, 22]. Virtualreality has also been used for virtual tourism [16, 5] to situatepeople in other places without actually being present.

    In this paper, we discuss our experiences in designing a virtualreality game for learning language. Building off of Crys-tallize [9, 10], an existing 3D video game for learning theJapanese language, we created a new version of this game thatworks with the Oculus Rift, a virtual reality headset system. Inparticular, we explored whether we could use virtual reality todesign game mechanics around culturally-relevant embodiedphysical interaction, such as bowing in Japanese greetings.

    To evaluate the effectiveness of our VR game design process,we conducted a formative user evaluation with 68 participants.Our results provide initial evidence that porting to VR andadding VR-specific game mechanics to Crystallize was usefulfor increasing involvement in Japanese culture and teachingplayers how to bow. However, players encountered some chal-lenges, including feeling sick while using the virtual realityheadset. The impact on learning itself was also inconclusive.Nevertheless, this formative evaluation indicates that we wereable to leverage some benefits of VR, which will inform fu-ture development of the game. Furthermore, the integrationof physical cultural artifacts like bowing has implications forthe design of language learning technology and virtual realitygames.


    Virtual reality and educationVirtual reality has great potential for education across a widespectrum of fields. One such field is surgical education. Cur-rently, aspiring surgeons learn surgical skills in the operatingroom, and while this provides the most realistic environment,there is much left to be desired from a pedagogical perspective.The focus is on the patient rather than the learner, and steps inthe surgery can not be adapted to the students needs. However,a VR surgical environment would allow students to practiceand learn without any danger to an actual patient. The VRenvironment can cater to students by allowing them to makemistakes, repeat steps, and pause as needed [17]. Further-more, surgical competence can both be taught and measuredautomatically with VR software [22].


  • Construct3D is an augmented reality mathematics and geom-etry learning tool that allows students to draw and visualizevirtual three-dimensional objects in real three-dimensionalspace. Student participants in a pilot study by Kaufmann et al.were able to learn how to use the VR tool quickly and thoughtthat it created a good environment for experimentation, whichsuggests that tools like Construct3D can be effective supple-ments to a traditional classroom curriculum [19].

    The sense of presence is the defining experience of virtualreality [28]. The main factors thought to influence presenceare: high-resolution information, environmental consistency,possibility for navigation and interaction with other avatars,and similarity of a virtual avatar to ones own body [34]. Im-mersion and involvement are both necessary for experiencingpresence, and presence leads to increased learning and perfor-mance [38]. Therefore, we focused our design of the virtualreality version of Crystallize around enabling presence.

    Language learning toolsThere are many computer-based tools for learning languages.MicroMandarin utilizes the users real world location to sug-gest vocabulary words [14]. Tip Tap Tones trains users torecognize tones in Chinese [13]. Dearman et al. used a desk-top wallpaper to teach vocabulary [12]. We build on theseideas and make them more engaging by creating a holisticexperience in a 3D virtual reality video game.

    Rosetta Stone [26] is a highly successful tool that mainlyteaches language through a series of pictures. A typical taskfeatures a set of four or more pictures that each show a certainsituation, such as a boy eating or a girl running, and asks theuser to identify the picture that most closely matches a phrasein the target language. Rosetta Stone offers many advantagesover traditional curricula: the learner receives immediate feed-back, information is presented in a visual context, and meaningis often learned through inference. Our work builds on theseideas but also features an interactive 3D environment that pro-vides a deep, situated, virtual reality context and prioritizesexperimentation and completion of tasks and goals in thatenvironment.

    DuoLingo [36] is another successful tool that teaches languageby asking the learner to translate sentences. DuoLingo also hasa highly structured learning progression and has achievementsand point systems to motivate users. Although 34 hours ofDuoLingo has been shown to be equivalent to a first-semestercollege course in Spanish, many students stopped using itwithin 2 hours [35]. Our work also uses game-like elements toincentivize users, but it teaches linguistic concepts in a situatedphysical context rather than through pure translation.

    There are some existing video games that focus on teachinglanguages. A good summary can be found in [29]. For exam-ple, Sanjigenjiten uses a 3D environment to teach vocabularywords [18]. This game makes use of visual context to teachvocabulary meanings, but it does not provide a deep learningprogression. Zengo Sayu [25] is a virtual environment forlearning Japanese in which the player explores a location andhears audio cues of words. However, it is not really a game.

    Crystallize is a 3D language learning game that teaches lan-guage material in its physical context, motivated by the theo-ries of situated cognition [8] and encoding specificity [32]. Thegame provides opportunities for players to learn words fromcontext and gamifies language learning through a quest systemthat requires players to solve challenges that involve learn-ing new words and using them to construct target sentences.The conceptual components of this game involve understand-ing specific vocabulary words and understanding grammaticalconstructions. Two evaluations of Crystallize have been con-ducted: a preliminary lab study with 42 participants [9] andan in the wild study with 186 participants from Reddit [10].Both studies showed statistically significant learning gains.

    Interaction des