standard Effectiveness Usability - user interface ... · formative evaluations focus on the design features of the applica-tion (i.e., usability and fun, quality of the visual representation,

Paper ID Title Description of the SG quality assessment according to paper Mapping with ISO 25010 standard

[P01] SMILE: an immersive learning game for deaf and hearing children

To evaluate efficiency, effectiveness, and user satisfaction, formative evaluations focus on the design features of the applica-tion (i.e., usability and fun, quality of the visual representation, and quality of the signing motion), the summative evaluation tests learning and knowledge acquisition.

Effectiveness Usability - user interface aesthetics Usability – operability Satisfaction-pleasure

[P02] A Game Scale to Evalu-ate Educational Com-puter Games

The aim is to identify the quality of any educational computer game in terms of three main categories which are enjoyment, learning and usability. Enjoyment (1) Challenge; (2) Curiosity & Mystery; (3) Clear goals; (4) Social Interaction; (5) Diversion; (6) Fantasy; (7) Arousal; Learning (1) Learner; (2) Curriculum; (3) Feedback; (4) Context; (5) Direct experience;

Effectiveness Satisfaction - pleasure

[P03] The implementation of team based assessment in serious games

Questionnaires focus on:

player´s game enjoyement

team assessment (has a potentially major role to play in seri-ous games, allowing individuals to evaluate the role they play in a team, what team they fit into best and how their teams operate as a whole).

Effectiveness

[P04] Assessment in Serious Games. An Enhanced Approach for Integrated Assessment Forms and Feedback to Support Guided Learning

To evaluate the effectiveness through assessment rules in order to guide students through dynamic feedback

Effectiveness

[P05] Evaluation of the Learn-ing Effectiveness of Game based and Hands-on Gear Train Laboratories

Effective in: Teaching specific concepts Increasing knowledge of field Better skills as engineer Satisfaction with: overall satisfaction, feeling of immersion, ease of use, total time required, clarity of instructions, ability to control assembly, ability to set parameters and ease of collecting data

Effectiveness Satisfaction – pleasure Usability - operability

[P06] Evaluation of usefulness of the Elinor console for home-based stroke rehabilitation

To explore the usefulness and effectiveness of a home-based system for rehabilitation of the upper extremity for persons suf-fering from stroke.

Effectiveness Satisfaction – usefulness

[P07] MOW: Augmented reality game to learn words in different lan-guages - Case study: Learning English names of animals in elemen-tary school

Progressive domain of oral recognition of words and concepts and their corresponding written form

Effectiveness

[P08] Enhancing the Educa-tional Value of Video Games

Objective evaluation: to whether (and how) the learning mecha-nisms embedded in an educational game have a (negative) impact on the overall entertainment (here we assume that entertain-ment is the driving factor for playing state-of the-art VGs).

Effectiveness

[P09] Game-like language learning in 3-D virtual environments

To evaluate student learning and motivation Effectiveness

[P10] An implementation of a structured and highly engaging learning envi-ronment on educational games for elementary education

1. Was it easy to start the game? 2. Were the graphics clear? 3. Were the graphics appealing? 4. Was the audio clear & appealing? 5. Were the controls easy to use? 6. Were you able to understand the instructions? 7. Was the game too difficult? 8. Was it easy to learn the Science concepts? 9. Did you enjoy playing the game? 10. Was the game exciting? 11. Was the game too fast?

Usability – operability Usability – user interface aesthetics Satisfaction – usefulness Satisfaction – pleasure

[P11] Characterization and evaluation of serious games: A perspective of their use in higher edu-

Ergonomic user interface, graphics richness, complexity, clear purpose and attractive game to players.

Usability - appropriate-ness recognizability, user interface aesthetics


cation Satisfaction - pleasure

[P12] A mixed methods as-sessment of students' flow experiences during a mobile augmented reality science game

Evaluate flow experience (features: flash of intensity, a sense of discovery and the desire for higher performance) using flow as a measure of student engagement with educational computer games and mobile learning games. Post surveys Flow Experiences: I was challenged and I felt I could meet the challenge.(Flow ele-ment: challenge–skill balance) I did things naturally without thinking too much. (Flow element: action–awareness merging) I had a strong sense of what I wanted to do. (Flow element: clear goals) I felt in control of what I was doing. (Flow element: sense of con-trol) I felt I was on track towards my goals. (Flow element: unambigu-ous feedback) I was totally focused on what I was doing. (Flow element: total concentration on task) It felt like nothing else mattered. (Flow element: loss of self-consciousness) I lost my normal sense of time. (Flow element: transformation of time) I really enjoyed what I was doing. (Flow element: intrinsically rewarding experience) I was in the zone. (Flow element: overall experience of being in flow)

Satisfaction - usefulness Satisfaction – pleasure Usability - operability

[P13] Implementing digital game-based learning in schools: Augmented learning environment of 'Europe 2045'

A1. Overall evaluation of Europe 2045 by students A2. What is the impact of the game on learning (in your opinion) A3. Do you think you have learned during the course of playing? B1. Was the game complicated for you? B2. Was it complicated for you to understand the rules of the game? B3. Was the video game interface complicated for you? C1. The most interesting part of the game is (Discussion, negotiations Simulation Encyclopaedia reading) C2. I gained most information from Teacher Simulation (3) (4) Encyclop. (6) C3. How interesting was the text material (the encyclopaedia and the news) for you? D1. Is it important for you that you can choose your project? D2. I identify myself with the role (i.e. the project) I play in the game Yes Probably yes I don’t know Probably not No E. Is it important for you that the game is based on real data

Usability – user inter-face aesthetics Usability – operability

[P14] Are educational com-puter micro-games engaging and effective for knowledge acquisition at high-schools? A quasi-experimental study

Engagement/feeling questions: “Do you think that you have learnt something?”, and “How did you like the game in particular?”


[P15] Case study of FISS: Digital game based learning for a broad range of ages

To enhance learning Effectiveness

[P16] A serious game for learning ultrasound-guided needle place-ment skills

To evaluate effective training Effectiveness

[P17] Becoming chemists through game-based inquiry learning: The case of legends of Alkhimia

To evaluate efficacy curriculum Alkhimia in helping students strengthen their understanding of chemistry

Effectiveness

[P18] The Influence of an Educational Computer

To ascertain the relative effectiveness of gameplay as a form of cultural learning:



Game on Children’s Cultural Identities

(MtlCI) 1. I prefer Taiwanese traditional food to western fast food. 2. Comparing with imported computer games, I lack confidence in games that are designed in Taiwan. 3. I feel that imported clothes are better than locally made clothes. 4. I think, there is no need to keep those historical relics located in valuable city land. 5. I feel that Taiwan is not as beautiful as many other countries. 6. We may find and import cheaper goods from abroad, thus, there is no need to develop our traditional industries. 7. I feel that Taiwan has convenient transportation. (SocCI) 1. I feel that we need to preserve and protect different cultures of different people in Taiwan. 2. I am interested in reading and knowing information concerning local customs and practices. 3. I hope I can taste local specialties all over Taiwan. 4. I feel there is no need to preserve traditional cultures in my society for they are outdated. 5. I believe that cultures of different ethnic groups in Taiwan are equally fine. (SpiCI) 1. If I could, I would like to let more people in different parts of the world know more about Taiwan. 2. Respect cultures of different ethical groups in my country will make us more united. 3. To worship our ancestor is a superior traditional virtue. 4. All cultures of different societies need to be preserved. 5. Invading of foreign cultures through media will eventually ex-tinguish our local culture. (SciCI) 1. I am interested in doing science activities. 2. Learning science can help me to solve my future problems. 3. I consider that science in my country is not well developed as other countries. 4. Technology products’ quality of my country is no lower than those of other countries. 5. I hope to be a scientist when I grow up. 6. Science was invented in the western world, and then, passed to us.

[P19] Quest Island: Develop-ing quest-driven learn-ing model by blending learning tasks with game quests in a virtual world

To evaluate its effectiveness Effectiveness Satisfaction - pleasure

[P20] An Exploratory Study of Game-based M-learning for Software Project Mana-gement

To evaluate the effectiveness of m-learning games Effectiveness

[P21] Exploring Newtonian mechanics in a concep-tually-integrated digital game: Comparison of learning and affective outcomes for students in Taiwan and the Unit-ed States

To evaluate the degree to which learning in the game reflects academically desirable learning

Effectiveness

[P22] An alternate reality I enjoyed playing the game Satisfaction – pleasure


game for language learning: ARGuing for multilingual motivation

I enjoyed using a foreing language The goal in the game were clear The game managed to keep my concentration The game kept me sufficiently challenged The game provided appropiate feedback on success or progress towards completing game tasks I would be willing to play the game over a prolonged period of time I would be willing to play the game a part of a foreing language course The game was engaging I would like to play the game again The game motivated me to use a foreing language I felt distracted while playing the game Playing the game provided me with a challenge Playing the game allowed me to compete Playing the game allowed me to cooperate Playing the game gave me a feeling recognition Playing the game gave me a sense of control Playing the game allowed me to enter a fantasy world Playing the game simulated my curiosity Playing the game helped me to fill my leisure time Playing the game gave me pleasure Playing the game helped me to relax Playing the game helped me to ovoid other activities Playing the game mademe feel good Playing the game stimulated me emotionally Playing the game allowed me ti escaoe the stress of life Playing the game helped to stop me from being bored Playing the game helped me to release tension Aspects of the game : the story, the narrative, the characters, the collaboration, the help and sipport, the realism, solving quest

Satisfaction – usefulness Usability – operability Usability – user interface aesthetics Context coverage – con-text completeness Fuctional suitability – functional completeness

[P23] Explore! Possibilities and challenges of mo-bile learning

Collaboration, Fun, Challenge, Artefacts, Game master, Learning Evaluate the effectiveness of the electronic version of the excur-sion-game as a learning technique

Effectiveness Satistaction - pleasure

[P24] Game Based Spelling Learning

Evaluation of learning effectiveness and the degree of amusement of the game

Effectiveness Satistaction - pleasure

[P25] Is Game-Based Learning Suitable for Engineering Education?

To assess the effectiveness and efficiency of an educational game Interest/Enjoyment I enjoyed playing this game very much This game was fun to play. I thought this was a boring game. This game did not hold my attention at all. I would describe this game as very interesting. I thought this game was quite enjoyable. While I was playing this game, I was thinking about how much I enjoyed it. Perceived Competence I think I am pretty good at this game. I think I did pretty well at this game, compared to other students. After working at this game for a while, I felt pretty competent. I am satisfied with my performance at this task. I was pretty skilled at this game. User Experience I enjoyed learning through the game I could identify myself with the main character. After playing the game I can better imagine the era presented in the game. When I started playing the game I got enough information to start playing it. It took me some time to figure out how to play the game. There was enough feedback to know what I did wrong during the game and was able to do it correctly afterwards.

Effectiveness Satisfaction - usefulness Satisfaction – pleasure Usability - operability Usability - user interface aesthetics Usability - appropiate-ness recognisability Functional suitability - functional appropiate-ness

[P26] A methodological pro-posal for learning games selection and quality assessment

Learning objectives Clear definition of objectives Correspondence between established objectives and the objec-tives that can actually be reached by using the learning game

Effectiveness Usability - user interface aesthetics Usability – operability


Context of usage Clarity of practical instructions for the use of the learning game Indications/suggestions on the context in which the learning game can be used Coherence of the game with the targeted context Coherence between the learning game structure and the planned training and learning context Link between the learning game activities and the profession-al/working context Didactic strategy Indication of the average play time Incentives and support to motivation Support to engagement and fun Coherence between the game strategy and learning objectives Quality of the game strategy with the individual player character-istics Clarity of the game environment/setting Organisation and structure of the learning game Clarity of the rules to be followed and decision making process Coherence between rules and consequence Constant focus on the player experience Clear definition of roles (eg, player, instructors, animators, etc) Coherence of the social and collaborative activity with the objec-tives Communication and media Clear and user-friendly tone and language Quality of the interaction between the learning game and the user/player Quality of the interaction among users/players, etc Coherence between the media used in the learning game and the contents, the established objectives and the target group Structure and organisation Easy to be installed (for off-line digital games) Modularity of the design Modularity in the use Aesthetic and usage of the media Quality of user/game interface Possibility of intervention on the use of materials (stop, rewind) Positioning of the different elements on the screen Technical quality Quality of image definition Quality of image composition Rhythm of images Quality and definition of audio Integration between audio and image elements Synchronism between audio and image elements Quality of typographic characteristics and clarity of texts Quality of typographic characteristics and clarity of texts Technical quality of drawings Technical quality of pictures Technical quality of graphic animations

Usability - user error protection Satisfaction – pleasure Functional suitability - functional completeness Functional suitability - functional correcteness Reliability – fault toler-ance

[P27] Experimental evalua-tion of an educational game for improved learning in introductory computing

To test learning at all levels of Bloom’s taxonomy (increase stu-dent learning and attitudes)

Effectiveness

[P28] A learning objective focused methodology for the design and eval-uation of game-based tutors

To maximize the chances for learning and detection of that learn-ing

Effectiveness

[P29] Game interaction state graphs for evaluation of user engagement in explorative and experi-ence-based training games

Evaluation of a player’s engagement in different behaviours and situations in a gamebased training task.

Satisfaction – pleasure Satisfaction - engage-ment


[P30] Assessing performance competence in training games

To obtain a quantitative assessment of performance competence in serious training games based on an evaluation of behavioral patterns of the player


[P31] Evaluation of serious games in mobile plat-forms with QEF: QEF (Quantitative Evalua-tion Framework)

Functionality reflects the characteristics of the educational software related to its operational aspects. It aggregates two factors:

easy of use content’s quality Efficiency measure the system’s ability for presenting differ-

ent views on its content with minimum effort aggregates, in the case of educational software, four factors:

audiovisual quality technical and static elements navigation and interaction originality

use of advanced technology

Usability – Operability, Usability – User interface aesthetics Efficiency Performance efficiency – time behavior

[P32] A serious heritage game for art history: Design and evaluation of ThIA-TRO

To increases motivation to learn and raises interest in art history and cultural heritage in general

Effectiveness

[P33] Design and evaluation of a serious game for immersive cultural training

Significant learning Potential Is the game fun to play and is the design of the 3D environment considered as convincing


[P34] ARTournament: A mo-bile casual game to explore art history

¿Is the play successfully (player performance on various levels)? This answer demonstrate that the play transfers knowledge about art history to the players

Effectiveness

[P35] EGameFlow: A scale to measure learners’ en-joyment of e-learning games

Concentration The game grabs my attention The game provides content that stimulates my attention Most of the gaming activities are related to the learning task No distraction from the task is highlighted Generally speaking, I can remain concentrated in the game I am not distracted from tasks that the player should concentrate on I am not burdened with tasks that seem unrelated Workload in the game is adequate Goal Clarity Overall game goals were presented in the beginning of the game Overall game goals were presented clearly Intermediate goals were presented in the beginning of each scene Intermediate goals were presented clearly I understand the learning goals through the game Feedback I receive feedback on my progress in the game I receive immediate feedback on my actions I am notified of new tasks immediately I am notified of new events immediately I receive information on my success (or failure) of intermediate goals immediately I receive information on my status, such as score or level Challenge I enjoy the game without feeling bored or anxiousa The challenge is adequate, neither too difficult nor too easy The game provides ‘‘hints” in text that help me overcome the challenges The game provides ‘‘online support” that helps me overcome the challenges The game provides video or audio auxiliaries that help me over-come the challenges My skill gradually improves through the course of overcoming the challenges I am encouraged by the improvement of my skillsa The difficulty of challenges increase as my skills improved. The game provides new challenges with an appropriate pacing The game provides different levels of challenges that tailor to different players

Effectiveness Satisfaction- pleasure Usability - User interface aesthetics Usability - operability Usability – user error protection Functional suitability - functional completeness Functional suitability - functional correcteness Functional suitability - functional appropiate-ness Reliability – fault toler-ance


Autonomy I feel a sense of control the menu (such as start, stop, save, etc.) I feel a sense of control over actions of roles or objects I feel a sense of control over interactions between roles or objects The game does not allow players to make errors to a degree that they cannot progress in the game The game supports my recovery from errors I feel that I can use strategies freely I feel a sense of control and impact over the game I know next step in the game I feel a sense of control over the game Immersion I forget about time passing while playing the game I become unaware of my surroundings while playing the game I temporarily forget worries about everyday life while playing the game I experience an altered sense of time I can become involved in the game I feel emotionally involved in the game I feel viscerally involved in the game Social Interaction I feel cooperative toward other classmates I strongly collaborate with other classmates The cooperation in the game is helpful to the learning The game supports social interaction between players (chat, etc) The game supports communities within the game The game supports communities outside the game Knowledge Improvement The game increases my knowledge I catch the basic ideas of the knowledge taught I try to apply the knowledge in the game The game motivates the player to integrate the knowledge taught I want to know more about the knowledge taught

[P36] Evaluating social games for kids and teenagers diagnosed with cancer

Game likeability and player experience Usability – user interface aesthetics Satisfaction – pleasure

[P37] Assessment of motiva-tion in gaming based e-learning

Assessment of motivation in real time by taking into consideration indicators such as: competence, effort, source of motivation (e.g. amotivation, intrinsic, extrinsic), and interest.

Satisfaction - pleasure

[P38] Serious games and learning effectiveness: The case of It's a Deal!

Effective learning – to gain knowledge of intercultural business communication, in instructive because it helps users to learn business English letter writing rhetorical conventions. To have enjoyed the experience of learning through a game

Effectiveness

[P39] Using serious games in computer science edu-cation

First question of the questionnaire considered the usefulness of the game session in general. Second question of the questionnaire considered students' moti-vation and learning during the games.

Effectiveness

[P40] Usability testing with children: Laboratory vs field studies

To gives insight different perspectives children bring to computer products and such basic elements as navigation and productivity

Effectiveness

[P41] Conceptual framework for a heuristics based methodology for inter-face evaluation of edu-cational games

Interface issues: Characteristics of the educational games that have an effect on its usability. Pedagogical issues: Characteristics that facilitate learning. Multimedia issues: Characteristics of the educational games that consist of multime-dia delopements and have an effect on its usability. Playability issues: The game study support the game play and it is meaningful, chal-lenge, strategy and pace in balance, player is in control and play-ers see the progress in the game and can compare the result.

Interface: Usability - User interface aesthetics Usability - operability Pedagogical: Effectiveness Satisfaction- pleasure Portability - adaptability Multimedia: Usability – user interface aesthetics Playability: Satisfaction- pleasure


Usability - operability

[P42] Development and po-tential analysis of Heu-ristic Evaluation for educational computer game (PHEG)

Interface

Uses aesthetic and minimalist design

Maximizes consistency and matches standards

The uses of space, color and text are according to the princi-ples of screen design

The uses of text, color and font follow the principles of read-ability

The quality of multimedia elements (text, image, animation, video and sound) used is acceptable

The use of multimedia elements support meaningfully the text provided.

The integration of presentation means is well-coordinated

The uses of multimedia elements enhance the presentation of information

The interactivity of the courseware is suitable to learners/students level

Provide specific and self- identified key for specific task (exit, glossary, main, objective)

Educational/Pedagogical

Clear goal and learning objectives

The activities are interesting and engaging

The design and the contents are reliable and proven

Offers the ability to select the level of difficulty in games Content

Reliable content with correct flow.

Clear and understandable structure of contents.

Navigation is easy and accurate.

Supporting materials are sufficient and relevant.

Materials are interesting and engaging. Multimedia

Usage of multimedia elements are acceptable

Combination of multimedia elements are adequate

The presentation of multimedia elements are well manage

Suitability of multimedia elements for specific use MM5 Playability

Challenge provided are up to the users standard

Users able to strategies

The pace of the game are in balance

Players able to control the game Players able to know the progress of the game

Effectiveness Usability - User interface aesthetics Usability - operability Satisfaction- pleasure Portability - adaptability

[P43] Heuristics evaluation in computer games

Interface































[P44] Analyzing Critical Usa-bility Problems In Edu-cational Computer Game (UsaECG)

Interface





























Usability - User interface aesthetics Usability - operability Effectiveness Efficiency Satisfaction- pleasure Portability - adaptability

[P45] Methodology to evalu-ate interface of educa-tional computer game







Usability - User interface aesthetics Usability - operability Usability –learneability Usability - user error protection Portability - adaptability Effectiveness Efficiency









Offers the ability to select the level of difficulty in games





Materials are interesting and engaging




Suitability of multimedia elements for specific use MM5





Satisfaction- pleasure

[P46] Investigating the effects of educational game with Wii remote on outcomes of learning

Four aspects are evaluated upon this educational game: Memorability of learning activity, enjoyment level during learning process, interesting level of learning process and motivation ef-fect after learning.

Effectiveness

[P47] A knowledge engineer-ing approach to devel-oping educational com-puter games for im-proving students' dif-ferentiating knowledge

Questionnaire of satisfaction and acceptance for educational computer games. Three dimensions of the questionnaire: perceived ease of use (five items); perceived usefulness (six items); and learning satisfaction (seven items).


[P48] A comparison of gains between educational games and a traditional ITS

Increase self-efficacy, interest, engagement, and self-regulation Improve reading comprehension by teaching self-explanation in combination with effective reading strategies. Improve understanding of difficult science texts

Effectiveness

[P49] An Experimental Evalu-ation of Scaffolded Educational Games Design for Programming

To improve teaching and learning programming Effectiveness

[P50] Evaluation of an immer-sive learning pro-gramme to support triage training: In-game feedback and its effect on learning transfer

Perform better in terms of the accuracy in applying the triage protocol

Effectiveness

[P51] An evaluative study on VISOLE-Virtual interac-tive student-oriented learning environment

Students’ advancement in the multidisciplinary subject knowledge and enhancement on their generic skills for problem solving

Effectiveness

[P52] Improving literacy in rural India: Cellphone games in an after-school program

To evaluate improving Literacy Effectiveness Satisfaction - pleasure

[P53] Studying Player Experi-ence in a Collaborative Embodied Interaction Game

To evaluate the learning experience and also how much children enjoyed playing the game. “User engagement” questionnaire: Which game of all in the “Polymechanon” did you like the most? What did you like most about the Wobble Board? Is there something that you did not like in the Wobble Board? Which is the easiest thing about the Wobble Board? Which is the most difficult thing about the Wobble Board? If you were to play the Wobble Board again, what do you think

Effectiveness Usability - user interface aesthetics Usability - operability Satisfaction-pleasure


you should do in order to drive the balls into the holes more easi-ly? Do you believe that the Wobble Board is just a game or helps you learn something and, if yes, what is that? Effectiveness To examine the deeper ways that children of this age think about the concepts we asked in physics

[P54] Enhancing learning through an online sec-ondary school educa-tional game

To evaluate the type of learning enhanced by online educational game through five indicators.

Effectiveness

[P55] Development of a walk-ing game for the elderly using controllers of hand buttons and foot boards

To test the efficacy of the game Effectiveness

[P56 ] FreshUP - Implementa-tion and evaluation of a pervasive game for freshmen

To effectively conveying learning content such as information about university processes in a motivating manner.

Effectiveness

[P57] Pedagogical agent in Multimedia Interactive Modules for Learning - MIMLE

Testing of learning efficiency Effectiveness

[P58] Results of user interface evaluation of serious games for students with intellectual disability

games enjoyable Using the software was tiring The content of the software was engaging Playing with the software was boring The games were easy to play Activities in game play were predictable I used the software willingly usability of the software The software displays realistic situations The presented situations were relevant and important Are you satisfied with the quantity and diversity of the questions used? Using the software was easy Software’s manageability There is sufficient opportunity to correct or revisit responses given in the game The content areas between levels are distinct It was easy to follow the activities in the game The noises or music used in the games were disturbing The speech used was clear and understandable Failures were typically results of disorientation Starting the software was an easy process Closing the software/Quitting at the end of the game was easy Closing the software/Quitting at the end of the game was easy graphics The software’s look is realistic The software’s look is likeable The software’s look was appropriate to its aim The pictures used were easy to recognize The illustrations and backgrounds used were helpful The connection between the pictures and the actions rendered to them were unambiguous Failures were typically results of not being able to recognise pic-tures Failures were typically results of not understanding instructions The software was rich in (visual and audio) stimuli The screens of the software were detailed

Usability - User interface aesthetics Usability - operability Usability -appropriateness recog-nizability Satisfaction – Pleasure

[P59] Vicarious learning with a digital educational game: Eye-tracking and survey-based evalua-tion approaches

Participants’ perceived qualities of the game: enjoyability; ease of use; duration; understandability. How easy would you find this Mission? How enjoyable would you find this Mission? How would you find the duration of this Mission? How easy would it be for you to understand the instructions in

Effectiveness Satisfaction – pleasure Usability – operability Usability - learneability


the Mission?

[P60] Evaluating user experi-ence of adaptive digital educational games with Activity Theory

General gameplay experience To observe general gameplay expe-rience along seven dimensions: immersion sensation of being surrounded by a different form of reality that captures all of a player’s attention; challenge refers to the difference between the skill level required by a game activity and the player’s existing skill; tension refers to the emotional or physical threats resulting from the failure to respond appropriately; competence is the ability to perform a specific task or an action in the game successfully; flow is a notion to describe those moments when a player is completely and totally absorbed in a game activity; positive and negative affect are related to other personality states and traits, such as anxiety and happy. Learning experience: goals whether the game provides the players with explicit, under-standable goals at appropriate times to help them concéntrate on the game activities feedback is offered in response to incorrect actions and provide an opportunity for learning content appropriateness means that the game content should be consistent with the goal of the educational context where the game is employed. integration refers to the notion that a game activity should draw directly on the knowledge and skills that the game is designed to foster in its players, and promote reflection on or application of such knowledge and skills media matching indicates that contents should be rendered using the most appropriate media(e.g., animation, sound,picture,text). Usability problems: game interface - e.g., button, navigation game mechanism - e.g., ease of use/control, learnability gameplay - e.g., goal, responsiveness

Satisfaction – Pleasure Satisfaction – Usefulness Effectiveness content appropriateness Usability – appropriate-ness recognizability Usability - User interface aesthetics Usability - Operability Usability - Learnability

[P61] Learning by gaming - Evaluation of an online game for children

What did you like about the game?, What did you dislike about the game?, What did you think of the story?, What did you think of the Characters?, If you could only change one thing about the game, what would it be?, Is this a game that you would play only in school or would you play it in your own free time?, Play more than once?, Recommend to friends?, Do you have any other comments about the game?.

Satisfaction - pleasure

[P62] Development and de-ployment of traffic control game: Integra-tion with traffic engi-neering curriculum for teaching high school students

How would you rate the difficulty of these activities? How much would you say these activities were fun and interesting to you? Would you enjoy activities like these in your high school science classes?

Satisfacton - pleasure Usability – user interface aesthetics Usability - operability

[P63] An Approach to Evalua-tion Component Design in Building Serious Game

To evaluate component design effectiveness in building serious games

Effectiveness

[P64] Serious Games for Movement Therapy after Stroke

To evaluate if the provision of serious games based interventions improves motor outcome after stroke

Effectiveness

[P65] Video games as a medi-um for software educa-tion

Rapid code comprehension with visualisation software (the game teaches basic programming)

Effectiveness

[P66] Fun and learning: The power of narrative

To investigate how the games affect learning experiences and understanding of the physics concepts

Effectiveness

[P67] A 3D educational mo-bile game to enhance student's spatial skills

OVERALL REACTIONS TO THE SOFTWARE terrible / wonderful difficult / easy frustrating / satisfying inadequate power / adequate power dull / stimulating

Satisfaction – pleasure Usability – operability Usability – user interface aesthetics Usability – learneability Performance efficiency –


rigid / flexible SCREEN Characters on the computer screen hard to read / easy to read Highlighting on the screen simplifies task not at all / very much Organization of information on screen confusing / very clear Sequence of screens confusing / very clear TERMINOLOGY AND SYSTEM INFORMATION Use of terms throughout system inconsistent / consistent Computer terminology is related to the task you are doing never / always Position of messages on screen inconsistent / consistent Messages on screen which prompt user for input confusing / clear Computer keeps you informed about what it is doing never / always Error messages unhelpful / helpful LEARNING Learning to operate the system difficult / easy Exploring new features by trial and error difficult / easy Remembering names and use of commands difficult / easy Tasks can be performed in a straight-forward manner never / always Help messages on the screen unhelpful / helpful Supplemental reference materials confusing / clear SYSTEM CAPABILITIES System speed too slow fast / enough System reliability unreliable / reliable System tends to be noisy / quiet Correcting your mistakes difficult / easy Experienced and inexperienced users’ needs are taken into con-sideration never / always

time behavior Reliability – maturity Reliability – fault toler-ance

[P68] Usability testing for educational computer game using observation method

Usabilidad:

Effectiveness: The accuracy and completeness with which customers achieve specified goals.

Efficiency: The accuracy and completeness of goals achieved in relation to resources.

Satisfaction: Freedom from discomfort and positive attitudes toward the use of the system.

Effectiveness Efficiency Satisfaction- pleasure

[P69] Towards a Comprehen-sive Methodology for the Research and Eval-uation of Serious Games

1. What are the requirements and design principles for a compre-hensive social-scientific methodology for the evaluation of SG? 2. To what extend does SG contribute to advanced learning? 3. What factors contribute to, or determine this learning? 4. To what extend and under what conditions can SG based learn-ing be transferred to the real world? Pre-game Socio-demographics: e.g. sex, age, nationality, culture etc. Previous experiences / skills: with computers, with games, with virtual learning environments, etc. Attitudes: e.g. change, conflicts, intrinsic and extrinsic motivation, learning styles, etc. Skills: e.g. personal competence (with games, with learning, with professional skills, etc.) Behaviors (behavioral intentions): Group, team, organizational characteristics: team/group conflict, psychological safety, psychological collectivism, team and organi-zational commitment, etc. In-game Game performance: e.g. based upon in-game scores like time, avoidable mistakes, etc. Game play: effort; dominance, influence, power etc. Game experience: flow, immersion, presence, etc. Post-game Game experience: e.g. engagement, fun while playing the game Player satisfaction with: 1. the game (e.g., clarity, relevance); 2 the game digital environment (e.g., attractiveness, ease of use); 3. Interaction with other students; 4. The facilitator First order learning (short term, individual, participants). Player

Effectiveness Usability - user interface aesthetics Usability - operability Usability – user error protection Satisfaction - pleasure Functional suitability - functional completeness


learning satisfaction, self-reported, self perceived learning, e.g., broad range of items .Measured changes in knowledge, attitudes, skills, behaviors (behavioral intentions). Second order learning (mid, long term, collective, participants and non-participants). Self-reported, casebased, reconstructive: e.g. asking clients, participants etc. how the GBL results have been implemented. Measured changes in team, group, and organizational characteris-tics: e.g. safety, commitment, performance.

[P70] Speech-enabled card games for incidental vocabulary acquisition in a foreign language

To assess learning gains. The students enjoyed the speaking mode of the game: On a scale from 1 (not at all) to 5 (very much), to what degree did you find interacting with this system enjoyable?”


[P71] Purposeful by Design? A Serious Game Design Assessment Framework

How to structure the assessment of serious games in terms of their design to achive pedagogical effectiveness Purpose It is critical to acknowledge that players bring their own intentions and purposes to the gameplay experience and might understand a game differently than intended by the designers Content& information the content contains all the data and words provided in the game that are visible and made approachable to the players Game mechanism pivotal in-game goal of the game, the operation of the reward system, the main playful obstacles/challenges, the difficulty bal-ancing and the win condition. Fiction&Narrative created fictional space and how it relates to the games purpose Aesthetics & Graphics aesthetic characteristics, imagery, style preferences, artistic me-dia, and the computer graphic techniques Framing the play literary of the players can essentially influence the game-play experience

Usability - operability Usability - user interface aesthetics Usability - appropiate-ness recognisability Functional suitability - functional completeness

[P72] Usability Testing for Serious Games:Making Informed Design Deci-sions with User Data

USER Learning. The user figures out how to perform an action that was unclear before (learn to play), or when the user is actively engag-ing in consuming content (learn content). Reflecting. The user pauses or wonders what to do next. Unlike when the user is confused and does not know what to do, reflect-ing events indicate pause to create action plans within the game space. Satisfied/excited. The user displays a remarkably positive reac-tion. Pleasantly frustrated. The user expresses frustration in a positive manner. A pleasantly frustrating moment urges the user to try to overcome the obstacle again. Frustrated. The user voices or displays negative feelings at not being able to complete the game or not knowing how to do some-thing. A frustrating moment urges the player to stop playing. Confused. The user does not know how to perform an action, misinterprets instructions, and/or does not know what he/she is supposed to do. Annoyed. The user performs properly a task in the game (knows how to do it), but feels negatively about having to do it. Unable to continue (fatal). This is usually the consequence of one or more of the above, or of a fatal technical error. An event is related to when the user becomes definitely stuck and/or cannot continue without the help of the researcher. Such events are highlighted because the origin of these events must always be resolved. SYSTEM Functionality. An event is related to prototype’s functionality when it is the result of the user activating a control item and it is related to one specific action Layout/UI. An event is related to layout/UI when the user makes

Effectiveness Satisfaction – usefulness Satisfaction – trust Satisfaction – pleasure Functional suitability - functional completeness Functional suitability - functional correctness Functional suitability - functional appropriate-ness


a wrong assumption about what a control does, or when the user does not know how to do something (negative events). It is also a layout/UI positive event when a user appreciates the design (fig-ures, attempts, colors, etc.) or having specific information displayed. Gameflow. An event that is caused not by a single specific inter-action, but as a consequence of the game sequences interactions and outputs and the specific gameplay design of the game. Content. A content event is related to text blurbs and other forms of textual information provided by the game. Technical error. A technical error event is related to a noninten-tional glitch in the system that must be corrected.

[P73] Adding features of educational games for teaching physics

Enhancement of student motivation, knowledge and understand-ing.


[P74] Experimental feedback on Prog&Play: A serious game for programming practice

Enhancement of programming skills (serious side)

quantity of work achieved - is evaluated using the game progress (number of missions completed)

acquired knowledge - is evaluated with students’ results of midsemester and final examinations. These examinations are designed by teachers external to the research team and are the same for all students

students’ future courses. System usability - allows identifying bottlenecks which could hinder students (installation problems, hardware requirements, software bugs, etc.). Post questionnaire with questions to evaluate the hierarchy of the players’ needs to identify which part of our serious game needs improvement: Rules need, players seek information to understand the basic rules of the game; Safety need, players need guidance about the game software; Belongingness need, players need to become familiar with the game to feel capable to achieve objectives; Esteem need, players need to be motivated by the game (feed-backs, scores, competition, etc.); Need to know and understand, players need to understand and know more information about the game (e.g. different strategies or hidden items) to reuse it during play time; Aesthetic need, players need good graphics and visual effects, appropriate music, sound effects, etc; Self actualization need players need to be able to transfer their creativity and imagination into the game as long as it conforms to the game rules. Entertainment (game side) has been evaluated with a questionnaire distributed to students at the end of experiments.

Do you appreciate the campaign story (missions)?

Do you think that using programming in Kernel Panic increas-es entertainment?

Teachers’ assessment - allows obtaining a qualitative evaluation of the serious game by teachers To observe teachers’ activities during experiments, we propose to film sessions

Effectiveness Usability – user interface aesthetics Usability – operability Satisfaction – pleasure

[P75] Multi-site evaluation of SimSE

Enjoyable, easy to play, inspection enjoyable, inspection learning, reinforce knowledge, helpful to learning, helpful in course


[P76] Shades of Grey: Playing games in the classroom to enhance student learning

Competing in a team made the game a more valuable learning experience I find it easier to concentrate on the learning when actively involved I find it easier to learn when I’m having fun I feel that my classes would be more interesting if they includ-ed learning games I was motivated to learn by playing the game I enjoyed this learning activity I feel that the technology used enhanced the game I feel that the technology used enhanced the learning experi-

Satisfaction – pleasure Usability – user interface aesthetics Usability - learneability


ence

[P77] PlayLOGO 3D: A 3D interactive video game for early programming education: Let LOGO be a game

Set of evaluation criteria related to usability, game play and edu-cational effectiveness. 40 heuristics were developed Game Usability Audio-visual representation supports the game GUH Screen layout is efficient and visually pleasing GUH Indicators are visible GUH The player understands the terminology GUH Navigation is consistent, logical, and minimalist GUH Game controls are convenient and flexible GUH The game gives feedback on the player’s actions GUH The player cannot make irreversible errors GUH The player does not have to memorize things unnecessarily GUH The game contains help GUH Players do not need to use a manual to play. HEP The interface should be as non-intrusive to the Player as possible Game Play The game provides clear goals or supports playercreated goals GUH The player sees the progress in the game and can compare the results GUH The players are rewarded and rewards are meaningful GUH The player is in control GUH Challenge, strategy, and pace are in balance GUH The first-time experience is encouraging GUH The game story supports the gameplay and is meaningful GUH There are no repetitive or boring tasks GUH The players can express themselves GUH The game supports different playing styles GUH The game does not stagnate GUH The game is consistent GUH The game uses orthogonal unit differentiation GUH The player does not lose any hard-won possessions GUH There is an interesting and absorbing tutorial that mimics game play.HEP The game is enjoyable to replay. HEP Player should not experience being penalized repetitively for the same failure. HEP Easy to learn, hard to master. HEP Challenges are positive game experiences, rather than a negative experience. Educational-Pedagogical Clear goal and learning objectives PHEG The activities are interesting and engaging PHEG Clear and understandable structure of contents PHEG Can be used as self-directed learning tools PHEG Medium for learning by doing PHEG Considers the individual differences PHEG Performance should be an outcome-based. PHEG Offers the ability to select the level of difficulty PHEG Player is taught skills early that you expect the players to use later, or right before the new skill is needed.

Effectiveness Efficiency Usability – user interface aesthetics Usability – operability Usability – learneability Satisfaction – pleasure

[P78] Digital Game-Based Learning in high school Computer Science edu-cation: Impact on edu-cational effectiveness and student motivation

Attributes of the application: (a) Overall appeal, (b) quality of the user interface, (c) accessibility of the contained learning material and questions, and (d) per-ceived educational value. Degree to which they found the application: (1) was interesting, (2) was enjoyable, (3) was engaging, (4) was simple to use, (5) was easy to navigate, (6) comprised satisfactory graphics, (7) comprised subject matter easy to understand, (8) comprised questions easy to answer, (9) was useful, (10) was conducive to learning about computer memory, (11) was condu-cive to the understanding of computer memory concepts, (12) was preferable to traditional teaching within CS courses. Computer Memory Knowledge Test

Effectiveness Satisfaction – pleasure Usability – operability Usability – user inter-face aesthetics Functional suitability – functional appropriate-ness

[P79] Effective design and User experience evaluation Usability - User interface


evaluation of serious games: The case of the e-VITA project

How easy the system is to use for the intended purpose, as determined mainly by the human-machine interface design (usable).

The extent to which the user views the system as useful and satisfactory (acceptance).

The extent to which the SGs are compelling/engaging for the target audience (realism,satisfaction).

aesthetics Usability - operability Satisfaction– usefulness Satisfaction– pleasure

[P80] A teachable-agent arithmetic game's ef-fects on mathematics understanding, attitude and self-efficacy

To evaluate in terms of student performance, attitude and self-efficacy.


[P81] Mobile gaming - A seri-ous business!

To test the usability, functionality and perceived educational value of the continually evolving game design and the software knowledge, comprehension, and application and the higher levels (analysis, synthesis and critical thinking). Individual flow experience, concerning control, enjoyment, and engagement

Effectiveness Satisfaction – usefulness Satisfaction - pleasure

[P82] ICT based Education for Students with Special Educational Needs in Sri Lanka

To evaluate the improvement of the performance of students with special needs.

Effectiveness

[P83] The playability evalua-tion of virtual baby feeding application

Interface





























Satisfaction- pleasure Usability - User interface aesthetics Usability – operability Usability – learneability Portability - adaptability

[P84] Learning blood man-agement in orthopedic surgery through game-

Learn the skills in the orthopedic-surgery context

Improve learning of the psycho¬motor skills

To provide a fun experience

Effectivenes Usability – operability Usability – user inter-


play face aesthetics Performace efficiency – time behaivor

[P85] Facilitating Comprehen-sion of Basic Concepts in Computer Language Implementation Cours-es: A Game-Based Ap-proach

Pedagogical utility of the tool It helps me understand the attribute grammar processing It helps me understand the role of semantic attributes It helps me understand the role of semantic equations It helps me realize the differences among the different types of grammar attributes I do not find it hard to realize my own mistakes when using the tool Context adequacy Evaluators suitably complements lectures Evaluators motivates me to study attribute grammar processing Learnability I can rapidly start working with the tool without a long period of training Mazes are a straightforward metaphor for representing attributed syntax trees Using tables to collect information about attributes seems natural to me Interaction I have used the problem statement several times I have used the oracles It is easy to identify which attribute instance corresponds to each attribute copy in the inventory It is always easy to know the state of problem solving Overall usability judgment Evaluators is easy to use I enjoyed using Evaluators I would recommend Evaluators to other Compiler Construction students

Satisfaction – pleasure Satisfaction – usefulness Usability – operability Usability - learneability

[P86] Toward operationaliza-tion of ‘flow’ in mathe-matics learnware

To see if it facilitates the flow experience while learning mathe-matics: Affective questions: I like to learn math from computer games like Super Tangrams. I like the way motion geometry was presented in Super Tangrams. Compared to other ways of learning math, how much fun was learning math through Super Tangrams? How much did you like the colors and patterns of the puzzle piec-es in Super Tangrams? How much did you like having background music in Super Tan-grams? How much did you like having sound effects in Super Tangrams?

Effectiveness Satisfaction – pleasure Usability – user inter-face aesthetics

[P87] A framework to im-prove evaluation in educational games

To improve upon traditional basic assessment methods (i.e. goals, scores & reports) and to provide information to help improve interaction with games (e.g. discovery strategies).

Usabilty - operability Effectiveness

[P88] Effects of game tech-nology on elementary student learning in mathematics

1. I know how to play the math game very well 2. I like the color and sound of the math game 3. I like to know my scores of the game 4. The scores help me to understand how well I did in the game. 5. I can control the math game options (eg. speed, level of difficul-ty) 6. I wish we would do more math game 7. My most favorite thing about the math game is:

Motivation : playing game, having fun

Learning (eg. improving scores, moving up levels, get-ting 100% correct)

Game features : various game tasks in GameBoy, mak-ing character, learner control

Do not know and no response 8. My least favorite thing about the math game is .........

Motivation : too difficult, losing games (eg. incorrectly answering questions), cannot play more

Game features : too easy (eg. some level is too slow), creating a character, repeat

Effectiveness Usability – operability Usability – user interface aesthetics Satisfaction - pleasure


Program errors (eg. software errors)

Nothing

[P89] A collaborative game-based learning ap-proach to improving students’ learning performance in science courses

Assessing the students’ knowledge of identifying and differentiat-ing the plants on the school campus (learning attitude, learning motivation, eagerness of the students to participate in group learning activities)

Effectiveness

[P90] Participatory evaluation of an educational game for social skills acquisi-tion

Captivation of interest, Meeting learning needs, Building confi-dence, Self-assessment.

Satisfaction- pleasure Usability – operability

[P91] Empirical study upon software testing learn-ing with support from educational game

Learning effect on the remembering and understanding Effectiveness

[P92] Usability of serious games for health

I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system I found the various functions in this system were well integrated I thought there was too much inconsistency in this system I would imagine that most people would learn to use this system very quickly I found the system very cumbersome to use I felt very confident using the system I needed to learn a lot of things before I could get going with this system

Satisfaction – trust Satisfaction - pleasure Usability – operability Usability - learneability

[P93] Game-like simulations for online adaptive learning: A case study

Effectiveness: To measure the effectiveness of the HCT videogame in terms of motivation towards learning and improvement of the effective-ness of the lab sessions Satisfaction: Do you think the HCT game has helped you to recognize the equipment needed? Do you think the HCT game eased the practical session for you? Would you like to use more games like this in this module? Would you like to have at your disposal the game during the prac-tical session as a reference material?


[P94] COgnition-based design rules enhancing deci-sionmaking training in a game environment (code red triage: Doc-toral consortium paper)

To improve the effectiveness of (serious) games Effectiveness

[P95] Learning with serious games: The SELEAG approach

Assess if the game is motivating Assess if the graphical environment / usability is adequate Questionnare of adapting heuristics for: evaluating playability of games Heuristics for usability Evaluation for history educational games Educational effectiveness factors Functionality dimension reflects the characteristics related to its operational aspects. It aggregates two factors: easy of use and content’s quality. Efficiency dimension measure the system’s ability for presenting different views on its content with minimum effort. It aggregates five factors: audiovisual quality, technical and static elements, navigation and interaction, originality and use of advanced tech-nology. Adaptability dimension is the aggregation of five factors: versatil-ity, pedagogical aspects, didactical resources, stimulates the initi-ative and self learning and cognitive effort of the activities.

Satisfaction – pleasure Usability - operability Usability – user interface aesthetics

[P96] Assessing nurses' ac-ceptance of a serious

INTERACTION Moving through the virtual environment was simple

Effectiveness


game for emergency medical services

I quickly learned how to move through the environment I could always concentrate on the actions in the game rather than the commands I had to give to activate them I quickly learned how to select tasks to perform the procedure VIEW I had a complete view of the environment I could observe well objects and characters in the environment from various points of view SPECIFIC APPLICATION DETAILS The sound of the shortness of breath of the patient is realistic The cyanotic complexion of the patient is realistic List virtual patient’s aspects you liked or disliked (open question) The application shows three animations, listed below, related to actions belonging to the procedure. Order them from 1 to 3 based on your subjective preference (1 best, 3 worst). Oxygen mask application. Vein cannulation. Patient’s transfer to the stretcher. The characters helped me understand what was happening in the environment and what I was supposed to do USEFULNESS AND EFFECTIVENESS AS A TRAINING TOOL This kind of application could increase my knowledge This application would integrate well with the training methods currently employed to train EMS nurses In what phase of the training should it be employed? (open ques-tion) GLOBAL SATISFACTION I felt involved in the experience I liked the graphics of the application I liked the audio of the applications I would be willing to use this kind of application

Satisfaction – pleasure Usability – user interface aesthetics Usability – operability Usability – learneability Functional suitability - functional completeness Functional suitability - functional correctness Functional suitability - functional appropriate-ness

[P97] On the usability and likeability of virtual reality games for educa-tion: The case of VR-ENGAGE

Characteristics of the students’ interactions with the game : 1. Game User Interface Acquaintance, which showed the player’s

level of understanding of the Virtual Reality user interface of the game.

2. VR Navigational Effort, that showed how well the user could navigate through the Virtual World.

3. VR Environment Distractions: There were many cases when the Virtual Environment drew the player’s attention to such an ex-tent that s/he may have missed the main point of the educa-tional game (which was learning a specific topic).

Among the questions about likeability asked during the inter-views were the following: 1. Which application was more motivating? 2. Which one was more interesting? 3. Did you get tired of any of the applications? 4. What do you think of VR-ENGAGE as a game? 5. What would you add to VR-ENGAGE? 6. What do you think of learning while

Usability – operability Satisfaction - pleasure

[P98] DELIVER! - An educa-tional game for teach-ing Earned Value Man-agement in computing courses

Sub-component motivation The game design is attractive (Attention) There was something interesting at the beginning of the game that captured my attention (Attention) The variation (form, content or activities) helped me to keep attention to the game (Attention) The game content is relevant to my interests (Relevance) The way the game works suits my way of learning (Relevance) The game content is connected to other knowledge I already had (Relevance) It was easy to understand the game and start using it as study material (Confidence) Passing through the game, I felt confident that I was learning (Confidence) I am satisfied because I know I will have opportunities to use in practice things I learned playing this game (Satisfaction) It is due to my personal effort that I manage to advance in the game (Satisfaction)

Effectiveness Satisfaction – pleasure Satisfaction – usefulness Satisfactionn – trust Usability – operability Usability – user interface aesthetics


Sub-component user experience Temporarily I forgot about my daily; I have been fully concentrat-ed on the Game (Immersion) I did not notice the time pass while playing; when I saw the game had already ended (Immersion) I felt myself more in the game context than real life, forgetting what was around me (Immersion) I was able to interact with others during the game (Social interac-tion) I had fun with other people (Social interaction) The game promotes moments of cooperation and/or competition between the players (Social interaction) This game is appropriately challenging for me, the tasks are not too easy nor too difficult (Challenge) The game progresses at an adequate pace and does not become monotonous – offers new obstacles, situations or variations in its tasks (Challenge) I had fun with the game (Fun) When interrupted at the end of the class, I was disappointed that the game was over (Fun) I would recommend this game to my colleagues (Fun) I achieved the goals of the game applying my knowledge (Compe-tence) I had positive feelings on the efficiency of this game (Compe-tence) I would like to play this game again (Fun) Sub-component learning How much do you think the game contributed to your learning in this course? (Short-term learning) How efficient was the game for your learning, comparing it with other activities of the course? (Short-term learning) Do you think the experience with the game will contribute to your professional performance in practice? (Long-term learning)

[P99] Empirical evaluation of an educational game on software measurement

Measurement competency (pre/post)

Measurement knowledge on the remembering level (pre/post)

Measurement knowledge on the understanding level (pre/post)

Measurement knowledge on the application level (pre/post)

Subjective learning effect: Learning effect on measurement concepts and process, Learning effect on measurement ap-plication

Appropriateness: Content relevancy, Correctness, Sufficiency, Difficulty, Sequence, Teaching method, Duration.

Engagement: Satisfaction, Fun

Effectiveness Functional suitability – functional completeness Functional suitability – functional correctness Functional suitability – functional appropriate-ness Satisfaction - pleasure

[P100] DOGeometry: Teaching geometry through play

Learning potential of the game Effectiveness

[P101] Experiences from im-plementing a face-to-face educational game for iPhone/iPod touch

Does The KnowledgeWar game has a positive effect on learning? Is the game a fun way of practicing knowledge? I thought the game was easy to use I think that i would need the support of a technical person to be able to use the game I would imagine that most people would learn to use this game applicaction very quickly

Effectiveness Satisfaction – usefulness, Satisfaction – pleasure Usability – operability Usability – learneability

[P102] Learning construction procurement negotia-tion in an educational game

Help students reach a comparatively better grasp of construc-tion procurement procedure Improved the efficiency of stu-dents’ knowledge

acquisition

Enhanced students’ learning motivation and meanwhile boost their thinking ability

Enhance the overall learning motivation of students

Effectiveness Satisfaction – usefulness

[P103] The Development of Computer Multimedia Game-Based Instruction on English Idioms for the 2nd

To evaluate contents, media technology and satisfaction Effectiveness Satisfaction – pleasure Usability – user inter-


Educational Range at Bannakhao School

face aesthetics

[P104] Developing a film-based learning system with english verbal reduced forms for supporting english listening com-prehension

Effectiveness of the learning system (learning attitude) as well as the user's attitude and satisfaction with the system System utilization, Information quality, Quality of HMI

Effectiveness Usability – operability Usability – user inter-face aesthetics Satisfaction - pleasure

[P105] Design and Evaluation of a Physical Interactive Learning Environment for English Learning

To evaluate English learning achievement, motivation and atti-tude: System Design: Learning content, Game design, Interface design Learning Motivation and Attitude: English learning motivation Attitude of technology integrating into English learning English learning attitude General assessment: Classmate interaction System support learning Improving self learning motivation

Effectiveness Satisfaction – pleasure Usability – user inter-face aesthetics

[P106] Work in progress - Using a computer gam-ing strategy to facilitate undergraduates' learn-ing in a computer pro-gramming course: An experimental study

To evaluate interest/enjoyment, perceived competence Effectiveness Satisfaction - pleasure

[P107] Usability evaluation for history educational games

User interface: ¿Is the game easy to learn, fluent to use and support the interac-tions which is general for game evaluations? Gameplay ¿Is the game as fluent and as fun posible? Game mechanics ¿Are the rules, which define the operation of the game world, fine and the interactions that occur in the game are fluent and meet the game’s objective? Playability ¿Do players not feel frustrated when playing games because of por graphics, sound effect, difficulty level and stages? Feedback Players should always be able to identify their score and their status in the game Inmersion ¿Are the players really participating and engaging in the game?

Interface: Usability - User interface aesthetics Usability - operability Gameplay: Satisfaction- pleasure Game mechanics: Effectiveness Playability: Satisfaction- pleasure Feedback: Usability - operability Inmersion: Usability - User interface aesthetics Satisfaction- pleasure

[P108] A conceptual frame-work for serious games

Measurement of achievement in meeting their learning outcomes engagement, interaction, feedback


[P109]

Playing a game in civil engineering - The inter-nal force master for structural analysis

To improve learning outcomes Effectiveness

[P110] Research on the design and evaluation of edu-cational games based on the RETAIN model

Relevance - Learning activities should be related to leamer's existing knowledge and experience

Embedding - effective integration of embedding content and fantasy

Effectivenes Satisfaction - usefulness


Transfer - The learning of the targeted content should make the players effective users of knowledge

Adaption - Learners develop their cognitive structure and skills through the process of assimilation and accommodation

Immerse - immerse in the fantasy, can maximize the learning efficiency.

Naturalization - automaticity and spontaneous knowledge. Learners can use existing knowledge habitually and don't need to devote mental resources.

Usability – appropiate-ness recognisability

[P111] Virtual Network Mara-thon with immersion, scientificalness, com-petitiveness, adaptabil-ity and learning

To evaluate the usability of the features of Virtual Network Mara-thon: Immersion: The system needs to generate the feeling that users who are playing the game are immersed in it. Scientificalness:The exercise time and intensity in the game strict-ly conform to the scientific guidelines for physical exercise pre-scription Competitiveness:The technique of modeling non-player competi-tors by learning from demonstrations, which learns competitive policies through mimicking demonstrated running behaviors with various styles of real runners, is introduced to create life like competitive non-player competitors Adaptability:To ensure that players at different fitness levels can enjoy similar engaging game play experiences Learning: allowing players all over the world to learn the history and culture of China and gain knowledge of the Olympics

Effectiveness Satisfaction – pleasure Usability – user interface aesthetics

[P112] Using a Pleasure Framework to Critique Tangible Learning Games

Creation – learner’s ability and desire to create something Discovery- a learner following clues in order to solve a specific intelectual problem Difficulty - learner is not turned off because the game is too diffi-cult or bored because the game is too easy Competition- learners compete against each other and/or against time or resource depletion. Captivation- learner is “spell bound” by the game. Sensation- learner faces physical activity, arousing stimulation, vértigo and faces sensory curiosity. Simulation- learner forms an analogy in the game from some-thing pleasurable in life Camaraderie- learner feels a social sense friendship and relaxa-tion Recognition – learner is publically recognized for their perfor-mance. Control – learner feels that working harder will impact their suc-cess. Feedback – learner is provided with an immediate feedback.

Satisfaction- pleasure

Documents

standard Effectiveness Usability - user interface ... · formative evaluations focus on the design features of the applica-tion (i.e., usability and fun, quality of the visual representation,