Serious Games Assessment - Ontario Tech U › kapralos › csci5530 › ...Effectiveness of Game-Based Learning (cont.) Hays conducted a review of the literature (105 articles) in

CSCI 5530: Serious Games Development (Simulations and Serious Games)

1

Winter 2012

Bill Kapralos

CSCI 5530, Winter 2012 Bill Kapralos

Winter 2012

Serious Games AssessmentFriday, April 13 2012

Bill Kapralos

Overview (1):Before We Begin

Administrative details

Brief review from last week

Learning and Serious Games

Overview

Assessment

Overview

Serious Games Assessment

Overview (1):Design Continued (Time Permitting)

Overview

Demonstrate-Practice-Test

Procedures

Miscellaneous


2

Winter 2012

Bill Kapralos

Administrative Details (1):

Last Lecture of the Semester

Today (Friday, April 13 2012) is the last lecture of the

course

Final Course Reports

For those officially enrolled in the course, a reminder that

your final course reports are due on or before April 23

I can’t extend beyond this as final grades are due first

week of May and I am out of the country as of May 1

Learning & Serious Games

Overview (1):Learning in Games

Although some have expressed the opinion that video

games are “mindless” and don’t offer any learning aside

form possibly some hand-eye coordination, whenever

one plays a video game

Learning is constantly occurring whether the player

wants it or is even aware of it

Despite this inherent learning → shortage of studies that

have methodically assessed learning via gameplay

(“entertainment” games or serious games)

Some studies have challenged the usefulness of

game-based learning


3

Winter 2012

Bill Kapralos

Overview (2):Studies That Have Questioned the

Effectiveness of Game-Based Learning

A meta-analysis conducted by Kulik in 2002 of six

studies that examined the use of simulation in the

classroom discovered limited learning effects → no

evident increase in two of the six studies

A review of articles related to assessment that appeared

in the journal Simulation & Gaming, in 2004 reports little

evidence on effectiveness of experiential learning with

insufficient evidence to conclude that simulations and

games result in meaningful learning


Effectiveness of Game-Based Learning (cont.)

Hays conducted a review of the literature (105 articles)

in 2005 on “instructional games” focusing on the

empirical research on the instructional effectiveness of

games and found that

Research on effectiveness of instructional games is

fragmented, filled with ill-defined terms, and

methodological flaws

Players should have access to support to help them

understand how to properly use the game



Hays study (cont.)

Some games can provide effective learning for a

variety of learners for different tasks → can’t

generalize on effectiveness of one game in a

particular learning area for one group of learners, to

all games, in all learning areas for all learners

No evidence showing games are the preferred

instructional method in all situations

Games should include debriefing and feedback to

provide learner with understanding of what happened


4

Winter 2012

Bill Kapralos



A review of articles related to assessment that appeared

in the journal Simulation & Gaming in 2004 reports little

evidence regarding the effectiveness of experiential

learning with insufficient evidence to conclude that

simulations and games result in meaningful learning

Also reports that this conclusion results from studies

that do not meet “the highest research design and

measurement standards”



Gosen and Washbush point out that there has been little

done to design studies and develop instruments that

could competently assess teaching approaches and

attribute this to three primary reasons

Careful, rigorous research dedicated to developing a

valid instrument and reflective of thought-out learning

objectives is extremely time-consuming



Gosen and Washbush (cont.)

The criterion variable being used, which is learning

from a computer-based simulation or experiential

exercise, is illusive → we believe we know what it is,

but what it looks like so it can be measured lacks

form

Our present world does not appear to care enough for

this work to be successful, and perhaps it should not


5

Winter 2012

Bill Kapralos



Many of the past studies on effectiveness of games and

simulations were conducted several years ago and the

studies they focus on date back several decades →

even in last 10 years, unprecedented development

within the videogame field and it has been suggested

that games more than five years old are “old news”

Can’t ignore fact that simulations and serious games

may be most effective means of safely and cost-

effectively learning skills and attitudes which are hard

to acquire by rote learning



Also many examples of studies that have demonstrated

that “learning games” that are designed properly do

produce learning, and plenty of it while engaging players

Livingston et al. [1973] evaluated seven years of

research and over 150 studies to examine the

effectiveness of gaming → found that “simulation

games” are able to teach factual information although

they are not more effective than other methods of

instruction

Overview (10):Studies That Have Shown Effectiveness of

Game-Based Learning

Effective “learning games” (cont.)

Livingston et al. [1973] evaluated seven years of

research and over 150 studies to examine the

effectiveness of gaming → “simulation games” are

able to teach factual information although they are not

more effective than other methods of instruction

Students preferred games and simulations over other

classroom activities → “gamed simulations” can lead

to changes in their the attitudes toward education,

career, marriage…


6

Winter 2012

Bill Kapralos




Recall the Lightspan Partnership that developed

educational games for Playstation console platform

for students at the elementary school level (K-8) →

results revealed that standardized test scores for

those using the games increased by 24% and 25% in

the vocabulary and language arts and increased by

51% and 30% with respect to math procedures and

algorithms respectively when compared to control

groups who didn’t use the games




Click Health developed video games for kids to self-

manage various health issues including diabetes.

Clinical trials funded by the United States National

institutes of Health, revealed that with respect to

diabetes, the kids that played their games (in contrast

to a control group that played a pinball game),

showed measurable gains in self-efficacy,

communication with parents, diabetes self-care, and

resulted in a 77% decrease in urgent doctor visits




Effectiveness of virtual reality and games in the

treatment of phobias and in distracting patients in the

process of burn treatment or chemotherapy has been

scientifically validated with the use of functional MRI

(fMRI) → differences in brain activity in patients who

were experiencing pain with and without the use of

virtual reality and games


7

Winter 2012

Bill Kapralos




The serious game The Matrix, developed to enhance

self-esteem, was subject to rigorous scientific

evaluation and was shown to increase self-esteem

through classical conditioning

Assessment

Overview (1):What is (Educational) Assessment ?

According the Higher Learning Commission educational

assessment of student learning is a participatory,

iterative process that

Provides data/information on students’ learning

Engages you and others in analyzing/using this

data/info. to confirm/improve teaching and learning

Produces evidence that students are learning the

intended outcomes

Evaluates whether changes made improve/impact

student learning, and documents the learning and

your efforts


8

Winter 2012

Bill Kapralos

Overview (2):What is (Educational) Assessment ? (cont.)

According to the University of Oregon, Teaching

Effectiveness Program assessment is

The process of gathering and discussing information

from multiple and diverse sources in order to develop

a deep understanding of what students know,

understand, and can do with their knowledge as a

result of their educational experiences; the process

culminates when assessment results are used to

improve subsequent learning

Overview (3):Two Forms of Assessment

Summative assessment

Assessment conducted at the end of a learning

process and tests the overall achievements

Formative assessment

Implemented and present throughout the entire

learning process and continuously monitors progress

and failures

Overview (4):Assessment and Serious Games

Learning with serious games remains a goal-directed

process aimed at clearly defined and measurable

achievements

Must implement assessments to provide an indication

of the learning progress and outcomes

“Serious games like every other tool of education,

must be able to show that the necessary learning has

occurred” (Michael and Chen, 2005)

Has been suggested that formative assessment is

particularly useful can be incorporated into the

serious game becoming part of the experience


9

Winter 2012

Bill Kapralos

Overview (5):Assessment and Serious Games (cont.)



achievements (cont.)

For serious games to be considered a viable

educational tool, they must provide some means of

testing and progress tracking and the testing must be

recognizable within the context of the education or

training they are attempting to impart

Assessment describes the process of using data to

demonstrate that stated learning goals and objectives

are actually being met within the serious game




achievements (cont.)

For serious games to be considered a viable

educational tool, they must provide some means of

testing and progress tracking and the testing must be

recognizable within the context of the education or

training they are attempting to impart

Assessment describes the process of using data to

demonstrate that stated learning goals and objectives

are actually being met within the serious game


Learning is a complex construct making it difficult to

measure and determining whether a simulation or

serious game is effective at achieving the intended

learning goals

In other words, has the desired effect, is a complex,

time consuming, expensive, and difficult process

Part of the difficulty stems from the open-ended nature

inherent in video games making it difficult to collect data

How do you show that students are learning what

they should learn and how do you know what you are

measuring is what you think you are measuring?


10

Winter 2012

Bill Kapralos


Learning is a complex construct making it difficult to

measure and determining whether a simulation or

serious game is effective at achieving the intended

learning goals

In other words, has the desired effect, is a complex,

time consuming, expensive, and difficult process

Part of the difficulty stems from the open-ended nature

inherent in video games making it difficult to collect data

How do you show that students are learning what

they should learn and how do you know what you are

measuring is what you think you are measuring?

Serious Games Assessment (1):Three Primary Types

Completion assessment

Concerned with whether the player completes game

In a traditional teaching environment → equivalent to

asking "Did the student get the right answer?" and a

simple criterion such as this could be the first

indicator that the student sufficiently understands the

subject taught albeit

Problems using this measure alone → players could

cheat and it is hard to determine whether the player

actually learned the material or learned to complete

the game

Serious Games Assessment (2):Three Primary Types (cont.)

In-process assessment

Examines how, when why player made their choices

Analogous to observations of the student by the

educator as the student performs the task or takes

the test in a traditional teaching environment

Teacher assessment

Focuses on instructor’s observations and judgements

of the student “in action” (while they are playing the

game)

Includes a combination of both completion

assessment and in-process assessment


11

Winter 2012

Bill Kapralos


In-process and teacher assessments can be

accommodated by the use of recent technology

For example → simple and cost-effective to obtain

screen recordings of the player’s game play, video

recordings of the players while they are playing the

game, and audio recordings to capture a players

voice for example during thinking aloud processes

which may happen unexpectedly or may also be

encouraged


With today’s technology, information from these

recordings can also be obtained automatically (without

the need for a camera operator, etc.) using a wide

variety of available tools

Recordings and info. obtained from the recordings

can also be used to facilitate debriefing sessions

More recent assessment methods include

“information trails” → tracking a player’s significant

actions and events that may aid in analyzing and

answering the what, how, when, who and where in

the game something happened

Serious Games Assessment (5):Pre- and Post-Testing

Although various methods and techniques have been

used to assess learning in serious games and

simulations in general, assessment is commonly

accomplished with the use of pre- and post-testing → a

common approach in educational research

Pre- and post-testing design is one of the most widely

used experimental designs

Popular in educational studies that aim to measure

changes in educational outcomes after modifications

to the learning process such as testing the effect of a

new teaching method


12

Winter 2012

Bill Kapralos

Serious Games Assessment (6):Pre- and Post-Testing (cont.)

Participants are randomly allocated to either a

“treatment” group or a “control” group.

Both groups receive identical pre-test → ensures that

the groups are equivalent

Each group receives a different “treatment” → either the

intervention being examined (playing the serious game),

or the “usual treatment” (don’t play the serious game but

rather rely solely on traditional instructional approaches).

Upon completion of “treatment” (experiment), both

groups are given (the same) post-test → results of the

pre- and post-tests are compared across both groups


Differences in scores across groups are assumed to be

a result of the “treatment” (the serious game)

Variant to the pre- and post-test design

No pre-testing performed and both groups are

exposed to the “usual treatment” and one of the

groups (the “treatment” group) will also receive the

“treatment” (playing the serious game)

Upon completion of the experiment, both groups

complete a post-test and once again, significant

differences across the test scores are attributed to the

“treatment” (the serious game)


Problems

Impossible to determine whether the act of pre-testing

has influenced any of the results

Impossible (and at times, unethical) to completely

isolate all of the participants → if two groups of child

participants attend the same school, they will

probably interact outside of lessons potentially

influencing the results while if the child participants

are taken from different schools to prevent this, than

randomization is not possible


13

Winter 2012

Bill Kapralos

Serious Games Assessment (9):Assessment… Not so Easy!

Although assessment can provide a measure of

learning, other factors that can indirectly lead to learning

Serious games captivate and engage

players/learners for a specific purpose such as to

develop new knowledge or skills

With respect to students, strong engagement has

been associated with academic achievement → level

of engagement may also be potentially used as an

indicator to the learning a serious game is capable of

imparting

Serious Games Assessment (10):Assessment… Not so Easy! (cont.)

Engagement according to Brockmayer

A combination of (a) immersion, (b) presence, (c) flow

and (d) psychological absorption

Various tools have been developed to provide a

measure of engagement including the Game

Engagement Questionnaire (GEQ) → scientifically

validated self-assessment tool shown to be reliable

metric for determining player’s engagement in games

Questions within GEQ chosen to be easy to

understand by gamers and non-gamers and that they

statistically relate to the concepts being examined

Serious Games Assessment (11):Assessment… Not so Easy! (cont.)

Another engagement measurement tool

Self-reporting questionnaire for measuring player

engagement known as the Game Experience

Questionnaire

In addition to questionnaires, physiological responses

such as heart-rate, skin conductance, cortisol, can

also provide an indication of player engagement


14

Winter 2012

Bill Kapralos

Serious Games Assessment (10):In-Game Assessment

Serious games (and games in general) can and

generally do contain in-game tests of effectiveness

As players progress through the game → accumulate

points and experience which make the next stages

and levels of the game easier and thus should score

higher if any learning has been imparted

Leads to “level-up” protocol of game testing…

Serious Games Assessment (11):In-Game Assessment (cont.)

“Level-up” protocol of game testing

Players divided into two groups with one of the

groups beginning the game at the first level for

example and the other beginning at the second level

If group that started at first level does significantly

better than other group, this is attributed to a

successful game that is capable of imparting the

intended instructional material → at least with respect

to the first level


Incorporating in-game assessments takes us away from

predominant, classic form of assessment comprised of

questionnaires, questions and answers, etc. → do

nothing more than test our memory rather than

measuring an understanding and/or creative use of the

acquired knowledge (albeit, this can be difficult to do)

and can interrupt and negatively affect the learning

process


15

Winter 2012

Bill Kapralos


Provides opportunity to take advantage of the medium

itself and employ alternative, less intrusive, and less

obvious forms of assessment which could (and should)

become a game element itself

Integrating the assessment such that the player is

unaware of it forms the basis of what Shute et al [2009]

describe as stealth assessment

A new but growing area of serious games → plenty of

opportunity here!

Design (cont.)

Overview (1):Well Done Needs and Task Analyses

Yield the basic material to determine content of game

You know what everyone is supposed to learn, the

priorities, many of the “tricks of the trade”, and critical

discriminations

Clear picture of your target audience → game players

Now we need to look at the content from an instructional

point of view

What are you actually teaching and how do you teach

within the context of the game

After you teach, how do you test → will test generate

a score or other feedback that’s shared with user ?


16

Winter 2012

Bill Kapralos

Demonstrate-Practice-Test (1):Traditionally…

Instructors have used the “exercise” model that follows

three critical steps in assembling a lesson plan

Demonstrate the skill → show users how to do it

Have users practice the skill → let them do it while

you give them advice based on their performance

Test them → check to see how well they can do it

without any aid

Good for skills-based training but demonstrating and

practicing cognitive tasks can be challenging

There are instructional strategies for doing so

Demonstrate-Practice-Test (2):Traditionally… (cont.)

This model fits right in with many of the principles of

good game design

If you want someone to do something in a game you

typically have a tutorial level → a “mentor” or a

“coach” that steps in and makes suggestions about

best practices

By end of tutorial level the mentor has been removed

and the skill has to be performed in the most difficult

of situations


Last step is really a test → if test is good enough, it

allows game to give realistic score of player’s

performance

Game even helps player transfer the skills back to the

real world

Transference happens when the final level is

challenging enough and contains the most important

distractions and complications of the real world

Transfer is the ultimate goal of training, instruction

and serious games → to be an effective game,

players have to carry the skills with them


17

Winter 2012

Bill Kapralos


Key parts of the model are often left out or minimized in

serious games:

Lack of adequately detailed instruction on how to

perform the task in the tutorial level

Not enough coaching responses to cover the variety

of situations where coaching is needed

Testing level that does not represent real-world

behaviour or have an effective scoring system

If attention is paid to the above areas and are not

minimized in final game design → truly effective serious

game is possible

Demonstrate-Practice-Test (4):Step Size

In the tutorial level, you must decide how much info to

present to participants in a single step

Don’t want to give users so much that they can’t

remember it all and you don’t want to give them too

little that the process becomes tedious

Step size → amount of info that should be presented

Step size solution

Present as much info you think can possibly be

digested in a particular step

Test your tutorial with the target population → if they

don’t “get it”, break tutorial down to smaller steps

Demonstrate-Practice-Test (5):Order of Presentation (“Shaping”)

In a game, order of presentation affects the way content

is introduced

Shaping

Going from simplest to most difficult

Suggests that its best to present an idea without all

the distracting complications of real life

Add more complications as you go along until final

level which is life-like as possible


18

Winter 2012

Bill Kapralos

Demonstrate-Practice-Test (6):Order of Presentation (“Shaping”) (cont.)

Issue with distractions or noise in the environment

Physical noise such as traffic passing by you as you

are changing a tire

Conceptual noise such as knobs and buttons on a

control panel that have nothing to do with the controls

that really matter

Adding a clock to an event can provide a critical and

very real distraction that also adds stress to the

situation → ticking clock can be noise too

Begin without noise and then add it as game

progresses

Demonstrate-Practice-Test (7):Order of Presentation (“Shaping”) (cont.)

Issue with distractions or noise in the environment (cont)

If your serious game is to have multiple levels, put the

fewest distractions at the lowest levels and then add

them as you move along

Increasing the cognitive load → if you want to present

a complex decision or procedure, ask the player to do

it in its simplest form at the lowest levels and add

complexity progressively

Procedures (1):Teaching Procedures is Tricky!

Often present instructional problems related to

“sequencing” → the key to performing a procedure

correctly is performing it in the right order

Sequencing

At lowest game levels, procedures should be isolated so

that there are no distractions or interruptions as the

player works through the task and then

Later on add distractions that represent the way the

real world would interrupt the performance of the

procedure


19

Winter 2012

Bill Kapralos

Procedures (2):

Sequencing (cont.)

Sometimes having to go through entire procedure goes

against principle of shaping → may be difficult to present

a simple version of a long and complicated process

But it is necessary to teach the entire procedure so

that user understands it entirely

Chaining

Teach the procedure backwards by starting with the last

step, then moving to the last two steps, then the last

three, last four, etc.

Procedures (3):

Chaining (cont.)

Structure of games allows chaining more than other

types of instruction

You can build out game scenario so that all player

has to do is complete the act in the first level etc.

Puzzle example → first time through almost entire

puzzle except last step (piece) is completed and next

time through all but last two steps are completed etc.

Chaining isn’t the solution for all teaching → learning

problems that are not sequencing/procedure problems

Procedures (4):Chaining (cont.)

Example from Fully Involved game


20

Winter 2012

Bill Kapralos

Procedures (5):

Discrimination Problems

Cognitive tasks

Medical diagnoses → is it skin cancer or eczema ?

Hostage negotiations, cultural awareness, and other

ill-defined domains → distinction between right and

wrong answers is often fuzzy

Competition and Discrimination

Although we use shaping in moving game from simple to

complex, sometimes there are other considerations and

The need to teach the entire procedure in its entirety

Dealing with discrimination problems

Procedures (6):

Competition and Discrimination (cont.)

The way to deal with two confusing or competing

concepts → present them together

Show them side-by-side during the tutorial when you

are demonstrating skills and practices

Show them together during game when simulating

the way people encounter them in the real world

Make it part of the game to tell competing items apart

→ will teach players to make critical discriminations

that they might miss if technique weren’t employed

Procedures (7):


Spark Island, Alaska kids fire safety game example

The issue of different types of waste → hazardous

waste, what is it? what isn’t it?

Task of sorting waste required discrimination between

hazardous waste and other types of waste and it was

a competition problem

Mini-game created called “trash pickup” where

players had to get right kind of trash into right barrel

Having players discriminate between competing items

fits nicely into a game


21

Winter 2012

Bill Kapralos

Procedures (8):


Spark Island, Alaska kids fire safety game example

Miscellaneous (1):

Job Aids Versus Instruction

Checklists or other notes that a performer can use to

complete a task accurately

Checklists are widely used in various disciplines and

they work very well → medicine/surgery and in

commercial aviation where pilots use a checklist

ensure plane is ready for a trip

Idea →so many things to do that it’s easy to forget some

No reason not to have a checklist because no one is

watching who would think the “pilots” were less

professional because they couldn’t do the whole pre-

flight check from memory

Miscellaneous (2):Job Aids Versus Instruction (cont.)

Checklists are far more dependable than unaided

memory and if possible are the preferred way for people

to get the job right

Problem → many reasons for not using the checklist

If the job is time-sensitive, or if it would be seen as

unprofessional to look things up as you move along

The rule for checklists → beneficial to develop them

whenever you can to help the performer


22

Winter 2012

Bill Kapralos

Miscellaneous (3):Job Aids Versus Instruction (cont.)

How do we incorporate a checklist in game ?

If in the real-world they are a critical part of the job

you’re simulating then build the checklist into game

so that in process of playing game, checklist is there

on screen to help player get things done → you’re

teaching users to use the checklist and in the end,

that is what you want them to do

Miscellaneous (4):Fading

A game version of “training wheels”

Instructional technique for removing support from

players so that they can stand on their own

If you build a checklist and don’t want the game

players to depend on it → remove it little by little

throughout the course of the game

Can also have partial checklists at higher levels of the

game and make them available when players need

them → a mini-game could have players fill in missing

steps

Documents

Serious Games Assessment - Ontario Tech U › kapralos › csci5530 › ...Effectiveness of Game-Based Learning (cont.) Hays conducted a review of the literature (105 articles) in