Genova December 6-7 2011 European Commission Information Society and Media 1 st Review Meeting GaLA Game and Learning Alliance The European Network of

GenovaDecember 6-7 2011

European Commission Information Society and

Media

1st Review Meeting

GaLAGame and Learning Alliance

The European Network of Excellence on Serious Games

Work package 6:Integration into Educational Processes

Work package leader UWS

Professor Thomas Connolly

Dr. Elizabeth Boyle

Dr. Thomas Hainey



Media

1st Review Meeting2

Reminder of WP6 partners and original allocation of tasks

T6.1 User and education stakeholder requirements (14)(Task leader: UWS, Thomas Connolly (7))INESC-ID, Joao Pereira (7)

T6.2 Metrics for SG in education (16)(Task leader: TUG, Michael Kickmeier (5))UNI-GRAZ, Elisabeth Friedrich (7)UNIGE, Francesco Bellotti (4)

T6.3 Integration methodologies (15)(Task leader: CNR-ITD, Rosa Bottino, Michela Ott (6))UCM, Pablo Moreno-Ger (8)UNIGE, Francesco Bellotti (1)

T6.4 Community of teachers and tutors (10)(Task leader: TU DELFT, Igor Mayer (3))UU, Remco Veltkamp (3) HWU, Theo Lim (4)

Total 55



Media

1st Review Meeting3

Recommendations in the review report and major changes to work plan

• The reviewers recommend that we change our approach in WP6.1 from surveys to a meta-analysis as “survey research is a weak methodology”.

The review panel “commends the notion of constructing a database to synthesise previous research and sees this as an important contribution to the serious games community but its value will be significantly diminished if a broad and deep look at established researchers around the world is not included.” We should take “a broad and deep look at established researchers around the world” including the work of Eva Baker, Sigmund Tobias and Dexter Fletcher.



Media

1st Review Meeting4

Publications

Tobias, S and Fletcher, J. D. (Eds) (2011) Computer Games and Instruction

Girlie C. Delacruz Gregory K.W.K Chung and Eva L. Baker (2010) Validity evidence for games as assessment environments. National Center for Research on Evaluation, Standards, and Student Testing

Baker, E. L., & Delacruz, G. C. (2008). A conceptual framework for assessment of learning in games. In H. F. O’Neil & R. S. Perez (Eds.), Computer games and team and individual learning (pp. 21–38). Amsterdam, Netherlands: Elsevier.



Media

1st Review Meeting5

Other recommendations in the review report

• “Take advantage of the EU Lifelong Learning Grant award and build in plans to measure learning impact and outcomes as part of the development of that game and report the finds as part of this WP when they are available – the panel sees this as a high priority contribution that this WP could make to the network.”

• “Since WP3 reported minimal penetration of SGs in engineering and manufacturing, it might be worth exploring how SGs might be integrated into engineering education, especially at the secondary level where there is probably a reasonable chance for effective use. The panel recommends an effort be made to coordinate further exploration of how best to integrate SGs into education with WP3 – use games identified in WP3?”

• “Coordinate taxonomies with WP1 – do not use something that is not being used consistently throughout the project and by other WPs.”



Media

1st Review Meeting6

Proposed year 2 activities for WP6.1 and WP2:Literature search and meta-analysis

• “A serious and formal meta-analysis associated with learning outcomes associated with serious games would be highly regarded by the global SG community” and “the review panel recommends taking up such an effort early in year two planning efforts”.

• The proposed meta-analysis of learning outcomes seems to fit better under task WP6.2 metrics than WP6.1. It is proposed that the meta-analysis would be a joint activity of WP6.1 and WP6.2 with UWS, TUG, UNI-GRAZ and UNIGE working together on the literature search and meta-analysis. INESC-ID to be redeployed in WP6.3?



Media

1st Review Meeting7

Meta-analysis step 1: Formulation of the problem

• “A serious and formal meta-analysis associated with learning outcomes associated with serious games”

• Will help to defragment (organise) the literature and provide evidence for effectiveness of serious games.

• How to organise “learning outcomes”– Cognitive, affective, motor, soft skills– Education, corporate training (mostly qualitative?)– SIGS



Media

1st Review Meeting

T6.2: UWS Literature Search terms and electronic databases

Time period of search: 1961-Feb 2011

Search terms: ("computer games" OR "video games" OR "serious games" OR "simulation games" OR "games-based learning" OR "MMOG" OR "MMORPG" OR "M.U.D." OR "online games") AND (evaluation OR impacts OR outcomes OR effects OR learning OR education OR skills OR behaviour OR attitude OR engagement OR motivation OR affect)

Databases searched:ACM (Association for Computing Machinery), ASSIA (Applied Social Sciences Index and Abstracts), BioMed Central, Cambridge Journals Online, ChildData, Index to Theses, Oxford University Press (journals), ScienceDirect, EBSCO (consisting of Psychology and Behavioral Science, PsycINFO, SocINDEX, Library, Information Science and Technology Abstracts, CINAHL), ERIC (Education Resources Information Center), IngentaConnect, Infotrac (Expanded Academic ASAP), Emerald and IEEE (Institute of Electrical and Electronics Engineers) Computer Science



Media

1st Review Meeting9

Search terms

Should we include other search terms? If there are other search terms UWS would prefer to carry out a separate search so we can still compare the results of a new search with previous results.

Literature

UWS will update the database using electronic databases to which we have access.

Do partners want other electronic databases?

Grey literature – volunteers?

Ph. D. theses – volunteers?

Proposed year 2 activities for WP6.1 & WP6.2:Update the searchable database



Media

1st Review Meeting10

Proposed year 2 activities for WP6.1 & WP6.2: Update the searchable database

Languages All the papers we identified were published in English. But are we missing some published e. g. in Spanish (Baltasar and Pablo’s paper), French, German journals? How would we access these? Volunteers?

Global reachWe need to include the work of American researchers - Eva Baker, Sigmund Tobias and Dexter Fletcher.



Media


Meta-analysis

• Slight concern that there may be a relative absence of high quality RCTs. Should we do a systematic or narrative review too? – The literature search would have to be done anyway.

• What quality of evidence do we accept in the meta-analysis? Do we focus only on RCTs? How about quasi experimental designs? surveys? qualitative studies?

• The search will identify many papers - from our previous experience over 7,000 between 2004-2009. It is a lot of work to search these for empirical evidence! Concern that we are doing work that was not budgeted for.

Proposed year 2 activities for WP6.1 & WP6.2: Meta-analysis



Media


Inclusion criteria for selection of papers?

• Papers to include empirical evidence relating to the impacts and outcomes of playing games.

• RCTS – but dearth of these• What time period are we searching? January 2004 to

March 2012?• Paper includes an abstract?



Media


Classification of learning outcomes

• Do we lump all learning outcomes together or categorise according to GALA metrics:

– Cognitive (knowledge and skill acquisition)– Affective – Motor– Soft skills

• Personal • Interpersonal

• Are we interested in different learning outcomes across different sigs? Ages?



Media


Rating the quality of the papers

• How appropriate is the research design?• How appropriate are the methods & analysis?• How generalisable are the findings of this study to the

target population? • How relevant is the particular focus of the study

(including conceptual focus, context, sample and measures) for addressing the question or sub-questions of this review?

• To what extent can the study findings be trusted in answering the study question(s)?



Media


Possible problem with the move away from surveys in WP6.1

• WP6.1 originally specified that we should look at “stakeholder perspectives (e.g. parents, teachers, professors, trainers, educators) … highlighting the challenges that need to be overcome, the expectations that need to be addressed and the suggested research lines to facilitate adoption of SGs/SVWs in education settings”

• A possible problem with not doing surveys is that it might make it more difficult to make or retain contact with schools and educational institutions.

• Would we also want to carry out a literature review on attitudes to and acceptance and expectations of games to provide evidence about stakeholder challenges of introducing games into different educational sectors.



Media


Proposed year 2 activities for WP6.1 & WP6.2: Meta-analysis

• We have already collected some evidence to form the basis for the meta-analysis: – UWS carried out a literature review of learning outcomes up to

Feb 2009. This paper was a narrative review which attempted to identify and categorise learning outcomes of playing entertainment and serious games. We need to update this to Feb 2012.

– We also have Tom Hainey’s analysis of outcomes of games– TUT provided a literature review on outcomes for playing games. – UniGRAZ provided results of a literature search but this was

mainly in the area of Neuroscience



Media


Can we get any papers from the year 1 metrics activities?

3 approaches

1. Review of previous attempts to categorise learning outcomes of games

2. Literature search of outcomes of educational games which have been reported in the literature

3. Literature search for papers including serious games and neuroscience (Unigraz)

Paper on metrics in serious games? Is someone else doing this?#

Liz, Francesco, Michael Kickmeier, Rui Prada and Maria are currently writing a paper about engagement in entertainment games, but given the directive about entertainment games is this still a GALA paper?

We will write a paper about engagement in serious games.

Possible papers from year 1 activities on metrics?



Media


T6.3 Integration methodologies (15)Task leader: CNR-ITD, Rosa Bottino, Michela Ott

CNR-ITD and UNIGE– primary

INESC-ID - secondary

UCM - tertiary



Media


Proposed year 2 activities for WP6.3

identify a selection of examples and best practices of the games which have been used successfully in educational contexts at the different curricular stages: nursery, primary, secondary and tertiary education.

Some GALA partners have already been asked to identify suitable games /projects and we would ask all GALA partners to provide examples of suitable game use experiences.

EU projects in the field of games and learning (in particular those financed in the 6th and 7th framework program-ICT and in other relevant EU programmes such as the lifelong-learning program) will be examined.



Media


Proposed year 2 activities for WP6.3

• The reviewers suggested we use CHERMUG project• Since WP3 reported minimal penetration of SGs in

engineering and manufacturing, it might be worth exploring how SGs might be integrated into engineering education, especially at the secondary level where there is probably a reasonable chance for effective use. The panel recommends an effort be made to coordinate further exploration of how best to integrated SGs into education with WP3.

• Are reviewers suggesting we focus on only a few games?



Media


WP6.3: Possible games/projects

Primary School• CNR: mind games• Harri’s game – edu-elements – mathematics and

chemistry – Harri Katemo• The Code of Everand – Ian Dunwell, Coventry Uni, road

safety game • Kristian Kiili’s brain training cell phones • Consolarium in Scotland: Brain training: reasoning

skills; Nintendogs: maths, resource management and communication skills



Media


WP6.3: Possible games/projects

Secondary• UCM: game to teach Resuscitation procedures 12-14

years; 334 students real world knowledge; paper to be published in English in Emergencies

• Me Tycoon (Kam) – extra- curricular• Consolarium in Scotland: Guitar hero: in transfer from

primary to secondary



Media


WP6.3: Possible games/projects:Tertiary

• UWS: Requirements collection and analysis game for IT students; paper published

• UCM: – games for medical students, operating theatre, labwork• ESADE: eFinance game teaching finance concepts to business

students• ESADE: Metavals – teaching statistics to students• BIBA: Risk management and supply chain management for Masters

students in engineering• AAU: investment game for business success• UNIGE Francesco – exploring entrepreneurship

Games for digital electronics /computer programming

UCM



Media


EU Lifelong learning game projects

Current– CHERMUG: teaching research methods to nursing students, Liz

Boyle– STARTUP – secondary schools entrepreneur game, Thomas

Connolly– Goblin project – developing a game for language; Dr Hen van

den Heuvel, CLST Radboud university, Netherlands– Magical: Michela Ott– Roman Breuer: RWTH Aachen – game to understand risk

management in water supply/flooding

Former– ARG: Language game, Thomas Connolly - secondary– ECLIL: Language game, Thomas Connolly - secondary



Media

1st Review Meeting

T6.4 Community of teachers and tutors (TU DELFT)

Task leader: TU DELFT, Igor Mayer (3)

UU, Remco Veltkamp (3) HWU, Theo Lim (4)

The community will emerge to some extent from GALA members, research and development contacts, stakeholders, association, outreach advisory board, ECGBL and other conferences

The reviewers want us to: “explore link to existing communities and networks in the education sector investigating the phenomenon of serious games, e. g. those linked to existing LLP projects and those initiated by EUN (European Schoolnet)”

There is a LLP network on serious games called SEGAN: Serious Games Network.



Media

1st Review Meeting

Presentation of D6.1: Year 1 time line

1-3 months 4-6 months 7-9 months 10-12 months

Task1: stakeholder needs

Identify stakeholders

Identify respondents; design questionnaire

Task 2:search and metrics

link to T1.2 & T7.2; agree search terms and scope of search

carry out search; agree metrics

Categorise games according to agreed metrics

Task 3: integration of games into education

Use 6.1 and 6.2 to identify studies

Task 4: community

Use WP6.1 to identify key stakeholders

Deliverables Interim activity report

Report 1: integration of SGs in educational processes



Media

1st Review Meeting

New 4 year time line

Year 1 Year 2 Year 3 Year 4

Task1stakeholder needs

Design survey of user and stakeholder needs; Identify stakeholders and respondents

Update search and database; identify relevant papers; carry out meta-analysis of learning outcomes of educational games

Update search and database; Publish meta-analysis of learning outcomes of educational games

Update search and databaseUpdate meta-analysis of learning outcomes of educational games

Task 2search and metrics

Search and database; link with WP T1.2; T7.2 - metrics

Update search and database; identify relevant papers; carry out meta-analysis of learning outcomes of educational games

Update search and databasePublish meta-analysis of learning outcomes of educational games

Update search and databaseUpdate meta-analysis of learning outcomes of educational games

Task 3integration of games into education

Use 6.1 and 6.2 to identify games; Examine lit and EU projects

Identify best practices and examples – case studies

Write up and publish best practices and examples – case studies

Develop guidelines; evaluation

Task 4community

Use WP6.1 & 6.2 to identify interested parties

Develop links with communities

Specific SG initiatives & activities within the community

Evaluate initiatives

Deliverables Report 1 Report 2 Report 3 Report 4



Media


Self evaluation of progress

• Include a WP evaluation plan that includes measures of success in future annual reports and present progress against those measures at the next annual review meeting

• Metrics –one measures is the amount of work carried out, papers searched etc.

Documents

Genova December 6-7 2011 European Commission Information Society and Media 1 st Review Meeting GaLA Game and Learning Alliance The European Network of