Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators:

A Follow-up Study

Immersive Reflective Experience-based Adaptive Learning

8th European Conference on Technology Enhanced Learning (EC-TEL 2013)

Gudrun Wesiak, Adam Moore, Christina M. Steiner, Claudia Hauff, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary

Donohoe, Gordon Power, & Owen Conlan

Overview

- Intro

- ImREAL

- Challenges and Approach

- ETU Simulator

- Simulator Augmentation

- Empirical study

- Methods and results

- Discussion & Conclusions

EC-TEL 2013 2

Intro

- Experiential Training Simulators

- For engaging and motivating learning experiences

- To create situational contexts to practice job-relevant skills

- Adult learning (Andragogy)

- Self-directed, experienced-based, goal-oriented, intrinsically motivating, relevancy-driven (Knowles, 1984)

- Influenced by learner’s physical, digital and social world

- Should adapt to learner’s previous experiences, knowledge, ….

EC-TEL 2013 3

4

ImREAL: Challenge

Simulated environment

Augment existing training simulators by...

Real-world activity modelling

Enriched user modelling

Affective-metacognitive scaffolding service

Develop services for different simulators

Real-world experience

Close the gap

Medical interview training

Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode

where the host has to decide about ordering food for his business guests.

The simulated scenario includes four episodes: Greetings (situations embed arriving

on time, different norms about greetings, first impression, and use of body language),

Dinner (situations embed use of body language and different preferences about food

and drink), Bill (situations embed use of body language and different norms about

payment), Goodbye (situations embed use of body language and different norms

about greetings). Figure 2 illustrates the interface and the interaction features provid-

ed to the leaner to select a response and read/write micro-blogging comments.

The simulator was used by 39 users who attended interactive sessions at learning

technology workshops or responded to invitations sent to learning forums in Europe.

The data was collected during the period 29 Oct 2012 – 15 Jan 2013. Micro-blogging

comments (total of 193) were provided by 25 of the users, and were semantically

augmented with ViewS for IC (see Table 3 for a summary).

Table 3. Summary of the ViewS annotation of the micro-blogging content in the study.

Episode Greetings Dinner Bill Goodbye All

Annotations WNAffect 82 84 18 8 192

Body Language 311 236 100 76 723

Distinct Ontology

Entities

WNAffect 36 36 11 5 57

Body Language 76 63 43 33 106

The output of ViewS was shown to two simulator designers (with background in

Psychology) who were involved in the creation and improvement of the simulated

scenario. The semantic augmentation output was visualised in the form of semantic

maps to enable exploration of both ontologies - WNAffect and Body language-, and

identification of ontology entities that linked to user comments and their location over

Business simulation & Buddy program

Simulated world

Real world

Fore

thought R

efle

ctio

n

Experiential Training Simulator: EmpowerTheUser - ETU

Medical Interview Training:

Two Psychiatric Interview Scenarios Mania & Depression

Basic idea: Watch patient video and select appropriate answer

2Training modes: Assess mode and practice mode

EC-TEL 2013 5

ETU: Assess modePerformance Prediction and Scores

EC-TEL 2013 6

Interview steps (Process)Interview skills

ETU scores

ETU: Practice mode

EC-TEL 2013 7

Visible Decision scores

Coach: feedback on decision path

Thinking prompts and scaffolding service

ETU: Practice mode Meta-cognitive scaffolding (MSS)

EC-TEL 2013 8

ETU: Practice mode Meta-cognitive scaffolding (MSS)

EC-TEL 2013 9

Prompts are based on MAI-ROC (Schraw, 194)Prompts are mapped to different interview situations

ETU: Practice mode:Affective Metacognitive Scaffolding

EC-TEL 2013 10

SAM:Smiley Affect Measurement technology

• Explicit assessment of current emotional state with SAM

• Selection of displayed prompts influenced by users’ emotional state

• Sentiment analysis of digital traces (U-SEM)

• Mappings of prompts to situations as weighted list

Empirical Study: Research Questions

• Simulator Augmentation:

• Is the simulation relevant for users?

• Are the services well integrated?

• Digital Traces:

• Usage, privacy & trust

EC-TEL 2013 12

• AMSS: Are there effect on…

• self-regulated learning?

• users’ learning experience?

• learning performance?

• motivation/affect?

Longterm Methodology:

Berthold et al., EC-TEL 2012

Participants

UT1: 143 TCD medical students, 22.7 years, 54.8% male, 97% real world interview experience (15.4% psychiatric)

UT2: 152 TCD medical students, 22.81 years, 50% male, 99% real-world interview experience (25.6% psychiatric)

• 2011

• Pure Simulator

Baseline

• 2012

• ETU with MSS

User Trial 1• 2013

• AMSS, SAM, U-SEM

User Trial 2

Methods

EC-TEL 2013 13

Methods: Procedure and Measures

EC-TEL 2013 14

Cohort survey (N=95)

Training in the simulator (N=143/152)

Post-sim survey (N=39/ 40)

Real-world interview

Post-int survey (N=7/19)

• Demographics• Social Network use & attitudes

• Log-data (Usage, scores)2 scenarios, 2 modes

• Real-World Relevance• SRL-strategies• Motivation & Affect• Learning Experience• Learning Performance

• Real-world relevance• Motivation• Fb on modes, AMSS, SAM, SN

Results: Log DataSimulator Usage & Training Times

15

Usage (solid lines) and mean training times (dashed) of assess and practice mode

UT1: very few users for Mania

UT2: short times for Depression

Assess vs. Practice:

Assess: more users and longer times

UT1 vs. UT2:

Same no. of users, but increased overall training times (both modes)

ResultsWhich aspects are (not) effected?

No differences in ...

Integration & perception of thinking prompts

Self-regulated learning strategies (QSRL)

Motivation to improve skills, to perform good, to apply what has been learned (post-sim & post-int)

No difference between UT1 & UT2, but change from Baseline

Perceived relevance of simulation

Usability (SUS)

Overall motivation

Overall workload (decreased!)

Deacrease from UT1 to UT2

Motivation to learn more about interviewing (post-sim)

EC-TEL 2013 16

Baseline

UT1/UT2

UT1=UT2

ResultsRelevance for Real-World Interviews

UT2 students feel better prepared for real-world interviews

Perception of relevancy & preparedness unchanged after real-world interview

17

ResultsLearning Experience

Flow (Flow short scale; Rheinberg et al., 2003)

increase in overall flow, technical flow & 7 single items

i.e. stronger involvement into the task, greater feeling of smoothness and fluency, smooth interplay between software components

18

Flow score means on item basis (left fig.) and overall means (and SE) for the FKS, worry, and tech scales (right fig.)

ResultsLearning Experience

Workload: Nasa-TLX (Hart & Staveland,1988)

Lower overall workload in UT2

UT2 users felt less frustrated & had stronger feelings of success

EC-TEL 2013 19

Mean scores (and SE) from the NASA-TX workload scales for the two user trials.

ResultsSelf-Regulated Learning

Reflection notes and AMSS prompts

• 1092 triggers provided (M=15/student) in UT2, 2001 in UT1

• MAI-prompts: 88.4% of 69 answers positive

• Increase in proportion of reflective notes

EC-TEL 2013 20

Table 1. Rounded percentages of content types for entries from the note-taking tools.

Note: Entries can be coded to more than one type, thus percentages may exceed 100.

Users Text entries Position Technical Notes Reflection

UT1 50 107 17 57 16 66

UT2 35 86 1 13 7 93

Results: Log DataETU Scores

ETU: predicted and achieved scores in UT1 and UT2

21

Higher scores in UT2 for introduction, eliciting and planning phase

Mainly underestimation of scores in UT2

ResultsAffect

Affective state reported after simulation: Total affective state scales (TASS, Nicolescu et al., 2006 )

Affect during simulation:

SAM was displayed 352 times (2.23 times / user) – used by 1 person

22Mean TASS scores on 7 subscales plus overall mean score across all scales.

NO

Yes

ResultsSocial Networks

Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn

Nobody (of 152) provided SN-ID in simulator (13 did in cohort survey; 11 fb)

For detailed results: see Moore, 2013 (Umap; LBR)

Reasons:

EC-TEL 2013

23

ResultsSummary

• Changes from pure (BL) to augmented (UT) simulator

↑ Perceived relevance, usability, state motivation

↓ Overall workload

• Changes from MSS (UT1) to AMSS (UT2) simulator

Relevancy: ↑ stronger feeling of preparedness

SRL: ↑ effort (times), feeling of success, reflective notes;

Performance: ↑ ETU-Scores

Learning Experience: ↑ flow↓ decrease in frustration

Motivation: ↓ to learn more about interviewing

EC-TEL 2013 24

Conclusions & Challenges

• Metacognitive scaffolding supports learners on different levels

• Evaluation of user model augmentation

Low user acceptance for exploiting their social digital traces

Need to increase willingness to share SN-IDs

• Study learning activity rather than purely subjective reports

Analysing learners’ real activities with the simulators

• (Non)-invasiveness vs. (un)-visibility

SAM, thinking prompts

EC-TEL 2013 25

Conclusions & Challenges

• (Non)-invasiveness vs. (un)-visibility

SAM now with triggers and presented in main window

EC-TEL 2013 26

NO

YES

Thanks for listening!

EC-TEL 2013 27