Is virtual/augmented reality beneficial for learning? EEG mental load indices

associated with learning in 2D compared to 3D displays

Miriam Reiner

The VR and NeuroCognition lab Technion, Israel Institute of Technology

http://vrneurocog.wixsite.com/vrneurocogMiriamr@Technion.ac.il

Amit Rozen

Alex Dan

Anat Dahan

Rotem Bennet

Nehai Farag

Darian Ryder

Nir Segal

Vered Halevi

Tal Pik

Guy Zuckerman

Boris Yazmir

Lulu Watad

Avivit Dolev

Postdocs:

Dr. Chen Ryder

20-Jun-18

The lab: Technology and methodology

Sig22 Neuroscience and Education

Eye

tracker

Robotic arms

EEGPerformance/learning

Brain correlates

Virtual Reality

(stimuli)

Virtual reality

20-Jun-18

Participants perform a task in a highly immersive virtual reality,

while connected to EEG, eyetrackers etc.

Projects• The Presence group (1999) – EU initiative to plan future calls on Presence.

• PresenCia

• PRESENCCIA

• IMMERSENCE

• EUROVERCITY (current)

• BEAMING

• Emphatic

• Beaming enhanced by machine learning for remote interaction• Smaller projects:

• Remote Presence and memory modification • BITE- Brain-research Inspired Technologies for Education• Educational neuroscience: What neuroscience tells us about learning in educational contexts

(funding from the Chief scientist, Ministry of Science)

• BBC on BEAMING : BBC's Rory Cellan-Jones. www.bbc.com/news/technology-18017745

• In a related piece Laurence Peter aof the BBC talks to Mel Slater from the University of Barcelona's and Ray Purdy from UCL about "Real-world beaming: The risk of avatar and robot crime“ www.bbc.co.uk/news/world-europe-17905533

Sig22 Neuroscience and Education20-Jun-18

A typical experiment…• immerse participants in an immersive multisensory virtual world

•ask them to perform a task (e.g. a motor task, solve a quiz/problem, demonstrate a concept, etc),

• connect participants to EEG and measure the resulting EEG signals -- frequencies, or event-related-potential;

• correlate EEG/ERP’s with behavioral patterns, stimuli/ environmental situational cues.

Sig22 Neuroscience and Education20-Jun-18

Look for correlations between:

• Purpose: attempt to model conditions for learning.

Sig22 Neuroscience and Education

Indicators of learning –

behavioral (action &verbal

Brain correlat

es

Sensory, environmental

situational cues

20-Jun-18

Will talk about:

•VR/AR, brief intro.

• Is V/AR better for learning compared to a flat screen? Why?• EEG mental load indices associated with learning in

2D compared to 3D displays

Sig22 Neuroscience and Education20-Jun-18

What is virtual reality?

•Virtual Reality (VR), can be referred to as the subjective sensory experience of

being immersed in a computer-mediated world

Sig22 Neuroscience and Education20-Jun-18

Augmented reality –virtual and physical

Sig22 Neuroscience and Education

• ….special gloves, earphones, and goggles, all of which receive their input from the computer system.

• .....senses are controlled by the designer. The system monitors the user's actions.

20-Jun-18

Design of the CAVE HTC-VIVE

https://www.youtube.com/watch?v=tAr7xjKoM5A

Setup

Sig22 Neuroscience and Education

• Virtual environment

20-Jun-18

A new type of remote learning?Face-to-face like e-learning

Only about 10% is conveyed via words… What conveys everything else? Bodily cues, physiological changes – blush, pupil fluctuations… rhythm of voice, bodily dynamics….

Four examples:

• Real-time brain measures of mental load while performing a learning task under varying conditions; (or– Is VR/AR better than flats screens?

• The brain mechanism behind Insight / incremental problem. Can we enhance insight?

• The shape of misconcept in science: what are neural responses when ‘you’ see the world behaving according to your own misconcepts?

• And how is it different from error-EEG signals?

20-Jun-18 Sig22 Neuroscience and Education

• Expedited memory consolidation and implications for a potential neural model of enhanced memory;

20-Jun-18 Sig22 Neuroscience and Education

EEG measures of

mental load

during problem solving in 2D and 3D VR. Alex Dan PhD thesis

• Two groups:

• Observed a remote instructordemonstrating two origami tasks –simple/complex- in 2D flat display Thenin VR 3D.

• The second group followed the sameprocedure, reversed order

• EEG was measured in real time.

• Participants were asked to repeat thefolding of the simple/complex task.

• EEG was not taken during actual folding

20-Jun-18 Sig22 Neuroscience and Education

Method

Viewsonic 3D ready Projector

Screen Emmiter Vision Pro

EEG recording software

WinEEG 2.96.63

Pa

rtic

ipan

t

Researcher

Mistar-EEG 201

Wired monitoring device

3D NVidia GlassesDell M6800 Laptop with

Nvidia Quadro K500M Graphic

Card

And proprietary software

HDMI to DVI Connection

Nvidia Graphics setup

Stereo Mode

120 Hz. Refresh Rate

Resolution 1024/768

Only to USB port

Procedure

20-Jun-18 Sig22 Neuroscience and Education

EEG File

• select learning interval, epochs

Artifacts reduction

• ICA and Analysis

Set Fourier analysis and parametrers

• Lenght of epoch

• Polynomial trends paremetr

es

• Haning time

window

Analyze

• Table ouput of

EEG spectra, amplitud

e and average

frequency

𝐶𝐿𝐼𝑛 =𝑇ℎ𝑒𝑡𝑎 𝐹𝑧 𝑛

𝐴𝑙𝑝ℎ𝑎 𝑃𝑧 𝑛The Cognitive load

index was calculated

based on the Theta

Fz/Alpha Pz ratio

(Holm et al., 2009). For

each participant, Alpha

Pz, Theta Fz, and the

average cognitive load

index for the

observation sessions

was assessed.

Results

20-Jun-18 Sig22 Neuroscience and Education

Scenario 2D 3DS

Relative

CLI

Red dot -Average

power over four

sessions –.

Green dotted line -

total average power

of all participants;

lue and red dotted

lines - the confidence

interval of 95%. The

y axis --absolute

power in (𝛍𝐕𝟐).

Results

Task+display

Box 2D Box 3D p-value

CLI (SDE) 0.8385 (0.024) 0.4596 (0.096) < 0.0001*

Task+display

Crane 2D Crane 3D p-value

CLI (SDE) 1.2483 (0.128) 0.500 (0.094) < 0.0002**

p-value p < 0.0292* = 0.8

A significantly larger

CLI in 2D compared

to 3D

for both tasks for all

participants

(matched pairs t-test, N

= 17, t-ratio = -3.0681,

SDE = 0.09483, p <

0.0037**).

20-Jun-18 Sig22 Neuroscience and Education

Performance –folding scores

20-Jun-18 Sig22 Neuroscience and Education

Folding Test scores of participants (N=14) after observing

and performing 2D/3D instructions of folding.

The y-axis represents Folding Test scores.

The x-axis represents the groups.

The bars represent the SDE.

The box represents the distribution of participants

Meaning what…?

• Higher mental load is correlated with higher ambiguity – 2D is more ambiguous than 3D. As if– it is more difficult to decipher the semantics behind the visuals.

• Found more Mu rhythms in the 3D condition. Associated with the mirror neuron system activations –learning through mimicking is better in 3D.

• Higher mental load is associated with a larger number of items during ‘short-term-memory’. Need to remember more in 2D. 3D supports memory by reducing the number of items to be memorized

20-Jun-18 Sig22 Neuroscience and Education

Implications for V/AR in social-emotional learning

Suggests that the face to face interaction is crucial

V/AR can be designed …..To enhance emotional interaction

by mimicking the emotional expression of the teacher/student

20-Jun-18 Sig22 Neuroscience and Education

Thanks!

Questions?

20-Jun-18 Sig22 Neuroscience and Education

This project has received funding from the European Union's Horizon 2020 Research And Innovation Programme under grant agreement No. 769872