Brain Computer Interface for Virtual Reality Control

Brain Computer Interface

for Virtual Reality Control

Christoph Guger

www.gtec.at

GRAZ

Mozart

VIENNA

g.tec

MOZART

Emperor„s castle

Musical

Empress

Elisabeth

www.gtec.at

Research Projects

#) EC project: ReNaChip - Synthetic system integration

Rehabilitation of a discrete sensory motor learning function by a

#) EC project: Sm4all – Smart Home for all

Brain-Computer Interface for smart home control

#) EC project: RGS – Rehabilitation Gaming System

faster recovery from stroke with games

#) EC project: BrainAble

BCI with VR and social networks

#) EC project: Decoder

BCI for locked in patients

#) EC project: CSI - Central Nervous System Imaging

#) EC project: BETTER

BCI for Stroke rehabilitation and rehabilitation robots

#) EC project: VERE –Virtual Embodiment Real Embodiment

Dissolving the boundary between the human body

and surrogate representation in virtual and physical reality.

#) EC project: ALIAS – Adaptable Ambient Living Assistant

Robot system interacting with elderly people providing

cognitive assistance and social interaction and inclusion

www.gtec.at

Our co-operations partners

• University of Barcelona, Spain

Mel Slater, Chris Groenegress

• IDIBAPS, Barcelona, Spain

Mavi Sanchez-Vives

•University College London (UCL), UK

Anthony Steed, Angus Antely,

• University of Technology Graz, Austria

Robert Leeb, Gert Pfurtscheller

• Wadsworth Center, New York, USA

Gerwin Schalk, Eric Sellers

• Tel Aviv University

Matti Mintz

Brain-Computer-Interface (BCI)

“A system for controlling a device e.g. computer, wheelchair or a

neuroprothesis by human intention which does not depend on the

brain‟s normal output pathways of peripheral nerves and muscles”

[Wolpaw et al., 2002].

HCI – Human Computer Interface

DBI – Direct Brain Interface (University of Michigan)

TTD – Thought Translation Device (University of Tübingen)

www.gtec.at

Subject/

Patient

Brain-

Computer

Interface

Device

Feedback

EEG/

ECoG control signal

Some examples of BCI applications

BCI

BCI_

www.gtec.at

MATLAB and Simulink environment

User-System #1

Subject,Patient

Real-time system

Real-time blockset

Custom featureextraction andclassification

MATLAB/Simulink

DAQ board16-32 channel

Biosignal amplifier

Custom hardwaree.g. orthosis

Stimulation unit

User-System #2Personal Area Network (PAN)control unit

biosignals

feedback

technical issues

Influencing components

adaptation

to subject

www.gtec.at

How to record brain activity for BCI ?

Functional imaging techniques: FMRI, SPECT, PET

Magnetencephalogram (MEG, SQUID)

Near Infrared Spectroscopy (NIRS, fNIR)

Electrocorticogram (ECoG)

Electroencephalogram (EEG)

www.gtec.at

Measuring brain electrical activity

Electro-corticogram (ECoG)

closely spaced multi-electrode grids or

strips applied directly to the cortical

surface, electrode diameter ~ 4mm, up

to 500 µV, 1 – 100 Hz

high signal-to-noise ratio, high spacial

and temporal resolution

highly invasive and limited study

opportunities

modified from University of Michigan

www.gtec.at

Electroencephalogram (EEG)

1 – 64 (128) channels, 1 µV – 100 µV, DC up to mV-range, 0 – 40 Hz, low signal-to-noise

ratio, moderate spacial resolution, high temporal resolution

Surface electrodes: 8 ...12 mm, mounted with conductive gel/paste

www.gtec.at

Changes of brain electrical activity and mental

strategies

- Slow cortical potentials (anticipation tasks)

DC-derivation, artifact problem, difficult strategy, feedback method

- Steady-State Evoked potentials (SSVEP, SSSEP)

Flickering light with specific freuqency

- Event-related, non-phase-locked changes of oscillatory activity

ERD/ERS (motor imagery tasks)

Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas,

imagination of hand- ,foot-, tongue- movements

- Evoked potentials (focus on attention task)

Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory, ...),

P300

Communication for the ´locked-in´

ALS patient in Germany using a BCI system for communication

Birbaumer, Kübler, Hinterberger,… Tübingen www.gtec.at

www.gtec.at


strategies




Flickering light with specific freuqency







P300

Physiological Background – why does it work

Imagination of hand movement causes an ERD which is used to classify

the side of movement. The desynchronization occurs in motor and related

areas of the brain. Therefore, for analyzing and classifying ERD-patterns

the electrodes must be placed close to sensorimotor areas.

Left hand

movementRight hand

movement

www.gtec.at

C4

GND

REF

RIGHT

C3

The ``Finger Movement Task``

Brisk movement of right index finger

C3

www.gtec.at

Oscillatory Activity

Paradigm for a simple motor imagery BCI experiment

left

right

Fixation cross CUE

0 2 5 6

beep

7431

motor imagery

8 s

Offline data classification

Recording of 40 trials minimumTRAINING

left

right

Fixation cross CUE

0 2 5 6

beep

7431

FB

8 s

feedback (FB)

classifier

Oscillatory Activity

Paradigm for a simple motor imagery BCI experiment

FEEDBACK

www.gtec.at

Patient Tom (C4/C5 lesion)

Case study: Electrocorticograms, ECoG

www.gtec.atWith permission from Schalk 2007

Direct Brain Interface, rhythmic activity, Albany, USA

www.gtec.at

With permission from Schalk 2007

www.gtec.at

Augmentation of neuronal

population activity using a brain-

computer interface

g.tec's Spike & ECoG Workshop

November 14th, 2010

Kai J. MillerPhysics, Medicine Neurobiology and Behavior, Neural Systems LabUniversity of Washington

Basic spectral changes with movement

“ERD”

Real-time representation of cortical activity

www.gtec.at

Active Electrodes

Advantages:

- No preparation (abrasion) of the skin required

- High signal quality and less 50/60 Hz noise with high impedance

- Reduced artifacts from electrode and cable movements

www.gtec.at

Active versus passive electrodes

Active

Passive

www.gtec.at

EYE MOVEMENTS

Channels closer to the eyes (1 and 4) show higher EOG artefacts than

central and occipital channels.

Both passive and active electrodes show a similar EOG contamination

which is also clear because both pick up the same source signal.

www.gtec.at

BITING

Biting produces EMG contamination almost equally on all channels

No difference between active and passive electrodes because both pick

up the same source signal.

www.gtec.at

CABLE ARTEFACTS

Cable artefacts are produced by touching or shaking the cables.

Active electrodes are almost unaffected while the passive electrodes

show large movement artefacts.

www.gtec.at

ACTIVE HEAD MOVEMENTS

Active head movements produce fewer artefacts with active electrodes

compared to passive ones.

Artefacts for both electrodes can occur because of skin-electrode

movements.

Passive electrodes are mostly affected by the cable movements initiated

by the head movements.

www.gtec.at

PASSIVE HEAD MOVEMENTS

Passive head movements have lower accelerations than active head

movements and therefore the artefacts are smaller and mostly visible

with passive electrodes.

www.gtec.at

BCI AND DRY ELECTRODES

BCI use P300, motor imagery or steady-state visually evoked potentials

(SSVEP) measured with the electroencephalogram (EEG) to control

external devices.

Evoked potentials,

event-related desynchronization,

power spectrum

accuracies

were calculated for dry and gel based electrodes to compare them.

www.gtec.at

www.gtec.at

www.gtec.at

www.gtec.at

www.gtec.at

Discussion:

First dry electrode system that works for motor imagery, SSVEP and

P300 (same accuracies reached for all)

Whole frequency range available: 0.1-40 Hz

First dry electrode system that covers extended 10/20 system on frontal,

central, parietal and occipital sites

More low frequency components in the EEG spectrum below 3 Hz

Careful montage required and more sensitive to surrounding noise

Group studies submitted in March 2011 to Journal of Neural Engineering

and BCI conference in Graz 2011.

Biosignal Analysis and Recording System in VE

• The recording system has to work in noisy environments

• CAVE system: creates a 3D Virtual World

TRIMENSION ReaCTor

3 back projected screens (3m x 2.2m)

1 floor screen projected by ceiling mounted projector

• 3D effect with shutter glasses www.gtec.at

Movie: “Walking through a Virtual City by Thought”

BCI Award 2010

www.gtec.at

www.gtec.at

Categorization of the 10 nominees

Title Control signal Application

fMRI Spikes N200/

P300

SSVEP MI Stroke Spelling/

internet/

art

Algorithm

development

A high speed word spelling BCI

system based on code modulated

visual evoked potentials

X X

Motor imagery-based Brain-Computer

Interface robotic rehabilitation for

stroke

X X

An active auditory BCI for intention

expression in locked-in

X X

Brain-actuated Google search by using

motion onset VEP

X X

Brain Painting - "Paint your way out” X X

Thought Recognition with Semantic

Output Codes

X X

Predictive Spelling with a P300-based

BCI: Increasing Communication Rate

X X

Innovations in P300-based BCI

Stimulus PresentationMethods

X X

Operant conditioning to identify

independent, volitionally-controllable

patterns of neural activity

X X

Neurorehabilitation for Chronic-Phase

Stroke using a Brain-Machine

Interface

X X

Total 1 1 6 2 2 7 1

www.gtec.at

Categorization of the 57 submitted projects

Property Percentage

(N=57)

Property Percentage

(N=57)

Real-time BCI 65.2 Stroke 7.0

Off-line

algorithms

17.5 Spelling 19.3

P300 29.8 Wheelchair/rob

ot

7.0

SSVEP 8.9 Internet/VR 8.8

Motor imagery 40.4 Control 17.5

EEG 75.4 Platform/Techn

ology

12.3

fMRI 3.5

ECoG 3.5

NIRS 1.8

Stroke

Potential Users Worldwide

Cerebral palsy – 16,000,000

Brainstem stroke – 10,000,000

Other stroke – 60,000,000

Spinal cord injury – 5,000,000

Postpolio syndrome – 7,000,000

Amyotrophic lateral sclerosis – 400,000

Multiple sclerosis – 2,000,000

Muscular dystrophy – 1,000,000

Guillain-Barre syndrome – 70,000

New York State Department of Health

Wadswor t h Cent er

Brain Computer Interface (BCI) for

Stroke Rehabilitation – Virtual Reality

• BCI in combination with VR for stoke rehabilitation

– People can be motivated via VR to activate their motor cortex

• Simple bar feedback:

– Patients should try to increase the length of a virtual bar just

by motor imagery tasks

• Rehabilitation games (e.g. RGS – Rehabilitation Gaming System)

– Patients play games activated by the motor imagery BCI

– Patients can activate the virtual arm (picture: RGS project)


Stroke Rehabilitation – Virtual Reality

• BCI - VR System setup:

EEG cap

www.gtec.at

• Motor imagery-based Brain-Computer Interface robotic rehabilitation

for stroke (Cuntai Guan et al.)

www.gtec.at


Stroke Rehabilitation – Recent Results



• Study for comparison between BCI guided rehabilitation and non

BCI guided rehabilitation (MANUS – manual user)

– 26 patients were recruited, and randomized into two groups

(15 in MANUS group, 11 in BCI group). The protocol of the

rehabilitation is summarized as follows

• Patients performed 4 weeks‟ rehabilitation training, 3

sessions per week, and each session lasted around 1

hour

– Clinical evaluation was done at the beginning of the training

(week 0), mid of the training (week 2), end of the training

(week 4) and following-up assessment (week 12).

www.gtec.at

• Clinical evaluation measures various outcomes for patients

– Fugl-Meyer Assessment (FMA) score is reported here as it represents

an overall recovery of motor impairment.

– Patients in MANUS group performed 960 repetitions, while BCI group

only performed 160 repetitions.

www.gtec.at



P300 – Approach (EEG)

A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

visual stimulationTime (ms)

Amplitude (µV)

51

www.gtec.at

The 6 x 6 matrix speller, single character flash

A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

concentrate on „W“Individual character intensifies for 60ms with 10ms between each

intensification

www.gtec.at


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

www.gtec.at


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

www.gtec.at


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

www.gtec.at


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

www.gtec.at


Target:1

5 µV

Non-target: 1 µV

Letter W Presentation

NON Target

Target

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7-10

-8

-6

-4

-2

0

2

4

6

8

10

time [s]

[µV]

P300

www.gtec.at

A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9

Rows- and Columns- Flashing

A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9


A B C D E F

G H I J K L

M N O P Q R

S T U V W X

Y Z 0 1 2 3

4 5 6 7 8 9


www.gtec.at

Performing real-time BCI experiments

Hands on seminar

P300 based speller Video

BCI goes

www.gtec.at

• BCI for social networking

• P300 based BCI – twitter interface

• Tasks:

• Definition of the users needs

reduced set of full functionality

• Adaptation of P300 interface

http://twitter.com/

www.gtec.at

Twitter Control

www.gtec.at

Japanese version

www.gtec.at

www.gtec.at

intendix - g.tec integrates Brain-Computer Interface (BCI)

technology into patients‟ everyday life

intendiX® is designed to be installed and operated by caregivers/patient‟s family at home

enables the user to sequentially select characters from a keyboard-like matrix on the screen

requires some training but most subjects can use intendiX® after only 10 minutes

a spelling rate of 5 to 10 characters per minute can be achieved by the majority of healthy users

the system can trigger an alarm, let the computer speak the written text, print out or copy the

text into an e-mail or to send commands to external devices

put on the cap, inject a drip of gel into each of the electrodes and start to spell!

intendiX® includes the active electrode system, a portable EEG acquisition system and a

notebook or netbook computer with the intendiX® software installed.

www.gtec.at

Live Experiment:

Smart Home Control XVR and BCI

To use the P300 potential for smart home control we developed:

Virtual Reality apartment in XVR (extreme VR) – open source package

to build VR

The Smart home needed several controllable elements such as TV,

music, windows, doors,...

Special icons to control all the devices inside the apartment

Portable BCI system to be inside the 3 D environment

Reliable and fast BCI system

Requirements

www.gtec.at

Concept

www.gtec.at

Concept

www.gtec.at

Subject

BCI-System

Electrodes and

biosignal amplifier

Signal processing

VR smart home

with controllable

elements

Head-tracker

UDP interface

Switch on the BCI

system Extract the EEG

information

Send commands

Translate BCI

commands

Highly immersive

feedback

www.gtec.at

The Virtual Reality apartment

Designed by Chris Groenegress, Mel Slater

www.gtec.at



www.gtec.at



www.gtec.at

Control matrix for smart home with special icons

Select music

www.gtec.at

Goto specific position – The Beamer

12 subjects, 8 EEG channels recorded

Fz, Cz, P3, Pz, P4, PO7, Oz, PO8

Referenced to right mastoid, grounded to the forehead

Data recorded with g.MOBIlab+ via Bluetooth

Fa = 256 Hz, bandpass 0.1 – 30 Hz

1st training run -> 7 icons per mask (7)

Application runs -> 23 icons

„Spelling Device“ Application

Single character flash experiment

Total of ~ 2.5 hours incl. electrode montage and instruction of the

subject

Study Design

www.gtec.at

www.gtec.at

Control the Smart Home by your thoughts

In cooperation with the

Virtual Environments and Computer Graphics, UPC, Barcelona

www.gtec.at

Results

- tested with 8, 4 and only 2 flashes per item

- best result: subject 8 with 100 % accuracy for 8 and 4 flashes

- worst result: subject 5 with only 30 % for only 2 flashes

Best and worst selection

www.gtec.at

Comparison of different masks: accuracy and EPs

-> Accuracy depends on arrangement of characters, background,...

and not on number of icons flashing in the matrix

Best accuracy

Worst accuracy

11 µV max15 µV max 8 µV max

The P300 works very well for spelling

This is based on 85 subjects so far !

Discussion

www.gtec.at

Why does the GoTo perform

worse?

What makes the difference?

www.gtec.at

www.gtec.at

Controlling Second Life with the BCI

Allows patients to take part in social networks

Control avatars just by thinking

Patients appear as healthy person

www.gtec.at


strategies




Flickering light with specific frequency







P300

www.gtec.at

Steady-State Visual Evoked Potentials

(SSVEP)

Frequency of stimulation Brain response

0 ... 2 Hz transient (single) VEP

3 ... 5 Hz undefined response

6 ... 24 Hz SSVEP

www.gtec.at

MethodologyVisually Evoked Potentials (VEP)

single VEP

www.gtec.at

MethodologySteady State Visually Evoked Potentials

(SSVEP)

SSVEP

7

Hz

Steady-State-Evoked Potentials

Higher

Frequency(e.g. 17 Hz)

Lower

Frequency(e.g. 14 Hz)

EEG Power Spectrum EEG Power Spectrum

www.gtec.at

Steady-State Visual Evoked Potentials

(SSVEP)

up to 48 different frequencies possible!

A B

C D

E F

Robot with video camera control

www.gtec.at

Saumitra Dasgupta, Mike Fanton, Jonathan Pham, Mike Willard

Faculty advisors: Deniz Erdogmus (BCI) and Bahram Shafai

(iRobot)

Equipment used: g.USBamp with g.Butterfly electrodes, iRobot

Four checkerboards flicker at:

13Hz (top left: means turn left)

11Hz (top right: means turn right)

9Hz (bottom left: means go forward)

7Hz (bottom right: means stop).

Robots sends video back from webcam via Skype to operator

Zero-order-hold fashion control - until a new/different command

comes the robot continues to perform the latest received

command.

Welch periodogram + SVM classifier of frequencies

+_

less training

up to 48 classes

fast response

up to 40 ... 55 bit/min

reduced responses for

higher frequencies

annoying stimulation

the adaptation problem

SSVEP

BCI Award 2010

www.gtec.at

Documents

Brain Computer Interface for Virtual Reality Control