Measuring emotions: New research facilities at NHTVExperience Design Intervening with people,...

Measuring emotions: New research facilities at NHTV

Dr. Ondrej Mitas Senior lecturer, Tourism, NHTV

experiences are keycentral concept in tourism management

one of three guiding research themes of NHTV

emotions make experiences valuablebrief, intense reactions to specific things

complex (mind, body, behavior)emotions are the value in tourism experiences

fuel memory, (re)purchase decisions, loyalty, reputation, health

Emotion mangementWhich emotions do you want which customers to feel at which touchpoint?

Emotion measurementWhich emotions do which customers actually feel when?

Experience DesignIntervening with people, interfaces, and setting to match the promised experience

Experience MarketingUsing the right emotion to that triggers decisions

Emotion mangementWhich emotions do you want which customers to feel at which touchpoint?

Emotion measurementWhich emotions do which customers actually feel when?

Experience DesignIntervening with people, interfaces, and setting to match the promised experience

Do you plan long enough into the future?

Are you really open to finding out?

Is there anything you can actually change?

Experience MarketingUsing the right emotion to that triggers decisions

Do you add value beyond price?

Emotion mangementWhich emotions do you want which customers to feel at which touchpoint?

LEGO® Serious Play® workshops: Facilitation to inspire and formulate customer experience statements

• Contact: Xander Lub

Experience DesignIntervening with people, interfaces, and setting to match the promised experience

CRADLE: Digital and blended experiences

Experience MarketingUsing the right emotion to that triggers decisions

Storytelling: Creating emotion from your product DNA

Emotion measurementWhich emotions do which customers actually feel when?

Self-response questionnaires+ inexpensive, flexible - validity problems

Narrative interviews+ precise, very flexible - validity problems

Facial coding+ time- and emotion-specific, highly accurate- clear view of the face and expression required

Neuro and bio techniques+ very time-specific, accurate, also un-conscious/-expressed emotions - less emotion-specific, limited by context

A laboratory at the leading edge of emotion measurement

A three-part laboratory in which partners can measure, design, and evaluate experiences

Measuring emotional reactions

Mobile emotion recording

Virtual reality simulations

Part 1: Mobile emotion recording

• Measure GSR (sweating), heart rate, and facial expressions using wearables and cameras

• Record emotional experiences during leisure/tourism, as they happen

• Contact: Ondrej Mitas

Tourist Info Center as an experience A mobile emotion measurement study

• The Gent TIC facilitates quality experiences of this beautiful medieval city

• Employees recently completed soft selling training, further developing the experience of visiting the information center itself

• Thus, an interesting context for testing and refining experience measurement tools

Using the tools of the NHTV Experience Lab

• Empatica E4 wristbands on 7 student mystery visitors • A variety of action and professional cameras pointed

at students and TIC employee • FaceReader facial expression analysis software

students fitted with wristbands at

beginning of day

one student at a time sent on a mystery visit;

face also recorded

for realism, students then enjoyed day based on advice

Experience lab tools reveal the Gent TIC experience

• Facial expressions of visitors were neutral about half of the time, especially when looking at maps; positive about a quarter of the time, especially when starting and ending the conversation

• Facial expression of employee was most positive when talking with visitors

• Heart rate often showed a local peak during conversation, suggesting positive valence (Example participant below; orange line is heart rate)

EntersofficeBeginsconversation

Endsconversation

Facial expression analyses show the importance of the beginning and end of the visit––these are most positive

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

firstquarter secondquarter thirdquarter fourthquarter

Valence(positivity)

participant1 2 3 4

Higher recommending visitors are more positive at beginning (TIC and Gent in general)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

firstquarter secondquarter thirdquarter fourthquarter

Facialexpression positivity ofmoreandlessGent(thecityingeneral)recommendingvisitors

recommend==8 recommend==9

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

firstquarter secondquarter thirdquarter fourthquarter

Facialexpression positivity ofmoreandlessTICrecommendingvisitors

recommend==8 recommend==10

Part 2: Measuring emotional reactions

• Measure high-quality EEG (brain activity), GSR (sweating), and heart rate

• Accurate picture of physical aspects of emotional responses

• Neuroscience techniques for testing marketing

• Contact: Marcel Bastiaansen

What is Electroenceophalography (EEG)?

• Electrodes placed on the scalp in an elastic swim cap • Amplifier strengthens the small changes in electrical

potential due to brain activity • Responses in this signal are averaged over many

pictures and participants to show differences

When averaged over many pictures, differences due to emotion show up

Time: 0 Time: 1 secondWe use facial coding or electrodes to determine which emotionBuilding AI algorithms at EUR to classify “desire” emotionBe careful with unsupported promises of commercial firms

This difference shows valence (+/-) vs. emotionally neutral

This difference shows strongly arousing vs. less arousing

User generated and commercial photos in Zeeland An in-lab emotion reaction measurement study

• Zeeland destination marketing organization traditional marketing with TV commercials (Land in Zee) and stock photos, as well as user-generated photo and video competitions (Fan van Zeeland)

• Strategic decisions among these approaches? Are TV commercials still effective, and are user-generated or stock photos more powerful?

Using the tools of the NHTV Experience Lab

• Consumer neuroscience approach using EEG • Experimental design with different combinations of

commercials and picturesGroup 1TV commercial Zeeland

Group 2TV commercial Honda

UG pictures of Zeeland

Stock pictures of Zeeland

UG pictures of Zeeland

Stock pictures of Zeeland

UG pictures of Zeeland

Stock pictures of Zeeland

EEG EEG

Experience lab tools reveal the emotional impact of TV commercials

Before Zeeland commercialAfter Zeeland commercial

Component 3: Interaction with technology

• Use VR and eye tracking to design and evaluate virtual or simulated experiences

• Combine with EEG or wearables for accurate emotion measurement

• Contact: Marnix van Gisbergen

How we experience VR A virtual reality movie study (for PhD of Wim Strijbosch)

• For marketing and experience design in attractions such as amusement parks, a fundamental question is how people remember experiences

• Specifically, how do we pick “experiences” from our stream of consciousness to remember?

Using the tools of the NHTV Experience Lab

• Watching two versions (one suspenseful) of a VR film while EEG, skin, and heart rate signals are recorded

50participants

Virtualrealitystimulus

Virtualrealitystimulussusp.

Retelling theexperience

Retelling theexperience

Self-reportonretold experience

Self-reportonretold experience

Experience lab tools reveal the VR experience

Experience lab tools reveal the VR experience

Correlating self reported emotions to physiology to refine physical detection of emotions

Experience lab tools reveal the VR experience

Comparing physical signals at key moments in film to determine which emotions mark “beginning” and “end” of experience

Manage customer experiences with us

Excellent customer experiences require managing, measuring, and designing for emotions

The future of emotion measurement is physical

Our lab offers the most up-to-date emotion measurement and management process in leisure and tourism

To the demo…Ondrej Mitas ondrejmitas@gmail.com

Wilco Boode boode.w@nhtv.nl