Transcript
Page 1: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

1

DR. VASSILIS-JAVED KHAN W O R K S H O P : P U T T I N G M O B I L E S E R V I C E S I N T O C O N T E X T

W A R S A W J U N E 2 0 1 1

EVALUATING USER EXPERIENCEOF MOBILE, CONTEXT-AWARE APPLICATIONSI N A U G M E N T E D V I R T U A L I T Y

Page 2: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

2

Dr. Vassilis-Javed Khan Dr. Paul Ketelaar Dr. Marnix van Gisbergen Dr. Andrés Lucero

Dr. Koos Nuijten Arief Ernst Hühn

Nils Desle

FINANCIAL SPONSOR: RAAK Pro scheme of:

RESEARCH PARTNERS:

Page 3: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

3

ME

• Senior Lecturer, Academy of Digital Entertainment, NHTV Breda University of Applied Sciences,The Netherlands

• Philips Research, Eindhoven• Vodafone R&D, Maastricht

• MSc., CEID, UPATRAS.GR• PhD., Dept. Industrial Design,

Eindhoven University of Technology, TUE.NL

Page 4: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

4

INDEX

• Position the approach we take compared to Lab & Field studies

• Present our setup

• Present 3 studies of the evaluation of a mobile, location-based app in our setup

• Conclusions

• Challenges / Future work

Page 5: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

5

ULTIMATE GOAL OF CONTEXT-AWARE APPS

• Provide added value to user

• How to evaluate user’s experience while taking context into account?

• Lab vs. Field test

Page 6: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

6

LAB

• Highly controllable• Useful for usability-related

studies

• Ecological valid?• Context?

Page 7: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

7

FIELD TEST

• Context

• Difficult to conduct• Logistic & technical challenges• Difficult to observe• Few participants• Qualitative

Page 8: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

8

EVALUATION IN AUGMENTED VIRTUALITY

• Context• Easy to manipulate• Large numbers of participants• Can include mobile phones,

tablets, other interaction devices• Can observe behavior

Khan, V.J., Nuijten, K. & Deslé, N. (2011). Pervasive Evaluation Application within Virtual Environments. Proceedings of International Conference on Embedded Computing and Communication Systems, PECCS 2011, 261-264

Page 9: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

9

MIXED REALITY

Real Environment Augmented Reality Augmented Virtuality Virtual Environment

Milgram, P. & Kishino, F. (1994) A taxonomy of mixed reality visual displays. IEICE transactions on information systems, E77-D(12)

VIRTUALITY CONTINUUM

Page 10: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

10

OUR SETUP: CAVE

• 4 back-projection screens• 3.5 x 2.6 m.• Illusion of a continuous 360o view

• Head-tracking with 4 Wii-Motes• User wears a headband with

infrared LED• ”Human joystick”• V.E. created in Maya & rendered

using the OGRE 3D engine• Two V.E.

• Supermarket,• City in an island• More to come

Page 11: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

11

Page 12: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

12

Page 13: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

13

Page 14: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

14

OUR SETUP

• The virtual supermarket is a generic supermarket, based on common denominators found in typical Dutch supermarkets

• Contains: isles, freezers, checkout counters, etc., all rendered with photo realistic graphics

• Audio: typical supermarket music, typical announcements

• ~20% of the shelves are filled with products

• Android app, Bluetooth communication with V.E.

Page 15: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

15

NOVELTY

• Immersive User Experience• Nearly photo-realistic vs. low resolution graphics• CAVE vs. V.E. presented on monitors

• Research focused on UX, going beyond effectiveness and efficiency

• i.e. perceived intrusiveness, shopping behavior, feeling of presence, etc.

• Capturing actual behavior vs. opinion• Marketing studies used exclusively scenario-based methods

Page 16: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

16

3 STUDIES

• Location-based ads

• Study 1: 27 participants (pilot)• Study 2: 70 participants• Study 3: 70 participants

• Wanted to measure: Perceived intrusiveness of ads• Has been reported:

• Perceived intrusiveness of ads leads to ad irritation1

• High values of ad irritation lead to ad avoidance1

1Li Hairong, S. & Lee, J., 2002. Measuring the intrusiveness of advertisements: Scale development and validation. Journal of Advertising, 31(2):37–47

Page 17: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

17

STUDY1• Ad: Chewing gum• Push• Controlled for:

• Product involvement• Attitude towards advertising in general

• Measurement: Perceived intrusiveness• Results: Location-congruent ads lead to less ad intrusiveness

than location-incongruent ads (virtual location)

GROUP1: SAW AD WHILE AT SOUP SHELF GROUP2: SAW AD AT COUNTER (NEXT TO CHEWING GUM)

Hühn, A.E., Khan, V.J., Ketelaar, P., Nuijten, K., Gisbergen van, M., Lucero, A. (2011-to appear).The Effect of Location on Perceived Intrusiveness of Mobile Ads. Proceedings Chi-Sparks 2011

Page 18: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

18

STUDY2• Differences with study 1:

• More participants• Open task• Exact same location for both groups• Products on shelf changed for

groups• Soup ad for both groupsHühn, A.E., Khan, V.J., Nuijten, K., Gisbergen van, M., Lucero, A., Ketelaar, P. (2011-to appear).

The Effect of Location on Perceived Intrusiveness of Mobile Ads. Proceedings ICORIA 2011

• Same result with study 1• Location-congruent ads

lead to less ad intrusiveness than location-incongruent ads (virtual location)

Page 19: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

19

STUDY3

• Same setup with Study 2• Ad of Soup• Location-Based• Measurement: perceived intrusiveness

• Differences:• Same products on shelves for both

groups• One group saw a social

recommendation along with the location-based ad

• Results: ad with social recommendation is perceived as more intrusive

GROUP1:Along with the ad they saw a social recommendation

GROUP2:Saw just the ad

Study conducted by: Stephan van de Kruis

Page 20: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

20

CONCLUSIONS

• Location does positively affect perceived intrusiveness of mobile ads

• Social recommendations would not help mobile, location-based ads

• Demonstrated a V.E. where UX research of mobile, context-aware systems can be performed

• Adjustments of the V.E. and context are done easier, are more controllable and faster compared to an experiment in a real life setting

• Reproducibility• If there is an initial halo-effect it does not affect the

actual perceptions

Page 21: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

21

CHALLENGES / FUTURE WORK

• Fill supermarket with a representative product set(~20K products)

• Head tracking with MS Kinect• Have a more complete phone app

• Shop with mobile phone• Include personalization profile

• Ability to interact with products• Use log and video input for analyzing behavior

• Social context: Avatars• Corroborate qualitative findings of previous research

endeavors & explore new research directions• Validation studies

Page 22: 1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS

22

SUMMARY

• 3 studies of mobile, location-based ads evaluation in augmented virtual supermarket

ACKNOWLEDGEMENTSPhotos & Graphics: Arief Ernst Hühn, Paula Isphording

CONTACT

[email protected]• vjkhan.com• @v_j_khan

Research partners:

Financial sponsor: RAAK Pro scheme of:


Recommended