World Usability Day - Measuring Engagement

User Experience of Interactive Educational Systems

An iPad StudyHuman Factors in Information Design

Design and Usability CenterBentley University

Amanda Davis Vignesh KrubaiDiego Mendes

© Copyright

Agenda

• What We are Researching

• Technology and Methodology

• Qualitative Findings

• Quantitative Analysis

• What Worked Well

• Challenges / Recommendations

© Copyright Diego Mendes Amanda Davis Vignesh Krubai

Education and Tablet Technology


• Pilot programs on using tablets in schools and higher education are underway

– NYC, Chicago, Virginia, California, Boston and Internationally– Intended to be used at home and school in place of textbooks

• Benefits include– Light-weight– Variety of applications– Interactive Learning

• Not many studies done to verify if students learn more, faster or better with tablets

What is happening now?

What We Are Researching– Engagement is a key part of learning effectively

• Is the increased use of technology affecting engagement?

– Do new or traditional technology engage students differently?

• Digital Textbook v. Paper Textbook

– Measuring engagement is complex because it involves biological and cognitive processes

– Requires an array of methods to be applied


Technology and Research Methods

Pearson Education E-bookPearson Education- One of the most innovative leaders in educationInkling Interactive Digital Textbook Technology- Same book in digital version and paper version- Allows for the same interaction (highlighting, post it notes etc…)


Digital Book Paper Book

Research Methods

What methods did we choose to apply?

Combination of different methods:

- Affectiva’s Q Sensor

- SMI Eye tracking

- Product Reaction Cards


Q Sensor: Measuring EDA

Q SensorCreated by Affectiva, a company founded by MIT Media Lab faculty and students

What is the Q-Sensor?A wireless biosensor that measures emotional arousal via skin conductance (Electro Dermal Activity).

High arousal = High User Engagement

Low arousal = Low User Engagement


Q Sensor: Research TechniqueChallenges in using the Q SensorThe data in the Q Sensor tells you there is some sort of engagement, but not which type of engagement. High engagement could mean:

Frustration Excitement Stress Delight

The Analysis:

1. Calculate average EDA Value

2. Calculate Peaks per minute


SMI Eye Tracking: Background

• Created by SMI (SensoMotoric Instruments)

• Advantageous for use on mobile or tablet devices

• Binocular tracking delivers eye tracking data over all distances


SMI Eye Tracking: Data Analysis

• Combined with Q Sensor data, SMI gives us the ability to try to pin point engagement

• Ability to analyze areas of interest across participants


Product Reaction Cards: Background

– Developed by Joey Benedek and Trish

Miner (Microsoft)

– A set of 118 words

• 60% positive

• 40% negative/neutral

– Easy to administer and easy to analyze

– Does not rely on a questionnaire or

rating scales

– Users don’t need to generate words

themselves


Product Reaction Cards: Technique Used

Process:

– Users were asked to circle 10 words from the list of 118 words

– This process was repeated after each object of study:

• Paper Textbook

• Digital Textbook

– We used the response cards to understand and categorize Affectiva’s Q Sensor

EDA/skin conductance data

Challenges:

– Real time reactions not available

– Participant misinterpretation of word meaning


What we found:Qualitative Results

Qualitative Findings

Participants• 10 undergraduate students who owned iPads and used the

internet for at least 10 hours per week

Common iPad Activities• Most common: accessing the web, checking email, gaming and

social networking• Additional: eBooks, music, course maintenance, movies and

looking at powerpoints during class


Which do you prefer and why?


50% of students preferred each

Digital Paper

Versus

What Do You Like About The Paper vs. The Digital Textbook?

Preferences on Paper Textbook

• Concerns about getting tired looking at the iPad screen & getting tired of swiping to flip pages

• Pictures were clear & didn't require enlarging to see

• Preferred for financial reasons: reselling & cheaper to purchase

• Allowed for more freedom and were more comfortable

• More reliable, as not dependent on batteries


What Do You Like About The Digital vs. the Paper Textbook?

Prefer Digital Textbook

• Easier to search for content

• Easier navigation

• Ability to access internet to do supplemental research


Quantitative Analysis

Quantitative FindingsWe plan to use the following information to help us in understanding user engagement:

- Electro-Dermal Activity Information- Magnitude = a measure of peak height- Area Under the Curve = represents peak volume- Peaks per Minute = is a measure of peak frequency

- Heatmaps of user engagement throughout session


Quantitative FindingsWith the addition of SMI eye tracking, we can:

- Pin point engagement areas and moments effectively

- Get the number of fixations on screen and paper textbooks across participants

- Create heat maps to determine points of engagement


What Worked Well

What Worked Well• Ability to observe:

– top-down (Cognitive Perception – post moment) - “I felt…”– bottom-up (Q Sensor Spikes – at the moment) - emotional reactions

• Affectiva- Easy to set up - Extremely detailed data about skin temperature changes, skin

conductance

• Use of SMI Glasses - Easy to set up- Eye tracking – precise view of how things worked- Data analysis capabilities are an advantage - Intuitive


Challenges

Challenges

- Coordinating the technology during the tests

- Synching issues- Video loses small frames (due to editing)

- Combining the data- 4 cameras (videos)- Affectiva data- SMI Data


Recommendations

Other Methodology Applications

• Areas of Application: - User Experience, Usability, Product Design, Market Research, Education Research,

Psychology Research, Visual Perception Research

• Product Design- Use it to find what emotionally engages people

• Usability- Use it to find out what frustrates people

• Prioritization of design features- Find what is more frustrating/exciting for the user

Caution: Labor intensive process to combine three methods (but doable)


Questions?

Contact:Diego Mendes – [email protected]

Amanda Davis –[email protected] Vignesh Krubai – [email protected] © Copyright Diego Mendes Amanda Davis Vignesh Krubai

Thank you to our sponsors who are here and excited to meet you!

The Design and Usability Center