The Effectiveness of Haptic Cues as an Assistive Technology for Human Memory

Living Environments Lab

Human-ComputerInteractionInstitute

Carnegie Mellon

Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos{stace, anind, hudson, paulos} @ cs.cmu.edu

The Effectiveness of Haptic Cues as an Assistive Technology for Human

Memory

Human Memory


Carnegie Mellon



Carnegie Mellon



Carnegie Mellon



Carnegie Mellon



Carnegie Mellon




Carnegie Mellon

Memory Impairments




Carnegie Mellon

Visual and Audio Cues


DeVaul, R.W., Pentland, A. and Corey, V.R. The memory glasses: subliminal vs. overt memory support with imperfect information. (2005)

Lin, W. and Hauptmann A.G.A Wearable Digital Library of Personal Conversations. (2002)

Disruptive

Prominent



Carnegie Mellon


Vibro-Tactile Cues

Unobtrusive

Discreet



Carnegie Mellon

Vibro-Tactile Interaction




Carnegie Mellon

Hypothesis


Vibro-tactile cues can improve human memory recall and recognition



Carnegie Mellon

Priming


Effective cross-modal priming for visual and haptic stimuli

Ballesteros S., Reales J.M., Mayas J. and Heller M.A. Selective Attentionmodulates visual and haptic repetition priming: effects in aging and Alzheimer'sdisease. (2008)



Carnegie Mellon

Attention and Navigation


Haptic feedback aids attention and navigation

Young, J.J., Tan, H.Z., Gray, R. Validity of haptic cues and its effect on primingvisual spatial attention. (2003)Kildal, J. and Brewster, S.A. VibroTactile External Memory Aids in Non-Visual Browsing of Tabular Data. (2006)



Carnegie Mellon

Multimodal Priming


Retrieval of audio and video information is assisted with vibro-tactile cues



Carnegie Mellon




Carnegie Mellon




Carnegie Mellon




Carnegie Mellon




Carnegie Mellon




Carnegie Mellon

Wearable Haptic Feedback Device




Carnegie Mellon

Cue 1: two medium pulses Cue 2: strong oscillating pulses

Cue 3: short soft pulse


Example Cues

Training Phase



Carnegie Mellon

Procedure

Training Phase

Pause

Test Phase


Pause



Carnegie Mellon

Method




Carnegie Mellon

Four User Studies

“Blending”

Audio Recognition

Visual Recognition

Free Recall




Carnegie Mellon

Blending

24 participants, 15 male, 9 female, ages 21-65

Shape Recognition Test




Carnegie Mellon




Carnegie Mellon

Auditory Recognition

20 participants: 12 male, 8 female, ages 21-65 Mandarin word translation test




Carnegie Mellon




Carnegie Mellon

Visual Recognition

22 participants: 16 male, 6 female, ages 21-65 Face memory test




Carnegie Mellon

Visual Recognition Test




Carnegie Mellon

Free Recall

24 participants: 16 male, 8 female, ages 21-65

Ebbinghaus nonsense syllable test (“hik”, “lup”)Ebbinghaus, H. Memory: A Contribution to Experimental Psychology. Teacher’sCollege (1913). Dover Publications.




Carnegie Mellon




Carnegie Mellon

Results

Audio recognition

Shape recognition

Face recognition

Free recall

+11% (5 trials, p = 0.075)

no effect

no effect

-20% (7 trials, p = 0.009)




Carnegie Mellon




Carnegie Mellon




Carnegie Mellon




Carnegie Mellon

Haptic Cues and Free Recall

Significantly lower performance with cues

Declarative vs. Perceptual memory

Our cues did not have semantic meaning




Carnegie Mellon

Haptic Cues and Cognitive Load

Cues were ‘distracting’

Perceptual differences between users




Carnegie Mellon

Haptic Cues as a Memory Aid

Memory aid for people with poor or impaired memory

Memory enhancing treatments often target low performers

Aldenkamp, A. P., et al., Effect of Sabeluzole (R 58 735) on Memory Functions in Patients with Epilepsy. (1995)

Mattay, V. S. et. al.. Effects of Dextroamphetamine on Cognitive Performanceand Cortical Activation. (2000)




Carnegie Mellon

Opportunities

Cue vocabulary


Maximum number of cues

Effectiveness of cues for people with functional memory impairments



Carnegie Mellon

Pulse Duration


Pulse 1 Pulse 2



Carnegie Mellon

Pulse Intensity


Pulse 1 Pulse 2



Carnegie Mellon

Delay Between Pulses


Pulse 1 Pulse 2



Carnegie Mellon

Cue Vocabulary Study

14 Participants, age 18-45Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos

{stace, anind, hudson, paulos} @ cs.cmu.edu



Carnegie Mellon

Ratings of Pulse Duration Differences




Carnegie Mellon

Ratings of Pulse Intensity Differences




Carnegie Mellon

Ratings of Differences in Delay Between Pulses




Carnegie Mellon

Ongoing Work

Systematically pick cues from the sampled space


Determine the maximum number of cues that can be remembered

Recruit subjects with functional memory impairments



Carnegie Mellon

Future Work

Adjust cue intensity to individual’s perception

Strive for context-awareness

Add semantic meaning to cues




Carnegie Mellon

The Effectiveness of Haptic Memory Cues

Haptic cues as a memory aid for poor performers

Assistive technology for memory-impaired users




Carnegie Mellon


Vibro-Tactile Devices



Carnegie Mellon




Carnegie Mellon

Roberta KlatzkyNSF Intel

Thank you


http://isiria.files.wordpress.com/2008/04/foreign_language.jpghttp://internetservices.readingeagle.com/blog/gavel/archives/cellphone.jpg

http://www.blendfilms.com/movies/handshake_small.movhttp://spiddlement.files.wordpress.com/2008/05/todolist.jpg

http://static.howstuffworks.com/gif/brain-1.jpghttp://www.sytechcorp.com/new_site/SytechCorp/SyTechCorpXY/images/Pager_AA021263.jpg

Image Credits



Carnegie Mellon

Questions


Kuznetsov, S., Dey, A. K., Hudson, S. E. The Effectiveness of Haptic Cues as a Ubiquitous Memory Aid. Pervasive Computing, 2009



Carnegie Mellon


Sixth Sense, MIT Media Lab, 2009



Carnegie Mellon


Point localization thresholds for males for different areas of the body(Weinstein, 1968)



Carnegie Mellon

Questions


Kuznetsov, S., Dey, A. K., Hudson, S. E. The Effectiveness of Haptic Cues as a Ubiquitous Memory Aid. Pervasive Computing, 2009

Documents

The Effectiveness of Haptic Cues as an Assistive Technology for Human Memory