Upload
lorena-harris
View
221
Download
0
Tags:
Embed Size (px)
Citation preview
A Framework
for
Perceptive, Attentive, Cognitive,
and
Social-Interaction Enhancement
From User-Centred
to
Ego-Centred Design
Morris Goldberg
PACSIE
Sreekar KrishnaProf. Panchanathan
Perc
ep
tion C
og
nitio
n
Attention
Social Interaction
•Perception:• Hear a baby cry.
•Attention:•Listen so as to hear one’s baby cry
•Cognition:•Identify the reason for crying
•Social-Interaction:•Rock the baby to stop the baby from crying
Friday, April 21, 2023 PACSIE: Ego-Centred Design 3
Perception, Attention, Cognition, Social-Interaction
•Movement:•Turn head to better hear the baby cry
•Move towards the baby, inspect baby
•Pickup baby, rock the baby
•Autonomic:• Posture: as baby may cry
•Heart-rate: if baby cries
Friday, April 21, 2023 PACSIE: Ego-Centred Design 4
Behaviour Responses:Movement and Autonomic
•Applications• Cerebral: injury, disease, and aging
• Autism, prosopagnosia
• « Extra eyes »
•Training, drugs, invasive• Practice makes perfect,…
•Assistive device• Non-invasive, customized, adaptive, lifelong
Friday, April 21, 2023 PACSIE: Ego-Centred Design 5
Enhancement
Friday, April 21, 2023 PACSIE: Ego-Centred Design 6
Motivation: The Carpenter Who Lost His Tools
Visual agnosia for toolssees tool but lost knowledge of tool
name, function, memory
Haptic tool recognition feels tool, knowledge recovered
Infancy: body(hands) calibrates vision
Maturity:
vision: dominates
body:mostly below conscious awareness
•CO poisoning heater•« Blind » to many objects•Recognized her mother through voice•Recognized colours, texture•Name some surafces
–Red plastic, shiny metal•Spatial acuity intact
–Gratings, but not directions•Basic problem recognizing shapes
•MA Goodale, AD Milner, Oxford Univ. Press
–Sight Unseen: An Exploration of Conscious and Unconscious Vision, 2004Friday, April 21, 2023 PACSIE: Ego-Centred Design 7
Patient DF: Visual form agnosia
Friday, April 21, 2023 PACSIE: Ego-Centred Design 8
Patient DF: Tests
• Object recognition (haptic)
• Copying objects
• Yellow blind to orientation
• Grasped any orientation
Friday, April 21, 2023 PACSIE: Ego-Centred Design 9
Visual processing streams: what, where, how
•Visual agnositics• rare, encumbered, difficult, non-stable
•The blind• visualize objects by touch, stable, intelligent
•Reader• NSF, focus groups
Friday, April 21, 2023 PACSIE: Ego-Centred Design 10
Surrogates: The visually impaired
•Vibro-tactile glove •Sensory substitution: distal visual to proximal haptic
•Haptic rendering not possible
•Observe haptic manipulation
•Uncover distinguishing haptic features/descriptors
•Deliver features by haptic cueing
•Supervised user learning: haptic cues
•Object visualization: user imagination
Friday, April 21, 2023 PACSIE: Ego-Centred Design 11
Haptic object visualisation
Kahol K., Panchanathan S.,“Neurocognitively inspired haptic user interfaces”,
Springer J. of MM Tools and App, v 3, pp 15-38 2007.
Kahol K, Smith M., “Neuro-cognitively Designed Dynamic Simulations For Laparoscopic Surgical Skills”. MODSIM. Virginia Beach, 2007. pp. 34-39.
•Design methodology •Participatory design
focus groups -scenario building
•Prototypic User •Average user
•Learning •Designer discovery
usage patterns, blog, competition..
•User discovery
short-cuts, hidden functionality
Friday, April 21, 2023 PACSIE: Ego-Centred Design 12
User-Centred Design
•Assistive device• white cane.. extension of self…peripersonal space
• left-neglect…stroke…line bisection..finger, laser-pointer, ,stick
•Haptic exploration • haptic glove 6 blind people pattern varies
• depends on degree and onset of blindness, other factors
•Ego-centred design• understanding of the individual
perceptive, attentional, cognitive, social (PACS)
Friday, April 21, 2023 PACSIE: Ego-Centred Design 13
Framework
•1st Principle: Sensory-Motor Coupling
•2nd Principle: Polymodal Reinforcement•Edelman, GM, « Neural Darwinisim: The Theory of Neuronal Group Selection », 1989.
•3rd Principle: Graft Onto Existing Scaffolds•Thelen and Smith, “A Dynamic Systems Approach to the Development of Cognition and Action”, 1996.
•4th Principle: Self Evaluation
Friday, April 21, 2023 PACSIE: Ego-Centred Design 14
Ego-Centred Design: Principles
Active sensing sensory adaptation to constant signal
adapt to pressure on body, smell, even vision
shift body, sniff, eye saccade, ….
« What » is being sensed? behavior is visible
deduce « what »
haptic manipulation- descriptors
Haptic rendering motor-sensory coupling
individual moves, assistive devices provides stimuli
Friday, April 21, 2023 PACSIE: Ego-Centred Design 15
1st Principle: Sensory-Motor Coupling
Modes vision: colour, shape, orientation, motion….
somatosensory: light and heavy pressure, pain, vibration…
auditory, proprioceptive, olfactory:…….
Existence-pathological cases Agnosia: loss of mode/function
Synaesthesia: extra function-e.g see sound, hear color
Friday, April 21, 2023 PACSIE: Ego-Centred Design 16
2nd Principle: Polymodal Reinforcement
Examples texture: somatosensory, visual, and auditory
taste: smell and somatosensory(tongue)
Associative memory Bartlett: memory is a re-creative process
Move hands, visualize, hear , feel
Friday, April 21, 2023 PACSIE: Ego-Centred Design 17
2nd Principle: Polymodal Reinforcement
Exploitation Wagner: musical identities
rehabilitation: colour code objects, audio prompts
Haptic glove audio and haptic cues for textures
Friday, April 21, 2023 PACSIE: Ego-Centred Design 18
2nd Principle: Polymodal Reinforcement
Child development reaching: months of effort, muscles, vision proprioceptive
orienting to people: instintive attraction to face-like patterns
Continuumscaffolding: basis for new functionalities
development nevers stops
Blind and body languageauditory cues: nervous movement, breathing, voice inflection
assistive device: correlate/integrate
Friday, April 21, 2023 PACSIE: Ego-Centred Design 19
3rd Principle: Graft Onto Existing Scaffolds
Gait variesload, terrain, physical condition
trial and error: self evaluation (pain, comfort)
Continuous self-evaluation
Innate Behavioral Responses attraction to eye-like objects
move eyes- get reward (smile,hug)-
emotional-based value system
Friday, April 21, 2023 PACSIE: Ego-Centred Design 20
4th Principle: Self Evaluation
Everyday Social InteractionsWhat do we know about them?
Friday, April 21, 2023 PACSIE: Ego-Centred Design 21
Social Interactions: Complex Visuo-Motor Interplay
Social Interactions
SensorySensory
PerceptualPerceptual
CognitiveCognitive
MotorMotor
Handshake
Eye Contact
Move towards each other(Keep in mind Proxemics)
Intent to interact
Shake hands
Conversation distance
Friday, April 21, 2023 PACSIE: Ego-Centred Design 22
Friday, April 21, 2023 PACSIE: Ego-Centred Design 23
Psychological Basis of Social Interactions• Emotions
• Interpersonal Attributes• Dominance• Trust• Like/Dislike
• Personality
• Communication• Turn Taking• Feedback• Attention• Iconic Cues, like Victory or Stop hand gestures.• Intrinsic Cues, like moving close to someone
Beh
avio
ral A
spec
tsB
ehav
iora
l Asp
ects
Com
mun
icat
ion
Asp
ects
Com
mun
icat
ion
Asp
ects
M. Argyle, The Psychology of Interpersonal Behaviour, Penguin UK, 1999.
Friday, April 21, 2023 PACSIE: Ego-Centred Design 24
Psychological Aspects – Dyadic Interactions
FeelFeel SendSend ReceiveReceive FeelFeel
M.G. Almore, “Dyadic Communication,” The American Journal of Nursing, vol. 79, Jun. 1979, pp. 1076-1078.
A. Teeters, R.E. Kaliouby, and R. Picard, “Self-Cam: feedback from what would be your social partner,” ACM SIGGRAPH 2006 Research posters, Boston, Massachusetts: ACM, 2006, p. 138.
ExpressionsBody MannerismsCommunicative Gestures
Friday, April 21, 2023 PACSIE: Ego-Centred Design 25
Insula and the Mirror Neurons
Mirror NeuronsMirror
Neurons InsulaInsula Limbic SystemLimbic System
Simulate Facial
Expression
Feel the Emotion
L. Carr, M. Iacoboni, M. Dubeau, J.C. Mazziotta, and G.L. Lenzi, “Neural mechanisms of empathy in humans: A relay from neural systems for imitation to limbic areas,” Proceedings of the National Academy of Sciences of the United States of America, vol. 100, Apr. 2003, pp. 5497-5502.
Derived out of fMRI studies on human viewing facial images Enhanced activity in the Mirror Neuron Area, Insula and Limbic System observed Findings are just anatomical patterns Exact Pathway is still not discovered
Everyday Social Interactions – The PACSIE Model
Bringing it all together
Friday, April 21, 2023 PACSIE: Ego-Centred Design 26
Friday, April 21, 2023 27PACSIE: Ego-Centred Design
Principle 1: Sensory-Motor Coupling
• Social Interactions are integration of complex sensory and motor actuations and responses.
Enactor (Encoding) Recipient (Decoding)
Visual EngagementVisual Engagement
Coordinated Motor Interaction – Eye Gaze,
Head Nod
Coordinated Motor Interaction – Eye Gaze,
Head Nod
Communication – Asynchronous Exchange
Communication – Asynchronous Exchange
Complex Social Engagement
Complex Social Engagement
PerceptionPerception
AttentionAttention
CognitionCognition
Social InteractionSocial Interaction
Ver
bal
Non
-ver
bal
Enactor (Encoding) Recipient (Decoding)
Body
Voice
Speech
65%
35%
27%19%
18%35%
Face Visual
Audio
Friday, April 21, 2023 28PACSIE: Ego-Centred Design
Principle 2: Polymodal Reinforcement
• Social Interactions are integration of multichannel information (from the enactor and the recipient’s perspective).
N. Ambady and R. Rosenthal, “Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.,” Psychological Bulletin. Vol. 111(2), vol. 111, Mar. 1992, pp. 256-274.
Friday, April 21, 2023 29PACSIE: Ego-Centred Design
Principle 3: Use existing skills (i.e. Scaffolding)
• Social Interactions capabilities are very individualistic based on when the person becomes blind.
Congenitally Blind
Congenitally Blind
Adventitiously Blind
Adventitiously Blind
BirthBirth Developmental YearsDevelopmental Years Professional YearsProfessional Years Geriatric YearsGeriatric Years
Social Skills
Social assistance need varies depending on when the person loses his/her sight.
Friday, April 21, 2023 30PACSIE: Ego-Centred Design
Principle 4: Recalibration by Self Evaluation
• Social Mirroring is the process by which effective social interactions are learned and propagated.
Hello !!!Welcome to
CUbiC
Hello !!!Welcome to
CUbiC
ActionAction
Social MirrorSocial Mirror
PerceptionPerception
Mirror NeuronsMirror
Neurons
Self Evaluate
Self Evaluate
M. Iacoboni, Mirroring People: The Science of Empathy and How We Connect with Others, Picador, 2009.
Social Solutions Through PACSIEBody Rocking Condition
Friday, April 21, 2023 PACSIE: Ego-Centred Design 31
Friday, April 21, 2023 PACSIE: Ego-Centred Design 32
Role of vision
Learning through
observation
Learning through
observation
Feedback on social
mannerisms
Feedback on social
mannerisms
Visual Impairment affects both aspects of social interactions
Friday, April 21, 2023 PACSIE: Ego-Centred Design 33
Stereotypic Mannerisms
Hand - 60.3%
Head- 13.1%
Trunk - 10%
Body - 6.9%
Leg - 6.4%
Face - 3.4%
Though body rocking is not the
most occurring mannerism, it is the
most noticed in a social scene
V.J. Eichel, “Mannerisms of the blind: A review of the literature,” Journal of Visual Impairment & Blindness, vol. 72, 1978, pp. 125-130.
Body Rocking in Adults
Rocking is a natural development behavior of the cerebellum’s balance system
Body rocking in adults is due to the lack of social feedback
Friday, April 21, 2023 PACSIE: Ego-Centred Design 34
Friday, April 21, 2023 PACSIE: Ego-Centred Design 35
Solutions
InterventionIntervention Self MonitoringSelf Monitoring
Immediate feedbackDoes not address the problemReduces the symptomChance of recurrence is high
Replace with Cognitive TaskProvides for symptom
substitutionChance of recurrence is low
Focus on Controlling the problem and not curing it !!!
M. Celeste, “Social Skills Intervention for a Child Who Is Blind,” Journal of Visual Impairment & Blindness, vol. 101, 2007, pp. 521-533.
R.L. Loftin, S.L. Odom, and J.F. Lantz, “Social Interaction and Repetitive Motor Behaviors.,” Journal of Autism & Developmental Disorders, vol. 38, Jul. 2008, pp. 1124-1135.
Friday, April 21, 2023 PACSIE: Ego-Centred Design 36
Social Solution
PACSIEPACSIE
Body Rocking
Body Rocking
Social Solution
Social Solution
Social Feedback
Social Feedback
Self Evaluate
Self EvaluateControlControl
Friday, April 21, 2023 PACSIE: Ego-Centred Design 37
Social Interaction Assistant: Work-inProgress
Friday, April 21, 2023 PACSIE: Ego-Centred Design 38
Person, Gaze, Expression, Identity Detector
•1st Principle: Sensory-Motor Coupling•Head movement controls sensing(far-near,left-right)
•2nd Principle: Polymodal Reinforcement•vibration and sound
•3rd Principle: Graft Onto Existing Scaffolds•mood recognition, voice recognition, footstep recognition..
•4th Principle: Self Evaluation•user evaluates/calibrates the signals from the SIA
•eventually SIA will also self-evaluate
Friday, April 21, 2023 PACSIE: Ego-Centred Design 39
Ego-Centred Design: Principles