Multimodality Alistair D N Edwards Department of Computer Science...
Preview:
Citation preview
- Slide 1
- Slide 2
- Multimodality Alistair D N Edwards Department of Computer
Science
http://www-users.cs.york.ac.uk/~alistair/teaching/AMSmasterclass/Multimodality.pptx
- Slide 3
- Multi-X MULTI-media, MULTI-modal, MULTI- channel... there are
many terms that can be prefixed by the qualifier 'multi', so that
there is a danger of getting diverted into debates about the
meanings of the terms and distracted from the real questions. At
the same time, these discussions often centre on computers; a
linguist or a playwright or a singer would not even use the terms,
far less wonder whether what they do should be prefixed by
'multi'.
- Slide 4
- Multi-X -Why is it that the debate arises in the context of
computer interaction? -Why is it that it does not arise in other
contexts of communication? -Are there are other contexts in which
the 'multi-X' question arises? -Are we progressing towards a
situation in which the question will become obsolete or
redundant?
- Slide 5
- How many senses do we have?
- Slide 6
- Five
- Slide 7
- Extra senses
http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain
2Balance 3Time 4Temperature 5Digestion
- Slide 8
- Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian
Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How
many senses do we have?
- Slide 9
- Class of 2008-10 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6
Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12
Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain
17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23
Tiredness 24 Thirst
- Slide 10
- What is a sense?
- Slide 11
- What is a human sense? (Lets not get distracted by questions as
to whether animals or even plants have senses in the way we
mean)
- Slide 12
- What is a human sense? An input channel? Only for external
information? or internal too?
- Slide 13
- What is a human sense? An input channel? Only for external
information? or internal too?
- Slide 14
- Objectives What modes do we use in interaction? Which others
could we use? Why? How?
- Slide 15
- Visual dominance Psychology Neurology User interfaces The
McGurk effect An example of a cross-modal effect
- Slide 16
- Visual veracity
- Slide 17
- Slide 18
- The rotating mask illusion
https://www.youtube.com/watch?v=sKa0eaKsdA0 We cannot help but see
the illusion The brain knows that it is impossible The eye is not
just a camera
- Slide 19
- Cross-modality Were looking at separate modalities but the
total is not necessarily the sum of the parts The role of auditory
cues in modulating the perceived crispness and staleness of potato
chips Journal of Sensory Studies, 19: 347363. doi:
10.1111/j.1745-459x.2004.080403.x
http://onlinelibrary.wiley.com/doi/10.1111/j.1745-459x.2004.080403.x/full
- Slide 20
- the perception of both the crispness and staleness was
systematically altered by varying the loudness and/or frequency
composition of the auditory feedback elicited during the biting
action. The potato chips were perceived as being both crisper and
fresher when either the overall sound level was increased, or when
just the high frequency sounds (in the range of 2 kHz20 kHz) were
selectively amplified
- Slide 21
- Some practical(?) possibilities Sounds Haptics Smell
- Slide 22
- Sounds Speech synthesized Non-speech earcons auditory icons
sonification data mapping audification programs
- Slide 23
- Earcons Symbolic sounds with arbitrary mappings to their
meanings Paint Open Open paint
- Slide 24
- Auditory icon A cartoon sound that resembles the thing it
represents Sonic finder
- Slide 25
- Spearcons A cross between speech and non-speech sounds Speech
speeded up fast but no need to learn
- Slide 26
- Spearcons Examples 1 2 3
http://sonify.psych.gatech.edu/research/auditorymenus/
- Slide 27
- Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res
earch/auditorymenus/ elephant
- Slide 28
- Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res
earch/auditorymenus/ elephant elevator
- Slide 29
- Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/researc
h/auditorymenus/ elephant elevator parking meter
- Slide 30
- Sonification
- Slide 31
- Sounds Why do we not make more use of sounds in
interfaces?
- Slide 32
- http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf
TermDefinition Proprioceptive Relating to sensory information about
the state of the body (including cutaneous, kinesthetic, and
vestibular sensations). Haptic Relating to the sense of touch.
Vestibular Pertaining to the perception of head position,
acceleration, and deceleration Kinesthetic Meaning the feeling of
motion. Relating to sensations originating in muscles, tendons and
joints Cutaneous Pertaining to the skin itself or the skin as a
sense organ. Includes sensation of pressure, temperature, and pain.
Tactile Pertaining to the cutaneous sense but more specifically the
sensation of pressure rather than temperature or pain. Force
Feedback Relating to the mechanical production of information
sensed by the human kinesthetic system. Haptics - definitions
- Slide 33
- Geomagic Touch http://www.sensable.com/ Formerly the Sensable
Phantom Omni
- Slide 34
- Trisenx Scent Dome http://www.trisenx.com/
- Slide 35
- iSmell
- Slide 36
- Slide 37
- smells Why do we not make more use of smells in
interfaces?
- Slide 38
- Conclusions Why do we not make more use of other
senses/channels/modes in interaction? Which ones could we
practically use more? What would be the benefits?