Slide 1

Slide 2

Multimodality Alistair D N Edwards Department of Computer Science http://www-users.cs.york.ac.uk/~alistair/teaching/AMSmasterclass/Multimodality.pptx

Slide 3

Multi-X MULTI-media, MULTI-modal, MULTI- channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.

Slide 4

Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant?

Slide 5

How many senses do we have?

Slide 6

Five

Slide 7

Extra senses http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain 2Balance 3Time 4Temperature 5Digestion

Slide 8

Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have?

Slide 9

Class of 2008-10 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst

Slide 10

What is a sense?

Slide 11

What is a human sense? (Lets not get distracted by questions as to whether animals or even plants have senses in the way we mean)

Slide 12

What is a human sense? An input channel? Only for external information? or internal too?

Slide 13

What is a human sense? An input channel? Only for external information? or internal too?

Slide 14

Objectives What modes do we use in interaction? Which others could we use? Why? How?

Slide 15

Visual dominance Psychology Neurology User interfaces The McGurk effect An example of a cross-modal effect

Slide 16

Visual veracity

Slide 17

Slide 18

The rotating mask illusion https://www.youtube.com/watch?v=sKa0eaKsdA0 We cannot help but see the illusion The brain knows that it is impossible The eye is not just a camera

Slide 19

Cross-modality Were looking at separate modalities but the total is not necessarily the sum of the parts The role of auditory cues in modulating the perceived crispness and staleness of potato chips Journal of Sensory Studies, 19: 347363. doi: 10.1111/j.1745-459x.2004.080403.x http://onlinelibrary.wiley.com/doi/10.1111/j.1745-459x.2004.080403.x/full

Slide 20

the perception of both the crispness and staleness was systematically altered by varying the loudness and/or frequency composition of the auditory feedback elicited during the biting action. The potato chips were perceived as being both crisper and fresher when either the overall sound level was increased, or when just the high frequency sounds (in the range of 2 kHz20 kHz) were selectively amplified

Slide 21

Some practical(?) possibilities Sounds Haptics Smell

Slide 22

Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs

Slide 23

Earcons Symbolic sounds with arbitrary mappings to their meanings Paint Open Open paint

Slide 24

Auditory icon A cartoon sound that resembles the thing it represents Sonic finder

Slide 25

Spearcons A cross between speech and non-speech sounds Speech speeded up fast but no need to learn

Slide 26

Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/research/auditorymenus/

Slide 27

Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res earch/auditorymenus/ elephant

Slide 28

Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res earch/auditorymenus/ elephant elevator

Slide 29

Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/researc h/auditorymenus/ elephant elevator parking meter

Slide 30

Sonification

Slide 31

Sounds Why do we not make more use of sounds in interfaces?

Slide 32

http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions

Slide 33

Geomagic Touch http://www.sensable.com/ Formerly the Sensable Phantom Omni

Slide 34

Trisenx Scent Dome http://www.trisenx.com/

Slide 35

iSmell

Slide 36

Slide 37

smells Why do we not make more use of smells in interfaces?

Slide 38

Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits?