Music and shape: Exploring cross-modal representation Examples, a model and analyses Professor Adam Ockelford University of Roehampton 11th March 2010

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.



Music and shape:Exploring cross-modal

representation

Examples, a model and analyses

Professor Adam OckelfordUniversity of Roehampton

11th March 2010



Music and shape

How can the two relate?

One set of dynamic relationships can be created through movement:

• more or less freely associated with sound

(eg, dance, gestures in performance)

• creating, causing or controlling sound(eg, playing instruments, conducting)



Music and shape

Another set of relationships can arise through the visual representation of sound – the focus of this presentation

• Seven examples• Attempt to construct common theoretical

framework• Indicative analyses• Discussion

Children’s ‘picture scores’

7 year old’s representation of a rhythm(after Bamberger, 1982)

Children’s ‘picture scores’

7 year old’s representation of a rhythm(after Bamberger, 1982)

Blind children’s drawings on ‘German Film’

After Welch (1981)

Blind children’s drawings on ‘German Film’

Conventional Westernmusic notation

Any features in common with children’s informal notations?

Braille music

Key differences with print?

Guitar chord tablature

What processes are at work here?

Similarities and differences with previous examples?



Graphic notation

Excerpt from Stockhausen, Electronic Study No. 2 (1954)

Jamie Roberts’s score of Jean Michel Jarre’s Oxygene (Track 4)

Score produced through synaesthetic response to

music



How does it all work?

A common underlying principle or principles?

Or fundamentally different processes at work?



Perceptual domains

• ‘Domains’ = way of modelling the fact that one sensation can have different ‘states’

• Touch: eg, temperature and pressure.• Vision: eg, hue, saturation and

brightness.







Perceptual domains

In music:

- Pitch: (chroma, octave), harmony, tonality

- Perceived time: onset, duration

- Timbre

- Loudness

- Perceived location of sound source



Different categories of domain

Can be conceived as being

single dimensional, bidimensional or multidimensional

and as being

linear or cyclical or both, or complex

Examples …

PitchQuickTime™ and a

TIFF (Uncompressed) decompressorare needed to see this picture.

Duration

‘Values’• Each state in which a domain is perceived to exist can be

conceived of as a ‘value’.• May be simple or complex (like the domains to which they

pertain)• Example:



Relationships within domains

• Musical structure is not to do with individual values, but the relationships between them within each domain - particularly pitch and perceived time

• We can assume that the relationships between values normally pass listeners by as series of qualitative experiences

• However, they may be conceptualised (culture specifically?) as differences or ratios, or they may reflect the complexity of the values to which they pertain

Examples of intra-domain relationships

‘Primary’ relationships

Examples of intra-domain relationships

‘Secondary’ relationships



Relationships between domains

• Two main types

– ‘Regular’ (isomorphic in the domain of ‘shape’)

– ‘Irregular’ (through association)



Regular inter-domain relationships

Guiding principle:to map values systematically

between domains,a relationship must involve

a function that isnot specific

to the perceptual domains concerned.

Regular inter-domain relationshipsFor example, differences are domain-specific, so

single values of difference cannot be mapped systematically between domains.

Regular inter-domain relationshipsWhereas, ratios are abstract, are not bound by the

context in which they occur. Hence they permit regular mapping between domains.

Regular inter-domain relationshipsRatios may occur between differences (as secondary

relationships). Hence the regular inter-domain relationships would be tertiary.

‘Irregular’ inter-domain relationships

Guiding principle:formed through association …

either through repetitionor indirect connection

Irregular inter-domain relationships

Different types:

(a) arbitrary (repetition)

(b) indirect (indirect connection)

(c) synaesthetic (indirect connection)

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Arbitrary inter-domain relationships

> •

Any relationship between a sound and its visual representation (shape) is possible.

For example:

accent staccato fermata

Example (‘staccato’) – relationship created through repetition (or verbal explanation = proxy repetition):

Arbitrary inter-domain relationships

Example (‘note cluster’) – relationship created through action / object as common visual and auditory source

Indirect inter-domain relationships

Can be reinforced through repetition

Example – green corresponds to Eb major (and vice versa)

Synaesthetic inter-domain relationships

Again, can be

reinforced

(for others)

through repetition

From theory to analysis

Children’s ‘picture score’

Simplest interpretation: arbitrary relationships – given shape (circle) maps onto note

Children’s ‘picture score’

More advanced interpretation: regular mapping of distance and duration; sequence and temporal order

Score on German Film of pitch glide

Coordinated regular mapping of vertical and horizontal distances with pitch and perceived time (implied tertiary relationships)

Print score

Arbitrary and semi-regular mappings combine(the latter reinforcing the former)

Braille score

Only arbitrary mappings survive transcription into Braille(which makes using this medium more challenging)

Guitar tablature

Combines both indirect and regular representations of pitch

Stockhausen score:

comprises regular

representations of pitch,

perceived time and loudness

Jamie’s score: comprises

regular and indirect

representations of pitch,

perceived time and timbre

Conclusions

• Evidence from children’s untutored representations of music suggests that sophisticated inter-domain mapping between sound and shape occurs early and intuitively

• This view is reinforced by blind children’s representations of changing pitch – which are also sophisticated, and exist in the absence of any visual model to guide them

Conclusions

• The precise nature of inter-domain mapping may vary from individual to individual, and according to cultural convention.

• However, it appears to be relatively easy to grasp regular mappings (consider, for example, widespread use of horizontal dimension for time and vertical dimension for pitch) or arbitrary symbols (as in standard notations)

Conclusions

• Some inter-domain mappings may be hard-wired through synaesthesia.

• These seem to vary from individual to individual.

• However, such representations can be learnt and appreciated by others

Conclusions

• What next?• More exhaustive search of different forms of

symbolic visual representation of sound to check broad applicability of the model

• Use the model to generate new forms of sound / shape installations

• More research into how mappings aries and are learnt, and into sonic/visual synaesthesia



Adam Ockelford

[email protected]

07818-456 472

mailto:[email protected]

Documents

Music and shape: Exploring cross-modal representation Examples, a model and analyses Professor Adam Ockelford University of Roehampton 11th March 2010