Music, Timbre and Time

XXII Congress of the National Association for Research and Graduate Studies in Music - João Pessoa - 2012

Music, Timbre and Time

Júlio dos Reis UNICAMP - [email protected]

José Fornari UNICAMP – [email protected]

Abstract: The influence of time in music is undeniable. As for our cognition, time influences how we sense acoustical and musical events. Here we divide the process of listening in three steps: Perception, Cognition and Affect. Each of them occur through different brain processes and are sensed by human audition in distinct time scales. As subjective processes, they present large perceptual differences from individual to individual. Nevertheless, it is still possible to study an average sense of time and duration for the perception, cognition and affect related to musical processes. This article presents an overview followed by a brief discussion of an ongoing research concerning the philosophical and pragmatic bases for the influence of time in the processes of perception, cognition and affect of music. In this early stage of research, we present a theoretical discussion about temporal processes in music. The aim here is to create the epistemological basis for a further empirical study using computational models to investigate the different time bases for the human perception of time and duration in music. Keywords: time, duration, perception, cognition, affect, timbre

1. Introduction

Sound is one of the most important channels of human communication. As such, it

has always been the subject of great fascination and phenomenological research on the

processes comprising it, from its formation to how it is perceived by the human mind. In

particular, musical sound events in nature are often obliterated by this mist of mystery.

Several researchers have been investigating physical, physiological and psychological aspects

of sounds that compound the art of music. However, despite the advances on this field of

investigations, there are still meaningful gaps of scientific understanding, yet to be solved.

1.1. Relation between time and music: In this project, we consider that time and

music are conceptually interveners and interwoven. However, one must realize that the

relationship between music and time is not purely physical. Acoustical studies always relate

to the sound propagation time in windows of seconds (s) or milliseconds (ms), but for

musicians such measures are unclear and sometimes are not well classified by the traditional

musical notation. For example, in commercial music handling time macrostructural is not

explored. Generally the tempo of a popular song remains the same throughout its execution.

In this study, however, the settings and extended concepts of musical time, and relate to better

analyse time and music. The physics of acoustics defines time in terms of its duration, as a

continuous function of time, with variant and measurable magnitude, although it is not

possible to address time, by itself, but rather, the modification of other variables and

processes. Music measures of time are often not described in the metric system, in seconds,

but in subdivisions of a pulse (beats per seconds, or bpm), previously established by the

1283


composer, conductor or performer. As the time for historians, music time is seen as the

element that determines the modification of a hierarchically organized network of sound

events (eg motives, phrases, passages, etc.). We can conclude that, for music, it is reasonable

to think of time as an organizing element of sound events, causing them to be both internally

consistent, within a small time scale, and externally segregated, for longer time interval

[TENNEY 1980]. In this study it was necessary to utilize these two time settings, because

some processes are strongly linked to music listening, or even inevitably linked to the musical

context and the musical work. It is also given here a brief citation to the concepts of duration

and time established by Henri Bergson, in his paper "Time and Free-will"[BERGSON, 1910],

where he defines Time as a sequence of simultaneity, divisible and therefore measurable, and

Duration as the single pass and pure time, being indivisible and immeasurable. We believe

that his concepts of time and duration are of great importance for further studies of musical

affection.

The first problem that we confronted is the definition of the boundaries between

disciplines that deal with information and musical sound. For instance, how to establish a

frontier between the physics of acoustic wave and the physiology of sound sensation? How to

separate sound perception – studied by psychoacoustics – from the human psychology of

music cognition? How to separate the evoked emotions from the appraised emotion, while

listening to or making music? These are questions that probably we are still far from solving,

and perhaps, as in modern conceptions of music, there is no single or definitive way to

formalize them.

1.2. Brief definitions of sound hearing process: Under an anthropocentric

perspective, we focus on the three mental processes that occur during music listening. They

are: Perception, Cognition and Affect. Aspects of perceptual or psychoacoustic, sound

phenomena are related to micro-time, context-free, that describe the way information is

captured sound like acoustic stimulus (external) for the binaural system (formed by the pair of

ears). Music Cognition deals with contextual aspects of sound information, describing with

internal, or mental, processes, knowledge and recognition of a given event or musical timbre.

These are formed by sound events that occur under a broad duration of time, enough to

constitute a memory of its occurrence, thus in music, corresponding to its musical meaning. It

is during this process that the identification of an instrument timbre occurs, for example,

through perception we notice the sonority of a violin, but it's through cognition that we

identify the contextual characteristics of this sonority in contrast to other known, as, in this

case, for instance, if this violin is (or not) a Stradivarius.

The third process that involves the sound heard (and its musical context) is

1284


Musical Affect. Some studies include this process in the field of music cognition, however,

cognition studies the emotion appraised, while musical affect studies the emotion evoked.

Affection has significant importance for musical composition and performance. Several books

and orchestration works define Timbre, specially the ones of musical instruments, based on

these possible emotional state that it can represent, such as: euphoria or solemnity. In fact,

musical affect is still a recent and yet to be explored area of human knowledge. Recently,

neuroscience has made significant progress in this field, albeit modest, in order to broad the

understanding of the complex intervenience of musical events on evoked emotions.

2. Epistemological Bases of Time and Music

To better understand musical perception, cognition and affect, we first need to

consider that they occur simultaneously, and the separation in different topics is merely to

facilitate the study. Next, it is thoroughly defined the concepts of each process and its

corresponding timescale, still based on the concepts of time, as mentioned above.

2.1. Perception, or Psychoacoustic process: The sound perception occurs in

minute time intervals, in a range known as the psychoacoustic hearing persistence; around

100ms (100 milliseconds) [ROOT 1928]. In this scale of time duration, there are no mental

models able to register (memorize) the sound event, thereby making them free of any musical

context, despite their great importance for the hearing process. It is in this small order of time

interval that occur the identification of the timbre of a sound. When an instrument generates a

tonal sound (a sound with clear pitch), the vibrations of the elastic medium (usually air) that

make up this signal, do not start with the maximum amplitude, but require some period of

time to rise. The mass inertia of the power oscillator which generates the sound signal in a

traditional musical instrument (e.g. a string, a membrane, a reed, etc.) requires a period of

time necessary for the establishment of the sound event which corresponds to the non-periodic

transitions, from the stationary lack of sound to its oscillatory steady-state. This initial

transition, from silence to peak amplitude, is named Attack Transient [MENEZES 2004].

Several researches prooved that removing this transient, the auditory identification of timbre

is impaired and a sound may even seems to resemble another one, or be considered artificial.

The technique of removing the attack of a sound is often used in electroacoustic music and

brings the sensation of an acoustic sound to seem like being purely electronic, i.e., generated

by mathematical or computational models.

Although we should require more experiments to proove it, evidences suggest that

the window (or, the duration of) time for the attack transient is under the auditory persistence

1285


(100ms). An interesting aspect to note is the relationship between tone (pitch) and rhythm

(beat) and how they are the extremes of the same continuous dimension of perception. The

continuity from one to the other is described by the duration of the musical event. It is well

known that if a rhythmic cell is accelerates after a given speed (or frequency), it is no longer

perceived as a rhythmic structure, but as a continuous sound event with a clear pitch, which

can correspond to a musical note. In fact, it is very difficult to get a drummer to play at such

speed, but it is possible to use technological resources to validate this perceptual phenomenon.

One can speed up the execution of an audio sample in looping – such as a short sample of a

speak phrase, a rhythmic cell, or a musical excerpt – until it is no longer recognized as before,

but arising into a continuous musical tone. It is important to observe that this effect will

happen even if the looped sample has no clear tonal sounds or steady partials. It is proved that

this effect happens when a sound sample is repeated at a rate faster than twenty cycles per

second (20 Hz), when this sound homogeneously turns into a constant pitch, which can thus

be represented by a musical note. Next figure depicts its frontier. F

ig. 1: Tempor

al sound

perception's

frontier. In

about 20Hz,

or a period of 50ms, the perception of the interval between sound events goes from rhythmic (time domain) to tonal (frequency domain).

2.2. Cognitive process: In Cognitive Sciences there are already a great influence

of the individual attention, which is related to short-term memory. We can say that it is the

moment of transition between an unaware process (Perception) of hearing, to semi- or fully

aware (Cognition) process of listening, where we also gain some control to focus our

attention, at some extent, according to our will. It is also in the Cognition process that we

complement the identification of the timbre of the sounds we hear, which involves the

physical events observed during the perceptual process with the individual knowledge

acquired in previous life experiences. This is due to the identification of sound aspects made

by the musical mind, in contrast to listener's previous sonic experiences, such as memories of

the same sound aspects previously listened. Cognition works within a time frame greater than

the one of purely perceptual or psychoacoustic processes. According to the studies attributed

to the psychologist William James, when listening to music, we have embedded the concept

of Specious Present; the reference of a cognitive time scale where most listeners have the

sensation of hearing sound events that are not simultaneous, but within a cognitive time

1286


window where they consider to be the musical "now". We argue that the musical "now" is

related to short-term memory, which can vary from individual to individual, from situation to

situation, as well as according to the complexity of musical or sonic information [POIDEVIN,

2000]. Some experiments have shown that the sensation of musical "now" occurs in durations

in the order of about one to three seconds [LEMAN, 2000].

This also enables the cognitive ability to focus the attention to a particular sound

event, while ignoring other sonic threads around us, at least partially. Thus, we are able, for

example, to have a conversation with another individual in an environment with other

conversations occurring simultaneously, as in a cocktail party. This is due to the partial

conscious control we have over the process of cognition, or thought, in contrast to the

mechanismsof perception that occurs involuntarily. For example, in the same cocktail party

example, if there is an unexpected explosion, we involuntarily pay attention to this sonic

event. In general, reactions to the processes of sound perception are involuntary, while the

reaction to cognitive processes are mostly voluntary.

2.4. Musical Affection and affective process: Musical Affection is, among all

three mental processes discussed here, the one most correlated to music context, and thus has

such a range of natural (genetic) and nurtured (learned) variables that makes any sort of

statement about it speculative and error-prone. The subjective nature of music, as well as

listener's age, gender, sociocultural background and training; are, perhaps, aspects that most

influence the Affection. For being too large, this field might yield to differences and

unpredictable results. Anyway, we might infer that Musical Affection handles the emotion

evoked by music through mechanisms related to its long-term memory. Affective aspects are

the ones described in terms of emotional dimensions, such as the Circumplex Model, with

brings the dimensions of Arousal and Valence, as constituents of a broad range of emotions

appraised or evoked by music listening [RUSSEL, 1980]. Emotions in music are evoked by a

prosody of Expectations and listeners' mental capabilities of recognize and establish them,

forming a discourse of anticipations [HURON, 2006]. This establishes the boundaries

between Cognitive and Affective aspects that describe emotions, as cognition deals with

emotion appraisal, while affection deals with emotion arousal. Of course, there are plenty of

possible discrepancy of emotions evoked by the same musical passage in different

individuals. This is due, among others, to the differences between previously acquired

(learned) information, distinct sociocultural backgrounds, which are normally all rounded to

the subjective label of personal taste. Given these facts, it is expected that there might be a

significant variation in the size of these time windows, especially those ones related to

musical affection, while for cognition, as it is usually related to the identification and

1287


understanding, short-term memory tends to suffice to describe it in, which is lastly constituted

of aspects occurring within the time windows of musical "now" (from 1 to 3 seconds).

Fig. 2: Russel's Circumplex Model.

The time scale of the emotion evoked by music is particularly explicit in the

expectation theory of David Huron. [HURON 2006] developed this theory from the studies of

Leonard Meyer [MEYER 1956]. This research deals with the idea that the emotional content

of music arises from mental dynamics constructed by the listener predicting the musical

events. According to Huron, while listening to music, we create expectations about upcoming

musical events and passages, embedded in the musical discourse. In simple terms, the

confirmation of a correct prediction causes a feeling of satisfaction, while attesting that a

prediction was wrong, cause discomfort feeling. According to this theory, the affect is

generated by the musical prosody of correct and incorrect predictions that the listener

automatically makes with respect to musical events of any order, such as: melodic, harmonic,

rhythmic, orchestration, processing and sound effects. Different expectations can even occur

simultaneously in the same, different or intertwined orders.

Some researches suggest that emotional arousal is related to long-term memory,

which is the process that allows us to compare larger passages of musical information (above

specious time) and, for instance, to discriminate between musical genres, styles and any other

specific information about a particular performer or composer.

As musical affect is related to the process of listening to a musical discourse, we

can relate it with Henri Bergson studies about Time. His definitions seem quite appropriate for

us to understand how emotions evoked by music are generated. We start from Bergson's

concept that Time is a succession of simultaneities, which, our consciousness gathers and

organizes in a virtual space, so that it can be measured. In this sense, Time differs from

Duration that is immeasurable and indivisible. Based on that, wee can say that the process of

Affection is based on the one of a multiplicity of states of consciousness given by Time. In

1288


short, during music listening, there is the possibility of emotion arousal if the listener's

conscience is able to capture moments of that music, and arrange them as to make sense of

them altogether. Following Bergson reasoning, our consciousness is placing musical

simultaneity in the virtual space of our mind, and thus, music that previously permeated

consciousness, starts to interact with other sensations and memories we have, for all states of

consciousness dwell the same homogeneous medium of time "transposed" into a form of

virtual space. From this concept comes the understanding of the interaction between emotions

evoked by long-term memory, which involves also listener's sociocultural "baggage". As we

feel this way where these emotional interactions occur, as a homogeneous medium, we can

also reckon some sort of magnitude for its evoked emotion.

3. Conclusions

Given the facts here theoretically investigated, we can foresee some of the

challenges of this study, in the further experimental investigation, so we might reach here

some valuable contribution for future investigations. We first checked the interdisciplinarity in

which this study is placed, which brought out the need of intersection of areas of knowledge

that usually do not interact, such as Music and Physics, despite their interconnections in the

process of music generation (e.g. luthiers, sound design, sound synthesis, etc.). Another

problem that arose was the difficulty to provide a clear definition of what is the time. That still

remains, to a certain extent, inconspicuous, to just in music, but for all areas of knowledge

that dare to study it. Therefore we had to use more than one definition of time along this

study. Finally, the conclusion that the processes here studied do not occur separately but

simultaneously, can lead to a wide variations in experimental results may lead to the difficulty

of seeking and predicting patterns of occurrence.

It is more effective to separate the processes of Perception and Affect, respectively

as: unaware and aware, passing through the intermediate process of Cognition, which seems

to be semi-aware. We have no control over the perception process, because our auditory

system picks up all sounds in the environment that are within our audible spectrum (20Hz ~

20000Hz) and sends this information to the auditory cortex, a specialized area of our brain

that identify sonic information, sort it and allows us to focus attention on a specific sonic

event. During the Affection process, we are able to organize this musical information, as we

are dealing with the conscious activity of music listening, where we will be subject to the

evocation of a feeling or mood. The study of Affection is important not only to research the

process of evoking specific emotions on a listener, but, it's in this process that we can evaluate

a musical performance, classify a song as how well structured is it, as well as finding the

musical style of a song. Clearly, it all depends on relationships made between the information

1289


received and the knowledge previously acquired during a lifetime of music listening, meaning

the sociocultural background of each individual, alone or as part of a community, so we

believe that the notion of time given by Bergson is meaningful, where he states that we

arrange moments, or episodes, within a virtual space of time that are retrieve by our

conscience.

We conclude by saying that the time scales related to the process of music

listening can be more personal than we might have expected, and can even undergo

variations, for each specific case. Even though physically we can set limits for each process,

psychologically, these limits are not always respected by consciousness, which, in a large

share of cases, this can even be consciously manipulated, in order to enhance the emotional

experience of musical appreciation, since that music, as an art, is in fact unpredictable and

subjective, given by a prosody of anticipations with deep perceptual, cognitive and emotional

effects on individuals. And, at the end, we quote Wilhelm Leibniz, in what concerns physics,

mathematics and music in a deep synthesis, when he mentions that: “Music is the pleasure the

human mind experiences from counting without being aware that it is counting.”

References:

HURON, D. Sweet anticipation: music and the psychology of expectation.Cambridge, MA: MIT Press, 2007. MENEZES FILHO, F. The Musical Acoustics in Words and Sounds.Editorial Atelie. 308 pp. 2004. TENNEY, J., Polansky, L. Temporal Gestalt Perception in Music. S T. Journal of Music Theory, vol. 24, no. 2, pp. 205-241. Autumn, 1980. ROOT, A. R. Auditory persistence, Summation, and fusion in impulse-Successive periods.Psychological Review, vol. 35 (6), pp. 507-514. November 1928. RUSSELL, J. A. The circumplex model of Affect. Journal of Personality and Social Psychology, vol. 39. pp. 345-356, 1980. BERGSON, H. Time and Free Will: An Essay on the Immediate Data of Consciousness, translated by F. L. Pogson, M. A. London: George Allen and Unwin. 1910. MEYER, L. B. Emotion and meaning in music. Chicago University Press, 1956. POIDEVIN, R. L. The Perception of Time. In: ZALTA, Edward (org.), The Stanford Online Encyclopedia of Philosophy, 2000. http://www.plato.stanford.edu (visited in 2012). JAMES, W. The Principles of Psychology. New York: Cosimo, 2007. LEMAN, M. et al. Correlation of Gestural Musical Audio Cues and Perceived Expressive Qualities. In: Gesture-Based Communication in Human-Computer Interaction. 5th International Gesture Workshop, pp. 40-54. Berlim / Heidelberg: Springer. LNCS Series

1290

http://www.plato.stanford.edu/


“Lecture notes in artificial intelligence”, v. 2915. 2004.

1291

Documents

Music, Timbre and Time