November 4, 2000SMPC Conference1 Technology and Film-Music (Multi-modal) Research Dr. Scott D. Lipscomb Institute for Music Research University of Texas

November 4, 2000 SMPC Conference 1

Technology and Film-Music(Multi-modal) Research

Dr. Scott D. LipscombInstitute for Music Research

University of Texas at San Antonio

Interdisciplinary Perspectives on Film Interdisciplinary Perspectives on Film MusicMusic


Organization of Presentation

Types of audio & visual stimuli used in film music researchMethods of A-V stimulus presentationFuture directions & possibilities


The Phonograph


Research StudiesTannenbaum (1956); dramatic presentation (live, studio tape, or recording of live performance) w/phonograph “accompaniment”


Videocassette Recorder (VHS)


Research StudiesThompson, Russo, & Sinclair (1994); investigation of perceived “closure”

Experiment 3Excerpts from Clue (1985) w/newly-composed MIDI scores

Bolivar, Cohen, & Fentress (1994); “congruency” between audio & visual

Experiment 1, 2, & 3Videos showing “friendly” or “aggressive” social interactions between wolves w/audio selected from commercially-used excerpts for broadcast

Marshall & Cohen (1988)


Research StudiesLipscomb & Kendall (1994); reliability of composer intent & subject SD ratings

Excerpts from Star Trek IV: The Voyage Home with intended music & “other” music from same film

Bullerjahn & Güldenring (1994); qualitative content analysis

Excerpts from “The Joker” (film school project) with newly-composed music by three German composers


Research StudiesKrumhansl & Schenck (1997); can dance reflect structural & expressive qualities of music?

VHS presentation of filmed ballet performance (single camera) w/Mozart’s Divertimento No. 15 in Bb major (K. 287)

Iwamiya (1994); audio-visual interactionsLaserdisc excerpts transferred to VHS in “matched” and “mismatched” conditions

Boltz, et al (late 80s & 90s); filmed narrative & memory for events/time estimates


Laserdisc Player


Research StudiesLipscomb (1995); synchronization of audio & visual

Laserdisc excerpts accompanied by digital audio files


DVD Player


Incredible Potential forFuture Investigations

Substantial increase in data capacityDVD-R to create “compilation” of excerpts from a variety of resources


Microcomputer


Stimulus Presentation

Now you see it …


Television Monitor


Large-Screen TV


Data Projector


Viewing DVD source asProjected Image


Computer Control

Complete Randomization inStimulus Presentation


Interfaces

VHS Interface (RS-232)

Laserdisc Interface (serial)


Research StudiesThompson, Russo, & Sinclair (1994); investigation of perceived “closure”

Experiment 13-D animation w/MIDI audio

Experiment 2Original film footage digitized into Quicktime format w/MIDI audio


Research StudiesSirius & Clarke (1994); effect of music on perceived meaning of images

Original 3-D animations w/newly-composed MIDI audio in various “styles”

Romantic, sci-fi, comic, Spanish, chase, disco, thriller, western


Research StudiesLipscomb (1995); synchronization of audio & visual

Laserdisc excerpts accompanied by digital audio filesAllowed complete randomization of stimuli

Datteri (1998); influence of audio on ambiguous visual stimulus

black & white vertical bars and sine signal, either single tone or scale


Music Experiment Development System

(MEDS)

Dr. Roger A. KendallUniversity of California, Los

Angeles


MEDS’ Control Panel


MEDS’ Experiment Module


Stimulus PlayList


Stimulus Creation

Synthesize complex signalsMIDI data


Data Analysis

A closer look at the stimuli


Fast Fourier Transform


FFT Displays


RMS Calculation


RMS Display


MEDS Data Editor


Statistical Analysis

Exporting Data


SPSS, Systat, Sygraph, etc.


Descriptive Stats


Graphic RepresentationAudio Four

Visual Five--Evaluative


Cluster Analysis


Multidimensional Scaling


Future Possibilities

The Internet as a Research Tool


Difficulties in A-V Presentation

Full-screen, full-motion A-V presentation is processor-intensive

Especially w/16-bit, 44.1 KHz stereo audio

A-V synchronization can be problematicEven with streaming technologies, internet presentation of A-V stimuli is unreliable over the internet

Quality varies radically at the receiving endMay be due to a number of unpredictable factors


Broadband Connections

Modem 28.8/56 kbps

ISDN 128 kbps

DSL 500+ kbps

T1 1300+ kbps

T3 ??


Streaming TechnologiesApple QuicktimeReal MediaMPEG videoMacromedia Shockwave & FlashMicrosoft Media TechnologiesMedia 100 iFINISHAvid ePublisherothers


Future Experiment?Subject pool world-wide

Rather than freshmen taking General Psych

Stimuli viewed over broadband internet connection

full-screen, full-motion streaming video with (relatively) hi-fidelity sound

Subject responses are fed directly into a database at the researcher’s locationData are analyzed & interpretedResults published in a peer-reviewed ejournal

Complete data set is made available online


Potential AdvantagesGreater generalizeability of results

Larger “N”Less heterogeneous groups

Facilitates cross-cultural researchFosters collaborative research and confirmation of data analysis


Contact Info

Dr. Scott D. LipscombInstitute for Music Research

The University of Texas at San Antonio

[email protected]://imr.utsa.edu/lipscomb/


Model of Film Music Perception

(original, 1994)

AssociationJudgment

Accent StructureRelationship

Perception

Aural stimulus Visual stimulus

YES

YES

NO

NO

IMPLICITPROCESSES

No Shift ofAttentional

Focus

Shift ofAttentional

Focus

Documents

November 4, 2000SMPC Conference1 Technology and Film-Music (Multi-modal) Research Dr. Scott D. Lipscomb Institute for Music Research University of Texas