Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
SoundsfromTheGardenButtMusicfromHell
AlBiles
Overview
• AudioinInteractiveMedia• CaseStudy:TheGarden,including…• TheButtMusicfromHell• Advicefordoingaudioinyourproject
Audio
• Criticaltoimmersiveuserexperience– Nolongeranenhancement,nowanexpectation– Barapproachingqualityofmoviesound
• Lastsetofassetsaddedtomostprojects– Getleftoverresources(memory&machinecycles)– Absorboverflowsfromearlierphases(money&time)– Havetoreacttowhat’salreadydone
• Audiogetssqueezed
Linearvs.InteractiveMedia
• Linear(movies,videos,etc.)– Onlyonepaththroughcontent,fixedtiming– Canfinetuneaudiotofitvisuals
• Interactive(simulations,games,VR,installations)–Multiple,semi-unpredictablepathsthroughcontent– Timingdependsonuseractions– Audiohastobeagile,morereactive
ConventionalAudioAssets
• Soundeffects/Foleyart– Visiblesourceinthenarrativespace
• Ambientsound– Can’tseesource,butcanhearit&expectit
• Voicework– Dialog,narration
• Music(cues)– Theme,background,source
• Interface(UI)sounds– Auralfeedbackforuseractions
Diegesis
• Diegetic:Heardbycharactersinnarrativespace– Soundeffects,ambient,dialog,exclamations– Sourcemusic:Musiciansplayinginscene,radio,etc.– Placeinnarrativespace:L-Rpan,level,reverb,etc.
• Non-diegetic:Heardonlybyuser/player– Narration,voiceover– Theme,background,incidentalmusic– Interfacesounds– Placeinuser/player’shead
PointofPerception(PoP)
• PoP =PointofView+PointofAudition• 3rd-person(Objectivecamera)– PoP isobserverofnarrativeworldthroughwindow– Diegeticsoundsallcomefromwithinthewindow
• 1st-person(Subjectivecamera)– PoP isuser/player’spositioninside narrativeworld– Soundscancomefromanywhere,evenoffscreen
• Almostallmoviesare3rd-person• Lotsofgamesare1st-person
SoundEffects/FoleyArt
• Trigger– User/playeractions,gameevents– Simple:Justcallplay()
• Localization– Diegetic=>placesoundatvisiblesource– UseLRpan,gain,EQ,reverb,Doppler(ifmoving)– Devenvironmentmightprovidereal-time3D
• Placesoundatsourcelocationinvirtualworld• Placemicatcameralocation• Physicallymodelwhatmichears
AmbientSounds
• Trigger–Whenenteringnewsetting,changinggamestate– Looptextures(croakingfrogs,wind,streetnoise)– Randomlyplaydiscretesounds(birdcalls,siren)
• Localization– Diegetic=>placethemsomewhereintheworld– Loopedtexturesplaced“everywhere”– Discretesoundsplaced“anywhere”
VoiceWork– Narration
• Trigger– Usuallyatagamestatechange– Playspeechtocompletion
• Localization– Non-diegetic=>Usuallymonaural,minimaleffects– Supposedtosoundinsideuser’shead
VoiceWork– Dialog
• BetweenNPCs(NonPlayingCharacters)– Triggeralinewhenplayerspeaks– LocalizetoNPClocation,withappropriateeffects– Oftensynchedtoanimation(especiallyincutscenes)
• BetweenNPCandplayer– NPCaddressingplayernoproblem(sameasabove)– Player’sspeechisaproblem
• Speechrecognitionnotreadyforprimetime• Mostdialogsaretext-based,atleastforplayerinput• Messy,tosaytheleast
Music
• Thememusic– Foreground:Makememorablefirstimpression– Usuallyastand-alone“tune”
• Background/underscoring– Establishsetting,enhancemood– Nota“tune”:Shouldn’tdrawattention
Music
• Incidentalmusic– Literallyaccompaniesanincident/event– Can“magnify”theevent(stinger)–Maybeforeshadowtheevent?
• Sourcemusic– Diegetic–Musiciansinnarrativespace,musiconradio– Usuallylocalizedtooriginofmusic
InteractiveMusic
• Musicshouldchangewithin-gameevents– Triggeredbynarrativeorbyuser/playeractions– Can’tjustplaycueonceandbedone
• Usuallyrequiresdeconstructingthecue– Horizontally:intoloopablephrases– Vertically:intolayerable parts
• Eventstriggertransitionswithin thecue• Differentcompositionalprocess
CaseStudy:GardenofEarthlyDelights
TheTriptych
TheGame• GamesetintheHellpanel
“Trappedinaworldtowhichtheydonotbelong,playersmustfindawaytoescape.Hell isastrangeplace,withlostsoulslockedintohellsoftheirowncreation.”
• PlayerGoalBecomethenewoverlordoftheregion
• Mechanics:2ModesSpirit:Movefreelybutcan’tinteractwithNPCsFlesh:PossessNPCbodies, interact,usestuff
• Gameplay:1st personPoPExploreandsolvepuzzlestounlockNPCbodiesandpossessthem
TheProject
• 2years:Mostworkinspring2014&spring2015• 12+faculty,30+students from5+academicunits• Lotsofteams– Gamedesign– Conceptart– 3Dmodelling&animation– Textures&UI– Development(programmers)– Audio– Research
QuickProjectCritique
• Greatlearningexperience…• Goalmorphed– From:serious/educationalgameaimedatthePrado– To:anadult-ish gamethatembracedthehellpart
• Scope:Tooambitious(Whatelseisnew...)• Semester:Artificialtimelineconstraint– Studentsdoingfinalprojectsforaclass– Allcameinatendofsemester:Notimetointegrate
• Bottomline:Needsmorepolish
AudioAssets
• 140+individualassetsused,alloriginal– LotsofSFX,voicework– Someambient,interfacesounds– Onlyonepieceofmusic
• Simpleinteractivity=>Nomiddlewareneeded• UsedUnitylocalizationforsomesounds
AudioTeam
• Facultylead• StudentsfromInteractive&GameAudioclass– 2studentsfromspring2014,4fromspring2015– Theirfinalprojects
• Idefinedinitiallistofneededassets,servedasconsultant,basicallystayedoutoftheway
• Studentsdidallproductionwork &deliveredassetstoprogrammers,except…
• …Ididthetune
AudioStudents!
AngelaMuscariello
DustinKochenspargerLindseyEllis TomFarrell
JohnMilliganRyanGochnauer
SFX/Foley
• Nolibrarysoundsinfinalgame• Recordedeitherinsoundlaborinfield(literally)• Soundsforalloccasions– Footsteps(walkcycles)– Cards,coins,knifesounds,mugsclanking– Openingdoors,crates;pickingthingsup,dropping– Heartbeats,dismembering,soulsucking,possession– Drinking,pouring,potionsounds– Interfacesounds
Forexample:WalkCycles
• Recordlotsoffootstepsindividuallyortogether• Editthemtogetherintoaloop–Maybehavealternativeloops–Maybehavealternativefootstepswithinaloop
• Differentsurfaces– Leaves–Mud–Water–Wood
MoreFoleyExamples
• ScrollOpening• FireonSpawn• Happiness-SuckingPotion• DeathGargle• SwanDeath
AmbientExamples
• Ghostwhispering• Wind• Demonhordeskittering• ScaryVoice
VoiceWork
• Dialogshandledwithtext– Userinteractionusesdialogboxes– DecidednottopronounceNPCtext
• Didlotsofvoiceactingforseveralmechanics– Reassembling/disassemblingwife’sbody– Bossbattlewiththerabbit
• Lotsofambientspeech:Voicesinamob
VoiceWorkExamples
• IntroAnnouncement• Wife(dis)assembly
• Crowdhecklingatbossbattle
• Oooohhh!
Music
• Nothemeorbackgroundmusic• Onlyonetune:Linear&diegetic• Usedinmechanicwhereplayersolvesaudiopuzzle tounlocktune&bestownewability
• Leveragedirresistiblefeatureofpainting…
TheBand!
HurdyGurdy
Drum
Shawm
Lute
Harp
Sacbut
CornettFlute
TheButtMusic
ButtMusicAligned
• PlainchantNotation• Four-linestafffornotepitches
• Nonotelengths• Nomeasurelines• Notimesignature• Supposedtohavelyrics
• GregorianChant
RealPlainchant
Cclef:ThislineisC
VerticalLine:PauseorrestLongerShorter
Climacus:aNeume
Punctum
• Neumes tellhowtosingasyllableofthetext
• Pitch[sequences]forasinglesyllable
• Severaldifferentsystems
• Replacedbyourfamiliar5-linestaff
• Stillusedinsomechurchestoday
BoschWasNoMusician• NoLyrics:Blasphemy!• Clefpaintedout?• Neumes don’tmatchanyknownsystem(justabrushgesture?)
• Neumes don’tlineupwellwithlines,sopitchesareambiguous
• Boschlikelyusingmusicas“decoration,”notasactualmusic
MusicasDecoration
WhattheHell…
• Transcribeitanyway!• Othershavedonethis• RecentlyinFebruary,2014– AmeliaHamrick– StudentatOklahomaChristianU.– Straightforwardinterpretation– Impose4/4timebydefault– Initialblogpostwentviral– Gotrecordedwithchoral“lyrics”
Amelia’sTranscription
ChoralRecording
• Gregorianchantinareally,reallybigcathedral• Onrepeat,imposed3/4timewithguitar&bass• Allegedlyrics:
Buttsongfromhell,Thisisthebuttsongfromhell.WesingfromourasseswhileburninginpurgatoryThebuttsongfromhell,Thebuttsongfromhell.Butts!
Al’sInterpretation
• AssumeCclef• Assumeallneumesarepunctums
• Interpretnotesasliterallyaspossible
• Handleambiguitiesasmusicallyaspossible
• Usebuttcrackasrestsinbottomtwostaves
• Useinstrumentsfromthepainting
Al’sScoreAssumeallquarternotesinCmajor(whitekeysonpiano)
SetdronestoC
Twobutt-crackrests
Addapitcheddrumlinetoprovideanunevenrhythm
Holdthelastnote
RenderedwithMIDI
HurdyGurdy
• DecidedtoplaythemelodiclineonaHurdyGurdy because:– It’sareallycoolperiodinstrument– Thisisthefirstdepictionofabuzzingdronestring(calledadog)
• Youcanallegedlybuyreplicasofthisonefor$5000(!)
AnatomyofaHurdyGurdy• Rosinedwheelexcitesstringsasit’scranked
• Keysshortenstringslikeanautoharp
• Keysdon’taffectdronestrings
• Buzzingbridgeaddsdifferenttimbretoonedronestring
• Basicallyastringversionofabagpipe(alsoexisted)
FakingaHurdyGurdy
• SynthesizedwithMIDIinstruments• UsedaYamahaMU-128tonegenerator(synth)• Melodiclinedoubledontwoinstruments:– Aweird,physicallymodeledbowedinstrument– Adetunedsitar
• DroneistwooctavesusingaChineseJingHu• DrumisaJapaneseTaikoDrum(pitched)• RecordedaMIDI“performance”
Al’sRecording
Monauralmixwithastereoreverb
Notlocalizedingame,justplayedback
Gamemechanic:Playerdiscoversfirstfewnotesofthetune,whichtriggersentiretune& unlocksnewability
SeguetoAdvice:AudioWorkflow
• Workflowiscentraltoproductiveproduction– Especiallywithmultiplepeopleonateam– Especiallyespeciallywithmultipleteams– Especially3withbig,complex,multi-facetedprojects
• Helpsgetahandleonsoftwarefordifferenttasks• Connectionsamongtasks
GardenAudioWorkflow:Simple
ProgrammersUnityC#
GitHub
AudioFolksAudacityProTools
Ableton Live
MusicDude(Al)MIDIEnvironment
AudacityMuseScore
mp3files
mp3fileVoiceActors
ScriptWriters
Linestorecord
Performlines
FoleyProps
Foleyart
AGeneralizedAudioWorkflow
ProgrammersInteractiveDevEnvScriptingLanguage(s)
VersionControl
SFX/FoleyDAW
Composer/ArrangerScoreEditor
Soundfiles
SoundEditorDAW
MusicEditorDAW
VoiceDAW
RecordingEngDAW
Soundfiles
DAWProjectsStems&Triggers
Soundfiles&Triggers
AudioWorkflowwithMiddleware
ProgrammersInteractiveDevEnvScriptingLanguage(s)
VersionControl
SFX/FoleyDAW
Composer/ArrangerScoreEditor
SoundfilesSoundEditor
DAW
MusicEditorDAW
VoiceDAW
RecordingEngDAW
Soundfiles
DAWProjectsStems&Triggers
MapSoundstoTriggers
MiddlewareWwise orFMODTriggers
&Localization
Scores&/orMIDI
Low-RoadAudioWorkflow
SimpleDevEnvGameMaker orProcessing
AudioDudeAudacity
Loop-BasedMusicGarageBand
mp3files
Individualloopfiles
VoiceActors
ScriptWriters
Linestorecord
Performlines
FoleyProps
Foleyart
LibrarySounds
LotsofAudio-RelatedSoftware!
AudioProductionTools
• AudioEditors:Audacity(Opensource)– Goodnews
• Easytouse• Lotsofeffects• Largeusercommunity• FineforsimpleSFX,voiceacting
– Badnews• Noeffectschains:Verycumbersometomix• Noloopfeature:Havetocopy/paste• NoMIDI:onlyaudioformusic• Verylimitedformusic
AudioProductionTools
• DigitalAudioWorkstations:ProTools,Logic– Goodnews
• Industrialstrength• Effectschains• MIDI• Almostrequiredfornon-trivialmusic
– Badnews• Licensed(notfree)• Learningcurve• Overkillforsimplethings
AudioProductionTools
• Loop-basedDAWs:FLStudio,Ableton Live– Goodnews
• Loop-basedcomposition/production• Full-featuredDAW• Can“compose”withminimalmusicalexperience
– Badnews• Steep learningcurve• Licenseissues
– GarageBand:Goodforbeginners• Flatterlearningcurve• LimitedasaDAW
AudioTools
• ScoreEditors(Finale,MuseScore)– Goodnews
• Generatesheetmusicforperformancegroups• GenerateMIDIversionsofthescore• Greatfortraditionalcomposers
– Badnews• Musiconly,traditionalmusicatthat• Noaudio• Notveryusefulfornon-musicians
AudioTools
• AudioMiddleware(Wwise,FMOD)– SitsbetweenDAW&game/audioengine– Goal:Connectcomposers,audiofolks,programmers
• Composerscanbuildinteractivemusicw/oprogramming• Audiofolkscanedit/mixaudioingamew/oprogramming• Programmerscanplacetriggerpointsincode
– Powerfulwaytostreamlineaudioworkflow– Greatonprojectswithcomplexinteractiveaudio– Overkillonsmallprojectswithsimpleaudio
AdviceforYourProjects
• Dospottingsessionsasearlyaspossible– Fromstoryboards,prototypes,earlyversions– Listallthesoundsyouneed– Determinetriggerpoints&sequences– Timethevisuals&synchpointswithin
• Preproductionwork– Decidewhichsoundsyou’llbuyvs.make– ExperimentwithFoleyprops–Writescriptsforvoicework– Findvoiceactors(amongyourteam,mostlikely)
Advice– Recording
• Recordingsession– Atleasttwopeople
• Atleastonetomakesounds• Atleastonetotwiddleknobs&pushbuttons
– Quietplace(turnofffans,appliances,etc.)– Decentmic(USBmicorstereorecorder)– Setlevelscorrectly(useheadroom,butnever clip)– Saveeverythinguncompressed(wav,aiff)– Archiveoriginalsandwrite-protectthem
Advice– Editing
• Startwithcopiesofarchivedoriginalrecordings• Workuncompressed(atleast44.1kHz,16bit)• Don’tbestingywithtracks• Savethemulti-trackproject,notjustmixedresult• Savefrequently,makealternativeversions• Deconstructcomplexsoundswhenpossible–Mayneedtodeliverstems,notfullymixedsound– Keepambienttexturesanddiscreetsoundsseparate– Interactivemusicrequiresindividualparts,phrases
Advice– VoiceWork
• Alwayshaveascript,atleasttostart!• Recorddialogsinsequenceinrealtime• Recordmonaurally,preferablyonemicperactor• Directingtheactors– Leavespacebetweenlinesunlessdialogmustoverlap– Goforgoodenunciationwithappropriatefeeling– Getdifferentemotionaltakes– Recordmultipletakes,butremember…– Youarenot StanleyKubrick!!!
Advice– Effects
• Whentoapply?– BakedintoassetatproductiontimeOR– Appliedin-gamebyaudioengine(activelocalization)
• Orderinwhichtoapplyeffects(guideline)– Compression(squeezedynamicrange)– Level(reallycouldbeanytime)– EQ(Treble/bassonsteroids)– Chorus/phase/flange/distortion– Panning(L-R)– Reverb
Advice– PuttingItAllTogether
• Lotsofaudioassetsatsametimeisaproblem– Canmaskeachother– Canbedistracting,overwhelming,breakimmersion
• Goal:Hearmostimportantsoundsclearly• Solutions– Ducklessimportantsounds(dialthemdownorout)– Useaudiodimensionstospreadsoundsout
AudioPriorities
1. Soundsneededforuser/playeractions2. Soundsthatmatchvisuals3. Soundsthatyou’dexpecttohear4. Soundsthatenhanceevents5. Soundsthatconveyemotions6. Soundsthatsetamood7. Soundsthatfillgaps
AudioDimensions
• Location:L-Rpan– Pancompetingsoundstodifferentlocations– Notalwaysfeasibleiftiedtoon-screensources
• Frequency:Centerpitch,bandwidth– ControlfrequencyrangeofsoundswithEQ– Importantsoundonlyoneusingaparticularband– Avoidloud“noise”sounds(theyeatallfrequencies)
• Timbre/envelope/origin– Soundswithdifferentattributescancoexistbetter
DevelopmentProcess
• Shouldbespiralmodel,notwaterfall– Fixingoneissueoftencreatesotherissues–Whack-a-Moleproblem– KeepDAWprojectsaroundtoremix
• Middlewarecanhelp– CanaddWwise toUnitydevinterface– Tweaklevels,otherparametersofspecificassetsinrealtimeduringgameplay
Overall
• Listenwithfreshears– Takelotsofbreaks&dootherthings– Getotherfolkstolisten,playtest
• DocumentEverything!– Eachassetshouldhaveadevtrail– Blog/Wikicanbeuseful(iffolksuseit)– Contentmanagementveryuseful:Learnhowtouseit– Librarian:Veryusefulrollforsomeonetoplay
• Havefun!!!!
Questions?