The Implications of Generative Music Systems on Video Game ...€¦ · BA (Hons) Creative Music Technology ‘The Implications of Generative Music Systems on Video Game Composers:

BA (Hons) Creative Music Technology

‘The Implications of Generative Music

Systems on Video Game Composers: A

Case Study Approach’

by

Matt Gibson

Submitted 3/5/19

Presented as part requirement for BA (Hons) Creative Music

Technology validated by the University of Hull

Department of Music and Performing Arts

University Centre Doncaster

Doncaster College

The author would like to make grateful acknowledgements to the following:

My supervising lecturer, Michael Dunn, for his expert guidance & valuable insight throughout

all the staff at the University Centre Doncaster Music & Music Technology Campus

the participants of this study, whose invaluable expertise made this possible

my family, Jeff, Deb & Sophie, for their help both academically & spiritually

&

my partner Rosie-Mae, for her constant support, patience & belief

i

Abstract

From Spore to No Man’s Sky, generative music in video games is becoming

increasingly prevalent and is viewed as a field with much potential. But what

does this mean for the composers of video game music? This study aimed to

examine the implications of generative music on video game composers, from

the perspective of both the design of the systems and what it means to work

with them. To accomplish this a case study approach was taken, whereby two

specially selected experts in the field were interviewed, and the resulting data

was then analysed thematically using a combination of a priori and emergent

coding.

The key findings of this study show that while certain skills and

personality traits may be a boon to a composer – such as high musical and

computer skills, and an adaptable, progressive mindset – even more significant

is the impact these systems may have on their role: rather being something to

“fear” they can instead be viewed as a tool that can aid the compositional

process, both in terms of speeding up workflow and providing valuable

inspiration, and also improving skill sets. However, other results also show that

the use of generative systems largely depend on a multitude of game-specific

factors, which in turn means that a composer may or may not be required to

work with them over the course of their career.

ii

Contents

Introduction ......................................................................................................................... 1

1.1 Introduction & research focus ................................................................................. 1

1.2 What is adaptive music in video games? ................................................................ 2

1.3 Aims & Objectives ................................................................................................... 3

Literature Review ................................................................................................................ 4

2.1 Introduction ............................................................................................................. 4

2.2 Video game music and film music ........................................................................... 5

2.3 How adaptive music supports players ..................................................................... 6

2.4 Approaches to adaptive music ................................................................................ 7

2.4.1 The generative approach ................................................................................. 7

2.4.2 Implications for composers .............................................................................. 8

2.4.3 Project specificity ............................................................................................. 9

2.5 Refocus of research question and conclusions ....................................................... 9

Methodology...................................................................................................................... 11

3.1 Framework of inquiry and critical assumptions ...................................................... 11

3.2 Approach and research philosophy ....................................................................... 12

3.3 Data collection method and sampling.................................................................... 13

3.4 Coding and data analysis ...................................................................................... 14

Data Analysis .................................................................................................................... 15

4.1 Why generative systems are utilised ..................................................................... 16

4.1.1 Solution to problems ..................................................................................... 17

4.1.2 Game considerations .................................................................................... 17

4.1.3 Summary ...................................................................................................... 17

iii

4.2 How generative systems function .......................................................................... 18

4.2.1 Operation ....................................................................................................... 19

4.2.2 Design ........................................................................................................... 19

4.2.3 Summary ....................................................................................................... 20

4.3 Working with generative systems .......................................................................... 20

4.3.1 Desirable skills ............................................................................................... 21

4.3.2 Role ............................................................................................................... 22

4.3.3 Compositional aid .......................................................................................... 24

4.3.4 Summary ....................................................................................................... 25

4.4 Project specificity .................................................................................................. 26

4.4.1 Goals of the game ......................................................................................... 27

4.4.2 Aspects of generative systems ...................................................................... 28

4.4.3 Summary ....................................................................................................... 29

Conclusion ........................................................................................................................ 30

5.1 Research objectives and findings .......................................................................... 30

5.2 Limitations and further research ............................................................................ 30

5.3 Final remarks ........................................................................................................ 31

Bibliography ...................................................................................................................... 32

Appendix A ........................................................................................................................ 35

Appendix B ........................................................................................................................ 44

Appendix C ........................................................................................................................ 53

C H A P T E R 1 – I N T R O D U C T I O N | 1

CHAPTER 1

Introduction

1.1 Introduction & research focus

It is common knowledge that video games are a rapidly-developing growth industry; it was

estimated that this sector was worth a staggering $138bn (£105bn) by the end of 2018 (Ell,

2019). As the technology and capabilities of consoles increase with every generation, so too

do the budgets and production values of the games, and this in turn has caused the

expectations of the ever-discerning players to rise also. Cutting-edge, photo-realistic visuals

and vast immersive game worlds have become the expected norm, and so video game

music has had to follow suit; far removed from the simple monophonic soundchips of

yesteryear, today’s game soundtracks can rival that of Hollywood blockbusters. But unlike

their silver screen counterparts, games are fluid and malleable: they are reliant on the

players to play out the narrative and to direct the action. Moreover, players are

unpredictable, so how can the music accurately reflect what it doesn’t know is going to

happen? This is neatly encapsulated by Velardo:

“The whole history of video game music can be seen as a continuous

search for compositional techniques and audio tools that create a musical

experience tailored to the visuals and narrative.” (Velardo, 2019, paragraph

2)

This need to synchronise music with what is being displayed on player’s screens has given

rise to the paradigm of adaptive music, whereby various technical processes and systems

enable the music to do just this.

But what does this mean for the composers who work with these systems? Given

that the use of adaptive music is fast becoming standard practice it was surprising to

discover a distinct lack of research from a composer’s perspective, so the aim of this

dissertation is to address this question. It is worth noting at this point that the scope of the

research was subject to some evolution over the course of the study. What started out as an

investigation into the broader trend of adaptive music became more focused once the

significance of generative music systems was identified; thus, this angle became the main

thrust of the study.


1.2 What is adaptive music in video games?

Before continuing it is important to define what is meant by adaptive video game music.

Velardo (2017, paragraph 3) succinctly describes it as music that “constantly changes

dynamically by reacting to some type of control input coming from the game”, either by direct

player interaction or other variables. There is some interchangeability between terms

regarding this, with some sources referring to “dynamic” and “interactive” music, so for clarity

the author henceforth chooses to use “adaptive”.

Also important for contextual reasons is knowing how adaptive music is implemented

into a game, and as such there are three main approaches: horizontal sequencing, vertical

re-orchestration and generative. Horizontal sequencing involves the relevant music cues or

segments being played sequentially and then rearranged to correspond with inputs coming

from the game (van Nispen tot Pannerden, T. et al., 2011); essentially, a completed music

cue is played and then transitions to key points in that track based on what is happening in

the game. To conceptualise this, imagine a basic gameplay scenario consisting of

exploration followed by combat; a composer would write music for both states in the same

music cue, and upon changing from one state to the next the track would jump to the

appropriate part.

Virtual re-orchestration involves several tracks of music which are then combined in

accordance with game inputs to form the completed soundtrack (Collins, 2008); in this case,

the completed music cue is broken down into separate stems of its constituent parts and

then re-mixed together as appropriate. Imagine the same gameplay scenario, except this

time when the states change the music continues to play seamlessly and other elements are

introduced, such as percussion to add tension.

Generative music - the focus of this study - is defined as a system that creates

original music from limitless combinations (Wang, 2015); in other words, music is composed

on-the-fly using stochastic algorithms, based on various rules and parameters and often

rooted in music theory. Returning to the same example, as the gameplay states transition

the music would also change, except this time it would instantaneously and organically

create new material. Two of the most notable examples of this technique are featured in the

games Spore (2008) and more recently No Man’s Sky (2016).


1.3 Aims & Objectives

The research question that guided this study is as follows:

What are the implications of generative music systems on video game composers?

It became evident during preliminary research that the generative strand of adaptive music

was most compelling, as not only did it differ significantly from the other techniques – which

appeared to be grounded in “traditional” methods of composition – it also was lauded for its

potential. Thus, it was deemed most appropriate to focus on this specific area. To explore

any implications, the following objectives were pursued:

▪ To examine the literature and identify core concepts/gaps in the research

▪ To gather qualitative data that details how generative systems work

▪ To gather qualitative data that details the experience of composing and

working with generative systems

▪ To evaluate the qualitative data in terms of what impact these areas have on

composers

It is hoped that this research will contribute value to an area is somewhat lacking in

critical investigation, and also provide a more thorough understanding for composers who

are hoping to succeed in this field.

C H A P T E R 2 – L I T E R A T U R E R E V I E W | 4

CHAPTER 2

Literature Review

This chapter will evaluate the literature pertaining to adaptive music systems for video

games, giving the reader a broad overview of the state of knowledge in this area and

identifying core trends. The initial question that drove the review was:

What are the implications of adaptive music on video game composers?

2.1 Introduction

Research into video game music has been varied, including physiological studies examining

the impact of music on stress levels during gameplay (Hebert et al, 2004), psychological

studies investigating the effects of music on participants with different levels of extraversion

(Levy, 2015), and the effects of tempo on performance (Lawrence, 2012). Also, video game

music is becoming increasingly recognized - Bush et al (2009) highlights audio as a

significant component of video games which directly affects consumer’s preferences on

purchases - yet there has been little scholarly insight into this field from a composer’s

perspective.

It is noted that this area is still in infancy; a key point among the literature is the

scarcity of research (Collins, 2009; Lawrence, 2012; Young, 2012; Prechtl, 2016) – and

specifically within the domain of adaptive music it is rarer still. Lawrence (2012) notes that

despite this, there is a general greater recognition for video game music in non-academic

circles, so in light of this several sources are non-academic in nature. This means that

greater emphasis will be placed on reviewing the emergent paradigms as opposed to

specific studies.

However, it is apparent that despite little academic research adaptive music is

becoming increasingly prevalent in modern video game design. Young (2012, p.47) asserts

that “[adaptive] soundtracks move ever closer to being the expected norm in game music”,

perhaps because of increasing production values and more discerning players (Collins,

2009). This contrast warrants further investigation and, additionally, as far as the author is


aware there have been no studies that explore the implications of adaptive music on game

composers, so this provides a unique opportunity to contribute to this area.

2.2 Video game music and film music

Perhaps since the study of video game music in general is a relatively young field, many

sources have found it appropriate to draw parallels with film music. As such, there is

consensus that game music can perform similar functions to that of film music in terms of

development of themes, complimenting on-screen actions, moods and locales, and

heightening emotion (Jorgensen, 2006; Berndt and Hartmann, 2007; Collins, 2009; Phillips,

2014; Prechtl; 2016). It has also been noted that a general ambition to make games a more

cinematic experience renders the application of film music theories relevant (Lawrence,

2012). However, there comes a point where similarities between these two mediums clearly

diverge: the idea of linearity. Prechtl (2016, p.2) notes that “film narratives are linear in that

they progress exactly the same way every time”, whereas games “feature nonlinear

narratives which are fixed neither in structure nor in timing”. Film scores are tied inherently to

the final edit and can complement the action onscreen, while games are more unpredictable

in that their structure is heavily reliant on the player (Collins, 2009).

Also, there is the issue of interactivity: Collins (2009) differentiates film music from

game music in that the player is the active agent of change, and therefore the music must be

able to react accordingly. This view is shared by Pidkameny (2008), who expands by stating

passive films only require passive scores whilst games have the opportunity to utilise music

that can be directly affected by the actions of the player.

These key points of non-linearity and interactivity are relevant in that they are an

integral concept within video game music, and as such they contextualise the necessity for

adaptive music in games; Levy’s (2015, p.50) statement of a need for “dynamic adjustment

of music” coupled with Whitmore’s (2014, 30:35) point that the “nonlinearity of games calls

for flexible music” emphasises this. There is the implication that an adaptive soundtrack

aspires to serve in much the same way as its film counterpart, despite the fundamentally

different medium; this is neatly summed up by Clark (2007, paragraph 6) who says

“[adaptive] music is the game industry’s answer to the Hollywood film score”. In terms of

what this means for composers it suggests that although typical scoring approaches may still

be valid, considerations must be made to account for these differences.


2.3 How adaptive music supports players

In addition to performing similarly to film music, adaptive music aids in supporting the needs

of players. It appears to be a widely-held belief that adaptive music is effective in enhancing

a player’s overall experience (Collins, 2009; Peerdeman, 2010; Young, 2012; Phillips, 2014),

and can do so for different reasons:

▪ correctively, to avoid repetition and, as a result, tedium

▪ creatively, to more effectively immerse players and enhance the narrative

For more practical corrective purposes, adaptive techniques are utilised to avert

repetition in the music. There is suggestion that repetition is undesirable and can be

detrimental in retaining player’s interest: Hedges, Larson & Mayer (2010, 52:10) assert that

“repetition becomes annoying” and players “turn annoying music off”, while Plans & Morelli

(2012, p.1) make the point that “avoiding tedium is paramount”. However, in his presentation

discussing the implementation of adaptive music into the game Final Fantasy XV, Iwamoto

(2017) states that reducing repetition wasn’t a consideration; in fact, one core component

was to keep the music memorable - which repetition has been shown to enhance

(Hargreaves, D.J. as cited in Plans & Morelli, 2012) - whilst utilising the flexibility that

adaptive music provides. This is notable in that it relates the perspective of an Eastern game

developer (other sources are predominantly Western) which could indicate that culture is a

factor, and more crucially implies that adaptive music can be considered as more than solely

a platform for reducing musical stagnation through variety. Moreover, the term “repetition”

may be misleading, as when coupled with some degree of variation it can be an effective

musical technique; perhaps it is more appropriate to say that exact replication of game music

becomes monotonous.

Adaptive systems are also utilised for creative purposes. A major point of consensus

in the literature concerns adaptive music’s positive role in immersion, and as a device to

support a game’s narrative (Collins, 2009; Hedges, Larson & Mayer 2010; Peerdeman,

2010). It is stated by Minter (2012) that it benefits both “cognitive” (players are only focused

on in-game sounds) and “mythic” (player’s role-playing tendencies are satisfied by the game

world) immersion, while Phillips (2014) relates the pivotal role adaptive music plays in the

willing suspension of disbelief. An investigation conducted by Zhang & Fu (2015) found that

appropriate background music resulted in higher levels of participant immersion, although

issues were raised about how the skill levels of players may have impacted on the results.

One major reservation about this source is that the music in question was not implemented

in an adaptive way, and it is therefore impossible to generalise based on this study.


However, given the increased flexibility of adaptive systems one may logically infer that an

adaptive soundtrack could replicate similar findings.

Tied to this concept of immersion, it is also accepted that adaptive music can be

utilised to empower a game’s narrative. Whitmore (2014) discusses the importance of timing

and synchronicity, the need to present a nuanced emotional arc, and the necessity of

seamless, convincing transitions. Taking this further, there are the notions that adaptive

music can “better support an untold story” (Baldursson & Olafsson, 2016, 10:23) for games

that have no set narrative, while Deriviere & Kurlander (2014) mention how adaptive music

can be used to lead and influence the gameplay of a player. Given the importance of

nonlinearity and interactivity this may place an onus on composers to pre-empt the actions

and intentions of a player, and subsequently compose accordingly.

2.4 Approaches to adaptive music

In addition to the merits adaptive systems can provide there were numerous references to

the specific adaptive techniques themselves; it was apparent from the literature that

horizontal and vertical approaches are the most common. Phillips (2014, p.210) notes how

these can “[quickly react] to every dramatic moment”, and Sweet (2016) mentions this can

be accomplished with relative ease; with regards to specifics, he goes on to state that

horizontal is the easiest to implement and thus allows the composer more time to focus on

the music, whereas vertical allows for more subtle and musical transitions. Interestingly, a

study conducted by Eriksson & Lindau (2013, p.2) found “no significant difference between

the static [pre-rendered, non-adaptive music] and adaptive systems in terms of perceiving

the music”. This contrasts with previous assertions that adaptive music helps facilitate

greater immersion and, while it notes flaws in both data collection and implementation of the

system, prompts debate on the effectiveness of both horizontal and vertical approaches.

Inherent limitations of the pre-rendered music required by these approaches are addressed

in greater detail by Phillips, who states the “sonic quality [of a] highly refined audio mix” is at

the cost of “the ability to finely manipulate the individual instrument performances for the

purposes of interactivity” (2014, p. 222).

2.4.1 The generative approach

Perhaps due to these limitations it is stated in no uncertain terms that “true adaptive

music needs to be generated in real-time” (Folmann, T., as cited in Thomas, 2016, p. 84);


generative music systems allow this, as it allows for micro-control of the music down to the

very note level (Hedges, Larson and Mayer, 2010). A comprehensive study by Prechtl (2016,

p.4) also noted the “lack of granularity” in the more traditional adaptive techniques, often

resulting in a “superficial dramatization” of cutscene and gameplay sequences (Berndt and

Hartmann, 2008, p. 130), and proposed to solve this with a generative approach. It was

found to have a “positive impact on game experiences” and shown to be more effective at

immersing a player than standard techniques. This is also corroborated by the findings of

Elmsley, Groves & Velardo (2018), who mention that this heightened immersion increased

the time spent playing the game. Other advantages of generative systems are they can “play

for long periods while avoiding repetition fatigue”, and also create “vast quantities of music

without the need for a [large] music budget” (Phillips, 2014, p. 232); as such, these benefits

seem to affirm a general belief that generative systems will become increasingly prevalent in

the future (Hedges, Larson and Mayer, 2010; Young, 2012; Phillips, 2014; Baldursson &

Olafsson, 2016).

However, these systems aren’t infallible: technological limitations play a big part,

such as the processing power required, storage and RAM limitations, and issues with

instrument playback (Hedges, Larson and Mayer, 2010). This has ramifications both for

practical aspects, such as efficiently running these systems in-game, and for creative

aspects: Collins (2009) states that while more adaptive, these systems can suffer from lower

production values than adaptive techniques that rely on pre-rendered music. It is

hypothesised that as these technological constraints lessen then all music could be created

in this way (Lamperski & Tahouri, 2016), although currently the implication is a trade-off

between technology and creativity; perhaps in response to this, there is evidence of a hybrid

approach which combines techniques to counteract any shortcomings and meet the needs

of the game (Phillips, 2014). Also, their rarity of use appears to be due to the complex nature

of both “designing and implementing” them (Phillips, 2014, p. 233) and as such they remain

an “unproven method” (Van Geelen, 2008, p. 101).

2.4.2 Implications for composers

Although there is little research relating any immediate implications of generative

systems on composers, there is some mention of the need for a more diverse skillset

(Minter, 2012; Young, 2012). When discussing their own generative system Hedges, Larson

and Mayer (2010, 31:00) state that this technique “doesn’t lend itself to traditional

composition techniques” and therefore they had to find a composer who was “comfortable


with non-traditional and non-linear methods”; they also point out that their system blurs the

line between sound design and composition. As well as possessing less conventional

composition skills, familiarity with the more technical aspects of integration is recommended

(Deriviere & Kurlander, 2014), which again highlights a crossover between the role of

composer and programmer.

This shows that generative systems may have implications for both required skills

and the role of a composer, yet while a useful starting point these statements fail to probe

the issue in any sort of detail; for instance, what creative processes could be applied when

composing for a nonlinear medium? Is knowledge of adaptive music integration an accepted

standard, or does this quality vary from project to project in the same way as adaptive

approaches? Also, an increasing trend towards the more complex generative approach

could potentially blur the line between composer and programmer, or perhaps render a

composer’s role less critical to a project.

2.4.3 Project specificity

Sources also suggest that the approach taken is defined by the game. Pacaud & Wintory

(2016, 35:15) talk about finding a “happy medium [of] degrees of adaptivity” between linear

and adaptive music content in their game, which suggests that depending on the goals of the

development team adaptive music may not be appropriate (Phillips, 2014); the pace of the

game may not warrant it, for instance (Rohrmann, 2015). Baldursson and Olafsson (2016,

46:32) say that “no [adaptive] system can work without first thoroughly defining game

elements”, while Deriviere & Kurlander (2014) hint at this symbiosis between music and

game by stating that if a gameplay mechanic is flawed then it impacts upon the composer.

Furthermore, Hedges, Larson & Mayer (2010) suggest that both the required style of the

music and the length of the game may be a factor, while it is also stated that generative

music fits “abstract game narratives better than set narratives with traditional plot points”

(Plans & Morelli, 2012, p.3). This further reinforces the idea that the goals and stylistic

choices of the game are intrinsic to the decision to use generative systems.

2.5 Refocus of research question and conclusions

The available literature shows that there is a growing acceptance of the importance

of adaptive music in games, how adaptive music benefits the player, how various techniques

are used and why, and what the future of this field may hold. Particularly, it was the latter


topic which proved to be crucial: such was the weight of evidence suggesting that generative

music systems will become the prevailing adaptive technique that it prompted a narrowing of

the research question. It was ultimately decided to focus on this specific adaptive approach,

as both the potential and complexity of this burgeoning area may result in the most impact

on composers, therefore the working research question was refined to:


Regarding this, there is little research – either specifically concerning generative music or

more generally - that details the impact on composers. Have composers had to adapt to this

growing trend, and if so, how? To summarise, the questions that arose are as follows:

▪ Regarding the nonlinear and interactive nature of adaptive music are different

compositional approaches required?

▪ Given the technical nature of generative systems, what does this mean for

composer’s skill sets?

▪ Given the perceived inclination towards generative systems, will the role of composer

as we know it change?

▪ Will the increasing sophistication of generative systems allow composers to realise

the vision of adaptive scores more efficiently and to greater effect?

C H A P T E R 3 – M E T H O D O L O G Y | 11

CHAPTER 3

Methodology

This chapter will outline the methodology that was employed for data collection, with regards

to the following:

▪ framework of inquiry based on literature review findings and critical assumptions

▪ research philosophy and methodological approach

▪ data collection method and sampling considerations

▪ how the data will be interpreted and analysed

3.1 Framework of inquiry and critical assumptions

Based on the findings of the literature review, a theoretical framework of inquiry was

developed to focus on the research objectives identified earlier.

Fig. 1, framework of inquiry


The framework, (see Figure 1), was separated into two distinct areas: system-centric

fields of inquiry, which focused on the systems themselves, and composer-centric fields of

inquiry, which focused on working practices and skills. It was deemed necessary to

investigate both angles as questions from each area may inform each other, and any

interrelation would warrant examination.

Also, several critical assumptions were adopted to frame and justify this study, based

on theories found in the literature (see Figure 2):

This serves to illustrate that being as adaptive music is becoming the industry-

standard norm, and that generative music has been shown to enhance immersion most

effectively, it can be logically inferred that generative music will become increasingly

prevalent. Therefore, it is important that composers hoping to succeed in video game music

have an insight into this phenomenon.

3.2 Approach and research philosophy

When designing the method to obtain primary data a qualitative approach was taken. The

questions posed by the framework of inquiry required answers directly related to people’s

working experiences with generative systems, so in terms of epistemology an interpretivist

stance was adopted to greater understand this area from the practitioner’s point of view i.e.

their construction of reality (Creswell, 2003). This also meant that theories and ideas could

be generated inductively, which was especially useful due to the scarcity of research

surrounding this area.

Fig. 2, critical assumptions


3.3 Data collection method and sampling

In terms of the method itself, essentially this study represents case studies of two different

game audio professionals and the systems they developed. The choice of philosophy and

research approach led to unstructured interviews being chosen to generate primary data; it

was felt that this was the best way to realise the phenomenological aspect of the qualitative

paradigm i.e. examine in detail what professionals would experience when composing or

working with generative systems, and gather holistic, naturalistic data of their working

experiences whilst giving the participant the freedom to provide additional information.

Using ideas presented in the framework of inquiry, a loose plan was devised to guide the

interview (see Appendix C) and ensure relevant data was collected.

To obtain the best data possible it was necessary to carefully select experts in the

field, therefore a purposive sampling technique was employed. The general target population

was anyone with a working, practical knowledge of generative systems i.e. game audio

professionals, and criteria had to be established to narrow down the sample frame to yield

the most valuable data sources. Due to the emergent nature of this field it was unfeasible to

be too stringent with selection criteria, although essential characteristics were as follows:

▪ experience in game audio and music

▪ experience working with integration middleware

▪ working understanding of adaptive audio

▪ experience working with generative systems, ideally in a commercially

available game

Two eligible participants were deemed sufficient as this would provide a rich and

diverse data set, while also keeping the amount of data manageable. It is worth noting that,

while this study only focused on two examples, it attempts to derive a broader understanding

of working practices by examining different aspects of their experiences.

With these considerations in place, potential candidates were identified and narrowed

down until the best qualified were found: an audio director who worked on a critically

acclaimed massively multiplayer online role-playing science fiction game (henceforth

referred to as Participant 1), and an audio engineer who worked on a critically acclaimed

action adventure game (henceforth referred to as Participant 2). Not only did these two

experts fulfil the essential sample criteria but also their respective systems bore distinct


characteristics: Participant 1’s system was predominantly melodic in nature, while Participant

2’s dealt solely with rhythm.

After these participants had been contacted the interviews were arranged to take

place via Skype for logistical reasons, as both resided in different countries. The interviews

were conducted independently of each other on 07/03/19 and captured using Skype’s

recording function with the participant’s consent.

3.4 Coding and data analysis

Upon completion of the interviews the raw primary data was then transcribed and coded

using a combination of a priori coding, for which the framework of inquiry was used as

reference, and emergent coding, which allowed for any significant and unexpected topics to

be accounted for. These codes were then arranged hierarchically into themes to allow for

analysis. The coded transcripts in their entirety can be found in Appendices A and B

respectively.

C H A P T E R 4 – D A T A A N A L Y S I S | 15

CHAPTER 4

Data Analysis

This chapter will present the key findings of the study, which proved not only to corroborate

ideas present in the literature but also served to generate new areas of insight into the

subject area. General findings were as follows:

▪ Why generative systems are utilised: insight was provided into various reasons

systems are used.

▪ How generative systems function: human input is an integral part of generative

systems, and there are also implications for required skills of the user

▪ Working with generative systems: generative systems can be viewed as a spectrum,

and as an aid to the compositional process.

▪ Project specificity: the decision to use these systems are based heavily on several

game-related aspects, thus the impact on composers may vary.


4.1 Why generative systems are utilised

Firstly, to provide context it was deemed necessary to investigate the reasons why

generative systems are used instead of other adaptive techniques; this section will provide a

brief overview of those findings. Upon analysis two distinct strands were observed: solutions

to problems presented by the non-linear nature of game audio, and various game

considerations (see Figure 3).

Fig. 3, thematic hierarchy of why systems are used


4.1.1 Solution to problems

There were numerous references by both participants to “problems” and the subsequent

search for “solutions” (Appendix A, lines 34 & 38; Appendix B, line 106). One significant area

of “problem” relates to player experience; on multiple occasions Participant 1 refers to

making the “experience a better one for the player” (Appendix A, line 158). Generative

systems were seen to be the solution for both immersion-related reasons, such as the need

for music to reflect the gameplay in real-time (Appendix A, line 35) and support the narrative

where “there was no story” (Appendix A, line 36), and to avoid repetition fatigue through

additional variation (Appendix A, line 152), which would support theories posited by the

literature (Plans & Morelli, 2012; Whitmore, 2014; Baldursson & Olafsson, 2016).

However, new topics absent from the literature were problems relating to resources,

with mention of both financial issues relating to budget size (Appendix A, lines 128 – 129),

and manpower issues (Appendix A, line 75). It is worth noting, however, that these concerns

were only raised by one participant, therefore suggesting that these ideas may not be

applicable generally.

4.1.2 Game considerations

Game-related considerations also bear some significance. Specific game characteristics

such as genre (Appendix A, line 29) and the scale of the game (Appendix A, line 70) are

noted, as are musical requirements such as the quantity (Appendix A, line 71) and type

(Appendix A, lines 123 & 124) of music required. In light of this evidence it may be inferred

that not every game is a suitable fit for generative music - a sentiment that is echoed by

Participant 2 (Appendix B, line 247) and supported by the views of Phillips (2014) - or at

least certain types of game make it more challenging to implement (Appendix B, lines 216 –

217).

4.1.3 Summary

While this data alone does not serve to inform the research question it does provide some

valuable context as to why these systems are used; reasons concerning the enhancement of

player experience support ideas present in the literature while new insight was revealed in

terms of managing budgetary and staffing issues. This resource aspect is particularly

interesting as, although interpreted as existing within two separate strands, could link closely


with game considerations: a game of a particular size, style or genre may vary the allocated

budget, for instance. This concept of project-specificity became more prominent during

analysis.

4.2 How generative systems function

To gather system-centric data participants were asked to relate how their respective systems

function. Thematic analysis of the data revealed insight into both the operation of generative

systems, within which topics concerning their automated nature and the need for human

input were identified, and how they are designed, within which topics concerning the

system’s modular nature, flexibility and complexity were identified. This is illustrated in

Figure 4.

Fig. 4, thematic hierarchy of system functionality


4.2.1 Operation

In terms of operation, participants touched upon a defining characteristic of generative

systems: their automated nature, whereby new music is generated automatically (Appendix

A, lines 151 – 153) and can potentially continue “forever” (Appendix A, line 96), supporting

Plans & Morelli’s definition: “composition that evolves in real time according to a specific set

of rules” (Plans & Morelli 2012, p.1). However, more significant were topics tied to some sort

of human input. There are multiple instances where participants relate the necessity for

musical input, be it a “[simple] 8-note idea” (Appendix A, line 91) or full musical piece

(Appendix B, lines 81 – 82 & 143).

User agency is also required to define musical parameters that govern both the

music (Appendix A, line 92; Appendix B, line 150) and the game parameters that will control

them (Appendix B, lines 181 – 182). Furthermore, these parameters then need to be

“hooked up” (Appendix B, line 182) to function in-game. This suggests that these systems

are perhaps not as automated as they may appear to the uninitiated; some degree of human

input is essential to operate them.

4.2.2 Design

Regarding design there is evidence to suggest that generative systems are modular in

nature, comprising of a “brain” that houses the algorithm (Appendix A, line 88), proprietary

software that enables the user to define musical rules and parameters (Appendix A, line 89;

Appendix B, line 149), and the facility to incorporate this into audio middleware or game

engine (Appendix A, line 94; Appendix B, lines 179 – 180). This shows clear evidence of a

pipeline when working with these systems, which may have implications for a composer’s

skillset: if they are required to work with all “modules” then it is possible that more technical -

possibly non-musical - expertise would be required, although the fact that this modularity

enables the user to focus solely on individual aspects may render this theory moot.

Moreover, this modularity is indicative of the flexibility of generative systems, a

concept which is expanded upon in the data: there is evidence that these systems can

perform multiple functions (Appendix A, lines 176 - 177), and form part of a hybrid system

with other adaptive music techniques (Appendix B, lines 146 – 147). Participant 2 also

alludes to a certain versatility by musing “there’s always a way to make it work with the

game” (Appendix B, lines 213 – 214). This is significant in that it implies a wholly generative

music system is not necessary; rather, it can work alongside other techniques, which


incidentally is how it was utilised in Participant 2’s game (Appendix B, lines 136 – 143 & 148

– 152). This theory would also support ideas in the literature of a hybrid approach (Philips,

2014).

However, it is apparent that there is an inherent complexity in the design. Participant

2 makes note of how “complicated” (Appendix B, line 160) their system is, as well as the

“challenging” nature of using them in general (Appendix B, line 203). This further

substantiates Philips assertions of complexity, suggesting that specialist expertise may be

required, and has ramifications for workflow: time spent navigating these complexities may

result in a composer spending less time on their musical duties.

4.2.3 Summary

In summary, the functionality of generative systems does appear to have implications

for the composer who utilises them. The fact that there are various phases throughout the

pipeline that necessitate human input, and the evidence that their complex nature may call

for a level of additional expertise, implies that composers may have to dedicate additional

time to learning these systems to operate them. This may also result in added responsibility

for the composer in terms of duties, although the modular nature of the systems would allow

for certain tasks within the pipeline to be delegated to other personnel; hypothetically, a

composer could focus solely on creating musical ideas and defining musical parameters

while a programmer handles the implementation.

4.3 Working with generative systems

Composer-centric data was collected when participants discussed areas that related to how

composers work with generative systems. Analysis revealed several key themes: areas

linked to desirable skills, such as personality traits and background; how these systems

affect a composer’s role, in terms of a possible redefinition of the role and the existence of

various support networks; and how they serve to function as a compositional aid, in terms of

workflow and as a tool for skills development. This is illustrated in Figure 5.


4.3.1 Desirable skills

In accordance with ideas in the literature that suggest the need for a more diverse skillset

(Minter, 2012; Young, 2012) there is evidence that a background in music, audio and

computers may be advantageous: both participants possess a musical education (Appendix

A, line 52; Appendix B, line 11) and prowess in musical technology and programming

(Appendix B, line 62 & 63). This would support the idea expressed earlier that specialist

expertise is important; both musical and technical knowledge are required skills, and as such

appear fundamental. Also noteworthy is the mention of various personality traits such as

versatility (Appendix B, lines 98 – 99) and statements that infer adaptability (Appendix B, line

128) and open-mindedness (Appendix B, line 99). This is perhaps reflective of the emergent

and still-evolving nature of these systems and implies that the composer using them must be

able to change along with it, or, as Participant 1 puts it, to “evolve further with our minds”

(Appendix A, line 278).

Fig. 5, thematic hierarchy of working with generative systems


This concept of embracing change is further implied by Participant 1 who, when

discussing other composer’s perceptions that generative music will “take work away” from

them, states that these views come from “mainly older composers [who are] not so much

interested in new things” (Appendix A, lines 116 – 117). The fact that this is attributed to

more experienced composers suggests that a willingness to experiment - rather than a

reliance on more well-established techniques and ideas - may be necessary, which in turn

has further implications: a change in attitudes and difference of approach may be required

when composing with these systems.

4.3.2 Role

In terms of how generative systems relate to a composer’s role, one unexpected area that

emerged was that of support networks. “Help” from colleagues was a prominent term

(Appendix A, line 58), as was working in “team[s]” (Appendix B, line 68) and the mention of

social networks (Appendix B, line 53). This would further support the idea of complexity, as it

appears a communal effort is necessary to overcome the inherent intricacies, and also bears

new meaning in light of the evidence of modularity and necessary skills: the presence of

other skilled individuals lends credence to the idea that elements of the system pipeline

could be delegated to other personnel, and this in turn may reduce the need for advanced

technical knowledge on the composers part.

Perhaps more critical was the inference that generative systems may redefine the

role of a composer. In this regard the notion of “fear” was evident; either the intimidation of

working with these systems (Appendix B, line 99), or the fear that composers themselves

may be replaced by them (Appendix A, line 116). This would indicate a somewhat negative

perception, however there is evidence to suggest that these fears may be unwarranted, such

as the fact that human input at each stage is essential and statements that suggest an

increased accessibility for composers “on all levels” (Appendix A, 196), regardless of a

traditional background or training (Appendix A, line 200 – 202). Furthermore, Participant 1

seems to challenge the preconceptions of the role:

“What does ‘a composer’ mean? If I compose a line of music, am I

composer, or do I have to go to school?” (Appendix A, line 194)

This is especially interesting as it questions not only necessary expertise but also appears to

prompt debate about the required level of input, which is further reflected in the data:


“…putting [musical ideas] into [generative system], [generative system]

creating something out of it, something new, something similar but

something else, and you use part of it in the game and have [generative

system] compose part of it. Some of it you don’t want to be touched and

you use it unchanged from what you composed, or you don’t use

[generative system] at all in the game…” (Appendix A, lines 151 - 155)

This illustrates the usage of generative systems as a spectrum, in much the same vein as

Pacaud & Wintory’s remarks about “degrees of interactivity” (2016, 35:15), that allows for

greater or lesser levels of input where one pole represents absolute human input through to

the other which represents absolute generative system output. Participant 1 also suggested

expertise may be a factor:

“…[if] you don’t have the background of a composer but your ideas are

better than a composer’s, then why shouldn’t you be allowed to – as a

programmer, maybe – to do music?” (Appendix A, lines 200 – 202)

When this is also conceptualised as a spectrum, and used in conjunction with the spectrum

of input, it can be theorised that the level of input is inversely related to the level of expertise

required; essentially, users with less expertise can obtain musical results with greater

Fig. 6, theoretical model of spectrum of input vs. spectrum of expertise


reliance on a generative system (see Figure 6). While it does not necessarily follow that

more experienced composers would utilise generative systems less, it does demonstrate

their accessibility for less musically trained users.

This issue of human input is also subject to a stark contrast of opinion: Participant 1

is firmly of the belief that a “human touch” is not necessary to “make music really interesting”

(Appendix A, lines 236 – 237), whereas Participant 2 states they are “convinced” that “we’re

never going to replace composers” (Appendix B, lines 263 – 264). These radically polarised

views were felt so significant that they warranted further examination, as will be apparent

hereafter.

4.3.3 Compositional aid

Another key topic was the concept of generative systems as a compositional aid, with

numerous references made to them being a “tool” (Appendix A, 141) or “instrument”

(Appendix B, line 119) that “help[s] the composer just compose” (Appendix A, line 131), with

the added bonus of facilitating skills development: Participant 1 refers to their system as an

“educational tool” that will “help [them] become a better composer” (Appendix A, lines 142 &

159). This suggests that rather than being a means to directly replace composers, as has

been discussed previously, generative systems are instead intended to be harnessed as a

tool which can help the compositional process. This is demonstrated in the data by various

issues relating to workflow, which were discussed frequently: benefits to time-efficiency

(Appendix A, lines 125 & 134 - 135) and inspiration (Appendix A, lines 136 & 141) were

noted, although one drawback is the limited sonic palette that composers may have to work

with (Appendix A, lines 245 – 246; Appendix B, lines 122 – 123).

Perhaps most noteworthy, however, is that despite evidence that indicates a different

approach or mentality may be required there is evidence that shows these systems still allow

composers to retain a more “traditional” workflow, thus contrasting the views of Hedges,

Larson and Mayer (2010, 31:00) that generative systems don’t “lend [themselves] to

traditional composition techniques”. Participant 2 claims they wanted the composer to “be as

true to his process as possible” (Appendix B, lines 118 - 119) and so minimised the extent to

which the system affected this; in essence, the workflow remained practically unaltered, as

only the MIDI files and audio assets were required, around which the generative elements

were designed (Appendix B, lines 136 – 143). This further corroborates the idea of

generative systems as a spectrum: in this instance the usage was relatively low when

compared to the system of Participant 1.


Furthermore, Participant 1 explicitly states that their system “lend[s] itself to

traditional composition” (Appendix A, lines 214 – 215) and mentions ongoing development of

a VST so that it may be used in standard DAWs (Appendix A, line 165). This integration of

generative system into common DAW represents an endeavour to make these systems

more accessible without the need for additional expertise, training or software; as such, it

represents Participant 2’s hope that using these systems becomes more streamlined

(Appendix B, lines 255 – 256).

4.3.4 Summary

In addition to substantiating some theories posited by the literature, the acquisition and

analysis of composer-centric data also revealed fresh insights. Musical and technical

knowledge appears to be important, as does possessing a versatile, liberal and progressive

mindset, evidenced by the need to embrace new technology and techniques. Also, the

existence of support networks may help to offset the complexity of these systems, as a

composer would be able to lean upon the expertise of others.

Perhaps most striking, though, was the other evidence presented. It is intimated that

generative music systems are viewed with trepidation by some composers, even as

something of a threat to their vocation, which may be the result of either unfamiliarity or

unawareness. The data goes some way to address these fears; in fact, one could almost

regard them as a misconception when taken in conjunction with the evidence that a degree

of human input is essential. Indeed, it could be posited that the phrase “generative music” in

its current guise is something of a misnomer, a blanket term for something more nuanced

and multi-faceted: rather than being a take-it-or-leave-it musical approach it can instead be

thought of as a spectrum, where it can be applied as much or as little as required depending

on the goal, and even used in conjunction with other adaptive techniques to support them.

Also, there is the notion that these systems are a “tool” or compositional aid that can

be utilised to expedite the compositional process, either to “free up time” so that more critical

aspects of the music can be focused upon (Appendix A, lines 125 – 126] or to provide

inspiration. This again points towards generative systems as a utility to be benefitted from

and would suggest the only implication for the composer would be a need to enact a change

in perceptions and embrace them. One could also argue that evidence indicating retention of

a more traditional workflow suggests minimal impact on the composer, and evidence of

integration into DAW software suggests a streamlining process that results in even greater

accessibility; as such, this “tool” could eventually could become as commonplace as the

digital EQ or reverb plugins that we use today.


4.4 Project specificity

One area which upon further scrutiny emerged as crucial was the idea of project specificity;

this general theme was apparent in the literature (Phillips, 2014; Rohrmann, 2015) and

became more obvious when examining the data. The fact that the two participants had

contrasting opinions on the level of input proved to be a catalyst for understanding this; to

understand this disparity the spectrum of input model was applied, and despite their high

levels of expertise (evidenced by their musical and technical backgrounds) they both sat at

opposite ends of the spectrum (see Figure 7).

Possible reasons for this could be the fact that their systems are very different beasts

- one harnesses melody and harmony, the other is predominantly rhythmic - and the games

they are implemented in are also different; one is an online multiplayer RPG while the other

is a single player action-adventure game. This suggested that the goals of the project itself

have a bearing on whether or not generative systems are used; the data itself also supports

this, as Participant 2 debates whether all “game[s] really need” generative music (Appendix

B, line 211), and states there is “a chance where it’s just not going to work with the game”

(Appendix B, line 247). Using this concept as a basis for thematic analysis it was found that

there are implications that extend to all other areas discussed previously.

Fig. 7, illustration of participants opinions on level of input


4.4.1 Goals of the game

First it is important to realise that there are various factors that determine whether

generative systems are implemented; this is illustrated in Figure 8:

As noted to section 4.1 there are various reasons why these systems are

implemented: game considerations, such as game characteristics and musical requirements,

and solutions to problems, such as resource issues. A game’s characteristics and musical

requirements can also be seen to interrelate, as genre and scale of a game would directly

affect the style and quantity of music. Similarly, the same can be said of resources; a large

game of a certain genre may require a greater budget. However, these aspects alone don’t

determine a system’s use: these all appear to be dependent on attitudes, as participant 2

notes that it “comes down to people’s motivation to use it” (Appendix B, lines 214 – 215),

particularly the inclination of the audio director or whoever is overseeing the project

(Appendix B, line 211).

Fig. 8, why generative systems are utilised


4.4.2 Aspects of generative systems

Also, when certain aspects of generative systems are factored in they too interrelate and

have a bearing; see Figure 9 below.

This illustrates the following points:

▪ Flexible design: the flexibility and modularity of their design, as detailed in section

4.2, has implications for resources. For example, the size of team may be a factor:

the modular nature would allow either a team to work on separate modules, or one

skilled member to work with all.

▪ Flexibility of use: concepts such as generative systems as a compositional aid, the

fact that it functions as a spectrum, and the benefits to improving workflow (see

section 4.3) have a bearing: games with differing musical requirements could

potentially use as much or as little generative content as desired. For example, a

game that required a large amount of music would be better suited to using more

Fig. 9, aspects of generative systems that affect why they are utilised


generative content than one that requires little. There may also be economic benefits

– both financially and in terms of manpower.

▪ Bespoke nature: the fact that this area is an emergent field is significant. It was

mentioned how there are currently few solutions available (Appendix A, lines 39 &

49) and the subsequent experimentation that is required as a result (Appendix A, line

137; Appendix B, line 77); this suggests that resources would have to be devoted to

developing a bespoke proprietary system in the first place.

▪ Technological limitations: memory constraints (Appendix B, line 120) and instrument

playback issues (Appendix A, line 245 – 246), would have a bearing. This impacts on

game characteristics: a large game of a certain genre may not allow as much

processing power to be devoted to the audio system as a smaller one, for example,

and this in turn means that they may be utilised less, or not at all.

4.4.3 Summary

There is some credence to ideas in the literature regarding project specificity, although more

thorough examination has revealed the extent to which this is the case: the existence of both

various game-specific factors and aspects of generative systems themselves. As to how this

informs the research question, the answer is somewhat simple but far-reaching: the

evidence that the use of generative systems is based on such a myriad of factors would

suggest the chances of a composer encountering one could vary wildly. The game, it’s

musical requirements, the size and finances of the development team, even the limitations of

current technology, all play a part in determining whether these systems are deemed

suitable. Perhaps most significant was evidence that it likely comes down to the preference

of whoever is directing the project; even if a game is perfectly suited to incorporate a

generative system, the fact that the audio director may be unwilling to use one, or even

unaware of their existence, would result in them going unused.

C H A P T E R 5 – C O N C L U S I O N | 30

CHAPTER 5

Conclusion

5.1 Research objectives and findings

In conclusion, the aim of this study was to explore the following question:


To achieve this, four objectives were pursued: to examine the literature and identify key

areas and research gaps, to gather qualitative data on both how generative systems work

and what it is like utilising them, and then to evaluate this data in terms of impact for

composers.

It was found that both system-centric and composer-centric aspects have

implications for composers. In terms of how they are designed, generative systems comprise

of modular parts wherein exists a pipeline that necessitates some degree of human input,

which may require additional computer and music skills to harness. More significant however

were composer-centric findings, both in terms of how a composer works and what their role

entails. New insight was gained into how generative systems can be viewed as a

compositional tool that aids workflow, improves compositional skills and be perceived as a

spectrum of input rather than a threat to composers. Additionally, being as the use of these

systems depends on various project-specific factors means that the chances of a composer

working with one may vary.

5.2 Limitations and further research

While this study was successful in generating new insights into the research area, it wasn’t

without limitations. It is worth noting that the study is representative of two participants

viewpoints, and as such cannot be generalised. Also, due to the emergent nature of this

field, and the scarcity of research available, there is a danger that this study attempted to

overreach in terms of data it collected. While a broader understanding was attained, deeper

C H A P T E R 5 – C O N C L U S I O N | 31

knowledge of actual working practices may be required; to this end, first-hand accounts from

composers who have worked with generative systems would be beneficial, or perhaps a

practice-led study that focused on composing generatively.

5.3 Final remarks

While the findings of this study indicate fresh, new insights for how composers can view

generative systems it is apparent that this is an area of subjectivity; this can be seen in the

polarised views of the participants regarding the necessity of human input to create music,

and in the fact that the decision to use them may be out of the hands of the composer

entirely. Indeed, speculation about the future of generative systems in games showed

uncertainty (Appendix B, line 202) and a general lack of consensus (Appendix A, line 233).

However, this can perhaps be attributed to the fact it is an emergent field with no

industry standard. It is clear this is an area with a lot of promise - both for games and other

applications (Appendix B, lines 282 – 284) - that could potentially change not only how a

composer creates music but how we view the role itself, and one which will hopefully

continue to be investigated.

B I B L I O G R A P H Y | 32

Bibliography

▪ Baldursson, B. and Olafsson, K. (2016). Intelligent Music For Games. In: Game Developers

Conference. [online] Available at: https://www.gdcvault.com/play/1023004/Intelligent-Music-

For [Accessed 24 Oct. 2018].

▪ Berndt, A. and Hartmann, K. (2008). The Functions of Music in Interactive Media. In Spierling,

U. and Szilas, N., editors, Proceedings of the International Conference on Interactive Digital

Storytelling, pages 126–131, Erfurt.

▪ Bush, Tracy W., Scott Gershin, Martin H. Klein, Adam Boyd, and Sarju Shah (2007). ‘‘The

Importance of Audio in Gaming: Investing in Next Generation Sound.’’ Paper presented at the

annual Game Developer’s Conference, San Francisco, March 4-9, 2007.

▪ Clark, A. (2007). Defining Adaptive Music. [online] Gamasutra.com. Available at:

http://www.gamasutra.com/view/feature/129990/defining_adaptive_music.php [Accessed 21

Oct. 2018].

▪ Collins, K. (2008). From Pac-Man to Pop Music: Interactive Audio in Games and New Media.

Aldershot, Hampshire, England; Burlington, VT: Ashgate.

▪ Collins, K. (2009). Game Sound. Cambridge: MIT Press.

▪ Cresswell, J. W. (Ed.). (2009). Research Design: Qualitative, Quantitative, and Mixed

Methods Approaches (3rd ed.). Los Angeles: Sage.

▪ Deriviere, O. and Kurlander, J. (2014). Remember Me: Behind the Unique Interactive

Soundtrack of the Future. In: Game Developers Conference. [online] Available at:

http://Remember Me: Behind the Unique Interactive Soundtrack of the Future [Accessed 22

Oct. 2018].

▪ Ell, K. (2019). Video game industry is booming with continued revenue. [online] CNBC.

Available at: https://www.cnbc.com/2018/07/18/video-game-industry-is-booming-with-

continued-revenue.html [Accessed 30 Apr. 2019].

▪ Elmsley, A., Groves, R. and Velardo, V. (2017). Deep Adaptation: How Generative Music

Affects Engagement and Immersion in Interactive Experiences. In: Digital Music Research

Network. London, p.8.

▪ Eriksson, O. and Lindau, P. (2013). Evaluating an Adaptive Music System in an Adventure

Game Environment. Södertörn University.


▪ Geelen, T.V. (2008). Realising Ground-breaking Adaptive Music. In From Pac-Man to Pop

Music: Interactive Audio in Games and New Media, ed. Karen Collins, 93-102, Burlington, VT:

Ashgate Publishing Limited.

▪ Hedges, J., Larson, K. and Mayer, C. (2010). An Adaptive, Generative Music System for

Games. In: Game Developers Conference. [online] Available at:

https://www.gdcvault.com/play/1012710/An-Adaptive-Generative-Music-System [Accessed 22

Oct. 2018].

▪ Hébert, S., Béland, R., Dionne-Fournelle, O., Crête, M. and Lupien, S. (2005). Physiological

stress response to video-game playing: the contribution of built-in music. Life Sciences,

76(20), pp.2371-2380.

▪ Iwamoto, S. (2017). Epic AND Interactive Music in 'Final Fantasy XV'. In: Game Developers

Conference. [online] Available at: https://www.gdcvault.com/play/1023971/Epic-AND-

Interactive-Music-in [Accessed 23 Oct. 2018].

▪ Jorgensen, K. (2006). On the Functional Aspects of Computer Game Audio. In Proceedings

of the Audio Mostly Conference, Piteå.

▪ Lamperski, P. and Tahouri, B. (2016). Real-time Procedural Percussion Scoring in 'Tomb

Raider's' Stealth Combat. In: Game Developers Conference. [online] Available at:

https://www.gdcvault.com/play/1023215/Real-time-Procedural-Percussion-Scoring [Accessed

22 Oct. 2018].

▪ Lawrence, D. (2012). The Effect of Musical Tempo on Video Game Performance. Ma.

University of Jyväskylä.

▪ Levy, L. (2015). The Effects of Background Music on Video Game Play Performance,

Behavior and Experience in Extraverts and Introverts. Ma. Georgia Institute of Technology.

▪ Minter, N. (2012). The Application and Integration of Adaptive Music in Video Games. Ma.

Savannah College of Art and Design.

▪ Pacaud, C. and Wintory, A. (2016). Waltzing with Blades: The Music of 'Assassin's Creed

Syndicate'. In: Game Developers Conference. [online] Available at:

https://www.gdcvault.com/play/1023230/Waltzing-with-Blades-The-Music [Accessed 23 Oct.

2018].

▪ Peerdeman, P. (2010). Sound and Music in Games.

▪ Phillips, W. (2014). A Composer's Guide to Game Music. Cambridge: MIT Press.

▪ Pidkameny, E. (2008). Sound in Video Games. In M. J. Wolf, The Video Game Explosion (pp.

251-257). Westport, CT: Greenwood Press.


▪ Plans, D. and Morelli, D. (2012). Experience-Driven Procedural Music Generation for

Games. IEEE Transactions on Computational Intelligence and AI in Games, 4(3), pp.192-198.

▪ Prechtl, A. (2016). Adaptive Music Generation for Computer Games. PhD. The Open

University.

▪ Rohrmann, C. (2015). Creating Hyper-Adaptive Music on an Indie Budget. In: Game

Developers Conference. [online] Available at:

https://www.gdcvault.com/play/1021886/Creating-Hyper-Adaptive-Music-on [Accessed 24

Oct. 2018].

▪ Sweet, M. (2016). Top 6 Adaptive Music Techniques in Games - Pros and Cons. [online]

Designing Music NOW. Available at:

https://www.designingmusicnow.com/2016/06/13/advantages-disadvantages-common-

interactive-music-techniques-used-video-games/ [Accessed 30 Oct. 2018].

▪ Thomas, C. (2017). Composing Music for Games. Milton: Taylor & Francis.

▪ Van Nispen tot Pannerden, T. et al. (2011). The NLN-Player: A System for Nonlinear Music in

Games. In Proceedings of the International Computer Music Conference 2011. International

Computer Music Conference 2011. University of Huddersfield: International Computer Music

Association.

▪ Velardo, V. (2017). What is Adaptive Music?. [online] Melodrive Blog. Available at:

http://melodrive.com/blog/what-is-adaptive-music/ [Accessed 27 Oct. 2018].

▪ Velardo, V. (2019). Audiokinetic Blog. [online] Blog.audiokinetic.com. Available at:

https://blog.audiokinetic.com/players-have-spoken-video-game-music-must-be-adaptive/

[Accessed 30 Apr. 2019].

▪ Wang, A. (2015). Generative Music and Video Games – Five College Digital Humanities.

[online] 5colldh.org. Available at: http://5colldh.org/generative-music-and-video-games/

[Accessed 1 Nov. 2018].

▪ Whitmore, G. (2014). Peggle 2: Live Orchestra Meets Highly Adaptive Score. In: Game

Developers Conference. [online] Available at:

https://www.gdcvault.com/play/1020240/Peggle-2-Live-Orchestra-Meets [Accessed 25 Oct.

2018].

▪ Young, D. (2012). Adaptive Game Music: The Evolution and Future of Dynamic Music

Systems in Video Games. BSC (Hons). Ohio University.

▪ Zhang, J. and Fu, X. (2015). The Influence of Background Music of Video Games on

Immersion. Journal of Psychology & Psychotherapy, 05(04).

APPENDIX A

Coded interview with Participant 1, carried out on 7/3/19. Duration 39 mins 18 secs.

[Beginning of recorded material] Code

A P P E N D I X A | 35

Interviewer: So, could you start with telling me a little bit about yourself? Just a 1

little bit about your personal background, if you’ve come from a musical 2

background, things like that? 3

Participant 1: Right, so my background lies in computers and music, obviously, like 4

most people that are in audio, at least for games. I studied computer engineering 5

and music, and I worked as all kinds of things: radio, sound design, in the 90s I was 6

doing 3D audio. Not as a programmer, per se, or doing V.R.M.L, but just as a sound 7

designer. More like that, with programmers. Actually, before that I studied in the 8

U.S for three years: recording engineering. I wanted to be… well I wasn’t sure what 9

I wanted to be! I guess going there was like an excuse to go out of Iceland for a bit. 10

So while there I got, not so much… this is in the early 90s, we weren’t really doing 11

much with computers at all then. We were using computers in the studio, but that 12

was about it. When I came home the internet was coming in fast, and technology 13

was growing really fast so I got interested in sound for virtual reality, which actually 14

died around 2000. We sort of forget about virtual reality before 2000 actually; 15

computers weren’t fast enough. 16

So my background is music: I’ve played in, like all of us have, bands (electronic and 17

other kinds of bands), and I’ve been interested in synthesisers and stuff like that. 18

But then I had a family, I couldn’t really support the family with music and doing 19

stuff like that. I ran a recording studio with an SSL and all the bells and whistles, 20

and that didn’t work here in Iceland because it’s just too expensive, there’s too few 21

people here. So I started working in the computer business just a salesman, and 22

that was around 2005-2007, and it was quite nice actually. It was a good time be 23

alive, with a family and a business like computers, so I forgot about audio for a 24

while. 25

I came back to it in 2008 when I joined [company]. When I joined [company] I 26

joined as an audio director… they wanted to make audio more important in the 27

game than they had, so they hired me for some reason! When I came in, in 2008, I 28

saw this game, [video game], that had… it was an MMO and it didn’t really have… 29

well it did have some sort of audio engine from here and there, just stitches from 30

here and there, they weren’t really doing it like they should have really. So I saw an 31

opportunity to use this new system called Wwise, I’m sure you’ve heard about it, as 32

a middleware engine and we started using that in 2009. But really soon I started 33

thinking about the music, how could we solve that problem? Because we had this 34 Reasons systems are used:

Solution to problem

Reasons systems are used:

MMO

Additional skills:

Prereq. For game audio

Additional skills:

Background in computers

Additional skills:

Background in music

Additional skills:

Background in audio

Additional skills:

Background in audio

Additional skills:

Background in music

Additional skills:

Interest in technology

Additional skills:

Background in audio

Additional skills:


Specialist role


Immersion – player exp.

Audio engine inadequacy


game that was happening in real-time but the music wasn’t happening in real time. 35

There was no story, and why should the music be written before the story 36

happened? It didn’t really make sense to me, so that was the main reason why I 37

started looking for a solution for that. 38

So I started looking and I didn’t really find much, I found some examples of real-39

time music on the net, and a couple of interesting ideas. Actually, the only real 40

attempt at this was Electronic Arts with Spore, I’m sure you’ve heard about that, 41

with Brian Eno and six programmers… really expensive, I didn’t have the budget for 42

that obviously. So that was an interesting experiment I think, but it was really 43

simple though; I looked at it and listening to it and it was really… well, I’m a Brian 44

Eno fan like many of us are, but it was really simple like they were just using the 45

white notes on a piano. It was a totally fantasy world, with fantasy characters, and 46

it was easy to make something out of that. Well, not easy but not a huge problem, 47

so it kind of worked. 48

So I couldn’t find a solution out there, and I guess there isn’t a solution out there 49

yet, this is in 2009. So I thought about doing this myself instead of finding someone, 50

and just attacking the problem myself. I found… well, I didn’t find [colleague] right 51

away. I went to the music school here, I did my Master’s in composition and had 52

[colleague], as my mentor. He introduced me to his system, [system], and I started 53

experimenting with that and working with him on possible solutions for this. And 54

more and more I became aware that this might be a solution, something like 55

[system]. 56

So yeah, that’s really… when I graduated in 2011 I admitted this would be the thing 57

to look at, [system], and maybe help [colleague] (with him helping me as well), 58

taking this forward. We started getting grants from the government here, I helped 59

him get grants to work on this, while I continued work on [video game]. And he has 60

been working ever since on this, and today we have a programmer here in-house 61

working on the system… it’s really [colleague’s] product still, it’s his product it’s not 62

our product. But he has a programmer inside [company] working on the project, 63

creating a plugin for Wwise and right now it’s running in-game composing music. 64

And that’s a first I know of, at least, that’s in a game. Maybe there is another 65

solution, probably is, but I haven’t heard of it. 66

Interviewer: I see. So, just returning back to when you were talking briefly about 67

the decision to move from using Wwise into exploring what [system] could offer, 68

what was it about the game itself that prompted you to make that decision? 69


Immersion - reflect visuals


Narrative – lack of

Reasons systems are used: Expense/small budget


Convenience/personal (?)

Reasons: Solution to problem

Emergent: underdeveloped


Solution to problem


Solution to problem

Emergent area: Underdeveloped

Emergent area: Potential

Emergent area: Experimental



Fantasy game

Additional skills:

Background in music

Help and support

Emergent area: Still developing

Help and support

Emergent area: Experimental


Solution to problem



Participant 1: Well, mainly that… So, we had this huge game. And, of course, we 70

great system that is Wwise. And we can create music, many hours of music, and we 71

can just play it randomly in the music system in Wwise if you know how it works, 72

and it can work really well. However, since I was… the audio department here is 73

really small, it’s just me and the programmer. Actually, there’s a team in London 74

now helping me but back then it was just me, and one programmer. I couldn’t see 75

how I would find the time to make a bunch of music, hours of music, to play in the 76

game. So first of all that, and so that’s why I thought “well, couldn’t there just be a 77

system that could compose music for me?”. Secondly, the game didn’t have a story; 78

how do you compose music for a story that hasn’t happened? So that was the real 79

idea, that was real recent… I started searching for a solution because if there isn’t a 80

story what do you do? You compose music to maybe dangerous situations, and use 81

it as you need? But to me that was, obviously a possibility and we have done that, 82

but it’s not perfect and I just want to make this perfect. So that’s why. 83

Interviewer: I see. So, can we talk more in-depth about the [system] system itself? 84

You’ve mentioned that it was designed by [colleague], can you tell me a bit about 85

how it works within your project? 86

Participant 1: So, I cannot tell you the details of the system, since I’m not the 87

programmer behind it. But how it works is that we have a brain, a [system] brain, 88

that can be situated anywhere, and we have a program called [system] Composer. 89

You as a composer, you sit down with [system] Composer, you write an idea, like 90

an 8-note idea. Really simple, it can just be a few notes. And you tell [system] what 91

to do with these notes, what it can do and what it can’t do, and how long it should 92

be, and so on and so forth. What you do is then after using Composer is you export 93

it, export it as a cell – we call it a cell – and import it into the plugin in Wwise. In 94

Wwise the plugin will use the cell that you created in Composer and it will compose 95

how ever long you want. You want it to be a year? A day? Whatever. Forever, 96

maybe? So it’s not really… I don’t have the nitty gritty of the system itself, but you 97

can see a lot about it on the homepage of [system] and I can send you some more 98

info if you like. I know he was using, if you’re interested in math and stuff like that… 99

if that’s what you need? Is that something you think you’d…? 100

Interviewer: I think so, yeah, that could potentially be quite useful, yeah. 101

Participant 1: I can ask him to provide me with all of the info that he has on 102

[system], all of how it… well, I know he can’t tell you everything but I’m pretty sure 103

there’s some information you can benefit from really. I can send you the 104

documents. 105


Scale of game


Small audio dept.


Time restraint


Narrative – lack of




Solution to problem

How systems work:

Hum. agency – musical input

How systems work:

Define musical parameters

How systems work:

Modular

How sys. work: Musical inpt.

Types of system: Melodic

How systems work:



Lot of music


Lot of music


Inadequate alternatives

How systems work:

Modular

How systems work:

Flexible


Interviewer: Yes, thank you, that would be really useful! So, as you briefly touched 106

upon there, when using this system the composer writes the initial ideas and the 107

rest of the music is generated based on these musical ideas. I know in your 108

[presentation] you discussed how the composer sets the various intervals they’d 109

like to use, things like that, so with regards to looking at generative music from a 110

composer’s perspective how do you feel would affect their role, the traditional 111

composer’s role if that makes sense? 112

Participant 1: We’ve been obviously to [conference] with this, we’ve been to 113

[conference] in Brighton with this, showing it off to various people. We’ve met 114

various people, various composers, and we’ve had quite a lot of scared composers! 115

That this will take work away from them. These are mainly older composers that 116

are not so much interested in new things, but also younger composers, and then 117

we’ve had people who are just generally interested in whatever is there. I would 118

say that if you play games… do you play games? 119

Interviewer: I do, yes. 120

Participant 1: So you know that 70% of game music is exploration music? You play 121

Fallout, for example, the majority of it is just you walking somewhere in a desert or 122

whatever, wherever you are, and you’re hearing some exploration music that’s just 123

ambient. So I would say that this would greatly benefit… a system like this would 124

make the game a better game as it would free up time for the composer to write 125

more heroic things for the game, or cutscenes, or whatever you use in the game, 126

creating more good stuff for the game and making it a better experience, I think. 127

Because usually there’s only a certain budget for music in games, and it’s usually… 128

well, not always, but very often it’s not very high it seems. So taking that budget 129

and really making the most of it, a system like [system] can only benefit the game; 130

apart from making more important art or music for the game to make the 131

experience a better one for the player I think that also having a system like this, a 132

tool like [system] Composer, will help the composer just compose! Imagine putting 133

in an idea and making a fugue in a minute instead of two weeks, writing it out for 134

all the instruments and everything. You can use your time more efficiently, 135

obviously, and it will give you new ideas. 136

I always, I try to find new ideas using […] or Max/MSP or whatever, I tried all of 137

these systems and there’s a lot of good stuff that people have done with this. All of 138

it… I mean most of it is not extremely useful, but all of it is very interesting and 139

gives me ideas, and that’s very valuable. So even if you don’t use [system] live in 140

the game, having a tool like this that gives you ideas on this is extremely important, 141


Small budget





How they affect role:

Perceived negative impact


experience

Compositional approach:

Time-efficient


Time-efficient


Inspiration


Inspiration


Compositional aid


Compositional aid


Type of music needed


Type of music needed



not only for games but just for music. So I guess I’d say it’s also an educational tool 142

that will help the composer, all composers. But with games, I mean we’re talking 143

more quality in the game, a better experience for the player, I would say. 144

Interviewer: It’s almost like a hybrid approach then, if I’m understanding you 145

correctly, like the composer hypothetically could write the music in a more 146

traditional sense, and the generative system will complement that almost, like the 147

best of both. 148

Participant 1: Yes, I absolutely agree with this. So I think that’s extremely exciting: 149

having the composer sitting down with his favourite sequencer creating music, 150

taking the […] created with the sequencer, putting it into [system], [system] 151

creating something out of it, something new, something similar but something else, 152

and you use part of it in the game and have [system] compose part of it. Some of it 153

you don’t want to be touched and you use it unchanged from what you composed, 154

or you don’t use [system] at all in the game but you just have these ideas and you 155

continue using [system] just to help you compose. Or you can go the other way and 156

only use [system] to compose, I mean it’s just… I think there’s no one way to use it. 157

It’s just a tool that will help make the experience a better one for the player. I 158

mean it will help the composer become a better composer as well, I think, and I 159

think in the end it will help the game to become a better game, and the experience 160

will be a better one for the player. That’s the end goal, I think, and that’s what 161

we’re trying to do. 162

It’s kind of funny that you contacted us now because we are really making a lot of 163

good progress with this right now. We have this running in-game as a plugin in 164

Wwise, they’re starting to create a VST plugin with it so you could use it in any… 165

like Cubase, or whatever, and it’s there. It’s very amazing that you contacted us 166

now because it’s starting to move. And you can actually go to App Store and 167

download [system] Composer, a free version of it, to try it out and there’s always 168

some new stuff, we’re working on some new… But I mean when the VST plugin is 169

out you kind of don’t need the Composer, you can just use the plugin instead. But 170

now we are working with a Danish composer who’s coming in here, he’s going to 171

help us compose some new music. I think it’s really important to get someone new, 172

some new blood in this, and new ideas running with this work, and he’s going to 173

help us compose some […] music, some parts of the game are really dark and it’s 174

really unknown, the territories, and we think the music like clusters will work really 175

well. We’re thinking of using [system] either to make the clusters, or to make 176

melodies over clusters that we make, we don’t know. It can be either one. 177








Educational

Reasons systems are used (?):

Additional variation


Inspiration


Compositional aid

Compositional Approach:

Educational


Traditional workflow


Emulate trad. workflow (?)



How systems work:

Limited functionality (?)

Emergent area: Still developing


Help and support

Help and support

Help and support

How systems work:

Flexible


[System] is not, I would say, an answer to everything. It’s a tool to accompany 178

whatever you do, at least for us. And then there are… I was talking the other day to 179

an indie gamer, who’s making indie games really on a tight budget, and it’s really 180

difficult for them to find music. For them to find quality music that they can use, 181

and it’s really expensive to get someone to compose something decent, so I think 182

this would help that group really, a system like this that can compose something 183

simple, something decent for a small indie game. 184

Interviewer: Briefly going back to what you said about developing [system] as a 185

VST, that’s really interesting. I feel that… from some of the research that I’ve found 186

previously, they mention that these procedurally generative systems can 187

potentially blur the line between being a true composer and a programmer, if you 188

understand, whereas just using it as a VST in your DAW would allow you to retain 189

that more traditional composer role without going too deep into learning the 190

programming system per se. 191

Participant 1: I mean, so if we go back to the problem, if we can talk about it as a 192

problem, with composers being afraid. They’re debating how they should market 193

this. Is this a tool for composers? What does a composer mean? If I compose a line 194

of music, am I a composer, or do I have to go to school to become a composer? I 195

think the idea is to say this is a tool for composers on all levels, something like that. 196

I don’t think we should tie this to… just aim at composers who’ve gone to school 197

and learnt all the tricks in the book, because why should we? If you can make 198

something that works, and you’re just interested and you have some great ideas 199

but you don’t have the background of a composer but your ideas are better than a 200

composer’s, then why shouldn’t you be allowed to – as a programmer, maybe – to 201

do music? That’s my stance on this. 202

Interviewer: I see. There’s an argument… a source, when discussing their own 203

system in a similar vein to this, they talk about… well, in their own words: “doesn’t 204

lend itself to traditional compositional techniques” so therefore they had to find a 205

composer who was comfortable writing in a non-traditional, non-linear fashion. 206

What are your thoughts on that argument? 207

Participant 1: Where is that document from? 208

Interviewer: It’s from another [presentation], from Hedges, Larson and Meyer. Let 209

me see if I can find the source itself… 210

Participant 1: Can you repeat the argument? I sort of lost you a little… 211


Small budget


Expense


Compositional aid

How systems work:


How systems work:





Compositional aid


Accessibility


Accessibility


Redefine role


Interviewer: It’s talking about how their system “doesn’t lend itself to traditional 212

compositional techniques…” 213

Participant 1: Right, right, yes. So I mean, [system] actually does lend itself to 214

traditional composition. It uses old methods in the background to compose music. 215

It relies on… it uses the whole orchestra for orchestration, really. I don’t think… I 216

think we will have to maybe readdress some of these methods for [system] in the 217

future. I don’t know for sure but something tells me we might, because things 218

change, how music is made and how everything is made. We’re not afraid… I’m not 219

afraid… I don’t there should be one set way of doing anything. 220

I think it’s great there’s a system that doesn’t do that, like the one you were telling 221

me about, that doesn’t have the… that’s just great. I think that’s awesome. I think 222

that even composer that only knows how to traditionally… they should really try to 223

do it untraditionally. It’s gonna make them better composers, as well. They will be… 224

they greatly benefit from something like that, I think. That’s my stance on it. 225

Interviewer: Thank you. So, moving on to the future, if you like… a lot of sources 226

herald this procedural type of music generation as being the future of 227

interactive/adaptive techniques in general. What do you feel about that? 228

Participant 1: I was discussing this… not quite this, but something similar. We were 229

just up in the… we were just having our lunch and we were discussing music, me 230

and two programmers. Me, and these two programmers are also musicians. We 231

were discussing generative music, and machine-made music, and I was the only 232

one… I’m not a programmer, who was sure that machines would surpass humans in 233

creating music, new music. I was the only one. Even the programmers didn’t think 234

that would happen, they thought that a human touch would always… that’s what 235

they said, yes… a human touch would always be necessary to make music really 236

interesting. I disagree… I disagree, I don’t think that would be the case. 237

Why do I say that? Well, I mean we are evolving extremely fast, faster and faster 238

with all technology I think… and the generative music I’ve heard… there are all 239

kinds of algorithms out there. Just using [system] and having [system] compose 240

something new is more interesting, at least for me, than the majority of the music I 241

hear on Spotify. I have a really difficult time finding something interesting on 242

Spotify, or something, I don’t know why. I think it’s just how I feel about this. We’re 243

getting breakthrough after breakthrough now with [system] composing some new 244

stuff. I mean there are some obstacles to get over, for example dealing with 245

sounds, like instruments. So if you have Beethoven’s 9th Symphony played by a 246



Emergent area:

Still developing


Accessibility

Compositional Approach:

Educational

Future:

Lack of consensus

Future:

Human Input necessary

Future:

Human input necessary

Future:

Technology

Future:

Technological limitations

Future:

Uncertain

Future:

Uncertain



How systems work:

Flexible

Team

Additional skills:

Background in music/comp.

Future:

Progression


Emergent area:

Still developing


computer and MIDI, you wouldn’t think much of it would you? That’s fact. So we 247

have that problem to solve. That’s a huge problem. But once we’ve solved all these 248

problems of computers being able to play instruments like humans, and they will in 249

the end I think… I don’t see a reason why computers or machines would be any 250

worse than humans, creating music. 251

I think it’s sooner rather than later… I mean, I know we’re awaiting… we’ve been 252

talking about flying cars for many years, we still don’t have them in 2019, if you 253

watch Bladerunner… actually we do have flying cars, but it’s just too dangerous… 254

But I think that generative music is interesting and it’s going really fast. I had this 255

guy who was into machine learning in Canada, he came in for a visit, and he’s 256

creating a system to mix tracks together called RaveDJ, you heard of that? 257

Interviewer: No, no… 258

Participant 1: It’s really interesting, he’s all into math and algorithms and stuff like 259

that. I understand like… only 10% of it, it’s really complicated. But one of the 260

solutions he might have with this machine learning thing is creating… having a 261

computer play instruments in real time. That’s one of the things an algorithm could 262

solve. So I don’t have all the answers but it’s happening really fast now, and I think 263

it will happen in the next ten years. In the next ten years we will have a computer 264

playing without flaws. That’s what I believe. 265

Interviewer: A lot of the challenges… they sound like they’re to do with 266

technological limitations and things like that, so it’s almost as if you’re saying that 267

as the technology grows these limitations become less of an issue and that will 268

allow these systems to thrive. 269

Participant 1: So you’re saying as technology evolves it will be less of an issue to… 270

can you repeat the last thing you said? 271

Interviewer: As the technology evolves, as you say, it will allow these generative 272

systems to become more popular, would you say? Or more accessible, or more 273

sophisticated and effective? 274

Participant 1: Absolutely, I think so. But I don’t think it will ever stop, I think we will 275

continue evolving forever. I also think that once people understand that that’s 276

going to happen it’s going to also involve people… your mind, along with the 277

systems. I think we will be forced… we will be happy to evolve further with our 278

minds, creating music, creating all kinds of art with machines. I don’t think 279

machines can… well, yeah, maybe machines can do it on their own, but I don’t 280

Future:


Emergent area:

Potential

Future:

Progression

Future:


Future:

Progression

Future:

Progression

Future:



How systems work:

Complexity

Future:

Human element

Future:

Adapt

Future:

Automation


think that’s gonna be a problem really, it’s going to be awesome I think. It’s gonna 281

be great. I think we’re going to be one happy family, one big happy family, with us 282

and all the machines. 283

Interviewer: Right, thank you. Well, I think all my topics are covered, so unless 284

there’s anything you’d like to add that we’ve not discussed…? 285

Participant 1: So you’re investigating this as a musician, or a programmer, or what? 286

Interviewer: Yes, as a composer, yeah, musician. Just seeing what sort of… not 287

impact… but seeing how these systems would affect a composer who’s working in 288

the video game industry, or wanting to move into it. 289

Participant 1: I find that having a system like this, which is amazing by the way, 290

and… people don’t realise how amazing it is. If it doesn’t… if it’s composing on it’s 291

own, and if people don’t understand the composition no-one’s going to notice it. 292

So I think once we get the systems, these machines, to compose something that 293

people understand… normal people, not like us maybe… that normal people 294

understand, then it will really take off I think. But until then we just continue. 295

Interviewer: Thank you very much. 296

[End of recorded material]

Emergent area:

Potential

Future:

Balance

Emergent area:

Potential

Emergent area:

Unrecognised

Emergent area:

Potential

APPENDIX B

Interview with Participant 2 carried out on 7/3/19. Duration 31 mins 33 secs.

[Beginning of recorded material]

A P P E N D I X B | 44

Interviewer: Could you being by telling me a little bit about yourself, like your 1

background? Have you come from a musical background, for instance? 2

Participant 2: Yeah, so in undergraduate I was doing a psych degree, a psychology 3

degree. I guess... towards the middle of it I did not like it, and didn’t really know 4

what I wanted to do and tried some other majors, and I kinda stumbled into doing 5

a basic music theory course and I just loved it. So I fell in love with studying music 6

academically… I was always, since 11, I fiddled around… I had a guitar, and I would 7

play guitar, all of it was very informal, no lessons, very experimental. It was just 8

something I did, and then studying it formally made me feel… like this just made 9

sense, this is something I want to do. I did an undergrad degree in music, and then I 10

immediately went to grad school after that undergrad degree and did a Master’s in 11

music. That’s where I was at [institution], and they have… I didn’t really even know 12

that they had a really good procedural music… it’s called an A.I and Algorithmic 13

Music Doctoral Program, and they have really great people there like [colleague] 14

and [colleague], [colleague]… people like that. I didn’t really know what I was 15

getting into, I just knew I wanted to keep doing music school. I wanted to go to 16

California, so I just applied to some of these schools that my professors were 17

recommending and that’s how I wound up at [institution]. But, man, it was just 18

fantastic for studying things that… studying algorithmic music, and they have great 19

studios, and teachers and all that stuff, so I lucked out. 20

And I was really interested in that too, because I was more interested in the 21

electronic stuff. I did composition in undergrad so I wrote pieces for chamber 22

music, but I loved the electronic stuff more, and working with the machines. So 23

yeah, [institution] was a great fit, and that’s where I met [colleague], who you 24

should… you should definitely reach out to [colleague], he’s very cool. He’s the guy 25

who wrote the code for the system, and I can give you his email address, he’s 26

always open to talk and he’s really a cool guy. He did his doctorate at [institution], 27

so I was doing a Master’s, I met him… I was doing a lot of stuff with Max/MSP and 28

real time interactive music stuff… and I graduated, and decided I didn’t want to 29

keep doing academia because it didn’t seem like a lucrative financial, realistic kind 30

of thing. It just felt like that getting an actual job was really hard at that time, that 31

was during the 2008 financial collapse and it was very… things were looking… well 32

it was 2010, but that was not too far after the recession. So we were in a recession, 33

2010, and I remember thinking if I keep spending more time doing this doctorate or 34

something it’s gonna… I’m never gonna get anywhere! 35

Additional skills:


Additional skills:

Educated in generative music

Help and support

Help and support

Additional skills:


Additional skills:

Background in music

Additional skills:

Background in music

Additional skills:

Experimental

Additional skills:

Background in music

Additional skills:


Help and support

Add. skills: Interest in tech.

Add. Skills: Back. In comp.


It’s not for everybody - academia - some people they can’t think of anything else 36

and there’s nothing wrong with that, but I wanted to apply what I had learned and I 37

wanted to get paid well for it, so the closest thing to solving that problem was the 38

game industry, where you won’t make a lot of money at first, but if you get lucky or 39

you hang in there then there’s a chance that you can do alright. And you can also… 40

the more important thing with being able to apply what I had learned in school, 41

and a lot of it was this algorithmic, technical music and sound design stuff. I was 42

looking for almost two years trying to get a job in the industry, and I have a brother 43

who worked at the company that I now work at and even that wasn’t enough, it 44

was actually really difficult. I was friends with all the people, it wasn’t till I met 45

someone on the audio team, [colleague], who was nice enough to meet with me 46

and we had lunch, and I showed him some of the procedural music stuff that I had 47

been working on at that time. He was really interested, he was interested in the 48

idea of procedural music for games before he met me, so that was a catalyst to… I 49

think a year and a half after that lunch was when I got an internship, so it was like a 50

very long process of trying to figure out how to get paid to do the things that I’m 51

interested in. It works out eventually if you just stick with it, but luckily through 52

connections… my brother, and [colleague], and [colleague] who was working with 53

me at the time. He was still in school, he was still doing his dissertation on 54

procedural music. So all these great people and supportive things… I eventually 55

landed this internship thing and I’ve been at [company] for about seven years, I 56

think at the three or four year mark we had gotten the engine shipping in [video 57

game]. 58

Interviewer: So you mentioned your process for breaking in, have you had any 59

other experience with systems like this? Since, or perhaps prior to this? 60

Participant 2: Yeah, I always have a little side project at home that I work on. I built 61

some synthesizers in Max… I did a lot of work in Max, I built a visualiser using Jitter 62

that would visualise FM synthesis. Then I transitioned into Pure Data and I built this 63

really cool sequencer that I was gonna try and launch in an app which never… 64

didn’t finish it, but we made an album with it. I always have like little things… 65

synthesis is big, I’m getting into Python now. I think it’s good to have little creative 66

outlets that nobody can have a say in, for me personally, because when you’re 67

working on a big team or even on a thesis there’s usually somebody directing, to 68

some extent. That’s great too, but I just like to have my own little project, stuff to 69

do in the evenings. 70

Help and support

Help and support

Additional skills:

Background in music

Additional skills:




Additional skills:


Additional skills:


Team

Additional skills:


Additional skills:



Interviewer: Yeah, yeah. So, can we move on to talk a little bit about the project in 71

question that you spoke about in your talk, [video game]? What specifically was it 72

about the game that promoted the decision to use that system? 73

Participant 2: Yeah, the decision to use that in the game… well, one: like I said, 74

[colleague] at the time was really into it, and I remember at the time we were 75

figuring out how to… how do we bring procedural music into [video game]? That 76

was the question. At the time I was just playing around in Max with a Markov chain 77

program that was on the internet that I copied and pasted… I think it was on The 78

Algorithmic Composer.com, if that’s even around, but I was like that’s neat! 79

Markov chains for music have been done for billions of years, you know for a long 80

time, so it wasn’t a revolutionary idea – I just wanted to try it. So I tried it and put a 81

Debussy piece in, and a Bach piece, and I played it for [colleague] and he was like 82

“oh yeah, that kinda works, it kinda sounds like it”, but he had a great insight and 83

he said let’s just focus it on rhythm, because it’s getting the rhythms much more 84

correct than the pitches. And he’s like we can also just focus it on one little aspect, 85

and that was a great idea… that was… from the kind of thing that I presented he 86

was able to direct it. And then from there… it just… that was like, how we were 87

gonna use it was gonna be this combat thing where the percussion would go from 88

low intensity to high intensity, and that was kind of the initial pitch… it grew a lot 89

from there. 90

Interviewer: Ok. I remember you talking about… in your presentation, how 91

because of the ever changing combat and the multitude of different states that… 92

that was one of the biggest factors in deciding to focus on the rhythmic elements. 93

Was this already predetermined before [composer] was brought on board? 94

Participant 2: I think yeah… I can’t remember. I know [composer] and [colleague] 95

are good friends, they knew each other before I was in the picture or so. I think 96

[colleague] always had in the back of his mind “here’s a great guy I know for [video 97

game]”, and [composer] is just a fantastic composer to work with because he’s very 98

flexible and open minded so this didn’t scare him… if it did, he didn’t show! It was 99

great… I’m pretty sure… it wasn’t like [composer] was involved in whether or not 100

we would use it, he didn’t have much input. It was just “we’re gonna use this 101

system” and then we were looking for a composer, and we decided on him, and 102

that was just part of the deal which they were aware of. 103

Interviewer: I see, that’s interesting. 104



How systems work:


Challenge

Emergent area:

Experimental/Musical input

Emergent area:

Experimental

How work: Ltd functionality

Reason: Ltd functionality

Types of system:

Rhythmic


Reflect visuals

Role:

Fear


Pers. Trait – open minded


Pers. Trait - versatility

Network


Participant 2: But I will say, real quick, with the different states… that idea came 105

about later with [colleague’s] interpretation of the solution. The original idea was 106

just to have two states: low and high intensity for percussion. So you’d have two 107

states, and [colleague] turned that on it’s head by making it possible to author as 108

many states as you want. I just wanted to throw that out. 109

Interview: Yeah, thank you! Just going back to the point you made about 110

[composer’s] versatility and flexibility… something I’ve found in my research, hold 111

on I’ve got a quote here… from Hedges, Larson and Mayer. It was a talk they gave 112

at GDC about a similar sort of system, and to quote them it says when using their 113

system it “doesn’t lend itself to traditional compositional techniques” and 114

therefore they had to “find a composer who was comfortable with non-traditional 115

and non-linear methods.” So, how does that apply… does that argument apply to… 116

Participant 2: I think… our goal with the project was not to disturb… we didn’t want 117

to disturb the composer at all. We wanted him to be as true to his process as 118

possible, and not have to have him learn a new instrument, so I guess it’s different 119

in that sense. But the versatility comes from where we had memory constraints 120

and we had to remake these instruments he was picking to be mono, to be sixteen 121

bit, and to be severely limited in their variations. So he might… he had a very 122

reduced palette to work with for percussion for these pieces, and so that was the 123

challenge. It was like “here, take something that isn’t as flashy as you’re used to 124

working with but make it sing”, and he was able to do that. I think some composers 125

would be… they’d be like “oh well I’m not gonna use instruments that aren’t at the 126

quality that I want them to be at”, but he was totally cool with all that. And he was 127

patient with us, asking… I think he might have contracted people to actually 128

convert his instruments to the correct… to our spec, and our spec might have 129

changed. The fact that it was such an evolving thing, it always feels like he… he is 130

versatile in that respect. But, again, with his process… we always want to respect 131

his process as much as possible. 132

Interviewer: Cool. So, can we just talk in a bit more detail about the system itself? I 133

know you detailed it quite succinctly in your presentation, but could you just talk 134

me through it a little bit in-depth? 135

Participant 2: Yeah, so… the system has a lot of layers, and there’s a pipeline I 136

guess to using it, where it starts with the composer. He uses instruments that will 137

work for real-time playback in a game engine, so those instruments have to be… 138

they have to be up to whatever spec your engine is gonna use. At the time, ours 139

was… we only had about 20-30megs of RAM, and we had sixteen bit mono just to 140

How systems work:

Flexible

Reasons to use:

Solution to problem







Emergent area:

Evolving


Pers. Trait – adaptability



Future?:

Tech limitations



Role:

Comp. aid/tool

How do systems work?:

Human agency


Limited palette


Technological limitation


Memory restraints


save space, and didn’t have a lot of round-robin variations. So the instruments are 141

limited, you give them to a composer, the composer does his magic. He then gives 142

you a midi file, he gives you all his stems, and then he gives you the percussion 143

instruments that he made… or that his guys made. You then take the mixes, the 144

stems, and you mix different… you make different mixes out of his stems, so one 145

for stealth, one for alert, one for combat. So those can be used to be mixed in the 146

game, because they’re just loops. 147

You then bring the percussion midi file and the percussion instruments into the 148

Scheme Designer tool, which is the tool that [colleague] made, and that’s where 149

you can create these states or schemes… these percussion states or schemes that 150

will play back underneath the looping stems that will change according to the game 151

state. So you author these things offline, from the assets that the composer 152

provides and… let’s see… so that’s the offline authoring pipeline. And then the… 153

implementing them into the game you have to find all the hooks and the outputs 154

that are driving the combat, the combat narrative; so even a player hiding in a bush 155

was part of the combat narrative. If the player is perched on a tree ready to attack, 156

that was part of the combat narrative. So you gather a list of all those things, make 157

sure they’re firing correctly, and then you hook up to all the content you created. 158

You hook the events up to the music content. But you can ask me specifics too 159

about the system… the system as a whole is quite complicated so I’m happy to go 160

into detail in specific areas. 161

Interviewer: Yes, you were talking about the schemes… from my understanding, 162

from what I’ve seen… [composer] provided you with the assets, the instrument 163

assets and the midi. So, like in your demonstration video, these schemes were then 164

created based on these? Like your initial assets? 165

Participant 2: Yeah, so the schemes… you basically… once you load a midi file into 166

the scheme designer, you also load the instruments in. You hit the analyse button 167

and it will analyse the midi and it will… I think it creates… it basically start playing 168

the thing back, with all the instruments playing back. It models the original 169

percussion stem. So from there you’re able to say “I’m gonna mute the toms” or 170

“I’m going to mute the hi-hats” or whatever, and you can selectively mute things, 171

you can change their volumes, you can solo things, and you can also change the 172

energy of the rhythm. So if it’s a really fast rhythm track on the toms, you can pull 173

the energy slider down and it will make the toms play at a lower rhythmic 174

frequency. It’s not changing the tempo, it’s just kinda stripping away layers of stuff, 175

of rhythm, if that makes sense? 176


Limited palette


Hum. agency – musical input


Gather assets


Hybrid system

How systems work:

Modular

How systems work:

Complexity

How systems work:

Link to game parameters

How systems work:


Reasons to use:

Narrative - gameplay


Hybrid system

How systems work:



Human agency

How systems work:



Automated


Interviewer: Yeah, I think so. Making it less dense? 177

Participant 2: Yeah, yeah, exactly. So that’s your sculpting environment, then you 178

just save those schemes, and you can bounce them out into one file that gets read 179

back in the engine, and depending on what scheme you trigger it will transition to 180

it. So yeah, it was definitely like… making a list, and figuring out what we’d wanna 181

support in the game, and then making the schemes and hooking them up. 182

Interviewer: I think one of the most fascinating things for me was when you were 183

talking about energy and how that’s distributed evenly, so it’s not the simple case 184

of just muting a particular instrument, for instance, it’s more about changing the 185

energy… relationships, if that makes sense? 186

Participant 2: Yeah, yeah, that’s all [colleague], he loves to talk about that stuff. It 187

definitely uses a lot of music theory stuff… I think he does a Schenkerian analysis of 188

the rhythm itself, and then… that kind of analysis is just a fancy way of just saying 189

“let’s strip away the density and what’s the core structural framework of this 190

rhythm”. If it’s in four-four you’re gonna wanna hear beats one and three if you 191

pull the energy all the way down, and if you pull it all the way up you’re gonna add 192

back in the sixteenths… so it is really fascinating, what he built. I don’t fully 193

understand how it works when you solo multiple energies and change them 194

around… that is a really neat, original idea by him and he definitely would be happy 195

to talk about that stuff. 196

Interviewer: Cool, thank you. So I know… talking again about the talk that you gave 197

at [conference], you briefly touched at the end on what you think the future of 198

these procedural systems might be. I’ve found… there seems to be a bit of a 199

consensus, in the research that I’ve found, that these algorithmic systems are the 200

future of adaptive systems in games. What do you feel about that argument? 201

Participant 2: Yeah, I don’t… to be honest I don’t know. I think it’s a possibility, it’s 202

very challenging to use these kinds of systems in games. I think that when they are 203

done right they add a lot, but it does take a lot of effort and a lot of experimenting 204

and a lot of hard work to make them appear the way they did. That doesn’t mean 205

that they aren’t the future, but I do think there’s other ways where you don’t need 206

them. Like, I don’t know if you’ve played the games Inside or Limbo? 207

Interviewer: Yeah, yeah. 208

Participant 2: Those games aren’t using procedural sound and music. It’s really 209

like… I think it comes down to the preference of the sound designer, the preference 210

How systems work:

Modular

How systems work:

Define game parameters

How systems work:


Help and support

How systems work:

Complexity

How systems work:

Complexity

Future?:

Potential

Future?:

Uncertain

Future?:

Complexity

Future?:

Project specific (?)

Future?:

Challenging




of whoever is directing the project, and to… does the game really need it, or call for 211

it? I really think every game, like a good game… like a really good, well, hard 212

worked-on game… it’s gonna ask for what it asks for. I think there’s always a way to 213

make it work with the game, it’s just… it really does come down to people’s 214

motivation to use it. And yes, if it was streamlined and made very easy then yeah it 215

could be the future, but I still have a hard time seeing how that… how to make it 216

easily fit into every game. That’s the hard part. 217

Interviewer: Obviously on your system you focused on the rhythmic percussion 218

elements, and I’d imagine… how would it have fared with melodic, harmonic 219

elements? Would that have added exponentially more complexity? 220

Participant 2: Yeah, it was solely built and designed to only handle rhythm. If you 221

would have fed it more things, I don’t know what it would have done since it had 222

no idea how to harmonise a pitch or a harmony collection. So it probably wouldn’t 223

have sounded very good, but I think… also we just didn’t have the space for that 224

kind of thing. I think in the future that could be a thing that happens… if you had 225

enough space, and you had enough memory and bandwidth to handle all these 226

instruments, because they’re just massive, like gigabytes and gigabytes of data. 227

And then of course you’d have to redesign the whole… you’d have to write a whole 228

harmony engine to analyse what the composer writes, and then find ways of 229

stripping away the layers of it and finding ways to expand it without robbing… I 230

guess the goal for us was never to rob the composer of being a composer. It was 231

always to just be sound designers with a composer, where we just strip away, and 232

edit, and make things that don’t loop. And there were times where we would 233

subtly create new music from the composer, but it was always very close to what 234

the composer wrote. I know there are ways to easily remix a song in a DAW and 235

you can do all sorts of things with pitching things, and cutting things up, and even 236

adding your own elements, and that I think just doesn’t sound as good as when you 237

stay with one voice. So I guess to answer your question, there’d have to be a new 238

harmonic engine written and there’d have to be a way to handle how heavy a load 239

it would be. 240

Interviewer: One thing I picked up on then… it’s almost as if you’re alluding to 241

whether these systems are suited to the game, or vice versa, and it all comes down 242

to the project. 243

Participant 2: Yeah, it’s the project, and also I think it’s the will of the director, or 244

the designer, to be like “you know what…” …if you really think procedural music is 245

a great thing then I think you can usually find a way to make it work. There might 246

Future?:


Future?:


Future?:

Versatility

Future?:

Complexity

Future?:

Challenging

Types of system:

Rhythmic

Future?:

Tech limitations

Future?:

Complexity

Future:



Additional variation


Avoid replication



Future?:

Complexity

Future?:

Tech limitations

Future?:

Tech limitations



Future?:

Versatility


be a chance where it’s just not going to work with the game, so you definitely need 247

the component of being able to squint or to just want to get it to work somehow, 248

and then you can find a way most of the time I think. I think what can turn people 249

off is the challenge of it, and there’s also a lot of great… Wwise has a great music 250

system, there’s a lot of great things out there that allow people to come up with 251

faster solutions, or things that work well, or maybe just as well but differently. And 252

I like that, I don’t want there to be one procedural engine that like… because 253

everything will sound similar, if everyone is using that. I think it’s good that people 254

can choose to use it, and that it’s challenging… I hope that it become less 255

challenging some day, and quicker to use. 256

Interviewer: It’s interesting you should say that. I’ve found a source that stated 257

when they were showcasing their generative system they met, or encountered, a 258

lot of “scared composers” who thought they would be out of a job, but he was at 259

pains to state it’s not so much a system to replace composers but to work 260

alongside them. And also touched upon helping the composer, like you talked 261

about, in terms of workflow and things like that. 262

Participant 2: Yeah, yeah, I mean… we’re never gonna replace composers. I am 263

convinced that’s never gonna happen. It’s like, you can’t… I love Neil Young, you 264

can’t write an algorithm that’s ever gonna sound like Neil Young, or Thom Yorke, or 265

anything like that. And they are composers, and they compose for really big 266

things… I think Thom Yorke and Hans Zimmer did Blue Planet or something. It’s just 267

never gonna happen, I think the tools though… that’s what I’m really interested in. 268

My biggest interest is… you make a tool that’s fun, it’s like a new instrument. You 269

can learn the piano, but one of the things that discourages me from learning the 270

piano is that there’s so many amazing pianists already out there, or who have lived, 271

and it just feels like really hard to come up with anything new, whereas the 272

computer feels like with that you can do some things that are really inventive and 273

interesting. So that’s the goal, and it’s never gonna replace everyone because it’ll 274

all sound the same, and then the second somebody uses a composer, a human 275

composer, they’re gonna stand out because it’s gonna sound different and 276

amazing! So it’s just not gonna happen. 277

Interviewer: All right, thank you. I think we’ve just about covered everything, 278

unless there’s anything that we haven’t talked about that you’d like to mention? 279

Participant 2: Yeah, I’d like to say that games are great and all for this kind of 280

technology but I think stuff that I’m interested in these days and don’t get a lot of 281

time to do is more of assisted technology, like the application of having musical 282

Future?:


Future?:

Versatility

Future?:

Challenging

Future?:

Alternative solutions

Future?:

Streamlined

Future:


Future:


Future:


Emergent area:

Innovate

Role:

Comp. aid/tool

Role:

Comp. aid/tool

Future:



systems for self-expression being given to people who don’t have arms or legs to 283

play the drums. Using this kind of technology to allow a person with disabilities to 284

express themselves through music I think is a really amazing… I don’t even know if 285

anyone is doing work in that area, but to me I think that’s related to this because 286

you’re interacting with a controller with your hands to generate music, essentially. 287

When you play that stealth combat in [video game] everything you’re doing is 288

generating the music, so if you could make a program like that that you could put in 289

the hands of someone who can’t move their hands, but they could use their eyes to 290

move an interface that creates music, I think would be a really worthwhile 291

application. That’s just my thoughts on another topic but I thought I should say it 292

because I think it’s important. 293

Interviewer: Yeah, thank you! Well I think that’s everything, thank you so much for 294

your time. 295

[End of recorded material]

Future:

Additional applications

Future:

Additional applications

Emergent area:

Potential

Reasons to use

Narrative - gameplay

APPENDIX C

Loose Interview guide plan based on framework of inquiry.

A P P E N D I X C | 53

▪ Need to know about interviewee

o Personal background:

▪ Are they musical/composers etc

▪ Why did they choose their field?

▪ Technical skills

o Professional background:

▪ How much experience have with generative systems?

▪ Worked with other systems before?

▪ Need to know about game that the system was used in:

o Why did style/type/scope of game dictate the use of generative system?

o At what point was composer engaged with project – before it was decided to use generative

system or after? Did composer have a say in the choice?

▪ Need to know why these systems were used (i.e. in lieu of other adaptive systems)

o What could it offer?

o Financial/technological/creative reasons?

▪ Need to know about generative system interviewee used/developed

o What is it/name of system

o Is it their own bespoke system? Why design own system?

o How does it work?

▪ Establish interviewees role in relation to project that featured generative system

o What their role entailed/duties etc.

o If they composed music for it:

▪ Talk about workflow/approach

▪ Challenges?

▪ Differences (if any) to writing music “traditionally”?

o If they didn’t compose:

▪ How did they work with composer e.g. relationship etc.

▪ What audio elements/assets did composers have to deliver?

▪ Need to establish what they feel future of generative systems may be:

o How do they see future of generative music?

Documents

The Implications of Generative Music Systems on Video Game ...€¦ · BA (Hons) Creative Music Technology ‘The Implications of Generative Music Systems on Video Game Composers: