Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
µ-SCRIPT-SYSTEM : A FRAMEWORK OF
SYNCHRONIZED MUSIC CONTENTS USING A SCRIPT
LANGUAGE
µ-Script-System : スクリプト言語を用いた同期機能を持つ音
楽コンテンツのためのフレームワーク
by
Hidehiko Abe
阿部秀彦
A Senior Thesis
卒業論文
Submitted to
the Department of Information Science
the Faculty of Science, the University of Tokyo
on February 1, 2005
in Partial Fulfillment of the Requirementsfor the Degree of Bachelor of Science
Thesis Supervisor: Takeo Igarashi 五十嵐健夫assistant professor of Information Science
ABSTRACT
This paper proposes a new framework in which events described in a script language are embedded in
Music XML. As the Internet becomes popular and the network technology improves, interactive musical
contents rapidly and widely spread. In making musical content, it is important to synchronize other
elements such as animations and user interaction. When we make such contents, however, we usually
control timings based on general time units, like millisecond or microsecond, but not on rhythmical ones,
like measure, beat or tick. This paper presents a new framework in which the control information can be
described inside the musical information, which has been conventionally described separately, by using
an easy script language. This makes the timing control of events more flexible and intuitive. We chose
Music XML as a format of the musical information. Music XML is a text-based new musical format.
The extended Music XML, in which the script is embedded, is not a format intended for exclusive use
with µ-Script System, but can be treated as usual Music XML by ignoring the script tags. Furthermore,
this system meets various demands of users by the interaction with external applications. This would
facilitate the creation of music contents.
論文要旨
本稿は Music XMLに対応したイベントをスクリプト言語で埋め込む新しい枠組みについて提
案する。インターネットの普及、ネットワーク技術の向上によってインタラクティブな音楽コンテ
ンツが急速にひろまり、より身近なものになってきている。音楽コンテンツを作るうえで、曲の進
行と、例えば動画の表示やユーザインタラクションといったその他の要素との間の同期は大切で
ある。しかし現在、そのようなコンテンツを作るときの同期の取り方は、ミリ秒やマイクロ秒と
いった一般的な時間単位をベースとしたタイミング制御に基づくものであり、音楽の時間単位の
基本である拍子や小節に合わせたタイミングの制御を行うような機構はないように思われる。本
稿では、従来別々に記述していた情報を音楽情報中に簡単なスクリプト言語で記述できるような
枠組みを提案する。これによりイベントのタイミング制御をより柔軟かつ自然に行うことができ
る。また、スクリプトを埋め込む音楽情報として、本研究ではMusic XMLを用いることとした。
Music XMLは新しいテキストベースの音楽フォーマットである。本稿で扱うフォーマットはシス
テム専用のものではなく、埋め込んだスクリプトタグを読み捨てることで、通常のMusic XMLと
しても扱える。さらに、本システムは外のアプリケーションと連携をとることで、よりいろいろ
なユーザの要求にこたえられる。これによって音楽コンテンツの作成を支援できるようになる。
Acknowledgements
We thank assistant professor Takeo Igarashi for his great advice. This work would not be
completed without his help. We are also grateful to members of Igarashi group, especially Mr.
Shigeru Owada and Mr. Yasushi Maruyama, for their comments on our work. Many ideas in
this work stemmed from discussion with them.
Contents
1 Introduction 1
2 µ-Script System 3
3 Design 5
3.1 Music data format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Music XML Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3 Script Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.4 µ-script language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4.2 Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4.3 Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.4.4 I/O and Connections . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4.5 APIs (Application Programming Interfaces) . . . . . . . . . . . . . . 10
4 Implementation 15
4.1 Reading Music XML file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Conversion into MIDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2.1 Calculation of resolution . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2.2 Calculation of ticks at the beginning of each bar . . . . . . . . . . . . 16
4.2.3 Conversion from note tags to MIDI event . . . . . . . . . . . . . . . 17
4.3 Interpretation of µ-Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 Interpreter Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.5 An Example Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5 Result 21
6 Discussion 25
i
7 Related work 26
8 Conclusion 27
A Syntax of µ-Script Language 28
A.1 Lexical Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
A.1.1 Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
A.1.2 Punctors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
A.1.3 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
A.1.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
A.1.5 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
A.1.6 Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
A.2 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
References 33
ii
List of Figures
2.1 Image overview of coventional system. . . . . . . . . . . . . . . . . . . . . . 3
2.2 Image overview of µ-Script System . . . . . . . . . . . . . . . . . . . . . . 3
2.3 The image of a file treated µ-Script System . . . . . . . . . . . . . . . . . . 4
4.1 GUI of the implemented interpreter . . . . . . . . . . . . . . . . . . . . . . 19
5.1 The tempo controller by socket connecting . . . . . . . . . . . . . . . . . . . 24
iii
Chapter 1
Introduction
Today, music plays an important role in interesting and interactive digital contents. Two major
reasons are the following. First, the rapid development of the Internet technology enabled a
large amount of data to be sent and received. Second, various kind of data can be processed
in short time thanks to higher spec CPU.
Though famous applications like Macromedia Flash [9] are popular in creating digital
contents that includes musical element, they have a problem in that synchronization is not
realized between musical and unmusical elements in those applications. The music elements
synchronize with unmusical ones only at the moment when the sound begins, and no systems
keep synchronization between musical time lines and unmusical ones after that. When the ini-
tialization fails or these time lines get disaccorded, the gap can not be modified. The second
problem is that the time units used in those applications is not as same as the ones used in
music. The former units are millisecond, microsecond, or second and the latter ones are mea-
sure, beat, or tick – a general time unit in MIDI (Musical Instrument Digital Interface) [10].
This will be inconvenient when, for example, creators want to make contents in which a robot
dances to the music. They need to convert the musical time unit into the general time unit
by means of a special calculation. The performance of the result of the calculation, however,
depends on the execution environment. Therefore, they need to repeat tests and modifications
to get the best correspondence.
This paper proposes a new framework that is based on musical time units and constant
synchronization while the music is going on. As a result, it will be possible to play music
in perfect concordance. It will be easy for the user to execute an event at any bar, any beats
they want. For example, users are enabled to set the number of repetitions, branches of music,
conditions and connections to servers. This system can be connected to external applications
by sockets. This would faciliate the creation of various contents. For example, when a creater
1
requires treating an external device which cannot be connected directly to this system, a simple
application which treats both the device and a socket enables to make a connection between
the device and the system.
Section 2 discusses a overview of µ-Script System. Section 3 presents the design of µ-
Script Language and the way to embed the script in the Music XML. Section 4 presents an
implementation of this system. In Section 5, a small and simple sample is shown. Section
6 discusses this system in the present version of implementation. In Section 7, we examine
related work. Finally, the summary of the conclusions in Section 8.
2
Chapter 2
µ-Script System
The simple way to treat music in conventional systems is to send music data to a music player
and order it to play the music (see figure 2.1). The music player can be both hardware and
software. A music file bears little relation to even remotely related to unmusical files in an ap-
plication. Additionaly, the data are sent only from an application to a music player. Therefore,
music seems to be isolated in the system.
µ-Script System is a model of connecting unmusical application with a music player (see
figure 2.2). A file treated in this system includes both music and unmusical data (see figuare
2.3). Therefore, it is easy for a music creater to set timings of events which is related to
music. This system connects unmusical data to music more strongly and makes unmusical
data synchronize with music.
Figure 2.1: Image overview of coven-
tional system.
Figure 2.2: Image overview of µ-
Script System
This system is designed to create interactive contents based on music, for example, a danc-
ing robot. Progress of music determines the timing of actions of the robot, such as waving
3
Figure 2.3: The image of a file treated µ-Script System
its hands and executing a turn. The robot looks its best on a stage with backlights blinking in
synchronization with the music. The stage might have fountains shape-shifting in the synchro-
nization. Therefore, since music is a base of the actions and the blinking and shape-shifting
patterns, these events should be described in music. Moreover, when interactions are added,
synchronization of this system will make a significant contribution. Since this system keeps
dynamically synchronization differently from conventional systems, gaps will be modified.
Therefore this system may suitable to such contents.
4
Chapter 3
Design
3.1 Music data format
This paper proposes a new simple script language, µ-Script Language, to write unmusical
information, in music data. Although most of the music data formats are binary ones, such as
SMF (Standard MIDI-file Format) [10] or Wav, MP3 etc, textual data format is more suitable
to describe a simple script language inside the music data. MML (Music Macro Language),
which has some specifications, is a famous textual data format, but it got out dated, so we
choose Music XML [4, 5, 3] as a music file format to describe µ-Script Language. Music
XML is a very new format, the version 1.0 of which was announced in January, 2004. Many
famous applications in the field of treating digital music, like Finale [6], come to support Mu-
sic XML file format. Moreover, it is a data format suitable to be distributed in the Internet, and
it matches to the purpose of this study of easily making and distributing music synchronized
contents.
3.2 Music XML Overview
This section presents an outline of Music XML which is used in this thesis. Music XML
data structure is roughly divided into four parts. In the first part, only music notation infor-
mation is described; what clef and what rhythm sign is to be chosen, how slurs, ties and acci-
dentals are to be described, where bar-lines are to be described, or which stems, note heads,
beams are chosen? In the second part, only music performance information is described. This
is the information needed to play music; how high or low the real pitch level is in comparison
with the step in chromatic scale, how long the note duration is, or whether the note is tied
or not. The third part includes both music notation and music performance information, like
steps. A step element indicates both where the note head comes and how high the note pitch
5
is. The last part is a set of unmusical information, like the name of the composer, the title,
score groups, etc.
Music XML has two types. One is ”score-partwise”, which is a series of parts with mea-
sures inside. The other is ”score-timewise”, which is a series of measures with parts inside.
Only score-partwise is supported in this paper since there was especially no essential differ-
ence between them. We will here after treat only ”score-partwise”, if not particularly men-
tioned.
This paper focuses on the performance part of Music XML. There are many tags in Music
XML, but only some main tags are paid attention to in this paper. They are roughly explained
in the following. When details are needed, they will be supplemented.
• alter tag: shows how high or low the note is in chromatic scale.
• attributes tag: is used to set and change the new attribute from old ones.
• backup tag: means to return back for the specified duration from current position.
• divisions tag: is put in attributes tags. It means to change resolution from current posi-
tion. Resolution is the minimum unit.
• duration tag: means the length of the note or the rest.
• forward tag: means to go ahead for the specified duration from the current position.
• measure tag: is a unit of measure.
• note tag: is a unit of note or rest.
• octave tag: specifies the octave where the note is described.
• part tag: means one of the song parts. It has measure tags as children in the score-
partwise type.
• part-list tag: declares each part.
• part-name tag: declares the name of the part.
• pitch tag: assigns the pitch. This tag includes an octave tag and a step tag as its children,
and also is possible to include a alter tag. These children specify musical performance
pitch. It has a note tag as its parent. A note tag which has a pitch tag as its child will
signify a note, not rest.
• rest tag: The parent note tag which has rest tag signifies rest.
6
• score-part tag: declares a part.
• score-partwise tag: is a root of Music XML in the score-partwise type.
• step tag: includes any one letter of ’C’, ’D’, ’E’, ’F’, ’G’, A’, or ’B’, which respectively
represent the sound of do, re, mi, and so on. This element means the step of the grand-
parent node, namely note.
• tie tag: This tag is used in pair. One of the pair has the start attribute and the other has
the stop attribute. Their parent note tags should have the same pitch.
3.3 Script Tags
µ-script language is embedded in <script>·· ·</script> tags in the Music XML data.
It controls the execution timing by the position where it is described. There are two kinds of
the execution timing. One is the initialization at the beginning and the other is done during
musical performance with synchronization. The processes which may be a burden afterwards
during the synchronization should be done before the music starts. It includes such tasks as
creation and connection of socket, declaration of global variables, and to set handlers of input
queues or sockets. These tasks will be explained in detail in chapter 4.
To achieve these two types of execution, script tags are allowed to be written both in score-
partwise tags and attributes tags. The script written in score-partwise tags is interpreted as the
initialization when this system reads a Music XML file, since score-partwise tags should be
unique in a tune.
The other is for the events during performance. These scripts to go are allowed to be
written directly under the attributes tags, for the reason following. The row of duration tags
shows the flow of time in Music XML. Notes and rests have duration. A note is played
just after the end of the previous note, rest, etc. Since these notes and rests can only be
monophonic. Forward tags and backup tags solve this problem. The former tags set ahead the
current timing, and the latter push it back, with duration. Measures tag can have all these tags.
Measures tag can also have attributes tags. Attributes tags are written between those tags and
have no duration. Therefore attributes tag signifies a particular point of time in Music XML.
That is why script tags should be written in attributes tags.
7
3.4 µ-script language
3.4.1 Overview
µ-script language is a simple script language which is embedded in Music XML files.
Syntax of theμ-script language is almost equivalent to the Subset of JavaScript (see Appendix
A) [11].
3.4.2 Values
The value types in µ-script language are as follows:
• Signed integer of 64 bits
• Unsigned integer of 64 bits
• Floating point number of 64 bits and 32 bits
• Boolean values
• String
• Arrays
• Sockets
• Null
• Undefined
• Functions
64 bits floating point number is used as basic numerical value. Constant numerical values
are recognized as this. If users want to use other types of numerical value, it needs to add
a suffix, which is ’f’ or ’F’ for 32 bits floating point numbers, ’l’ or ’L’ for 64 bits signed
integers, and ’u’ or ’U’ for 64 bits unsigned integers.
A character is one byte, and String is an array of bytes. Constant string values are written
in the form of ”. . .” or . . .’. The difference between them is the terminal symbol. Strings
starting with a double quotation mark end with the same mark. Single quotation marks are
recognized as normal characters. On the other hand, those starting with single quotation mark
end with the same mark, too. Double quotation marks are recognized as normal characters. To
use single quotations in ’. . .’ or double quotations in ”. . .”, it is necessary for these characters
to be escaped by a special character \. BS, LF, CR, and TAB can be respectively represented
8
by the escaped b, n, r, and t. It is possible to describe the byte value directly in the form of
\ooo or \xhh. An o character is an octal number character and an h character is a hexadecimal
number character. These values must be positive and less than 25610.
Null value is used to clear handles or to set non-reference value. Undefined value is a
result of evaluating uninitialized identifiers or functions that return a void, etc.
Function type is a simple function. No closures are generated.
Sockets are generated by the function ”Socket”. This function has two parameters; the
first is a host name, the second is a port number. When it fails to connect the server, the value
of the Socket type is not generated.
Array values in µ-script language are fixed length array. No systems identify the value
types in this language. Therefore any values can be set in an array. The Boolean value, for
example, can be set as an element of array which is a function value before substitution. The
same is true for variables. Moreover, various types can be mixed in an array. For example,
the first element can be a socket in a three elements array, while the second is a Boolean value
and the third is a function. A multi-dimensional array is expressed by an array which contains
arrays.
All values can be evaluated as Boolean. Boolean entity is true or false. Arrays, functions,
and sockets are always true. Null and undefined are always false. The numerical value is false
when it is 0, NaN, Positive Infinity, or Negative Infinity; otherwise the value will be true. Zero
length strings are false and other strings are true.
3.4.3 Statements
Statements should be separated by semicolon characters. Semicolons of a last statement-
list can be omitted, according to the context.
When the script interpreter fails to evaluate an expression in a statement, it stops interpret-
ing the statement, then restart the interpretation by the following statement.
This language has four statements to control the programming flow. These are ’for’,
’while’, ’do-while’ and ’if’ statement. ’Continue’ and ’break’ statements can be used
in ’for’, ’while’ and ’do-while’ statements. ’Return’ statements are allowed to be used
anywhere in the function body.
This system is able to recognize common fundamental floating point arithmetic operators,
namely, +, -, *, /, %, <, >, <=, >=, ==, and !=. It seems uncommon that it is recognizes bitwise
operators, such as, ~, &, |, ^, <<, >> and >>>. In using bitwise operators, the interpreter
converts a number to an integer before the operation, and it again converts the result converts
to the original numerical type after the operation. All of the Boolean operators, namely, & &,
9
|| and ^^, can be applied to all values, since all types of values can be evaluated as Boolean
ones. Besides, this language has a substitution operator, which is =, and complex substitution
operators, that is, +=, -=, *=, /=, %=, ~=, & =, |=, =̂, <<=, >>=, >>>=, & & =, ||= and ^^=.
This script also has block statements. The scope of the identifiers declared in a block
statement is limited to the block.
Function application fails when the number of parameters of the function is not equal to the
number of actually given arguments. Otherwise, the expressions as arguments are evaluated.
Next, the interpreter creates a new environment that shares only global environment with
others, where the corresponding parameters refer the evaluated values. Then, the function
body statements get evaluated. Finally, the created environment is destroyed and the results
are returned.
3.4.4 I/O and Connections
This language has two input and two output mechanisms. Socket connection paly the role
of both input and output. This system is controlled by three primitive functions. These are
”Socket”, ”Recv”, and ”Send”. Each socket value has two byte-wise queues. One of them is
a buffer to send data to the server and the other is to receive from the server. When ”Socket”
is called, the interpreter creates a native socket and tries to connect the server specified by
the arguments. When this process succeeds, it creates a socket value of µ-script system and
creates a thread that polls the native socket and delivers the data from the native socket to
buffer and vice versa in the background. We adapt such a complex system for fear that the
connections interrupt the music.
Message queue is the other type of input. Any value can be added to this queue, such
as arrays, number, or string. When values are added, a handler is called with an input value
as an argument. Stream is the other type of output. This will be in use when ”PrintInt”,
”PrintDouble” or ”PrintString” function is called.
3.4.5 APIs (Application Programming Interfaces)
µ-script language has primitive functions as follows:
� CreateArray(size) :
This function generates a fixed length array, in which the number of elements is ’size’.
If size is not a number or is negative number, it will be an error and the evaluation stops.
All elements in generated array are Undefined.
� size: Number of elements of generated array.
10
� Return value: An array whose element size is ’size’.
� ArrayToString(array) :
This function converts an array of the numerical values into character string. The length
of the result is equal to the input array length. If there exists one or more value that is
not a numerical value in the input element, it will be an error. To convert a number to a
byte, each of number is cast to a byte modulo by 256.
� array: Number’s array to be converted.
� Return value: a converted string.
� StringToArray(str) : This function converts a string into an array of the numerical
value.
� str: a string to be converted.
� Return value: a converted array of numerical values.
� GetLength(in) :
This function returns the length of an input array or string.
� in: a string or an array whose length a user wants to know.
� Return value: Length
� GetResolution() :
This function returns the resolution of a tune installed in an interpreter.
� Return value: Resolution.
� PrintInt(num) :
This function converts the input value into 32 bits signed integer and output it to a
stream.
� num: A number to output to stream.
� Return value: Undefined.
� PrintDouble(num) :
This function converts the input value into 64 bits floating point number and output it to
a stream.
� num: A number to output to stream.
� Return value: Undefined.
11
� PrintString(str) :
This outputs the input string. ’\ 0’ character is recognized as a terminal symbol in a
similar way of C language.
� str: A string to output to stream
� Return value: Undefined
� Jump(ticks) :
This function sets the current time to the input position. Current statements get exe-
cuted. If a script is described in the newly fixed time, it is also executed.
� ticks: The destination position with tick unit.
� Return value: Undefined.
� Stop() :
This stops the performance.
� Return value: Undefined.
� Transpose(num) :
This sets a transpose as an absolute value. To set transpose as a relative value, a function
defined below is needed.
_transpose = 0;
function RelativeTranspose(num){
_transpose += num;
Transpose(_transepose);
}
� num: A number of transpose.
� Return value: Undefined.
� SetHandler(func) :
This function set ’func’ as a handler of an input queue. When the queue gets input
message, the handler is called with the message as argument. To clear the handler, this
function is called with ’func’ being null.
� func: The handler function of input queue.
� Return value: Undefined.
12
� ToTick(measure, beat, ticks ) :
This converts input measure, beat, and ticks into MIDI ticks.
� measure: Measure number to be converted.
� beat: Beat number to be converted.
� ticks: Tick number to be converted.
� Return value: Converted ticks.
� GetTicks() :
Current position is returned.
� Return value: Current position.
� Socket(hostname, port ) :
This function creates a socket connected to the server specified with arguments. When
it fails to connect to the server, it reports an error.
� hostname: A string that specify a server.
� port: The port number.
� Return value: A socket value connected to the server.
� Recv(socket, length) :
This reads data from the socket up to the ’length’ bytes.
� socket: A socket to read.
� length: The maximum length of to read data.
� Return value: A string buffering read data.
� Send(socket, msg ):
This writes data in the socket.
� socket: A socket to write data in.
� msg: A string buffering data to send.
� Return value: Undefined.
� SetTempo(bpm) :
This sets tempo.
� bpm: A number of beats per minute.
� Return value: Undefined.
13
� SetSocketHandler(sock, handle) :
The function is called to set a socket handler. The handler is called when data are in the
socket. If buffered data are not read in the handler, the handler continues to be called
with sock as argument.
� sock: A corresponded socket.
� handle: A function of handler.
� Return value: Undefined.
14
Chapter 4
Implementation
4.1 Reading Music XML file
This system uses ”expat”, one of the most famous and fastest libraries in the world of
parsing XML files of non-verification type. The present specification of this system disregards
unnecessary part when converting it into MIDI data. Tags which can be read in Music XML
at present are alter tags, attributes tags, divisions tags, duration tags, measure tags, note tags,
octave tags, part tags, part-list tags, part-name tags, pitch tags, rest tags, score-part tags, score-
partwise tags, step tags, and voice tags. Script tags can be added to this list of tags. The
script language can be described between the beginning script tag and the ending one. A
problem is caused here. µ-script language has less symbols ,’<’ and ’<=’, but the symbol is
recognized as beginning of tags in expat, and parsing fails. There are two ways to avoid this
problem. One is to use greater symbols, ’>’ and ’>=’, instead of the less symbols, since the ’>’
character is recognized as a normal character in expat library. The other is to describe scripts
in ”<![CDATA[.. . ]]>” tags. All of the characters are yet to be recognized as script in this
phase.
4.2 Conversion into MIDI
Conversion from Music XML to MIDI is divided in three phases, calculation of resolution
of the whole music, calculation of ticks at the beginning of each bars, and conversion from
note tags to MIDI event.
4.2.1 Calculation of resolution
Resolution is a unit of the time stamp in MIDI. It is easy to calculate the resolution. Music
XML expresses time by duration. This unit of duration is specified by the divisions of the
15
Attributes tag. Divisions are the index of how many equal parts there are in a given musical
time units. Therefore, resolution is the least common multiple of all divisions of music, which
is then multiplied by ten. This multiplation is indispensable, because the durations needs to
be an integer after multiplied by 0.9. The detail is explained in 4.2.3
4.2.2 Calculation of ticks at the beginning of each bar
Good showed a simple algorithm to calculate length of duration [2]. The time stamp of
each bar is determined by the duration included in the bar. Each bar has elements which have
their children of note, backup, and forward. After searching for durations in all parts, the
length of a measure of each part is calculated. The measure length is the maximum value of
those of each part. However, all the lengths of a bar of each part should be equal in general.
Although there are some rare cases in which they are not equal, for example, when time
signatures of each part are not equal, this system doesn’t support those cases. The length of a
note and a rest is determined by the current divisions and its duration, shown as follows:
ticks =resolution×duration
current divisions
Current divisions is the value of divisions in the attributes tag described immediately be-
fore. Since resolution is the least common multiples of the sets of current divisions multiplied
by ten, as previously shown, note duration is also an integer. The length of a bar of a part,
mp, is equal to the summation of these lengths subtracted by the sum of backup durations as
follows:
lengthmp = ∑note\chord∈mp
ticksnote + ∑forward∈mp
ticksforward − ∑backup∈mp
ticksbackup
The measure length is the maximum value of the length of a measure of each part.
lengthm = maxp
{lengthmp
}
At last, the timestamp of the beginning of measure is summation of all of the preceding
measure lengths.
timestampm = ∑m′precedem
lengthm′
16
4.2.3 Conversion from note tags to MIDI event
First, it is necessary to decide the height of the note. Note number, which is used as pitch
in MIDI, is calculated from the step, the octave, and the alter of a note as follows:
notenumber = (octave+ 2)×12 + alter +
0 if step = ’C’
2 if step = ’D’
4 if step = ’E’
5 if step = ’F’
7 if step = ’G’
9 if step = ’A’
11 if step = ’B’
If the alter is not defined, it will be zero. The length of each note is defined as follows:
notelength =resolution×duration
current division×0.9
Current divisions is the same as was above. The reason of the multiplation 0.9 is that a
note is not performed at the whole the length when actually performing. Since resolution is
multiplied by ten, note length is integer. In MIDI, a note is represented by thw events. One
is ”note on” event, which is message 0x90{note number}{note velocity}. The other is ”note
off” event, which is 0x80{note number}00. 0x90 or 0x80 are status bytes. The note on event
is sent at the beginning of the note, and the note off event is after the time passed for the length
of the note. In this paper, note velocity is constantly 0x5A. The current time advances by the
note ticks, not by the note length.
In the case of a note with tie, note off event of the note with starting tie is not sent, and
note on event of the note with stopping tie is not sent. If a note has both the starting tie and
the stopping tie, both events are ignored. Moreover, when a note which has a chord tag is
converted to events, the timestamp of the note on event will be as same as the previous note,
and that of the note off event is recalculated from the duration of that note. The chord note
does not change the current time. When the forward tag, which is not converted to MIDI
events, puts the current time forward for the ticks calculated from the duration of note. The
backup tag is not converted to MIDI events neither, and it puts back the current position for
the ticks calculated from the duration of note.
When a script tag is found, the lexical analusis and the parsing of the script language
occurs. If the parsing is succeed, the result can be found in the current position. The current
position does not move.
17
4.3 Interpretation of µ-Script
µ-Script system uses Bison and Flex to make its parser and lexical analyzer. For the actual
grammar, see Appendix A. This script language interpreter has GC (Garbage Collector). GC
uses famous algorithm, Mark and Sweep. It controlls all of the expressions, statements, and
values. GC is executed before a new XML file is read in the present version of implementation.
A synchronized queue as a message queue is the standard input. The array is added to this
queue. When there is input data, the interpreter passes the data to handler as an argument,
and the handler is executed. When handler is not set or the number of the parameter of the
handler is not equal to the length of the input data, the data is immediately ignored. On the
other hand, the output is C++ Stream. The stream is connected with the interpreter, and data
from the interpreter is received.
One of the main features of this system is a communication of ”Socket”. When ”Socket”
is created, it tries to connect the server host and port, which are taken as arguments. When
the connection fails, it will be an error. Otherwise, the thread is created for each socket and
watches the socket, while two thread-safe byte data queues are created. One of the queues is
used to buffer the received data. The other is used to buffer the data to be sent. When ”Send”
is called, the data is copied in a buffered queue. Then the flow immediately returns to the point
where the function was called.
4.4 Interpreter Behavior
A fast thread is generated at the beginning of the interpretation to control multimedia
without delay. Execution is done in the thread in the following order. First, the present timing
is calculated. The timing is calculated from Tempo and resolution as follows:
time[ticks] = passedtime[msec]× resolution[ticks/beat]×bpm[beat/sec]×60×1000
The passed time is the diffrence between the time of last phase and the current time. The
present position is the total of the passed-time ticks times.
Next, input message queue is checked. When data come, only a piece of data is read
per execution of thread. It is passed to a handler function as an argument and the function
is execution if the handler is set and the number of parameters is equal to the number of the
arguments. Otherwise, the input data is thrown away. The purpose of this restriction is not to
disturb the performance of music when a large amount of data suddenly comes. Input data is
thrown away when the handle is not set.
18
Then, it is checked whether handlers of sockets are set or not. When a handler is set in
socket, and when the buffer of socket has data, the handler is called with the socket as the
argument. Each socket is checked only once. At this moment, if the handler does not read out
the data, the data remains in the buffer, which makes the handler to be read again.
When all the call checks of handler end, the search and execution of a script is done if
there exist any scripts that should be executed in present time or the time immediately before.
When API of the script movement is called, MIDI Player assumes that time has passed while
ongoing the script is executed until last minute. The interpreter searches for scripts that exists
exactly at the same time stamp. If there is any, the script will be executed.
After processing all the scripts upto the end, the MIDI message is transmitted.
4.5 An Example Application
The figure 4.1 is the GUI of an example application.
Figure 4.1: GUI of the implemented interpreter
The ”Open” button is used to find an xml file and to open the file. When the file opens,
this system reads the data. First, it is parsed and is converted to MIDI data. Next, µ-Script
language is lexcally analyzed and parsed. Finally, the script for initialization is executed.
19
Clicking ”Play” button begins the performance and the script is executed at the given
timing. The ”Stop” button stops the performance. The text box at the upper left side is a
window for a simple experiment of script execution. When a script is written here and the
”Exec” button is clicked, the script is parsed and interpreted. Although the interpretation
has no effects on the environments which is used in performance at the moment, it effects to
outside devices, such as a music player or MIDI device.
The window at the upper right side is a text output window which is connected to a stan-
dard output stream of the interpreter. If ”PrintInt”, ”PrintDouble” or ”PrintString” function is
called, the data from the interpreter is displayed in the window.
20
Chapter 5
Result
A small and simple sample is shown. It is a fragment from the famous tune of a video game
Super Mario Brothers. The script embedded the sample controls the tempo and the flow of the
music.
the sample is as follows:
<?xml version="1.0" standalone="no"?><!DOCTYPE score-partwise PUBLIC
"-//Recordare//DTD MusicXML 1.0 Partwise//EN""http://www.musicxml.org/dtds/partwise.dtd">
<score-partwise><part-list>
<score-part id="P1"><part-name>Voice1</part-name>
</score-part><score-part id="P2">
<part-name>Voice2</part-name></score-part><score-part id="P3">
<part-name>Voice3</part-name></score-part>
</part-list><script><![CDATA[
sock7 = Socket("127.0.0.1", 10006);function sockhandler (sock){
str = Recv(sock, 4);array = StringToArray(str);tmp = array[0];if( 0 > tmp ){
tmp += 256;}SetTempo(tmp);
};ret = 0;function handler(){
ret = GetTicks();Jump(ToTick(38,1,0));SetHandler(null);
};SetHandler(handler);
21
SetSocketHandler(sock7, sockhandler);]]></script><part id="P1">
<measure number="1"><attributes>
<divisions>4</divisions><clef>
<sign>G</sign><line>2</line>
</clef></attributes><note>
<pitch><step>E</step><octave>4</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>E</step><octave>4</octave>
</pitch><duration>1</duration>
</note><note>
<rest /><duration>1</duration>
</note><note>
<pitch><step>E</step><octave>4</octave>
</pitch><duration>1</duration>
</note>::<measure number="38">
<attributes><divisions>12</divisions>
</attributes><note>
<pitch><step>C</step><octave>4</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>G</step><octave>3</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>C</step>
22
<octave>4</octave></pitch><duration>1</duration>
</note><note>
<pitch><step>E</step><octave>4</octave>
</pitch><duration>1</duration>
</note>::
<note><pitch>
<step>B</step><octave>4</octave><alter>-1</alter>
</pitch><duration>1</duration>
</note><note>
<pitch><step>F</step><octave>4</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>B</step><octave>4</octave><alter>-1</alter>
</pitch><duration>1</duration>
</note><note>
<pitch><step>D</step><octave>5</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>F</step><octave>5</octave>
</pitch><duration>1</duration>
</note><note>
<pitch><step>B</step><octave>5</octave><alter>-1</alter>
</pitch><duration>1</duration>
</note><note>
23
<pitch><step>F</step><octave>4</octave>
</pitch><duration>1</duration>
</note><attributes>
<script>Jump(ret);ret = 0;SetHandler(handler);
</script></attributes>
::
Two scripts are embedded in the sample. The first script is described in score-partwise
tags. It is are at the moment of initialization. It shows that the interpreter creates a socket
and then sets each handler to the socket and to the message queue, a standard input. The
socket handler needs a byte from the socket and sets it as beat per minute. In the message
queue handler, the interpreter stores the current position, then set 38th bar as playing position.
A sound effect played when Mario gets a Super Mushroom is described at this bar and the
interpreter does not accept the standard input while the sound is playing. It returns to the
stored position when the sound effect ends, and it comes to accept the input again.
Additionally, a simple Java application is implemented which is connected to the inter-
preter and controlls tempo.(see figure 5.1)
Figure 5.1: The tempo controller by socket connecting
This application is connected to the interpreter with a socket. When the value of its slide
bar changes, it sends a byte which is equal to the scale of the slider through the socket.
The current tempo can be controled by using the slider from the perspective of a user.
When the slider is moved to the left side, the tempo slows down. When the slider is moved
to the right side, the tempo quickens up. Additionaly, when the ”GO!(Message)” button (see
figure 4.1) is cliked, the sound effect interrupts the main music.
24
Chapter 6
Discussion
µ-Script System in the present version of implementation is able to treat simple and small
samples, such as Mario. It is able to read script tags and a subset of Music XML tags and
converts from them to MIDI events. µ-Script Language is lexical analyzed and parsed by this
system. The system controls a flow of music, transposition, and a tempo by calling APIs of
the language. Moreover, socket connections are allowed in this system. They are powerful in
coordination with external applications and devices. The tempo is controled by using both a
socket and APIs in the Mario sample.
However, the system has three problems in the present version of implementation. The
first one is that it is difficult to describe complex scripts since the syntax of µ-Script Language
is small. Additionally, a kind of the API is not enough. It is necessary to extends the syntax
and APIs, so as to support class and closure or to enable to control a window. The second is
that the system cannot read all of the Music XML tags. Some unread tags should be converted
to MIDI events. Hence, the system plays only the piano, for example. It is necessary to
implement both a reader and a converter which treat these tags. The thrid is the GC is not
efficient. When this system runs for a long stretch of time, unnecessary objects might increase
and burden the software. They might prevent this system running at worst. It is necessary to
improve in the quality of GC.
25
Chapter 7
Related work
CAL (Cakewalk Application Language) [7] is a language that treats music. It is similar to Lisp
in that it is a function programming language. It is not appropriate to treat music, because
music is a proceedure, not a function. In this language, prefix operators are used, but infix
ones are not. For instance, ”(+ 1 1)” in Lisp means ”1 + 1” in mathematics, and ”(< x
y)” means ”x < y”. Additionaly, a pair of parentheses is a function application operator. It
makes easy to lexically analize and parse the language. Since function applications are nested,
however, there are too many parentheses and it is not friendly for users when the nest is deep.
Macromedia Flash is a famous application in creating contents which synchronize with
music. Music is treated as background music in this appliction. Therefore users set only the
beginning of music, but can not control it afterwards. Since files used in Flash are binary
format, it is difficult for the famous internet searching systems to find and make an index of
these files and the development kits is needed to create contents.
Music XML is fast growing new file format in digital music [1]. This format can be used
in many famous application, such as Finale or Sibelius [8].
Script language is embedded to many other languages, such as HTML, VRML97(Virtual
Reality Modeling Language) [12] and Maya [13].
Java Script is found most often in HTML files. It gives movement to objects, such as text
data, images and browsers and can also get and handle user interactions. VRML97 is a format
to describe interactive 3D objects for world wide web. Java Script customized for the format
can be used. It is written in a script node. It handles events, such as user interactions, and
controls objects and environments of VRML. Maya is a complete workflow solution of mod-
eling, animation, visual effects, and advanced rendering technologies. MEL (Maya Embedded
Language) can be used in Maya. It is such a language as Java mixing with a shell script. It
enables to customize user interfaces and to create original visual effects.
26
Chapter 8
Conclusion
This paper have proposed the design and the implementation of the µ-Script System. This
system treats Music XML files embedded the µ-Script Language. It makes unmusical ele-
ments synchronize with music during the performance of the music. Therefore, it repaires the
unexpected gap between music and unmusical elements which may be caused depending on
performance environment.
In chapter 2, this paper have proposed a framework of µ-Script System. This system
model is contrasted with conventional one. Then it have been shown that the system may be
suitable to contents based on and synchronized with music.
In chapter 3, µ-Script Language have been designed. It has a similar syntax to Java Script
and has APIs which controls a music player. Additionaly, it has a message queue and sockets.
Therefore, it connects to external applications, such as a tempo control sample application.
In chapter 4, µ-Script Interpreter have been implemented. It has tree functions; The first
one is a reader of Music XML files in which µ-Script is embedded. The second is a converter
from Music XML data to MIDI events. The third is the main interpreter of µ-Script. This has
a fast thread to execute main tasks and background threads to connect to servers through each
socket.
In chapter 5, a small and simple sample, a background music of Mario, have been shown
and have tried running on this system. It controls the tempo and the flow of music.
In future, we would like to make a program library of this system and would like to dis-
tribute it. Since it treats large files, it will be necessary to develop an authoring tool for this
system. Finally, it is necessary to create applications which enable more complex events, for
example, operating phisical objects.
27
Appendix A
Syntax of µ-Script Language
A.1 Lexical Syntax
A.1.1 Keywords
The keywords of this language are shown as follows:
abstract, as, break, case, catch, class, const, continue, debugger,
default, delete, do, else, enum, export, extends, false, finally, for,
function, get, goto, if, implements, import, in, include, instanceof,
interface, is, namespace, native, new, null, package, private, protected,
public, return, set, super, switch, synchronized, this, throw, throws,
transient, true, try, typeof, use, var, volatile, void, while, with,
However, there are keywords which are not implemented.
A.1.2 Punctors
The punctors of this language are shown as follows:
!, !=, !==, %, %=, &, &&, &&=, &=, (, ), *, *=, +, ++, +=, ,, -, --, -=, ., ..., /,
/=, :, ::, ;, <, <<, <<=, <=, =, ==, ===, >, >=, >>, >>=, >>>, >>>=, ?, [, ], ^, ^=,
^^, ^^=, {, |, |=, ||, ||=, }, ~
A.1.3 Numbers
The number of this language are shown as follows:
• HI : 0 [xX] [0-9A-Fa-f]+
• DI : 0 | ( [1-9] [0-9]*)
28
• FRAC : [0-9]*
• MANT : (DI (. FRAC?)?) | (.FRAC)
• EP : [eE] [+-]? [0-9]+
• 64bits floating-point number : HI | (MANT EP?)
• 32bits floating-point number : MANT EP? [fF]
• 64bits signed integer : (DI | HI) [lL]
• 64bits unsigned integer : (DI | HI) [uU] [lL]
A.1.4 Comments
The comments are the form of /* . . . */, and one line comments are the form of // . . . ↵
A.1.5 Strings
The strings are the form of " . . . " or ’ . . . ’.
A.1.6 Identifiers
The identifiers are shown as follows:
• identifier : [A-Za-z ] [A-Za-z 0-9]*
A.2 Syntax
simple.qualified.identifier : identifier
qualified.identifier : simple.qualified.identifier
primary.expression : null | true | false | NUMBER | STRING | paren.list.expression array.literal
| function.expression
paren.expression : ( assignment.expression )
paren.list.expression : paren.expression | ( list.expression , assignment.expression )
function.expression : function function.common | function identifier function.common
array.literal : [ ] | [ list.expression ]
postfix.expression : attribute.expression | full.postfix.expression
attribute.expression : simple.qualified.identifier | attribute.expression property.operator | attribute.expression
arguments
full.postfix.expression : primary.expression | full.postfix.expression property.operator | postfix.expression
++ | postfix.expression --
29
property.operator : brackets
brackets : { } | { list.expression }arguments : ( ) | paren.list.expression
unary.expression : postfix.expression | void unary.expression | ++ postfix.expression | -- post-
fix.expression | + unary.expression | - unary.expression | ~ unary.expression | ! unary.expression
multiplicative.expression : unary.expression | multiplicative.expression * unary.expression | multi-
plicative.expression / unary.expression | multiplicative.expression % unary.expression
additive.expression : multiplicative.expression | additive.expression + multiplicative.expression |additive.expression - multiplicative.expression
shift.expression : additive.expression | shift.expression << additive.expression | shift.expression >>
additive.expression | shift.expression >>> additive.expression
relational.expression : shift.expression | relational.expression < shift.expression | relational.expression
> shift.expression | relational.expression <= shift.expression | relational.expression >= shift.expression
equality.expression : relational.expression | equality.expression == relational.expression | equal-
ity.expression != relational.expression | equality.expression === relational.expression | equality.expression
!== relational.expression
bitwise.and.expression : equality.expression | bitwise.and.expression & equality.expression
bitwise.xor.expression : bitwise.and.expression | bitwise.xor.expression ^ bitwise.and.expression
bitwise.or.expression : bitwise.xor.expression | bitwise.or.expression | bitwise.xor.expression
logical.and.expression : bitwise.or.expression | logical.and.expression && bitwise.or.expression
logical.xor.expression : logical.and.expression | logical.xor.expression ^^ logical.and.expression
logical.or.expression : logical.xor.expression | logical.or.expression || logical.xor.expression
conditional.expression : logical.or.expression | logical.or.expression ? assignment.expression : as-
signment.expression
non.assignment.expression : logical.or.expression | logical.or.expression ? non.assignment.expression
: non.assignment.expression
assignment.expression : conditional.expression | postfix.expression = assignment.expression | post-
fix.expression compound.assignment assignment.expression | postfix.expression logical.assignment as-
signment.expression
compound.assignment : *= | /= | %= | += | -= | <<= | >>= | >>>= | &= | ^= | |=logical.assignment : &&= | ^^= | ||=list.expression : assignment.expression | list.expression , assignment.expression
statement.abbrev : expression.statement semicolon.abbrev | block | if.statement.abbrev | do.statement
semicolon.abbrev | while.statement.abbrev | for.statement.abbrev | continue.statement semicolon.abbrev
| break.statement semicolon.abbrev | return.statement semicolon.abbrev
statement.no.shortlf : expression.statement semicolon.no.shortlf | block | if.statement.no.shortlf |do.statement semicolon.no.shortlf | while.statement.no.shortlf | for.statement.no.shortlf | continue.statement
semicolon.no.shortlf | break.statement semicolon.no.shortlf | return.statement semicolon.no.shortlf
30
statement.full : expression.statement semicolon.full | block | if.statement.full | do.statement semi-
colon.full | while.statement.full | for.statement.full | continue.statement semicolon.full | break.statement
semicolon.full | return.statement semicolon.full
substatement.abbrev : empty.statement | statement.abbrev
substatement.no.shortlf : empty.statement | statement.no.shortlf
substatement.full : empty.statement | statement.full
semicolon.abbrev : ; | /* empty */
semicolon.no.shortlf : ; | /* empty */
semicolon.full : ;
empty.statement : ;
expression.statement : list.expression.allow.in
block : { directives }if.statement.abbrev : if paren.list.expression substatement.abbrev | if paren.list.expression sub-
statement.no.shortlf else substatement.abbrev
if.statement.no.shortlf : if paren.list.expression substatement.no.shortlf else substatement.no.shortlf
if.statement.full : if paren.list.expression substatement.full | if paren.list.expression substatement.no.shortlf
else substatement.full
do.statement : do substatement.abbrev while paren.list.expression
while.statement.abbrev : while paren.list.expression substatement.abbrev
while.statement.no.shortlf : while paren.list.expression substatement.no.shortlf
while.statement.full : while paren.list.expression substatement.full
for.statement.abbrev : for ( for.initializer ; optional.expression ; optional.expression ) substate-
ment.abbrev
for.statement.no.shortlf : for ( for.initializer ; optional.expression ; optional.expression ) substate-
ment.no.shortlf
for.statement.full : for ( for.initializer ; optional.expression ; optional.expression ) substate-
ment.full
for.initializer : /* empty */ | list.expression
optional.expression : list.expression | /* empty */
continue.statement : continue
break.statement : break
return.statement : return | return list.expression
directive.abbrev : empty.statement | statement.abbrev
directive.full : empty.statement | statement.full
directives : /* empty */ | directives.prefix directive.abbrev
directives.prefix : /* empty */ | directives.prefix directive.full
typed.identifier : identifier | identifier COLON type.expression
function.common : LP parameters RP result block
parameters : /* empty */ | nonempty.parameters
31
nonempty.parameters : parameter.init | nonempty.parameters , parameter.init
parameter : parameter.attributes typed.identifier
parameter.attributes : /* empty */ | constparameter.init : parameter
program : directives
32
References
[1] Good, M and Actor, G. ”Using MusicXML for File Interchange”. Proceedings of the Third Inter-
national Conference WEB Delivering of Music, IEEE Press, Los Almitos, CA, 2003, p. 153.
[2] Good, M, ”MusicXML for Notation and Analysis”. The Virtual Score: Representation, Retrieval,
Restoration, Walter B. Hewlett and Eleanor Selfridge-Field, eds., MIT Press, Cambridge, MA,
2001, pp. 113-124.
[3] Castan, G., Good, M. and Roland, P. ”Extensible Markup Language (XML) for Music Appli-
cations: An Introduction”. The Virtual Score: Representation, Retrieval, Restoration, Walter B.
Hewlett and Eleanor Selfridge-Field, eds., MIT Press, Cambridge, MA, 2001, pp. 95-102.
[4] Good, M. ”MusicXML: An Internet-Friendly Format for Sheet Music”. XML 2001 Conference
Proceedings(Orlando, FL, December 9-14,2001)
[5] Good, M. ”MusicXML in Practice: Issues in Translation and Analysis”. MAX 2002: Musical
Application Using XML, pp. 47-54.
[6] Finale, Coda Music Technology, http://www.codamusic.com/.
[7] Cakewalk Pro Audio, Cakewalk, Inc., http://www.cakewalk.com/.
[8] Sibelius, The Sibelius Group, http://www.sibelius.com/.
[9] Macromedia Flash MX, Macromedia Inc., http://www.macromedia.com/.
[10] MIDI Manufacturers Association, The Complete MIDI 1.0 Detailed Specification v96.1 (second
edition), 2001.
[11] Waldemar Horwat, ”Java Script 2.0”, 2003, http://www.mozilla.org/js/language/js20/.
[12] Information technology – Computer graphics and image processing – The Virtual Reality Modeling
Language(VRML), ISO/IEC 14772-1:1997 and ISO/IEC 14772-2:2002.
[13] Maya, Alias System Corp., http://www.alias.co.jp
33