Czech Technical University in Prague

Faculty of Electrical Engineering

Department of Computer Graphics and Interaction

Bachelor thesis

Electronic Keyboard Tutorial System

Petra Kalinová

Supervisor: Ing. Adam Sporka, Ph.D.

Study Programme: Software Engineering and Management

Field of Study: Web and multimedia

June 11, 2009

iv

Declaration

I hereby declare that I have completed this thesis independently and that I have listed allthe literature and publications used.

I have no objection to usage of this work in compliance with the act �60 Zákona£. 121/2000Sb. (copyright law), and with the rights connected with the copyright actincluding the changes in the act.

V Praze dne 11. 6. 2009 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

vi

Abstract

The purpose of this work is to design and implement an application used by beginnerpianists to play songs using an electronic keyboard connected to the computer via a MusicalInstrument Digital Interface (MIDI). The program can serve as an educational tool as wellas a means of entertainment for people who have already mastered playing the piano, dueto its style di�ering from standard musical notation by the way it symbolizes the toneswhich are being played. An educational element is represented by the set of editable lessonscontaining individual instructions and references to particular songs in Standard MIDI FileFormat. Each session may focus on a speci�c piano playing problem and while the song ispractised an evaluation is provided.

Abstrakt

Zám¥rem této práce je navrhnout a implementovat aplikaci umoº¬ující za£ínajícím pi-anist·m hrát skladby na elektronické klávesy p°ipojené k po£íta£i pomocí rozhraní MIDI.Tento program m·ºe slouºit nejen jako výuková pom·cka, ale také jako zdroj zábavy prouºivatele, kte°i uº základní techniku hry na klavír ovládají, protoºe zp·sob, jakým jeskladba zapsána se li²í od b¥ºn¥ pouºívané hudební notace tím, jak jsou symbolizoványjednotlivé noty, které se mají hrát. Výukový element je reprezentován formou editovatel-ných lekcí, které obsahují konkrétní pokyny a odkazy na skladby uloºené v MIDI souboru.Kaºdá lekce se tak m·ºe zam¥°it na speci�cký problém. Hodnocení hrá£e probíhá jen-dotliv¥ po kaºdé p°ehrané skladb¥ a slouºí jako rozhodovací faktor pro jeho postup nadal²í obtíºnostní úrove¬.

vii

viii

Contents

List of Figures xi

1 Introduction 1

1.1 Traditional music education . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Speci�cation of the goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Relevant techniques and technologies 3

2.1 Alternative music notations and music storage systems . . . . . . . . . . . . 32.1.1 Piano roll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Klavarskribo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1.3 Standard MIDI File format . . . . . . . . . . . . . . . . . . . . . . . 42.1.4 MusicXML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Embedded solutions in electronic keyboards . . . . . . . . . . . . . . . . . . 52.2.1 Electronic keyboard . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.2 Yamaha Education Suite . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Music video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3.1 Konami KeyboardMania . . . . . . . . . . . . . . . . . . . . . . . . . 72.3.2 Guitar Hero and Rock Band . . . . . . . . . . . . . . . . . . . . . . . 82.3.3 Synthesia - Piano for Everyone . . . . . . . . . . . . . . . . . . . . . 8

2.4 Research summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 System requirements speci�cation 11

3.1 System requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Requirements speci�cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.1 Evaluating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.2.2 Playback system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 User interface design 13

4.1 Representation of music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Representation of piano keyboard . . . . . . . . . . . . . . . . . . . . . . . . 15

5 System architecture 17

5.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.1.1 MIDI module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.1.2 System module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2 Management of input and output . . . . . . . . . . . . . . . . . . . . . . . . 185.2.1 Screen management . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.2 Data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2.3 Graphics management . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6 Implementation 21

6.1 Software development tools and libraries . . . . . . . . . . . . . . . . . . . . 216.1.1 Programming language . . . . . . . . . . . . . . . . . . . . . . . . . . 216.1.2 Graphics framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.1.3 Integrated Development Environment . . . . . . . . . . . . . . . . . 226.1.4 MIDI Application Programming Interface . . . . . . . . . . . . . . . 22

6.2 Implementation highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

ix

6.2.1 Application graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 236.2.2 PianoRoll library overview . . . . . . . . . . . . . . . . . . . . . . . . 246.2.3 XML object loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.2.4 Input from keyboard . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.2.5 External tools for lesson authoring . . . . . . . . . . . . . . . . . . . 25

7 Testing 27

7.1 Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.2 Testing process and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

8 Conclusion 29

8.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

9 Bibliography 31

A List of acronyms 33

B UML diagrams 35

C Application screenshots 39

D Usability testing questionnaire 41

E User manual 43

F CD Content 45

x

List of Figures

2.1 Original piano roll, source [16] . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Piano roll in MIDI software . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Example of Klavar notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.4 Yamaha's lighted keyboard, source [17] . . . . . . . . . . . . . . . . . . . . . 62.5 Screenshot of KeyboardMania video game . . . . . . . . . . . . . . . . . . . 72.6 Guitar Hero video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.7 Synthesia video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.1 Final design of playback UI . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 2D design of playback UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.3 Playback screen storyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.4 Other possible designs of playback UI . . . . . . . . . . . . . . . . . . . . . 15

5.1 Hierarchy of modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2 Diagram of application's menu �ow . . . . . . . . . . . . . . . . . . . . . . . 195.3 Diagram of objects in XML �le . . . . . . . . . . . . . . . . . . . . . . . . . 205.4 Graphics lifecycle in application . . . . . . . . . . . . . . . . . . . . . . . . . 20

6.1 Scheme of connection between computer and electronic keyboard . . . . . . 226.2 Orthogonal views of playback scene with camera viewports . . . . . . . . . . 246.3 Relationships between PianoRollLibrary namespaces . . . . . . . . . . . . . 246.4 Mockup of external authoring tool . . . . . . . . . . . . . . . . . . . . . . . 26

7.1 Illustration of test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

B.1 User's use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35B.2 Design of data model classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 36B.3 Interaction and relationships between PianoRollLibrary members . . . . . . 37B.4 Screen management classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

C.1 Main menu screen with static background image . . . . . . . . . . . . . . . 39C.2 Playback scene before tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 40C.3 Playback scene adjusted according to test results . . . . . . . . . . . . . . . 40

xi

xii

CHAPTER 1. INTRODUCTION 1

1 Introduction

Playing the piano is a skill usually learnt during childhood from professional teachers,typically in weekly sessions. Although the grand piano has limited mobility and is oftenexpensive, it is still one of the most familiar musical instruments and the most favoured tobegin with. In recent years, popularity of keyboard instruments has been boosted by theincreasing quality and the decreasing price of more portable versions of piano � the digitalpiano and the electronic keyboard.

Looking at the age of beginner pianists there is a signi�cant amount of adults who decideto learn to play the piano. However, most potential students give up the idea even withouttrying. The reasons might di�er from case to case but the majority are frightened bythe necessity of the traditional education system which could appear to be too pedantic.Furthermore, there is the musical notation which might scare o� another group of people,although reading music is not as hard as it seems at �rst sight.

Having a talent for music is not the key point to successfully learn playing any musicalinstrument as it requires lot of patience as well as regular exercising and nothing can expressthe whole principle more accurately than the phrase: Practise makes perfect. In additionthe gained skills must be maintained by occasional training and can be even completelyforgotten after some time.

1.1 Traditional music education

In the Czech republic, a type of school has been established called Základní um¥lecká²kola (ZU�) where the education of di�erent art forms including music is learnt. Theseschools have a long lasting tradition in every major town and are also responsible fororganizing various local competitions of young artists. Although they are administratedby the Ministry of Education, Youth and Sports (M�MT), the education expenses arepartly covered by students' tuition fees.

Most of the time spent in front of the piano keyboard during the �rst few sessions isused for learning the position of each key on the sta� and by the time the student gets thisinto practise, he might be already bored or fed up enough to discontinue further training.While children are usually forced to attend the lessons by their parents, the adults whodecide to play on their own are sacri�cing their free time and spending their money, and ifthey feel that both could be used for something more enjoyable, they simply stop and thenegative experience may make them possibly never try it again.

Before beginners �rst sit at the piano they have to attend a few music theory lessons andbecome familiar with speci�c terminology and the principal of notation. The student thencontinues with acquiring the basic playing technique and the identi�cation of the correctpiano key according to the notation. The learning process as a whole consists of the maintopics which are: ear training, rhythm, notation, technique, improvisation, sight readingand memorization [2]. The actual teaching method has to adapt according to student'sdi�erent personality, learning style, education level, and so on.

The requirement of individual education plus the high purchase cost of the instrumentitself makes piano training expensive and therefore is not a�ordable for everyone. The ideaof a complete solution for teaching oneself to play the piano using either printed or an

2 CHAPTER 1. INTRODUCTION

electronic tutorial is not the right one. As it applies mostly to motor skills; the bad habitsgained at the beginning are later hard to get rid of. At least an occasional supervision bya professional piano teacher who checks the technique and recommends suitable exercisesfor further training is necessary.

1.2 Speci�cation of the goal

The aim of this thesis is to implement a tool which would make the �rst steps of beginnerpianists more comfortable and enjoyable than it would have been while applying the classiclearning methods. The plan is to �nd another way of displaying the notes in the musicbeing played so the user can avoid the necessity of learning the common notation in the�rst place. The requirement for frequent practise highlights the importance of the system'sability to maintain the user's motivation to repeat the songs over and over. The motivationused for the same purpose in the computer games is to record an amount of points gainedduring the song playback in a form of score.

As stated in the project's assignment, the tutorial system requires an electronic keyboardconnected to the computer using the Musical Instrument Digital Interface (MIDI) whichwill be described in more detail further in the text. The possibility of communication be-tween the instrument and the application brings forward an essential interaction with theuser and provides them with feedback from the system to their actions which is a funda-mental principle of each learning method. These two elements, interaction and feedback,are the reasons for the decision to connect the devices together and bring the piano lessonsinto another level.

The development of our application will begin with background research of technolo-gies which have something in common with our system. According to the results of ourinvestigation, we will be able to de�ne the requirements of the system. Then we specifyits concrete functions which will be further designed and implemented. The qualities andlimitations of �nal product should be revealed during the tests.

CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 3

2 Relevant techniques and technologies

This chapter consists of an explanation of the situation regarding piano playing lessonswhich will help us with identifying the problems which we should be aware of. Then welook closer at the notation system and analyze the non traditional ways of displaying themusic. The description of existing either educational or entertainment systems follows.

2.1 Alternative music notations and music storage systems

Music notation is a system which describes music using written symbols. The way of nota-tion has been evolving since the prehistoric ages similarly to the writing system. Modernmusic notation has its origin in European classical music of 18th century and has beenwidely used throughout the world until now. However, there has always been the inten-tion to discover an alternative which would facilitate music-reading. With an increasingimprovement in the area of technology, music production has moved, during the centuries,from mechanical systems such as orchestrion to electronic devices capable of not only re-playing but also storing and editing the music. Therefore the need for translation of thepaper notation into its electronic form has appeared. Common audio formats used for thesound storage and by audio players usually manipulate the actual audio wave unlike thedescriptive music �le formats. Examples of mentioned technologies follow in this chapter.

2.1.1 Piano roll

Piano roll is the �rst industrially reproduced music storage medium used by self-playingpianos, i.e. Pianola. It is made of the roll of paper with perforations which determinethe position and length of the note played on the piano. The roll moves over a speci�cdevice with holes, one for each piano key. When a perforation passes over the hole, thenote sounds. This mechanism now belongs to the museum but the term piano roll is stillused in MIDI data editing software as a name for another option of showing the datain di�erent way from the standard musical notation. Piano roll typically provides colourcoding, variable-length symbols to indicate note duration, and a chromatic keyboard-stylelayout. The examples of the historical and modern look of piano roll style are shown inFigure 2.1 and Figure 2.2.

Figure 2.1: Original pi-ano roll, source [16]

HUD

Playing instructions

Keyboard representation

Time Score

Keyboard

Track representation

PCscreen

Moderator

Participant

00:25 17463

Time: 00:25

Score: 17463

00:00 0

00:25 0 00:22 134

00:19 421 00:16 650

Figure 2.2: Piano roll inMIDI software

4 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.3: Example of Klavar notation

2.1.2 Klavarskribo

Klavarskribo is a music notation introduced in 1931 by the Dutchman Cornelis Pot. Itdistinguishes from the conventional notation by being easily readable. However, it hasn'tmet with great success among music teachers and the only signi�cant group of people usingthe notation in practice nowadays are the members of the promoting organizations in theNetherlands [10]. Although the notation is universal for all instruments and for singing itsuits keyboard instruments the most since the design logic is based on the piano keyboard.The example of Klavarscribo can be seen in Figure 2.3.

The klavar notation itself is unique in the way the stave is placed on the page andcorresponds to the real piano keyboard more than the traditional layout which consistsof �ve horizontal lines and four spaces whereas in the klavar stave, the notes are writtenvertically and the music is read from top to bottom. The lines match the position of theblack keys on the keyboard thus each note has its own individual place and no sharps, �atsor even any key signature are needed. In other words, Klavarskribo presents a notation inits straightforward form, stripped of the music theory. Whether the note is played by rightor left hand is indicated by the orientation of note's stem. The Klavarskribo Foundationhas transcribed over 25,000 pieces and custom composition can be transcribed easily withthe aid of specialized software, the KlavarScript.

2.1.3 Standard MIDI File format

The Musical Instrument Digital Interface is a speci�cation which was �rst announced in1982 and de�nes a communication protocol used for transfer of music data in a form ofMIDI messages through MIDI devices. Although the standard was primarily designed forkeyboard instruments only, it was soon extended to special drums, guitars and even enablesthe lighting to be synchronized with the music during live performances [1]. Apart from thespeci�c protocol, the MIDI Manufacturers Association (MMA) also de�ned a speci�cationfor the format for storing MIDI messages, formally called a Standard MIDI File (SMF)format.

Remarkably this �le format is small in size in comparison with the Waveform audioformat (.wav) where the actual sound wave is written directly in the �le contrary to theMIDI �le format (.mid) which contains just the data describing the music and the speci�c

CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 5

representation of the sound depends on the current synthesizer. Another advantage overthe wave storage system is ease of editing, composing or recording the music data includingthe possibility of its conversion to the printable standard notation eventually.

SMF is di�erent from native MIDI protocol, because the events are time-stamped forplayback in the proper sequence. The data are stored in hexadecimal form and are struc-tured into two di�erent types of chunks, header and track. The �le does not contain onlythe notes but also stores the information about instrument used to replay the track ina form of speci�c instrument patch which is identi�ed by a Program Number parameterwith a range of 128 possible values where each number corresponds to one melodic sound.The portability between several synthesizers is ensured by the instrument table de�ned inGeneral MIDI (GM) standard.

2.1.4 MusicXML

This type of music notation is not meant to be used by the players directly but can serve asa storage format further used by music production software. The transcription is based onExtensible Markup Language (XML) which is the format widely used by automated toolsto be parsed and manipulated. MusicXML was designed for the interchange of scores.Although it can be possibly written by hand, a more practical approach is to use aninteractive score writing program. More details can be found in [12].

2.2 Embedded solutions in electronic keyboards

After the examination of the paper and digital representation of music we can move toits actual producing in a form of sound output from electronic device such as keyboard,which will serve as an input for our application. In addition, one interesting attempt ofan educational system which is built-in and shipped with the instrument as a completesolution will be presented.

2.2.1 Electronic keyboard

A keyboard is an electronic device resembling a grand piano by having similar lookingmusical keyboard. While it lacks not only the piano's size and weight but also its timbreand keystroke, it can actually sound like more than one instrument by a simple change ofthe settings. Although the generated reproduced audio limits the �nal credibility of theinstrument's sound, it o�ers another big advantage against the classic piano which is thepossibility of sound volume regulation or even the redirection of the output into a headset.

Today, a variety of di�erent types of keyboards exist on the market. From inexpen-sive instruments targeted at amateurs and children to truly professional synthesizers richin many features. Nonetheless even the mainstream is able to show o� many functionsand o�ers reasonable overall quality due to recent advances in computer and electronicstechnology.

In addition to a large tone bank, the instrument typically contains pre-programmedrhythm patterns for several musical styles presented as an auto-accompaniment. The keyfunction in the sense of piano imitation is the touch response, also called touch sensitivity

6 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.4: Yamaha's lighted keyboard, source [17]

and does not have to always be included. The optional parts of the keyboard are additionalsockets placed on the back which allow the user to plug MIDI or audio inputs and outputs.

2.2.2 Yamaha Education Suite

Yamaha belongs to the group of electronic piano manufacturers and its products are wellknown among the musical public. Various keyboard series contain the built in YamahaEducation Suite (YES) which is aimed at helping the user with piano playing. Severaldi�erent versions have been developed since 1998 [17].

The suite contains a system of lessons, built-in songs and a chord dictionary which isthere to display the notes and chord names on the LCD screen. The lesson itself can bedisplayed in numerous modes in order to provide more ways to practice. Short descriptionsof some modes follows. Timing mode allows the user to play the melody by pressing anykey and only the correct timing is important. Waiting mode stops the playback of a songuntil the player �nds the correct note and then continues. Minus One mode plays the songat the normal tempo minus the part the player has chosen to play. Each hand can betrained separately and in addition, the correct �ngering is indicated on the LCD displaywhile the lesson is running. The lesson system wouldn't be complete without the progressmonitor to present lesson grading.

The Yamaha EZ series o�ers a helpful feature which is a keyboard with LED embeddedkeys allowing interactive learning. Figure 2.4, shows such an instrument. On these modelsa red light operates as a guide to tell the player which note to play and when. In earlierversions of keyboards the lights were placed right above the keys while the latest ones aremade to have the e�ect of the key itself emitting the light. To facilitate the learning processeven more, the song stops and waits for the player to �nd the correct key thus letting themcomfortably play the song on their own. Not only Yamaha but also Casio, another wellknown brand producing electronic keyboards, o�ers similar illuminated keyboards.

2.3 Music video game

This section gives a brief overview of products of the gaming industry which belong to themusic-themed video game genre and can be closely speci�ed as a rhythm game, a subgenreof action game. These games are based on simulating the playing of musical instrumentswhere the player's accuracy is further evaluated. Achieving a high score is the main game-play factor, further boosted by the support of multiplayer mode. The speci�c instrument

CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 7

Figure 2.5: Screenshot of KeyboardMania video game

used has a function of game controller and is either in its convenient or simpli�ed version.The players follows the instructions on a scrolling display which in principle is similar topiano rolls. The short description of arcade and console games in this chapter is followedby a presentation of a software application which has its origin in the previously mentionedgames and the development of which was stimulated by their increasing popularity. Thesegames have a lot in common and can inspire us in the playability of the elements theycontain.

2.3.1 Konami KeyboardMania

Keybaordmania is a rhythm video game published in 2000 by Japanese company Konami,the producer of video games and toys and is actually an owner of several music game-related patents. Since 1997, the company's Bemani division, has released many di�erentseries of music games which have met with greater or lesser success but de�nitely served asinspiration for other products of the genre further described in the next section. Benami`smost successful rhythm game which, unlike the other Konami's products, has crossed theborder of Japan and was released in North America and Europe in 1999, is Dance DanceRevolution (DDR) where players control the game by the movement of their feet on aspecial pad, in the order dictated by on-screen instructions. it is remarkable to notice thatthis game is now used not only for its entertainment purpose but also as a an exercise tool[5].

The notes in Keyboardmania are represented on-screen by small bars that scroll down-ward above an image of the keyboard itself. The goal is to play the matching key whena note bar descends to the red play point line. The game has several Japanese releasesintended for arcade cabinets, two home versions for Playstation 2 which use a special key-board controller and a Windows PC version shipped with the Yamaha EZ-250i lightedkeyboard a screenshot of which can be seen in Figure 2.5. The last mentioned softwarebrings the concepts of Yamaha's YES and allows users to practice their playing in di�erentlevels and even load any General MIDI song. However, the game is not included with thelatest version of Yamaha's lighting keyboard, EZ-200 [17].

8 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.6: Guitar Hero video game

2.3.2 Guitar Hero and Rock Band

Guitar Hero, released in November 2005, and Rock Band, released 2 years later, both belongto the same category of modern music video games distributed on various video gameconsoles. Each of them requires special controllers in the shape of a musical instrumentto play built-in songs of various genres. The increasing popularity of these games led tothe development of several custom clones which mostly interact with MIDI �les and arecontrolled using a PC or MIDI keyboard, i.e. Synthesia [11].

The most widely known is a guitar peripheral with �ve or more colored buttons on theneck and a strumming controller. The mechanics of the game lies in pushing down theinstrument's color matching buttons according to the indicators rolling on the board inmusic's rhythm, at the right time. The rating is represented on a meter which should staygreen and only goes red when a player misses too many notes which can even cause thesong to stop completely if the performance is too bad. For better imagination, the pictureof the playback screen can be seen in Figure 2.6.

The game supports not only single player modes but also various multiplayer modeswhere the songs are performed by more players at the same time. Unlike most rhythmgames, Guitar Hero and Rock Band have a career mode that guides the player throughthe di�culty settings and o�ers decently conceived learning curve which makes the gamesuitable not only for experienced gamers but also for entire beginners. Another remarkablepoint witch makes those two games di�erent from most of their predecessor and clones, isthe graphics design suiting the genre and a party game mode has been made for it as well.Although the board with rolling notes dominates the layout, there are other elements suchas animated background which does not lead the player's attention away and only makesthe screen more interesting for observers.

2.3.3 Synthesia - Piano for Everyone

Synthesia is a computer game developed by Nicholas Piegdon with the �rst release pub-lished in November 2006 and the latest one which is dated from September of 2008. Itallows the user to play songs in a MIDI format using a musical keyboard connected tothe computer. The project was initially named Piano Hero due to the in�uence of GuitarHero; however, it soon had to be renamed as the author received a cease and desist letterfrom Activision, the publisher of the Guitar Hero [11]. A player determines which pianokey should be played from the position of the note falling from the top of the screen andtravelling towards a piano displayed at the bottom as is shown in Figure 2.7, thus no sheetmusic reading is required. The users can choose whether the track or the tracks of theMIDI song will be played by them or automatically by the program. A basic evaluation

CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 9

Figure 2.7: Synthesia video game

system in a form of per-song scoreboards which enables the players to keep track of theirprogress is included. While the basic functionality is still free, a Learning Pack key can bepurchased to unlock additional features, such as a sheet music display mode.

2.4 Research summary

Looking closely at all the described products the YES �ts our goals the most. Its userscan largely pro�t from o�ered modes while practicing their playing but are heavily limitednot only by the built-in selection of songs but also by the small screen with low resolution,placed on the keyboard. The lighting key feature can serve well when playing an easy songbut with an increasing di�culty the player would appreciate the knowledge of incomingnotes in order to prepare the �ngers in time.

Piano roll notation seems to be suited more to music editing than learning and musicperformance due to lack of guidance on which hand or �nger plays the note, the elementswhich are present in standard notation and Klavarskribo. The piano roll style is widelyused in the mentioned video games but those systems are not designed to teach the user toplay the real instruments unlike the educational purpose of our application. We will takeinspiration from the gameplay elements which we think should not be missing even in atutorial application.

Synthesia, which also uses a piano roll style for representing the music, seems to be theapplication which suits our ideas about the system. Simple and well-arranged 2D layoutwith the set of horizontal equidistant lines, following which the user should get used toplaying quite fast seems to be the right representation of the piano roll style. AlthoughSynthesia o�ers the mentioned optional learning pack, it consists of MIDI �les of severaldi�culties only and, apart from sheet music display, it does not contain any additionalinstruction info about piano playing technique.

While designing our application we will try to follow the user interface (UI) designconcept of previously mentioned systems together with an attempt to keep the programhighly educative and interactive at once with a regard of remarks gained in this research.

10 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

CHAPTER 3. SYSTEM REQUIREMENTS SPECIFICATION 11

3 System requirements speci�cation

In this section the requirements of the system de�ned in the project's assignment are listedand described in more detail, together with other requirements which follows the summaryfrom the previous section.

3.1 System requirements

Following functional requirements de�ne what the system should do.

• The system allows the user to play the songs without the player's knowledge ofmusical notation.

• The system provides the instruction information about piano playing in the form ofcomplete lessons.

• Lessons have a speci�ed number of songs which are required for their completion.

• Lessons consist of an individual amount of songs and it is up to user which songsthey choose to play.

• A song is considered completed when a required percentage of score points is gainedby the player.

• Before the song starts playing, the user can choose to play the song in a slower tempobut is penalized as the score is evaluated.

• While playing the song the user is informed about the remaining time of the songand the amount of score points gained.

• The song's playback can be paused by the user.

• When the song is �nished the user is informed of their achievement.

• An electronic keyboard and PC are connected together via a MIDI interface.

• Lessons data are de�ned in an external �le which is possible to edit.

• The songs are identi�ed by their names; the song's author might be added as well.

• The application can be exited at any point by pressing and con�rming the escapecommand.

The system's behavior from the user's point of view is described in a use case diagramplaced in Figure B.1. End users of the system are people of any age who are willing toconnect two devices together and spend their free time playing the keyboard in front oftheir computer.

12 CHAPTER 3. SYSTEM REQUIREMENTS SPECIFICATION

3.2 Requirements speci�cation

First of all, the user needs to have an electronic keyboard connected to the computervia a USB MIDI interface in order to be able to participate in the educational processwhich consists of several lessons where new players can register and participate. The non-educational mode is called Quickplay mode and while performed, no results related to theplayer are stored in the system. After the system's launch the user is informed if any MIDIinput device have not been recognized. The user controls the application using PC orpiano keyboard to navigate through the menu. The con�gurable options of the system arelocated in an options menu and consist of resolution, MIDI input and output settings anda toggle full screen mode entry. The song data are kept in the SMF while the additionalinformation like the song's name, its author, the default tempo, the di�culty and the trackbeing played are stored in the system's data structures. The user commands and external�les can be considered as system's input.

3.2.1 Evaluating system

The player's achievement is expressed in a form of score points. Several possible di�cultymodes of song exist. The simplest mode is when only the correct rhythm is evaluated andthe key user has pressed is not important allowing him to pay attention at the song itself,not the positions of played notes. Another mode is when the correct oncoming note ispressed but the exact time of its press is not evaluated. Normal mode is when both therhythm and correct note is rated with a tolerance which di�ers with the playback tempo.The tolerance is lowest at the slowest tempo. The di�culty mode currently used is de�nedin the lesson's data and the user is given a notice about the actual system of evaluationbefore playback starts.

3.2.2 Playback system

As already mentioned in Subsection 2.1.3, the SMF song consists of one or more paralleltracks where each track actually represents one part of the overall piece of music and isassigned to a speci�c instrument. For example the melody performed on the trumpet isstored in the �rst track while the piano accompaniment is stored in another. The systemcurrently supports only one graphical representation at a time.

The interpretation of the playback as it is presented and designated to be played bythe user lies in the conversion of a speci�ed track to its real time graphics representation.While the designs of various playbacks used in researched programs aimed to be performedon the keyboard are in two-dimensional (2D) view, we have decided to originate our play-back by using an attractive looking three-dimensional (3D) view. The aim is to makeour design intuitive and easily readable for the players who are not familiar with any ofdescribed similar tools. The challenge rests in representing the whole keyboard containingseveral keys unlike the described guitar or drums peripherals with maximum of �ve dif-ferent positions to watch and play. Although a few of designs presented further had beenoutlined before the actual application implementation, several details have been adjustedin the functioning program prototype and after the application usability tests described inChapter 7.

CHAPTER 4. USER INTERFACE DESIGN 13

4 User interface design

This part of the product design process usually consists of an activity called prototyping.A prototype is a form of early product's model with limited functionality the purpose ofwhich lies in verifying the suitability of a speci�c design approach and allows comparisonbetween alternatives. The advanced version of prototype is a mockup which provides atleast part of the functionality of a system and enables testing of an actual design. In thefollowing section a few drafts of our system which can further serve as a basis of the paperprototype will be presented.

Because the playback screen is the key point of the whole application, the following textis focused on its design. The arrangement of the playback screen's graphical user interface(GUI) depends primarily on the position of the keyboard model, since it is the largeststatic element on display. As mentioned in the previous chapter, the scene is planned tobe created in 3D space rendered onto 2D screen from a static position of a camera. Theperspective view of the piano keyboard has been chosen because of the more realistic lookand also because the object will take less space on the screen. Although the keyboardis not meant as the dominant part of the screen, its position in the lowest bottom partensures that it is still well visible and also as close as possible to the real piano keyboard,placed in front of the computer screen.

Apart from rendered 3D playback scene, the screen used for playback contains the Heads-Up Display (HUD), the static part of user interface which in video games is used to displayinformation important for player during the game. Our HUD contains only two elementswhich is the amount of score points and the remaining time of the song as can be seen inFigure 4.1. Figure 4.2 shows the 2D concept, used in Synthesia application mentioned inSubsection 2.3.3, to enable the reader straight comparison between two di�erent views.

4.1 Representation of music

Another part of the 3D scene is the graphical representation of the notes the user will playfor the musical part. As stated at the end of the research chapter our application will stickwith the piano roll style as it is easy to transform from the SMF and intuitive enough forthe player to read. Each note of the played track is shown as a box directly proportionalin length to the note's duration above the corresponding piano key. The box moves towardthe piano key model and when its near edge touches the key, the key should be pressed

Figure 4.1: Final design of playback UI

14 CHAPTER 4. USER INTERFACE DESIGN

Figure 4.2: 2D design of playback UI

Figure 4.3: Playback screen storyboards

and when the far edge leaves the screen the key should be released. The illustration of thejust described process can be seen in Figure 4.3.

The boxes appear on the screen in advance before they should be actually played togive the player time to prepare. When playing an advanced piece of music the techniqueof preparing �ngers to incoming notes applied while playing from the standard notationneeds to be used [2]. The beginner user should follow the lesson system carefully to getused to more di�cult pieces automatically. The success of the overall education systemlies in the lesson's design, which is not the exact purpose of this work and brings out asuggestion for further work.

The lines heading to the keys on which the boxes trace have a di�erent colour accordingto the octave where they end. Another line crosses the line where the keyboard begins atthe time where the downbeat appears in the song bringing in a replacement for regular barline known from standard notation. The position of the lines is in the same plane as the

CHAPTER 4. USER INTERFACE DESIGN 15

Figure 4.4: Other possible designs of playback UI

keys are and the perspective view makes the boxes get bigger when they are come closeror in other words are going to be played. This design has been chosen over the other twoconcepts shown in Figure 4.4 which were rejected due to their being less able to adapt tothe scene requirements.

4.2 Representation of piano keyboard

If the electronic keyboard is connected and the system is receiving the MIDI input messagesthe models of keys on the screen move up and down as the user presses or releases them.Timestamps of the notes in the songs and of the played keys are the data for evaluatingthe user's score. The number of keys which can be displayed on the screen is limited by thecurrent screen resolution which can be customized in the Options menu. The minimum ofoctaves displayed is the number to cover the actual song range and others are shown to�ll the space on the screen and since not used, they are shaded to express their inactivity.Already referenced Figure 4.3 also shows the various cases of the piano key states in thecombination with the di�erent position of corresponding note box.

16 CHAPTER 4. USER INTERFACE DESIGN

CHAPTER 5. SYSTEM ARCHITECTURE 17

5 System architecture

After we got through the speci�cation of the program's functions and we have a rough ideaof its graphical look, it is time to plan the solution from the software architectural viewand to build preliminary data models. The �rst steps involve the division of the systeminto its separate objects so the application can be later easily implemented using objectoriented programming. The goal of the design is extensibility so the new capabilities canbe added to the software without major changes to the underlying work. The ideas forfuture work are introduced in the �nal part of this thesis.

5.1 Components

Beginning with the design from the global view, we have decided to split the systemas a whole into three logical parts where each part has its own functionality capability.The hierarchy of those parts including the MIDI API is shown in Figure 5.1. The �rstmodule, which uses services of the MIDI API is the library managing the MIDI commands,the second is the system library containing the graphical and MIDI related objects andmethods. The last but not least is the part where the application's entry point is placedand which is responsible for the screen and data management.

Figure 5.1: Hierarchy of modules

5.1.1 MIDI module

The purpose of this library is to encapsulate the low-level functions from the system'sMIDI API library for their later manipulation in the program. It should be able to handlethe data retrieved from the library on the level of the objects as well as read the data fromthe SMF. All the abstract MIDI entities such as input and output devices, sequencers,sequences, tracks, clocks etc. should have their object equivalents in this library and themethods streaming the MIDI data should be provided. The discussion about usage of theexternal solution for this task is discussed in Subsection 6.1.4.

5.1.2 System module

This library represents a bridge between the MIDI objects and the application's maincomponents and as it performs di�erent functional assignments. It is essential to divide itinto more parts:

18 CHAPTER 5. SYSTEM ARCHITECTURE

MIDI � methods and objects which communicate directly with the MIDI library.

Recognizing, identifying and communicating with MIDI input and output devices.

Classes for sending and receiving various MIDI messages.

Song � methods working with objects storing the data from MIDI song �le.

Parsing the content of the SMF and saving them into tracks which consist of noteswhere each note has its pitch and duration.

Class which gathers needed data and perform the MIDI song playback with its ownMIDI clock. Also includes the displaying of 3D models.

3D model � contains parts of the playback scene.

Loading the graphical objects from �les and displaying them.

Input � handling the input from the user's electronic keyboard, if connected.

Exception � speci�c exception to be thrown in potentially unstable places further caughton the higher level.

Score � calculating the score points when the piano key is pressed. The amount dependson the actual timestamp and other parameters.

5.2 Management of input and output

5.2.1 Screen management

Screen manager is a class responsible for displaying various screens according to the currentstate of the system in which it occurs and which depends on players input mainly. The usercan navigate the menu either using the PC or the piano keyboard following the descriptiveicons which are part of the background image. The concrete menu �ow diagram [4] ofour system can be seen in Figure 5.2. Various types of screens with di�erent functionalitywhich are described further were arranged into a tree hierarchy. The class diagram of thesescreens can be found in Figure B.4.

Screen manager � maintains a stack of screens and calls their update and draw methodsand routes the input to the active screen.

Game screen � screen which has its own update and draw logic and serves as a parentscreen for all screens which description follows.

Splash screen � displayed as a �rst screen when the application is launched.

Message box � a popup screen which communicates with the user.

Info screen � a screen which enables displaying the data on several pages.

Menu screen � base class for screens that contain a menu of options which can be navi-gated and selected by the user

CHAPTER 5. SYSTEM ARCHITECTURE 19

Player #1

2 Players

- New Player -

Player #2

- New Player -

Back 1

6Training

1 Main Menu

Options

Quickplay

About

Exit

Continue

6 Lessons

Lesson #2

Lesson #1

Lesson #3

Back

LessonInstructions

7 Lesson #1

Back

Song #1

3 Songs

Song #3

Song #2

Song #4

Back

9 Playback

Back

10 Results

Replay

Back

Input device

4 Options

Resolution

Output device

Full screen

Back

5 About

Back

8 Loading

Tempo

Back

0

2

3

4

3Continue

3

5

7

8

11

2

8

36

6 / 1

6 / 1

9

10

SongInstructions

y

z x

z

Figure 5.2: Diagram of application's menu �ow

Background screen � stays behind all the menu screens and draws the background im-age.

Playback screen � the scene where the MIDI song is replayed.

Loading screen � coordinates transition between menu and playback screens which takesa longer time do to the data load.

HUD � a screen which displays graphics loaded in 2D textures onto the playback screen.

Whole concept of managing the screen states has been designed according to the samplecreated by Microsoft and published on their community web pages of XNA Creators Club[8].

5.2.2 Data management

According to the de�ned requirements, we are not going to operate with score writingtherefore we have decided to store the songs replayed in the sessions in a MIDI format foreasy data manipulation and communication with the electronic keyboard. Also, since theMIDI is older than MusicXML, more supporting software and music �les exist in the .mid�le format. The MusicXML is still a possible way and its usage might be reconsideredduring the application's further development when other functionalities may be added.

The data which the system uses as parameters for its object are stored in a �le systemin their appropriate form. The �rst types of data are the MIDI �les themselves whichare simply placed on concrete �le path such as .\Music\*.mid where the appropriateloading function seeks for them. Next information needed for the application run is theplayer's details such as the nickname and the table of completed songs so the programcan decide from what point continue the player's training. Those data should not beeditable externally by the user; therefore the byte stream of the internal application objectis suitable.

Last but not least are the lessons data. To meet the easy editable requirement, anystandardized structure should be used. We have decided to create a simple XML document

20 CHAPTER 5. SYSTEM ARCHITECTURE

Figure 5.3: Diagram of objects in XML �le

Figure 5.4: Graphics lifecycle in application

which can be parsed afterwards in the program by the object manager into application'sinternal objects. The set of entities is shown as a diagram in Figure 5.3. The strictsyntax which can be de�ned in Document Type De�nition (DTD) enables the possibilityof later connection with other applications such as an editor tool. The separate data alsomakes easier the additional upgrades of content and allows the possibility of interchangesof lessons between users.

5.2.3 Graphics management

The graphical content of the application is stored in the appropriate �le formats and is oftwo types. The 3D models representing the virtual piano keyboard and the other elementsof a playback scene are created in 3D modeling software. The next type is the 2D imageused as the background of the screens and for HUD components and can be drawn in anybitmap or raster graphics editor. Those objects are loaded in the program the way inwhich depends on the graphics library used in an implementation.

The approach of displaying the graphical information has been realized using the updateand draw methods that are usually implemented when handling the graphical data inapplications such as video games where they represent the part, so called game loop. Asthe names suggest, in the update method the changes of the graphical parameters whichare visible in the draw method are made. The draw method manages to render the modelsin 3D world into the 2D according to certain matrices and displays the �nal image on thescreen. The whole lifecycle of the graphical content in the running application describesFigure 5.4.

CHAPTER 6. IMPLEMENTATION 21

6 Implementation

6.1 Software development tools and libraries

Before we move toward the implementation part itself, the chosen technologies will be pre-sented together with the particular reasons for their selection from other possible options.The research has been focused on technologies supporting Rapid Application Development(RAD), a type of software development methodology. This technique is suitable for ourproject since its principle is based on minimal planning in favor of rapid prototyping andallows an iterative development process.

6.1.1 Programming language

Looking at the designing part, the class-based and component-oriented programming lan-guage with support for 3D graphics is essential. The data are manipulated on the objectlevel thus there is no need for any low-level programming language. A high-level languagewith integrated garbage collector and ability to assist the programmer with type checkingis suitable. According to the project's assignment, the platform-dependency has not beende�ned so there is no need for using a platform independent programming language suchas Java. Since the author's currently used operating system is Microsoft Windows we havedecided to use C# which responsibly ful�ls all the previously mentioned requirements.

C# is an object-oriented programming language developed by Microsoft to work underits .NET Framework. The syntax is based on C++ and contains elements similar to thoseof Java. A point worth mentioning is the modularity in C#, represented by support of anindependent part of code in a form of dynamic-link library (DLL); Microsoft's implementa-tion of the shared library which allows the programmer to import unmanaged code writtenin other languages such as C or C++. We will take an advantage of this functionalitywhile wrapping the low-level MIDI library in the code written in C#.

6.1.2 Graphics framework

Handling the graphics in the program requires a set of tools which provide the functionalityin the form of a framework or library. Graphical APIs commonly used are cross-platformOpenGL and Microsoft's Direct3D, the part of DirectX. Considering our aim of using RADtools and developing our application in C# it is essential to use Microsoft XNA Frameworkwhich is in fact a managed version of DirectX that is intended to assist development ofgames and also supports porting the code on the Microsoft's video game console Xbox 360.

Unlike the DirectX runtime, XNA Framework is not shipped as a part of Windows anddevelopers are expected to redistribute the runtime components along with their games orapplications which can be distributed free of charge under Microsoft's current licensing.There is also an o�cial site containing useful samples, making the framework easy to learnand understand. In addition, a community of XNA developers has grown on the Internetwith its increasing popularity so another portion of samples is available online acceleratingthe development by using solved solutions even more.

22 CHAPTER 6. IMPLEMENTATION

Figure 6.1: Scheme of connection between computer and electronic keyboard

6.1.3 Integrated Development Environment

When we decided to use C# together with XNA, the choice of integrated developmentenvironment (IDE) is pretty straightforward since the only IDE o�cially supporting XNAis XNA Game Studio which can be run only under its corresponding version of MicrosoftVisual Studio (MSVS). MSVS exists in many versions where the one supporting C# de-velopment uses its own compiler, Microsoft Visual C#. Apart from the integrated browserof o�cial Microsoft's online documentation [7], the IDE supports the C#'s documentationsystem, which is similar to Java's Javadoc, but based on XML. When present in the code,even the custom code notes and comments are visible in the tooltip of auto completionfuture of MSVS called IntelliSense.

6.1.4 MIDI Application Programming Interface

MIDI devices are of two main types. The �rst, MIDI input, is the device also calledcontroller and is capable of sending MIDI messages unlike the second, MIDI output, whichreceives MIDI messages from the input and plays them. MIDI keyboard or a digital pianoconnected to the computer through the USB MIDI interface can serve as both MIDI inputand output device. The way of connection describes Figure 6.1. USB MIDI interfaces donot pass any sound or audio data through them, just the MIDI data represented by MIDImessages.

Considering playing the electronic keyboard, when a piano key is pressed a signal de�ninga Note-on message is transmitted. When the key is released a Note-o� message is sent.These messages consist of a status byte which de�nes the message type and up to two databytes which in the case of Note-on/o� message are the number representation of the note'spitch and its velocity. Both mentioned messages along with the Program change messagebelong to the group of the channel voice messages and are processed by the MIDI outputdevice in some way unlike the Meta Events commands which do not result in any MIDImessage being sent [3].

The handling of the MIDI commands while developing a software product requires theuse of an application programming interface (API). In Windows-based computers thereare two APIs o�ered by the operating system which allow accessing hardware ports atthe low level. The �rst is a module of Windows Multimedia API, the winmm.dll libraryand the second is DirectMusic, a component of the MS DirectX API. DirectMusic hasbeen, together with other older APIs declared by Microsoft as deprecated, cannot be used

CHAPTER 6. IMPLEMENTATION 23

by 64-bit applications and is no longer part of the latest's versions of DirectX SDK [7].Another option for how to deal with MIDI data is the package javax.sound.midi providedby Java 2 Standard Edition Platform as a part of Java Sound API. While implementingour application we have decided to use the winmm.dll library regarding the selection ofother development tools discussed earlier in this chapter.

To encapsulate the function of low level library, wrapping a frequently used DLL functionin a managed class is an e�ective approach to follow. In our application we needed to�nd a way to realize communication between a computer and MIDI devices using MIDImessages on the programming level. We are going to use the winmm.dll library, written inC programming language. Instead of writing our own C# wrapper we will take advantageof the complete solution to speed up the process of development.

We have found two remarkable .NET MIDI libraries. The �rst is NAudio [6], the libraryof which development started in 2001, with currently released 1.2 version from June 2008published under the Microsoft Public License. NAudio is a complete solution for workingwith variety of audio �le formats, not only MIDI. The second is C# MIDI Toolkit [13]which has been developed by the author since February 2004 now in its �fth version fromApril 2007 licensed under the MIT License.

Both toolkits have their source code available to browse together with demo applicationsthus we could have made our decision according to the suitability of the projects capabilitiesto our application by looking at available objects and methods in detail and did not haveto rely on their documentation. Although NAudio is robust project it is still marked as inbeta version. At the end we have decided to go with C# MIDI Toolkit due to its MIDIorientation only, its complex structure and standalone sequencer.

6.2 Implementation highlights

Further text in this chapter covers the implementation details of the most important partsof our application which is called PianoRoll ; the name which has been chosen for theprogram.

6.2.1 Application graphics

The XNA framework supports both 2D and 3D graphics in the form of images loaded in the2D sprites and 3D meshes using the ContentManager class which handles the managementof the graphics content and the content pipeline. Content pipeline is used to convert thecontent assets into an XNA speci�c .xnb format. These objects are rendered in a Drawmethod where the sprites are passed to the graphics device using a SpriteBatch objectwhereas the meshes of 3D object are rendered one by one using a BasicEffect class whichsupports vertex colors and lightning [9].

Models used in our application have been created in Autodesk Softimage Mod Toolwhich is a free version of professional 3D computer graphics application recommended forusing with XNA as it natively includes an export function to Microsoft XNA Game Studio.At the current stage of the system's development the models of black and white key arelow-poly objects �tting the parameters of the real keys. The note boxes are simple cubeswith default unit size resized in the program according to the length of the representednote. The complete scene drawn in the program using cloned key, box and line models is

24 CHAPTER 6. IMPLEMENTATION

Figure 6.2: Orthogonal views of playback scene with camera viewports

Figure 6.3: Relationships between PianoRollLibrary namespaces

together with highlighted camera viewport shown in Figure 6.2.

6.2.2 PianoRoll library overview

Objects and methods from C# MIDI Toolkit used by our program are placed in Sanford.

Multimedia.Midi namespace. The relationship between toolkit's and system's namespacesis outlined in Figure 6.3 and can be seen in more detail in Figure B.3. The toolkit's objectsand methods used for sending and receiving the MIDI messages between MIDI devices arehandled within the PianoRollLibrary.MIDI namespace classes. The SongData class fromPianoRollLibrary.Song uses the toolkit's objects Sequence and Track to parse the datafrom the MIDI �le and the instance of MidiInternalClock is used as the main timer forthe playback. The need of the separate timer is because the song does not start playinguntil the �rst note's box hits the keyboard line.

The PianoRollLibrary.3DModel consists of classes which contains the update anddraw methods of playback scenes' graphics objects. Those methods are called inside the

CHAPTER 6. IMPLEMENTATION 25

SongPlayer's class method PlayTrack() since they are dependent on the parameters ofcurrently played track. Lines representing the boxes' path are static and drawn togetherwith the keyboard whereas the note boxes and bar lines are dynamically added or removedfrom the lists according the actual position of the timer in the song.

6.2.3 XML object loader

As mentioned in Subsection 5.2.2, we are going to load the lesson data from the XML �leto enable the possibility of editing and sharing the data. Therefore the requirement forthe object manger to change properties can be made without recompiling the program.An object manager used in our program has been taken from the online sample foundon Ziggyware web pages [15]. Unlike other existing object loaders this one does not usethe built-in XML serialization but de�nes its own re�ection API. It is able to instantiatea class de�ned in attribute type of node object and �lls the �elds with the content ofchild nodes. For example if we want to add a new song into the lesson's list of songs usingAddSong(Song song) method, we insert the <Song> and its child element into a <AddSong>tag. The object loader also supports loading resources therefore the images can be loadedinto lessons and displayed further in the program on the screen with lesson instructions.

6.2.4 Input from keyboard

The user's input performed by pressing and releasing the keys on electronic keyboard isused not only for playing the song but also for navigating the menu screen. For thatpurpose there is a set of classes which store the piano keyboard states and manage thecommunication between the piano keyboard 3D model and the real electronic keyboardrepresented by MIDI input device.

When the song is replayed, the MIDI input device sends the MIDI messages to the system.These messages contain data about currently pressed and released keys and are receivedin the MidiInput class where event handlers are declared which invoke the appropriatemethods in the PianoKeyboard class. These methods contain calls for other actions suchas piano key animation, score evaluation and passing the message to the MIDI outputdevice and use as parameters the data of the channel message received, i.e. the Data1

property gets the note's pitch and Data2 note's velocity.

6.2.5 External tools for lesson authoring

When the song data are created it is essential to know the parameters of the SMF which isgoing to be used like the default tempo or tracks the �le holds. For this reason there is aneed of an external tool. We have created a simple Windows Forms application using ourPianoRollLibrary and the C# MIDI Toolkit. The screenshot can be seen in Figure 6.4.For now the capabilities of this program are limited so while creating the testing data wealso used Music MasterWorks music editing application produced by Aspire Software [14].Since only the 30-day trial version is free to use, we would like to expand our editor in orderto become a complex tool shipped with our tutorial system with all needed functionalitiesincluding the lessons data editor.

26 CHAPTER 6. IMPLEMENTATION

Figure 6.4: Mockup of external authoring tool

CHAPTER 7. TESTING 27

7 Testing

While testing our application, we will focus on usability tests which examine if the productmeets the requirements from the user's point of view and how easy and intuitive its usageis. The aim is to discover errors and areas of improvement while observing people using theproduct. The tracked aspects are e�ciency, accuracy, recall and emotional response. Theperson who leads the test, a moderator, �rst �nds out information about the participantin the form of questionnaire, then he gives him concrete instructions and instead of askingquestions or showing what to do. Moderator watches the reactions and the actual stepsthe user makes. An interview about person's feelings and impressions of the system closesthe testing process. Participants are identi�ed in the report by alphabet letters since theirtrue identity is not relevant to the test and should stay annonymous.

7.1 Test setup

Our tutorial system was tested on the small group of users with di�erent piano playingskills and experience with similar products to cover the target group of users as complex aspossible from the sample. The tests were not run in one round and after each test changesto signi�cant problems were corrected. This procedure has been chosen to increase thee�ciency of upcoming tests since we assume that if the mistake was left, it would beharder to analyse other potential problems covered by the detected problem.

We expected the test to reveal the playback screen's defects as we were aware of a lackof preliminary prototype tests which would take extra time to create. Instead of earlyprototyping, we have decided to implement the program to the designed form and wait forthe state of development, where the system currently occurs, which allows us to make thechanges on the graphical design on the run.

To make the tests possible, the test data had to be created. These data contain thebeginning lessons with simple melodies and some advanced songs placed in Quickplaymode. For the �nal version of the application the data should be created in co-operationwith a professional music master in order to guarantee success in the system's purposesince our aim is to develop a tool which would allow it. The songs on which the userstrain his skills should be chosen according to certain criterion consulted with the expertsat pedagogy.

Before and after test questionnaire can be found in Appendix D. An explanation of thequestions and the summary of the results gained while testing our 5 test subjects in theaverage age of 22 follows. In Figures C.2 and C.3, there are the playback scene designsbefore and after testing to show the di�erences made to the program.

7.2 Testing process and results

Since our application is targeted at piano players of di�erent levels either to be used as aneducational or an entertainment tool, we asked the participants about their piano playingexperience. The reactions to the piano roll style of representing the notes from the playersused to the typical notation were observed. Also we considered it useful to know if theywere familiar with the concept of playback from similar products and score used in video

28 CHAPTER 7. TESTING

Figure 7.1: Illustration of test setup

games. When the participant answered these questions he was navigated to the song on alevel appropriate to his skill and asked to follow the onscreen instructions.

The questions from the after-test questionnaire were pointed at user's impressions andopinions about our application. We were interested if the score motivated the player toreplay the song which was con�rmed by users which played the video games. Accordingto answers to question about adequate lesson level we adjusted the test data before weapproached the next test. The participants were asked about their desired use of oursystem. Most of them would use the application for their entertainment, only a few becauseof its educational element. We suppose the �rst group to be the people who would learnthemselves without even noticing it. Some of the participants expressed their intentionto play their favorite songs. Participant A was sure that he would de�nitely prefer thisway of learning to the classic way which he would have never started by himself but oursystem seems to be more of an incentive. None of the participants would hesitate withbuying electronic keyboards capable of connection to their PC which they all own andwould invest from 2 000 to 5 000 CZK.

Most of the problems occurred with the playback screen. Participant A had di�cultieswith recognizing if the position of his hand corresponds to the range of virtual keyboardand was not sure when the key should be pressed. To correct this problem each octave wasdistinguished with a speci�c color and the key which was about to be played was highlightedby the re�ector from the note box which darkens when closer to the corresponding pianokey. Participants B and C did not recognize what key should be played therefore theadjustments in the colors of di�erent key states was made and the lines were prolongedeven in front of the keyboard. Participant D only had a problem recognizing when the twosame tones follows immediately but during the second replay he played it right. Becausethis user was the type of player which can play the songs by ear he suggested a replayfunction of the song which is going to be played. The last one, Participant E liked thekeyboard input when navigating the menu but needed some time to get familiarized withthe concept as he was without the previous experience with similar products. Although hestated that the incoming boxes are better than if the keys would just change colour.

CHAPTER 8. CONCLUSION 29

8 Conclusion

I have implemented a tutorial system which is targeted at users who are going to learnto play the piano using an electronic keyboard and a computer. The application supportsplayback of songs stored in MIDI �le format and o�ers the user the possibility of partici-pating either in Training or Quickplay mode. The player's knowledge of musical notationis not required since the program represents music in a visual form.

I have chosen the subject of this work because of my personal knowledge of the pianoplaying educational system. I found there is a lack of encouragement which prevented mefrom returning to playing after a few years of pause. Therefore my goal was to create apiano playing tutorial system which would be educative and entertaining at the same time.When designing the application itself I was mainly inspired by my experience with GuitarHero whose concept seemed to me to be applicable for educational purposes if applied tothe real musical instrument.

First of all I had to look closely at the current situation regarding available tools and havelearnt from their drawbacks and took advantage from their successful points. I studied indetail the MIDI technology to learn how to handle communication between an instrumentand a computer program. I also searched for the most suitable option of handling theMIDI commands in my program.

Through testing I have con�rmed that players who are not familiar with standard musicalnotation can use the program pro�ciently and with a clear fun element. Though thesystem's data needs to be precisely adjusted to ensure their adequate di�culty. I amaware of the necessity of cooperation with a professional teacher for creating the lessondata to ensure the e�ectiveness of the educational element.

8.1 Future work

In order to provide users of our educational system with additional services some sugges-tions for the future work follows. First of all the existing editor tool could be extendedwith the capability of preparing the MIDI songs and levels for the application. In theplayback scene the advanced graphics using particle e�ects and shaders is likely to makethe playback more intuitive and attractive to the players. Especially the visualization atthe moment the key should be pressed needs to be improved. To satisfy the largest rangeof target group of users I would suggest letting the user choose between di�erent genresof songs played to ensure their enjoyment of the application as a whole. The score adjust-ments in di�erent modes need to be revised and tested. Finally, an animated hand showinghow to play a certain song correctly can be used as a guiding tool for the user.

30 CHAPTER 8. CONCLUSION

CHAPTER 9. BIBLIOGRAPHY 31

9 Bibliography

[1] MIDI Manufacturers Associatio. MIDI.http://www.midi.org/.

[2] Jan Dostal. Dít¥ u klavíru. Supraphon, Praha, �R, 1th edition, 1977.

[3] Daniel Forró. MIDI komunikace v hudb¥. Grada, Praha, �R, 1th edition, 1993.

[4] Brent Fox. Game Interface Design. Course Technology PTR, Boston, USA, 1thedition, 2004.

[5] DDR Game. DDR Distributor.http://www.ddrgame.com/.

[6] Mark Heath. NAudio.http://naudio.codeplex.com/.

[7] Micfosoft. Microsoft Developer Network.http://msdn.microsoft.com/en-us/library/.

[8] Microsoft. Game state management sample.http://creators.xna.com/en-US/samples/gamestatemanagement.

[9] Benjamin Nitschke. Professional XNA Programming. Wrox, Boston, USA, 1th edition,2008.

[10] Klavar Music Foundation of Great Britain. Klavarskribo.http://www.klavarmusic.org/Explanation.pdf.

[11] Nicholas Piegdon. Synthesia game.http://www.synthesiagame.com/.

[12] Recordare. MusicXML.http://www.recordare.com/xml.html.

[13] Leslie Sanford. C# MIDI Toolkit.http://www.codeproject.com/KB/audio-video/MIDIToolkit.aspx.

[14] Aspire Software. Music MasterWorks.http://www.musicmasterworks.com/.

[15] Ziggyware team. Advanced Object-Loading through XML and Re�ection.http://www.ziggyware.com/readarticle.php?article_id=160.

[16] Dalian Piano roll.http://website.lineone.net/~agr/24.jpg.

[17] Yamaha. Yamaha lighted keyboards.http://www.yamaha.com/.

32 CHAPTER 9. BIBLIOGRAPHY

APPENDIX A. LIST OF ACRONYMS 33

A List of acronyms

2D Two-Dimensional

3D Three-Dimensional

API Application Programming Interface

DDR Dance Dance Revolution

DTD Document Type De�nition

GM General MIDI

GUI Graphical User Interface

HUD Heads-Up Display

MIDI Musical Instrument Digital Interface

MMA MIDI Manufacturers Association

RAD Rapid Application Development

SMF Standard MIDI File

UI User Interface

XML Extensible Markup Language

YMS Yamaha Education Suite

34 APPENDIX A. LIST OF ACRONYMS

APPENDIX B. UML DIAGRAMS 35

B UML diagrams

Use cases

Figure B.1: User's use cases

36 APPENDIX B. UML DIAGRAMS

Class diagrams

Figure B.2: Design of data model classes

APPENDIX B. UML DIAGRAMS 37

Figure B.3: Interaction and relationships between PianoRollLibrary members

38 APPENDIX B. UML DIAGRAMS

Figure B.4: Screen management classes

APPENDIX C. APPLICATION SCREENSHOTS 39

C Application screenshots

Menu screen

Figure C.1: Main menu screen with static background image

40 APPENDIX C. APPLICATION SCREENSHOTS

Playback screen

Figure C.2: Playback scene before tests

Figure C.3: Playback scene adjusted according to test results

APPENDIX D. USABILITY TESTING QUESTIONNAIRE 41

D Usability testing questionnaire

Questions asked before test

1. Age:

2. Piano playing experience (years active/years inactive):

3. Similar products experience:

4. Video games experience:

Questions asked after test

1. First impression about the tutorial system:

2. Adequate level of lessons:

3. Motivation to replay a song via score:

4. Negative feelings invoked:

5. Appealing things:

6. Clear principle of playback screen:

7. Purpose of using this application:

8. Electronic keyboard purchase, limit:

42 APPENDIX D. USABILITY TESTING QUESTIONNAIRE

APPENDIX E. USER MANUAL 43

E User manual

System requirements

• Supported operating systems: MS Windows XP SP3, MS Windows Vista SP1.

• A graphics card that supports DirectX 9.0 and Shader Model 1.1.

• Installed Microsoft XNA Framework Redistributable 3.0.

• Installed Microsoft .NET Framework 2.0 or higher.

• Electronic keyboard connected to the computer via MIDI.

Installation instructions

Application installation is not required, user can run the �le directly by opening the �lePianoRoll.exe.

Navigating the menu

Command PC keyboard / Electronic keyboard

Select Enter / two white keys next to each other

Cancel Esc / two black keys next to each other

Down Down arrow / any white key

Up Up arrow / any black key

Left Left arrow / two white keys separated by one white

Right Right arrow / two black keys separated by one black

44 APPENDIX E. USER MANUAL

APPENDIX F. CD CONTENT 45

F CD Content

CD

|-- PianoRoll - Executable file

|-- src - Source files of the application

|-- text

| |--latex - Source files of this text

| \--thesis.pdf - PDF version of this text

\-- readme.txt - User manual

46 APPENDIX F. CD CONTENT