COMPOSING WITH HYPERSCORE: AN INTUITIVE INTERFACE FORVISUALIZING MUSICAL STRUCTURE
Morwaread Farbood Henry Kaufman Kevin Jennings
New York UniversitySteinhardt School
Harmony Line, Inc.Cambridge,
Marino Institute ofEducation
Hyperscore is a graphical, computer-assisted compositionsystem that allows users to intuitively visualize and editmusical structures. It maps musical features to graphi-cal elements such as color, shape, and line texture. Thesegraphical elements allow users to control both high andlow-level musical features such as pitch, dynamics, melodiccontour, and harmonic tension. Hyperscore facilitates com-position by providing the user with a visual representationof the large-scale structure of a piece while simplifyingthe process of integrating diverse rhythmic and melodicmaterial. It is designed primarily for users with limited orno musical training.
Hyperscore is a computer-assisted composition system thatenables users to compose music graphically. The systembreaks down musical structure into two levels: 1) the cre-ation of rhythmic and melodic phrases and 2) the shapingof the large-scale progression of the piece. Graphical userinput is in the form of freehand drawing; the lines drawnare interpreted according to shape, color, and position andconverted into pitches and rhythmic values. Hyperscoreprovides a layer of abstraction between these two levels:the melodic, note-level input and large-scale, form-levelshaping .
It tries to push the concept of what you see is what youhear in a score as far as possible. While some musical el-ements are inputted in piano-roll notation, the score itselfis drawn freehand by the user. Thus the interface attemptsfind a balance between precise control and having a free-flowing and expressive visual interface. Making freehandline drawings into a useful music control system enablesusers with little no musical experience to explore musicalcreativity at many levels. The system provides high-levelcontrol over the dramatic arc of the piece as a whole aswell as the placement of individual melodic and rhythmicelements.
2. PRIOR WORK
In the late 1960s, the advent of the first computer ter-minals opened up the possibility for graphical interfaces.
Max Mathews and L. Rosler began using this new tech-nology to explore graphical applications for music. Uti-lizing a new computer called the GraphicI, they developeda compositional language that substituted graphical inputin the place of tediously punched data cards. The sys-tem allowed a musical score to be specified as a groupof graphs and provided the means to algorithmically ma-nipulate them by drawing curves on the computer screen.
A broad range of graphical computer-assisted compo-sition tools have followed Mathews and Roslers work.Some systems are suited for professional musicians; thesetend to use graphical objects to represent musical func-tions or tweak parameters. The former category includesPatchWork/OpenMusic , a visual, object-oriented pro-gramming environment designed by researchers at the In-stitut de Recherche et Coordination Acoustique/Musique(IRCAM) to encapsulate musical functions in graphicalobjects that can be dragged, dropped, and interconnectedto implement musical algorithms. The latter category in-cludes standard commercial applications such as DigitalPerformer, Cubase, and Logic. They are in essence multi-track sequencers that use graphical input to manipulateparameters such as volume, timbre, and attack and decayenvelops.
Systems such as David Zicarellis OvalTune, a pro-gram in which users create sounds and visual images si-multaneously by painting with a mouse, and Cyber-Band, developed at IBM, are designed for users who dontnecessarily have musical experience. CyberBand is sim-ilar to Hyperscore in some ways: it uses riffs, or the-matic fragments, as musical building blocks and higher-level modifiers to edit and refine the music. Its interface,however, is fundamentally different from Hyperscores. Itdoes not use drawing as a method of combining musicalmaterial and lacks the visual freedom of Hyperscores en-vironment.
Iannis Xenakis UPIC  is a system that uses a large,high-resolution graphics tablet for input. Its macrocom-positional level lets the user draw a time-frequency scoreconsisting of lines, curves, and points. Even though Xe-nakis himself used UPIC to compose music, he envisionedmusically inexperienced users like children using the sys-tem as well. Other applications accessible to musicallyuntrained users include Maxis/Iwais SimTunes , the
Figure 1. The initial Hyperscore prototype.
Macintosh program MetaSynth , and Morton Subot-niks Music Sketch Pads .
Hyperscore is unique from all of these applications forseveral reasons. While other programs like UPIC mighthave similar freedom in graphical input, Hyperscore goesa step further from a musical perspective by mapping graph-ical elements not just to one-dimensional features such asfrequency or pitch, but higher-level structures like func-tional harmony. It does not require users to be proficienteither musically or technicallythey do not have to playan instrument, or read music, or know how to program.Yet it also allows them to compose music in many differ-ent styles without taking away their sense of ownership inthe creative process.
3. THE EVOLUTION OF AN INTERFACE
The goals of Hyperscore have evolved considerably sincethe first prototype was completed in 2000. The earliestversion was almost entirely automated  . The userdrew a musical tension line and the program then parsedline and generated a piece of music according to the shapeand texture of the line (see Figure 1). The line was di-vided into parabolic segments and the smoothness of thesegments analyzed. If a segment was smooth, the corre-sponding rhythmic texture in the generated music wouldbe less dense (greater rhythmic activity was equated withmore tension). The user selected all or a subset of ninepre-composed melodic patterns which were used to gen-erate the piece. The system did not choose which melodicpatterns to use deterministically; each time a piece wasgenerated for a specific graph, the results were different.The graph could only influence the texture density pro-duced by the combination of contrapuntal lines.
Over the course of the next few years, the interfacewent through a series of iterations and the focus shiftedto allow more precise user control and less automation.
Figure 2. An intermediate version of Hyperscore.
In intermediate versions, users were given the ability tographically annotate the tension line to indicate at whatpoint in time melodic material would be used by the sys-tem (Figure 2). The graphical interface was redesigned and the concept of colored pens for drawing linesintroduced. Users could indicate where and what kinds ofmelodic material were used by selecting and drawing witha color that was mapped to the melodic pattern. The linesproximity to the tension curve influenced what motivicmaterial was selected by the generation algorithm. Even-tually, as the music generation became more user-drivenand deterministic, the tension line ceased to function assuch and simply became a time and volume-modulationline.
Another new feature was the automated harmony gen-erator. This algorithm was implemented using hierarchi-cal Markov chains to handle different layers of organi-zation. One set of Markov chains was used to generatea series of higher-level harmonic functions, and anotherset was used to generate the actual chords. The chordfunctions were simple, consisting only of three categories:tonic, dominant, subdominant. Chord function transitionprobabilities were selected based on the time at which thechord occurred and the function of the chord precedingit. The chords themselves were chosen according to timeand relative frequency at which the chord would appearregardless of the circumstances (i.e. not dependent at allon the preceding chord). Eventually this feature was aban-doned for an improved harmony control that allowed formore user input and more sophisticated chord progres-sions (see Section 4.2).
4. THE CURRENT SYSTEM
Hyperscore is written in C++ using DirectX and the Win32API. The current interface consists of an expansive, zoomablecanvas where users can create any number of musical frag-ments and whole pieces. Users can position these mu-sical objects anywhere on the canvas and can view theworkspace at any level of zoom for ease of editing.
Figure 3. The current version of Hyperscore. All three types of mo-tive windows are present (melodic, polyphonic, and percussion) as wellas a sketch window.
4.1. Interface and interaction
The basic musical building blocks in Hyperscore are calledmotives. Motives are relatively short musical phrases thatcan be built up together in a composition. Motives arecreated in special windows that allow the graphical place-ment of notes in a piano-roll style notation where time isthe X-axis and pitch is the Y-axis (see Figures 3 and 4).Notes can be placed by clicking anywhere in the motivewindow, and then stretched graphically to change the noteduration (using standard handles familiar to any GUIfor shape manipulation). A user-configurable grid ensuresthat the note onset and duration remains constrained. Notescan be multi-selected and the whole group can be scaledto speed up or slow down the notes. In that case, only thebeginning and ending of the whole phrase are constrainedto the grid, allowing a more flexible way of editing musi-cal phrases. Individual notes can be snapped to the gridlater if desired.
To enforce a melodic conception of a motive, the mo-tive window can dynamically constrain only one not