Soundium2: An Interactive Multimedia Playground

Simon Schubiger-Banz, Stefan MüllerDIUF, Université de Fribourg, CH-1700 Friburg

MultiMedia Lab, University of Zurich, CH-8057 Zürichemail: shoobee@corebounce.org

AbstractThis paper gives an overview of Soundium2.Soundium2 is a unique combination of different areasof computer science ranging from real-time signalprocessing over programming languages to automaticsoftware configuration. It not only serves as anexperimental tool for exploring new ideas incomputer science but also is frequently used in livemultimedia performances in order to expose it to real-world conditions. Although similar systems exist, fewwere designed explicitly for live multimedia-performances. In this sense, Soundium2 is more likea multi-user, audio-visual instrument than a softwaresystem. Soundium2 is introduced and an overview ofits architecture is given. The features of Soundium2are outlined and compared with related work.

1 IntroductionThis paper gives an overview of Soundium2,

supported by the MultiMedia Lab of the University ofZurich, the Parallelism and Artificial Intelligencegroup of the University of Fribourg and theCorebounce Association.

Soundium2 brings together different areas ofcomputer science in a unique multimedia system. Themain motivation behind Soundium2 is the quest for aframework, which allows us to test new ideas inaudio processing and visualization with minimaloverhead. Nevertheless, the system should be rocksolid and user friendly since we use it frequentlyduring live performances. The current client serverarchitecture in combination with data flow networks,signal processing, 3D graphics, versioned globalstate, and an advanced GUI manages to match thesecontradicting goals. The rest of this paper starts withbackground information on related work as well asthe origins of Soundium2, then continues with anoverview of the system and finally concludes with anoutlook.

2 BackgroundAlthough several excellent multimedia systems

exist today, none of them implement all features weconsider as crucial for successful rapid developmentand live performances. This section lists a smallselection of systems that perform well for certain

aspects and are in turn very inspiring for our ownwork.

2.1 The Interactive Dance ClubThe Interactive Dance Club (Ulyate and

Bianciardi 2002) was an installation built for the 25thannual ACM SIGGRAPH Conference and includesmany elements also found in Soundium2 (such asinteractive, music synchronized live-visuals). On theone hand the interactive dance club is a mix ofheterogeneous components, and thus lacks thehomogeneity and consistency of Soundium2. On theother hand, it goes further than Soundium2 bystrongly integrating the participants of an eventthrough interactive zones. Although possible withSoundium2, we consider an integration of theaudience more as an artist’s choice than an integralpart of a performance.

2.2 The Max Family and SuperColliderThe Max environment (Pukette 2002) has its

origins in audio processing. Different extensions suchas DIPS (Matsuda and Rai 2000) add multimediacapabilities. It shares with Soundium2 the splitbetween a small, optimized processing core and amodular GUI. The main difference is that Max is avisual programming environment. Soundium2 has anintegrated programming language too but it is mainlyused for system manipulation (e.g. system state, orGUI) and it is completely independent of the engines,which implement the processing core. Max allowsvery fine-grained composition and control, resultingin a highly flexible system. Despite that, typicallyMax developers tend to include larger building blocksas “externs” instead of constructing them from basicoperators. This is exactly the approach of Soundium2.Engines export only a set of “externs” that can beinterconnected. Simple operators such as arithmeticare excluded.

Another example is SuperCollider (McCartney2002) which is a highly evolved computer musiclanguage closely integrating processing and processdescription. The question of granularity, placement ofprocessing objects, and process description is noteasily answered; the deciding argument forSoundium2 is its main purpose: a live-performance

system where complexity (at least for the user) has tobe kept at a minimum

2.3 Parallel, Bomb, WinAmp pluginsAll these tools have in common that they visualize

aspects of an audio signal such as spectrum or leveland process the audio signal with DSP (Digital SignalProcessing) algorithms. User interaction is alsopossible to some degree but for both aspects theynever go as far as Soundium2. Despite theirrudimentary functionality compared to Soundium2 orMax, some of these tools produce very appealingoutput and are thus very inspiring from an artisticperspective. Especially the success of WinAmpplugins show that a large community is willing todevelop visualization and audio processing tools andan even larger group is consuming their output.

2.4 Soundium2In addition to the inspiring work in multimedia

research in general, each of the project team memberscontributes its own expertise in a specific field.Several predecessors focusing on various aspects ofsound processing and visualization were developedby individual members of the project. The experiencegained from its ancestors laid the foundations for theproject. Soundium2 was written entirely from scratchas an integrated multimedia framework. It retains thefeatures of its predecessors considered crucial for aninteractive multimedia tool (Juillerat 2001; Specht2000; Parish and Müller 2001; Schubiger andHirsbrunner 2003; Schubiger 2002):– State of the art signal processing– High-level information extraction e.g. beat-

detection– State of the art 2D and 3D visualization– Strongly typed dataflow network– RPC client/server architecture with clustering– Automatic software configuration for network

construction and media-format handling– Graphical user interface optimized for interactive

live-performances

3 A closer look at Soundium2Figure 1 depicts the overall architecture of

Soundium2. Basically, Soundium2 is based on amulti-tier client/server architecture with one clientcontrolling a cluster of servers called “engines”. Themotivation for this separation is twofold: the firstmotivation is code complexity and code maintenanceand the second is timing. The engines should handleall timing sensitive operations while keeping theircode complexity small. All complex operationsshould run on the client, who may or may not controlthe engines in real-time. This allows us to rapidlydevelop and fine-tune individual engines for specifictasks such as 3D rendering or audio processing while

performing more complex tasks like global-state andGUI management in the client.

Figure 1 - Overview of the Soundium2 architecture

The lowest architectural level is represented byspecialized hardware (hardware level) such as videoprojectors, DMX and MIDI equipment etc. All thesedevices are controlled by individual engines (enginelevel) which are optimized for a specific task. Eachengine holds a part of the global dataflow networkstate which is manipulated through an RPC interface.This RPC interface is mainly used by the client(soundium level) to configure and parameterize theengines. Only the client holds the entire globalsystem state composed of the per engine dataflownetworks as well as the system wide statemodification history. The client also renders a multi-user graphical user interface (GUI level) that can beaccessed via the remote frame buffer (RFB)(Richardson and Wood 1998) protocol over thenetwork for collaborative performances.

Features

EngineThe Soundium2 engine is a C++ framework based ona processing graph of building blocks (the nodes) thatcommunicate which each other (along the graphedges). It has been designed with two main goals inmind: First, provide efficient real-time processingfacilities to different media types. Second, provide ahighly abstracted SPI (System ProgrammingInterface) for the building blocks that allows non-experts to write signal and media processing code.

The first goal has been achieved by implementinga system that makes use of extensive multithreading.The building blocks are automatically assigned toschedulers, which handle processing characteristics ofa specific media type. Dataflow analysis determinesthe execution order inside the different schedulinggroups. The building blocks communicate troughconnected ports allowing arbitrary data types. Inter-scheduler communication is automaticallysynchronized.

The application of rigorous OO methods in C++almost completely hides scheduling andsynchronization issues from programmers of buildingblocks. For instance, ports can be accessed likenormal variables. To date, building blocks for

sequencing, audio signal processing, MIDI I/O,parameterized OpenGL rendering graphs, andOpenGL Performer has been implemented.

RPC interfaceThe RPC interface is extremely simple, basicallyallowing the manipulation of the engine’s dataflownetwork. The calls comprise creation and removal ofnodes as well as edges and reading and writing ofparameters. Every call takes a timestamp argumentoccurring in the future, which is used by the engine toproperly schedule the execution of the call. Twoauxiliary calls allow obtaining static engineinformation (supported node types and data formats)and querying the current engine time.

SL1The global system state (the union of the per-enginedataflow networks) is versioned and stored as arevision tree. Every state manipulation is saved in therevision tree and it is thus possible to rollback to anyprevious state and starting a new branch from there.Even forward propagation of state changes issupported if the changes are non-conflicting withlater revisions of that state. The integration ofversioning in a multimedia system ensures that noartwork is lost and that new artwork can be easilyevolved from existing work. The storage format iscalled Soundium Language 1 (SL1), a full-fledgedimperative programming language. Besides encodingthe system state, SL1 is also used for client sidescripting, GUI customization, and performancepreparation.

ResourcesThe management of resources such as 3D models,textures, and audio clips is entirely HTTP (HyperText Transfer Protocol) based. In addition to flat fileresources, a proprietary extended resource format(XRS) allows association of SL1 code with mediafiles, thereby transforming passive resources intoactive objects that can react to events like resourceloading.

Software ConfigurationResource classification1 is the foundation for theautomatic software configuration capability used inSoundium2. A modular scheme allows the addition ofresource transformations (such as format conversion),which are automatically aggregated based on theirsemantics in order to construct high-leveltransformations. Software configuration relies on anontology, which is planned to include metainformation beside the current media relatedinformation such as music style, mood, etc. 1 By classification we understand the process ofdetermining resource properties such as content type aswell as extracting features like beat positions.

Figure 2 - The Soundium2 graphical user interface

Graphical User InterfaceThe graphical user interface (GUI) allows interactiveinspection and manipulation of the system state aswell as editing SL1 code. Figure 2 shows ascreenshot of the Soundium2 GUI with a selection ofuser interface elements. The system state representsthe union of all dataflow networks on each engine asa graph. SL1 scripting is used for modifications ofSL1 code. The revision tree holds the trace of all statechanges in the system and allows arbitrary jumpsbetween system states2. Node parameters can eitherbe modified through dedicated GUI elements orassociated with hardware devices such as MIDIcontrollers.

Live Performance with Soundium2Using Soundium2 for live performances mainly

consists of preparing a set of global states for theevent in question. The preparation takes into accountinformation such as expected music-style, social andcultural aspect of the audience, spatial and technicalconstraints of the location (projection surfaces andon-site equipment). Hours before the event, theengines are hooked up to the input3 and outputequipment4 and clustered with the client in an IPnetwork. During the event, the artists use thegraphical user interface to change parameters,reconnect inputs, or add new nodes in order to adaptthe multimedia experience. The modifications are theresult of the artist’s perception of the event andconstitute his live contribution to the performance.Usually each artist focuses on an aspect like

2 The revision tree can also be seen as a multi-levelredo/undo tree.

3 Commonly used input equipment includes audio, video,MIDI, EEG (Electro-Encephalogram), environmentalsensors, etc.

4 Output equipment are e.g. video projectors, audio-system,laser projectors.

visualization or sound5 and the final multimediaexperience is the result of a strong interactionbetween the artists as well as the audience.

4 Conclusion and OutlookSoundium2 combines input from various domains

such as signal processing, client/server architecture,software configuration, programming languages anduser interface design. Besides being used for liveperformances (see figure 3) which is a verysatisfactory part of the work, it also serves as anexcellent playground for exploring new ideas. This ismainly thanks to its modular design and its strongvisualization and interactive manipulationcapabilities. For example, Soundium2 is also oftenused just as a framework for rapid prototyping likethe phoneme-recognition / facial-animationdemonstration given at Imagina’03 (Kalberer, Müllerand van Gool 2002). It is also used as a MMS (MultiMedia Messaging) demonstration system atSwisscom.

Currently the system is expanding in manydirections but the basic architecture proved to bestable for a long time. On the graphics processingpart, extensions are under the way to take intoaccount recent enhancements of OpenGL. To fullyimmerse into the on-site equipment, DMX 512support is currently being implemented. On the audiopart, we work on higher-level music analysis as wellas improving the interactive mixing and virtual DJ(Disk Jockey) capabilities. A long-term endeavor iscertainly the addition of media meta information suchas style, audience perception, cultural aspects, ormood to the ontology.

Figure 3 -Live visuals produces by Soundium2

5 Not all artists have to use Soundium2 to participate –thanks to the wide range of supported input equipment andsignal processing inside Soundium2, Soundium2 can belinked to almost any data source.

ReferencesN. Juillerat. 2001. SoundEngine “Un serveur d’effets

sonores en temps réel.” Master’s thesis, Université deFribourg, 1700 Fribourg, Switzerland

G. A. Kalberer, P. Müller, and L. van Gool. 2002. “SpeechAnimation Using Viseme Space.” Vision Modeling andVisualization 2002, pp 463–470, Erlange, Germany

S. Matsuda and T. Rai. 2000. “DIPS: the real-time digitalimage processing objects for Max environment.”Proceedings of the International Computer MusicConference 2000. International Computer MusicConfernece

J. McCartney. 2002. “Rethinking the Computer Language:SuperCollider.” Computer Music Journal, 26(4):61–68

Y. I. H. Parish and P. Müller. 2001. “Procedural Modelingof Cities”. Proceedings of SIGGRAPH’01, pp 301–308,Los Angeles, CA, USA

M. Pukette. 2002. “MAX at Seventeen”. Computer MusicJournal, 26(4):31–43

T. Richardson and K. R. Wood. 1998. “The RFB Protocol.”ORL, Cambridge, UK

S. Schubiger. 2002. “Automatic Software Configuration –A Model for Service Provisioning in a Dynamic andHeterogenous Environment.” PhD thesis, University ofFribourg, No. 1393

S. Schubiger and B. Hirsbrunner. 2003. “JRPC - AnEfficient RPC Implementation for Java.” JavaDeveloper’s Journal, submitted

M. Specht. 2000. “Paralleler Software-Synthesizer –Cumulus.” Master’s thesis, ETHZ, EidgenössischeTechnische Hochschule, 8092 Zürich, Switzerland

R. Ulyate and D. Bianciardi. 2002. “The Interactive DanceClub: Avoiding Chaos in a Multi-ParticipantEnvironment.” Computer Music Journal, 26:3:40–49