Soundpath : Real-Time Portable Control Framework for 3D sound system

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Soundpath: Real-Time Portable Control Framework for 3D sound

system

Soundpath: 3D音效系统的实时便携控制框架

2

1. Introduction

2. Motivation

3. System Architecture

4. Case Study

5. Conclusion

Outline

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Introduction

4

Soundpath is a project with the goal of developing real-time portable control framework for 3D sound system such as IOSONO.

The framework uses web service to link various portable devices to 3D sound system.

The portable devices are served as front sensing and control units to generate a 3D sound spectrum.

It eases the creation of installations and oeuvres based on 3D sound system.

An indoor localization system is implemented to link the personal tracking and sound spectrum to enable live performance. It demonstrates the advantages of Soundpath.

Introduction

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Motivation

6

IOSONO Studio

7

IOSONO system is a world leading audio technology which can provide a 3D Sound effect for interactive designs.To provide such an experience, scenarios are created on another software: the Nuendo Platform. We have to define a source which plays one sound and has one path to create a scenario in Nuendo Platform.Static Active Interaction

Motivation

8

Before we start our project, SCHM has already created a project to do some interaction work using IOSONO.They use an iPad or other IOS device to send standard OSC packets to Nuendo Platform using MAX/MSP and create the sound source.The iPad can provide a rotation matrix. When the user roll or pitch the iPad, the sound will follow its movement.

Motivation

9

We want to develop a framework which can help interactive designers to use various kinds of interactive input such as inertial sensors, smart phones etc to access Nuendo Platform easily.Designers can mainly focus on creating sound effect in art without worring the communication between interactive input and Nuendo Platform.New portable devices can be easily extended and control the IOSONO.Hence, the framework should be scalable, adaptable and easy to extended.

Our goals

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

System Architecture

11

A web application, why not?The aim of developing such a platform is to create an interface capable of being the relay between the interactive input and the 3D sound system.

The Advantages of Web Service□Standardization□Versatility□Cross-platform□Event-driven

The choice of the architecture

12

To meet the constraints, we decided to choose a “three layer” architecture: Controller layer => Handle Sensing data and Request

input, invoke the corresponding Service Service layer => Realize sound effect based on the

sensing data DAO layer => handle the connection with the databaseWe implement an indoor localization system for live performance. Next, I will use it to expain the whole three layer architecture.

A three layer architecture

13

o The smartphone gets two data coming from all the AP in the studio:□ The frequency□ The power of the signal received

o The smartphone sends all the data to the web application via a web GET request

o Each time one of the value changes, the smartphone recalls the request

Controller Layer

14

o Several web-services:□ Trilateration localization□ Centroid localization□ Fingerprinting localization□ Coordinates transformation

o Once receiving requests of webservice, the server will return resultso With the power of signals coming from the AP,

localization algorithm is called => Cardinal coordinates

o Then, transformation of the coordinates => Polar coordinates

Service and DAO Layer

15

o The webservice sents the polar coordinates to the patch Max/MSP

o The patch sends it to the IOSONO systemo One message for one sourceo Sources on IOSONO are directly piloted by the

application

Result

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Case Study

Indoor localization system with IOSONO for live performance

17

There are 5 APs set up in SiWei studio. The length and width of the room is 6.5m and

5.25m respectively.

Environment

18

A user carrying an Android Smartphone is walking around in the room.

The IOSONO system will generate a sound spectrum following the user’s track.

Expected Result

19

Experiments

Measure the AP signal strength under the condition with/without music

All the sheet of paper you can see represents the position of each data measurements.

20

Experiments Result

Results for one AP (upper left corner) are presented on the two following graphs

the X/Y axes represent the plan of the studio and the Z the signal power.

21

ArchitectureWeb ApplicationAndroid Smartphone

Database

IOSONO IPC

Patch MAX/MSP

With the data sent by the smartphone, the WA calculate the location of the user.Then, the WA gets the sources belonging to the user in the database. It calculates the polar coordinates of each sources and stores it in the database before sending it through the patch MAX/MSP to IOSONO IPC

Power information of each access point is sent to the WA

OSC packet contains the coordinates and ID of the sources

The patch gets the coordinates, ID for one source and sends all this information to the IOSONO IPC

22

http://192.168.0.*:8080/coordinate/real/centroid/-80.0/2452.0/-10.0/2437.0/-100.0/2422.0/-150.0/2412.0/-100.0/2412.0/

Communication format

/iosono/renderer/version1/src 1 0 6.28 -184 1 0 0 0 0 0 0

Android Application will request the Server in the following format

IP address of Server

Page name

Service name

AP1’s signal strength

AP1’s frequency

AP5’s signal strength

AP5’s frequency

OSC packet format for IOSONO system

Polar coordinate

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Shang Hai J

iaotong

UNIVERCITY

22/10/2

013

Conclusion

24

o We develop a framework for easy connection between IOSONO and interactive inputs.

o The framework uses web service to link various portable devices to 3D sound system.

o Designer can mainly focus on creating sound effect itself and building new web services for new interactive inputs.

o Easy to switch implemented web service o Implement a localization service to link the personal

tracking and sound spectrum for live performance. It demonstrates the advantages of Soundpath.

Conclusion

25

Thank you for your attention

无线与传感器网络实验室 Laboratory for Wireless and

Sensor Networks

Demo

27

Unlike the deployment in SiWei studio, the AP are placed in the Demo Area of this room.

Demo Area is a small scaled region of this room.

While the user is walking around in Demo Area, the sound will follow his track in the same scale of this room.

Demo

Demo Area

28

Weighted Centroid Localization