Development of Interactive Media Façade

Development of Interactive Media Façade Applications with a Flexible Hard- and Software Architecture

Development of Interactive Media Façade Applications with a Flexible Hard- and

Software Architecture

Stephan Bergemann, Julien Letellier, Jens Reinhardt, Jürgen Sieck

Research Group INKA

HTW Berlin

{stephan.bergemann, julien.letellier, jens.reinhardt, juergen.sieck}@htw-

berlin.de

Abstract

This paper starts with a short overview of the history of media façades,

gives an introduction to several visualization and interaction tech-

niques for buildings and architecture as well as discusses advantages

and disadvantages of different examples. It then explains some of

the technical background and inally it proposes an architecture that combines different hardware and software interfaces. This architecture

enables students, us and third-party developers to build applications for

our media façade, by using common standards.

6 Culture and Computer Science – Visual Worlds and Interactive Spaces

Figure 1 Schematic overview of the media façade at the FKI in Berlin

1 Introduction

1.1 Media façades

Media façades create new connections between digital and urban space.

Never before was there an interface between the physical and the digital

world that appeals not only to individual users, as in the case of a personal

computer, but also to a collective. [Tsch13] Existing media façades are often

temporary and mostly pure passive only playing back videos, pictures, and

other pre-rendered content. Interactive use cases for media façades are

currently focused by many research institutions.

Several installations have been described in the past allowing users to interact

with media façades. In the Tower of Winds in Japan environment parameters

like wind speed, direction and noise level are sampled and used to illuminate

the tower accordingly [Haeu09]. Given the 20th anniversary of the “Chaos

Computer Club” foundation, the Haus des Lehrers building in Berlin has been

transformed into a monochrome dot-matrix display that could be remotely

controlled by cell phones either sending text messages or calling a provided

number and then pressing number keys for interaction [Blin13]. Also using


text messages but in combination with a slingshot device, the Spread.Gun

[FZH10] and the SMSlingshot [FH12] projects were realised to virtually shoot

messages at a projection façade. The ARS Electronica Center in Linz created

an application to control their installed neon tube based media façade

[Bori11]. Using a smartphone and touch gestures users were able to draw on

the building. This was realised using camera tracking in the smartphone and

mapping the touch events on the building. All these projects used different

interaction methods and protocols for data interaction. These examples show

the usage of a wide range of input data and methods. There exist message

based systems such as Spread.Gun and SMSlingshot as well as button based

input at the Haus des Lehrers and inally live video streaming and touch gesture based applications like the drawing application for the Ars Electronica

Center.

1.2 Research Center for Culture and Computer Science (FKI)

The façade of the FKI located at Wilhelminenhof campus of HTW Berlin

(University of Applied Sciences) not only contains a photovoltaic system,

but also a media façade. This media façade will not be temporary and is not

designed to display only a single application, but is a permanent installa-

tion being open for students and staff members at the university, interaction

designers and video artists to realize their applications within an interactive

media development environment. One special feature of this media façade

is that it does not consist of one single screen area, but combines different

display technologies to gain a bigger area. Parts of the façade have LEDs,

rear projection with video projectors and ambient lights, see igure 1. About 30,000 RGB-LEDs cover an area of approximately 7 by 3 meters of the north

façade. The windows are projected by 12 video projectors on the north and

another 12 projectors on the south façade are planed with a full HD resolu-

tion. The remaining space between the windows of the irst and second loor and besides and the LED-Grid is covered by 36 LED-Spots that generate an

ambient light. The usage of different visualization technologies raises a few

key questions regarding the resolution, display of continuous content on the

heterogeneous sub systems, and inally capabilities to interact with either one application for the whole façade or different synchronized applications distri-

buted to a subset of output devices serving parts of the façade.


2 Visualization and interaction technologies

As of the characteristics of buildings and the diversity of display technologies

for buildings, it is not always possible to cover a whole façade with a single

projection area by one output device. In general, multiple output devices

have to be combined in order to cover the whole façade where each one

provides a part of the whole image to be displayed (segmentation). For images

and even more videos and interactive applications, synchronization of the

visualizations is a crucial requirement where its segments have to be played

back at the same time.

2.1 Display technologies

Light-emitting diodes (LED) are used on a variety of equipment and installa-

tions since they are small, eficient and can cover a wide spectrum of colors when using red, green and blue LEDs. Depending on the amount and the

arrengement of the LEDs a high resolution can be achieved. One drawback

are the high initial costs for large screens with many diodes.

With video projectors it is possible to project moving images onto a projec-

tion screen. The images can be projected either onto the front façade of the

building from the outside (front projection) or from the inside, typically using

the windows (rear projection). Depending on the projector, the resolution is

very high, but the brightness compared to LEDs is considerably low (or makes

the projectors very expensive, usually bigger and also noisy). So when using

standard projectors for front or rear projection, the ambient light level has to

be as low as possible.

2.2 Visualization technologies

Most façades will not only display prerendered content, but also live and

interactive media. When using multiple screens and display technolo-

gies, synchronization, scaling, and transition challenges must be solved.

ClusterGL21 is a library accompanied with some utility programs that tries to

solve some of these challanges by intercepting the rendering commands form

a given application and distributing them to other computers via network.

One drawback is that developers must have extensive programming and

basic mathematical skills, because ClusterGL2 can only work with OpenGL

applications using the overall implemented subset of OpenGL-Commands.

1 http://clustergl.org/


If all OpenGL-Commands deined by the Standard(s) would be covered by ClusterGL2, also existing Frameworks like OpenFrameworks2 or existing game

engines like Unity3 might become useful for implementing applications.

2.3 Interaction technologies

There are many hardware devices and sensors that can be used by visitors

to interact with a media façade: mobile devices, like smartphones or tablets;

individually developed hardware or do-it yourself (DIY) hardware, like custo-

mized Arduino4 devices or passive hardware, like microphones, cameras or

the Kinect5 are just a few. Mobile devices are the most promising because

they have a wide distribution and a lot of people carry smartphones or tablets

with them.

Some of these technologies have to be brought to the façade and others might

also be provided by the façade. In the case of the FKI we decided to install a

camera into the façade and add a wii hotspot for outdoor network.

3 Requirements engineering

After describing this technical background, ideas are presented that will focus

mostly on the question why people should interact with buildings, architec-

ture, or urban screens and how interaction methods can be enabled in form

of hardware and software interfaces. Therefor we prepared and conducted a

survey to evaluate the necessity of different sensors, supported programming

languages, primarily used software tools and more as we plan to establish

cooperations with other research institutes. Focusing on the local FKI media

façade, types of supported applications and interaction forms are described

as well as participation methods for developers, designers, media artists, and

visitors.

A major group of people developing for and interacting with our media

façade will be students. Several groups of students at the university of applied

sciences as well as our staff members have already developed prototypical

applications in the past few semesters, exploiting the different options for

interacting and visualizing their content. These applications were imple-

2 http://www.openframeworks.cc/3 http://unity3d.com/4 http://www.arduino.cc5 http://www.kinect.org


mented lacking the possibility to deploy it on the real façade as it was still

under construction but gave us an impression of what students expect in

terms of interaction interfaces. These use cases formed the foundation for the

inal system and where extended by further application concepts to fulill the needs of all deined target groups.Some of the interactive applications where using web and/or smartphone based applications to interact with the media

façade. This lead to the conclusion, that we would need outdoor wireless

network or have to rely on UMTS beeing available in the area around the

building. Other applications used camera and thereby gesture based interac-

tion input methods to control their applications.

Figure 2: Schematic overview of technical infrastructure

Development of Interactive Media Façade Applications with a Flexible Hard- and Software Architecture11

Therefore we installed a camera that currently is able to monitor the area

in front of the building when needed for the application to calculate on its

images. As of the different use cases for our façade (at night the whole façade

can be used, including the rear projection areas, whereas during day time

only the LED based areas are usable) there was also always the decision to

make for the students, whether to implement their application on the low

resolution but usable during daytime LED panels, or to implement their appli-

cation for the whole façade. This also implied the decision, to either think

about the proper distribution of their application (e.g. using ClusterGL2), or

being able to use whatever Software they liked but only display their applica-

tion on a very small part of the façade.

Already implemented and further possible applications are:

• Classical games, like Pong, Space Invaders and Tetris

• Multi-user and collaborative games (e.g. Paint It [BGHK13], Line-

Runner [GSVT13])

• Visualization of environmental information (e.g. sonic, weather, data

from the photovoltaic façade), conference and event information

4 Proposed architecture

Tracking down the complexity of the described ideas in terms of their interac-

tive needs, an application and library architecture was implemented to lexibly solve the above described occurring challenges of combining multiple output

devices, providing access to diverse sensor input data (the architecture allows

the integration of indoor (e.g. ambient light, sonic, temperature), outdoor (e.g.

camera, weather, sonic), and mobile sensors and devices (e.g. do-it-yourself

(DIY), smartphones including their numerous sensors)) and inally deploying applications on the facade.

One central element of the architecture is an application launcher, that is

capable of starting and stopping applications by either randomly switching

between non-interactive applications (like different visualizations) or selec-

tively starting interactive applications all in their appropriate environment.

This launcher also keeps track of the fact, that each application may only run

a certain maximum time.

As described in [BS13] there have been some prototypical applications

implemented showing the general capabilities of OSC and RTP for inter-


actively controllable applications. OSC is an easily adoptable and therefor

usable message format. RTP describes a protocol mainly used in conferen-

cing systems. It is therefore capable of transporting various multimedia data

formats with a clean focus on realtime processing. Combining OSC and RTP

enables application developers to implement a vast amount of different multi-

media application scenarios.

The architecture also includes a component that handles the interactive data

from the client applications. The communicator was implemented using

Node.js, a system that uses JavaScript as the programming language and

implements the concept of non-blocking IO, so each ile operation or network request is not handled in separate threads but stored in a form of a queue that

is processed sequentially. This allows for a large number of connections and

is an ideal solution for media façades. It is possible to connect to the Node.

js application via different protocols (e.g. OSC or RTP) and from various

platforms. This makes it easy to implement web applications for example.

[Lete13] describes this part of the architecture in detail.

5 Conclusion and future work

Media façades provide a way to transport art, information and multimedia

content. Even interactive installations can be integrated right into the façade

using plenty of different input devices and provide enhanced possibilities to

turn a regular building into a memorisable one. However, current research

only has investigated in different single application approaches that are

usually implemented only for one speciic purpose, e.g. festival or trade fair, and that are usually temporary installations. For the media façade of the FKI

we developed an architecture that works with a wide range of display tech-

nologies. The architecture combines several standards and enables external

developers to quickly build interactive applications for the façade.

Our future work besides implementing a wide range of different applications

exploiting this new architecture for our façade will include the development

of a web-based platform to upload applications, specify their settings and

capabilities to provide this information to users that want to use the applica-

tions on the façade with their own controllers.

Development of Interactive Media Façade Applications with a Flexible Hard- and Software Architecture13

6 Acknowledgement

This paper describes the work undertaken in the context of the project

SIGNAL hosted by the research group Information and Communication

Systems INKA that is generously funded by the European Regional Develop-

ment Fund (ERDF).

Literature

[BGHK13] Buff, Andreas; Gerlach, Thomas; Helbig, René; Kampe, Toralf; Kasimova, Gyuzel; Müller, Arend; Paeschke, Markus; Sieck, Jürgen: Interactive Media Facade Web Application with ClusterGL, 2013 (unpublished).

[Blin13] blinkenlights.net. Project Blinkenlights. url: http://blinkenlights.net, Last visited: April, 6th, 2013.

[Bori11] Boring, Sebastian et.al. Multi-user interaction on media façades through live video on mobile devices. In: Proceedings of the 2011 annual conference on Human factors in computing systems. CHI ’11. Vancouver, BC, Canada: ACM, 2011.

[BS13] Bergemann, Stephan; Sieck, Jürgen: Protocol- and Systemdesign for Interactive Media Façades. In: Informatik, Kultur, Technik 2013. Odessa, Ukraine.

[FH12] Fischer, Patrick Tobias; Hornecker, Eva: Urban HCI: spatial aspects in the design of shared encounters for media façades. In: Procee-dings of the 2012 ACM annual conference on Human Factors in Computing Systems. CHI ’12. Austin, Texas, USA, 2012.

[FZH10] Fischer, Patrick Tobias; Zöllner, Christian; Hornecker, Eva. VR/Urban: Spread.gun - design process and challenges in developing a shared encounter for media façades. In: Proceedings of the 24th BCS Interaction Specialist Group Conference. BCS ’10. Dundee, United Kingdom: British Computer Society, 2010, pp. 289–298. isbn: 978-1-78017-130-2.

[GSVT13] Gropengießer, Uwe; Sellmann, Marcus; Vermeulen, Fokke; Thiele-Maas, Michael; Sieck, Jürgen: Gaming with media facade, 2013 (unpublished).

[Haeu09] M. Hank. Häusler. Media façades: history, technology, content. avedition, [Ludwigsburg, Germany], 2009. isbn: 9783899861075 3899861078.

[Lete13] Letellier, Julien: Development of an Architecture for Interactive Media Façades. In: Modern Information Technology 2013. Odessa, Ukraine.


[Tsch13] Tscherteu, Gernot: Media Facades: Fundamental terms and concepts, http://www.mediaarchitecture.org/mediafacades2008/exhibition/basics-eng/, Last visited: April, 6th 2013.

Documents

Development of Interactive Media Façade