Mobile Augmented Reality: Exploring Content in Natural and Controlled Settings Using 3D Tracking

Abstract The advent of the mobile device has propelled education, adoption and implementation of Augmented Reality on a massive scale. The success and evolution of this event and the underlying technology over the last 5 years is nearly indisputable proof that the industry is progressing, though incredible examples of useful augmented reality applications are yet to be explored. The ability to recognize images, markers and 3D objects is one of the most important aspects of Augmented Reality. With present work we are proposing three application examples, which make recognition and visual search more intuitive, natural and accessible.

Author Keywords Augmented Reality; POI; SLAM; 3D tracking; Controlled Settings; Natural Settings; Information Visualization

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

General Terms Experimentation.

Permission to make digital or hard copies of part or all of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. Copyrights

for third-party components of this work must be honored. For all other

uses, contact the Owner/Author.

Copyright is held by the author/owner(s).

MobileHCI '13, Aug 27-30 2013, Munich, Germany

ACM 978-1-4503-2273-7/13/08. http://dx.doi.org/10.1145/2493190.2494653

Anton Fedosov metaio GmbH Infanteriestraße 19, Haus 4b Munich 80797 Germany anton.fedosov@metaio.com

Tobias Eble metaio GmbH Infanteriestraße 19, Haus 4b Munich 80797 Germany tobias.eble@metaio.com

MOBILE HCI 2013 – DEMOS AUGUST 29th, 2013 – MUNICH, GERMANY

460

Introduction Industry has made incredible advances in research efforts, having aligned tracking software with gravity and introduced commercially available 3D optical recognition and tracking systems that runs on off-the-shelf smartphones [6]. These achievements are bringing Augmented Reality (AR) applications from labs to a natural environment. Therefore there is an increase of interest from content creators to end-users to manifest their spatial ideas in true urban settings rather than in virtual spaces [7]. Our prototype is devoted to demonstrate potential of the Augmented Reality in controlled and natural settings (Fig. 1).

Tracking system Many existing AR platforms limit experiences to GPS mapping and the anchoring of graphics to simple, 2D surfaces [3, 9]. The platform we developed builds on these capabilities to deliver real-time tracking and recognition of entire 3D objects. In addition to making substantial improvements in a point cloud technology, we have been optimizing our prototypes for Simultaneous Localization and Mapping (SLAM) algorithms, e.g. [4], enabling the deployment of AR experience in almost any real-world environment. Developed technology is based on a state-of-the-art visual localization and tracking framework using local image features and 2D-3D point correspondences. These use cases rely on an offline-learned model of the environment, which in our case is a sparse representation comprising 3D points with associated feature descriptors.

Augmented Reality Experiences Surveys of Augmented Reality [2, 5] show us comprehensive outline for AR applications. We selected

three use cases in our project that we believe demonstrate an undisputable value of Augmented Reality technology over any other interaction type for a given scenario. The demo is a smartphone application utilizing several components as a tracking target, including:

§ A visually appealing high-definition poster of Marienplatz (Fig. 4), Munich’s primary landmark, that contains a number of points of interest along with billboard annotations adjusted accordingly to the façade of the building. Occluded objects that are not in direct view, positioned in such fashion to allow quick allocation of its real counterpart. The meaning of entire augmentation should be unambiguous for the user; it has to be clear what part of the scene is virtual and what is target for augmentation. We would like to benefit from such a use case to promote AR [1] experiences in an urban settings.

Figure 1. 3D tracking using SLAM system in an outdoor scene.

§ A city block miniature model that incorporates a 3D map with SLAM system allows content creators to

MOBILE HCI 2013 – DEMOS AUGUST 29th, 2013 – MUNICH, GERMANY

461

experiment with their design ideas to be deployed in a city structure. This use case features an interactive bird-view of an urban scene with possibility of natural zoom to unveil scene details. This setting can be useful in game design [10], applications in architectural domain and exhibition planning [8]. Upon target tracking an application triggers an animation with an airplane circulating around the city.

Figure 2. Tracked 3D map of the environment with a model.

§ A conventional photo printer. We believe that this simple Augmented Reality use case of hassle-free interactive step-by-step guide, showing a process of

replacing a cartridge in a printer save some time and effort for a number of end-users to complete that routine maintenance task at home or in office environments. We have used redundancy technique here superimposing wireframes of the printer 3D model over the real unit to increase human perception of the virtual and real parts of the same object.

Figure 3. Printer maintenance AR use case.

Infrastructural requirements The demo setting requires availability of enough space on the wall (4 x 2.35 meters) to accommodate high-definition printed poster of Marienplatz, primary tourist destination in historic city center of Munich (Fig. 4). A desk is needed to accommodate the photo printer (Fig. 3) and the city block model (Fig. 2). Wireless connectivity is required to enable real-time interactivity with demo application.

MOBILE HCI 2013 – DEMOS AUGUST 29th, 2013 – MUNICH, GERMANY

462

Figure 4. Marienplatz poster with superimposed POI annotations on a tracked building façade as seen through a smartphone.

References [1] AREngine. Introducing the Worlds First Augmented Reality Chipset - Metaio & ST-Ericsson http://www.youtube.com/watch?v=6br7NreTwD4

[2] Azuma, R.T. The Survey of Augmented Reality. In Presence: Teleoperators and Virtual Environments 6, 4 (1997), 355-385.

[3] Kato, H., Billinghurst, M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. Augmented Reality, 1999. (IWAR '99) Proceedings. 2nd IEEE and ACM International Workshop on , vol., no., pp.85,94, 1999

[4] Klein, G.; Murray, D. Parallel Tracking and Mapping on a camera phone. Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE International Symposium on , vol., no., pp.83-86.

[5] van Krevelen, D.W.F., Poelman, R. A survey of augmented reality technologies, applications and limitations – The International Journal of Virtual Reality, 2010, 9(2), 1-20.

[6] Kurz, D., Ben Himane, S. Inertial sensor-aligned visual feature descriptors. Computer Vision and Pattern

Recognition (CVPR), 2011 IEEE Conference on, vol., no., pp.161,166.

[7] Lonsing, W. Architectural models in urban landscapes: Synthesis of markers and virtual structures. Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH), 2011 IEEE International Symposium On , vol., no., pp.109,110.

[8] Miyashita, T., Meier, P., Tachikawa, T., Orlic, S., Eble, T., Scholz, V., Gapel, A., Gerl, O., Arnaudov, S., and Lieberknecht, S. An Augmented Reality museum guide. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '08), 103-106.

[9] Rekimoto, J. and Ayatsuka, Y. CyberCode: designing augmented reality environments with visual tags. In Proceedings of DARE 2000 on Designing augmented reality environments (DARE '00), 1-10.

[10] Wetzel, R., McCall, R., Braun, A.-K., and Broll, W. Guidelines for designing augmented reality games. In Proceedings of the 2008 Conference on Future Play: Research, Play, Share (Future Play '08), 173-180.

MOBILE HCI 2013 – DEMOS AUGUST 29th, 2013 – MUNICH, GERMANY

463