Natural exploration of 3d models ∗

Samuel A. Iacolina 1

samueliacolina@gmail.comAlessandro Soro 1,2

asoro@crs4.itRiccardo Scateni 1

riccardo@unica.it

1 Department of Mathematics and Computer ScienceUniversity of Cagliari

Via Ospedale, 72 09124 - Cagliari, Italy

2 CRS4 - Center for Advanced Studies, Research and Development in SardiniaPOLARIS Science Park - Ed. 1 - 09010 Pula - Italy

ABSTRACTWe report on two interactive systems for natural explorationof 3d models. Manipulation and navigation of 3d virtual ob-jects can be a difficult task for a novel user, specially witha common 2D display. With traditional input devices suchas 3d mice, trackballs, etc. the interaction doesn’t insist di-rectly on the models, but is mediated and not intuitive. Ouruser interface allows casual users to inspect 3D objects atvarious scales, panning, rotating, and zooming, all throughhand manipulations analogous to the way people interactwith the real world. We show the design and compare thetests on two alternative natural interfaces: multitouch andfree-hand gestures. Both provide a natural dual-handed in-teraction and at the same time free the user from the needof adopting a separate device.

Categories and Subject DescriptorsH.5.2 [User Interfaces]: Interaction styles

General TermsDesign

Keywordsmulti-touch, free-hand, interaction, natural, kinect, depth,gesture, 3d

∗an accompanying video for this paperhttp://www.youtube.com/watch?v=cTnhkKuwCDw

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHItaly-2011 September 13-16, 2011, Alghero, ITALYCopyright 2011 ACM 978-1-4503-0876-2/11/09 ...$10.00.

1. INTRODUCTIONNote for the readers: an accompanying video for this pa-

per is available athttp://www.youtube.com/watch?v=cTnhkKuwCDw.The design of the user interface is crucial to the develop-ment of hardware and software for exploration of 3D mod-els. Terms such as ’easy to use’, ’intuitive’, ’designed withyour needs in mind’, are often used to describe such tech-nologies. The recent explosion of off-the-desktop paradigms,such as virtual and augmented reality, ubiquitous comput-ing, etc, the design of a 3D user interface becomes even amore critical aspect to study for researchers and developers.

However, the public which uses these systems does notneed to be familiar with the use of computers, and evenif they are, many real world applications don’t allow anytraining to the use of a certain interface.

For example museums are evolving into one of the princi-pal components of the leisure and education industry and,in doing so, are faced with many challenges arising from thecompeting needs of attracting large masses of visitors, man-aging the ensuing logistics while preserving their specificitywith an appropriate balance between leisure and learning.

In the last few years, the classical concept of the museumspace as a room containing showcases full of objects is start-ing to give way to that of an environment in which the visitornot only reads and contemplates, but also interacts and in-terprets. This is leading to more and more frequent presenceof digital media displayed in interactive audiovisual instal-lations, playing now an important role among the contentsexhibited in museums.

This is why exhibition centers are actively looking for in-novative and creative installations, where the relationshipbetween user and content is more natural, thus forgettingthat there is a computer controlling their experience, ormaking them move away from preconceived ideas about it.

In this paper, we report on an interactive system for nat-ural immersive exploration of 3D models, such as those ac-quired by modern 3D scanning technology. A natural 3Dinterface allows casual users to inspect 3D objects at var-ious scales, while intuitively maintaining them within theoptimal display workspace.

2. RELATED WORKIn 3D interactive environments users can move and act

Figure 1: Automatic recognition of the shape of theclosed and open hand: the red area is the segmen-tation of the hand, the white region is the ConvexHull.

in a three-dimensional space, both the user and the systemwork on information based on the position of objects in 3Dspace.

The place in which the interaction is performed can beeither the physical space, a computer simulated representa-tion, or a combination of the above. When user input isperformed in the real space people can control the systemby means of gestures or movements, captured by a suitablesensor.

2.1 3D InteractionIn the 3D interactive model users perform their tasks and

share information with each other and with the system ina 3D space. Such scheme is intuitive since humans alwaysinteract in three dimensions in the real world [2].

Tasks can be classified in

• selection and manipulation of objects in the virtualspace

• navigation

• control of the system

Such activities can be performed in a virtual space by meansof different interaction techniques and using interactive tools.3D interaction techniques are schematized according to theabove classes of actions: techniques that support the ex-ploration of the virtual world are defined navigation tech-niques; those ones supporting the selection and interactionwith virtual objects are labeled as selection and manipu-lation techniques. Last, system control techniques supportthe activities of control of the application itself. In order forthe system to be usable and effective, interaction techniquesand devices must be tigthly and coherently interrelated[1].The power of interaction with a virtual model within thereal world allows the users to exploit their natural and in-nate ability of manipulation, and to set in real world the ex-change of information with real objects. Users, though, stillface difficulties in the interpretation of the virtual 3D sceneand in understanding the interaction paradigm[4]. Even ifmoving in a three-dimensional world is natural and intuitive,difficulties arise since, unlike in the real world, the virtualenvironment doesn’t allow the user to exploit all sensorialabilities: the ability to sense perspective and occlusions areprimary senses used by humans. Furthermore, though thevirtual 3D scene appears three-dimensional, it is still a pro-jection on a 2D surface, which causes inconsistencies in the

perceived depth, and misunderstandings in how the interac-tion should happen [8].

2.2 3d InterfacesThe user interface is the medium for user and system com-

munication, they provide a device for the representation ofthe three-dimensional state of the system, and devices capa-ble of acquiring the 3D input from the user (manipulation).The simple use of a 3D output device is not enough to pro-vide a 3D interaction. Users must be allowed to perform 3Dactions. To this end, specific input and output devices havebeen designed.

2.2.1 Hardware3D input devices vary in terms of degrees of freedom and

can be classified in standard devices , trackers, and gestureinterfaces[2]. Examples of the first type include keyboards,stylus, joysticks, mice, touch screens and trackballs. Evena simple 2D mouse can be used as a navigation device ifit allows the user to move within a virtual world. Trackersare capable of sensing and following the movements of thehead, hands, body of the user and, given the position overtime it is possible to update the viewpoint and the state ofthe virtual world. There are several types of 3D trackers,based on ultrasonic, mechanical, optical, hybrid inertial andmagnetic technologies. Other devices, such as wired glovesand bodysuits can sense the position of the hands and bodyand send such information to the system, enabling gesturalinteractions. Anyway these are still expensive solutions.

2.2.2 SoftwareUsers must be able to manipulate virtual objects. Manip-

ulation typically consists of selecting, moving and rotatingobjects. Direct-hand manipulation is the most natural tech-nique, because it is intuitive for people to use their ownhands to act on physical objects. Most techniques involve avirtual hand to select and relocate virtual objects and adopt3D widgets to modify the settings of objects or to searchand move objects [3]. Other techniques exploit Non-linearMapping for Direct Manipulation[10] and ray casting [9], inwhich a virtual beam is used to chose and select an object.Recent research focus on the design of interactive surfacesand whiteboards, to use e.g. in classrooms.

3. OUR PROPOSAL

3.1 Natural object explorationA variety of devices could be used for motion control, the

standard layout need a 2D vector, and a state (pressed /released). A simple approach without multitouch controlis to use a two button 2D (or 3D) mouse, use mouse dragto specify motion, pressing one button for rotate/pan andanother one for scaling. However, this standard scheme ofobject exploration, can be a difficult task for a novel user,even with a common 2D display . We introduce a 3D userinteraction technique which allows casual users to inspect 3Dobjects at various scales, integrating panning, rotating, andzooming controls into natural and intuitive operations. Ourexperiments concentrate on two innovative input devices:multitouch tables and vision-based gesture recognition, bothallowing a fully unencumbered interaction. Much of userinteraction is replaced by simple, natural hands motions,reducing user interface complexity and user burden.

(a) rotate (b) rotate Z-axis (c) zoom (d) pan

Figure 2: Comparison of the different manipulations performed on multi-touch table (row 2) and free handmanipulation (row 3).

3D ViewerOur research includes the design of a 3D viewer. We haveimplement several input filters for common file formats, bothbinary and ascii based, such as ply, off, x3d, vrml, etc. Theinterface is designed to support both precise input (suchthe ones the user performs with a traditional mouse) andmore rough, but also more natural, actions such as thoseones typically performed on multitouch tables. Tools forthe computation of bounding box, centroid, high resolutiontexturing, normals etc. are also provided.

Multitouch ExperienceMultitouch systems try to fill the gap between physical anddigital world, and provide a valuable support to the designof tools and environments in which the control is mainlyphysical. A multitouch system integrates the visualization(and consequently elaboration) of a model with its directmanipulation, providing an interaction paradigm in whichthe place where the action happens, and where its effectsare displayed are coincident. In other words the informationis transformed into a physical object, the 3D model, thatcan be manipulated to change some of its properties, suchas shape, size, position. We have readapted the traditionalmultitouch interaction paradigm to fit the exploration of 3dcontents.

Rotation by touching the model and moving the finger toa given direction the user can control a rotation hingedon the barycenter of the object.

Z-axis rotation by using both hands it is possible to causea revolution around the z-axis, where the X and Y axesare coincident with the vertical and horizontal sides ofthe screen, while the Z axis is perpendicular to thesurface of the screen.

Zoom by using both hands and moving them to oppositedirections the user can resize the model. Moving the

hands apart from each other enlarges the model, whilemoving hands towards each other shrinks the modelproportionally.

Panning hitting three or more points, even with the fingersof one hand, and dragging towards a given directiontranslates the model to the same direction.

Since the interaction is not limited to a single hand, the mul-titouch experience gives to the virtual objects some proper-ties of the real world, i.e. the possibility of direct manipu-lation by means of actions that are natural and intuitive forthe user, that is thus able to exploit his/her own abilities ofgesturing and shaping the world with the hands.

Free-hand InteractionTo further complete and improve the above paradigm forthe exploration of 3D models we have extended it to sup-port free-hand interaction. The model is displayed on a widescreen, under which a depth camera provides sensing of theuser movements and gestures. Just like the multitouch in-terface described above, our free-hand interaction schemeis based on a press/release paradigm, that in this case ismapped to the act of opening and closing the hand, whichresembles the act of grasping a real object. The algorithmthat recognizes the state of the hands is described furtheron.

Rotation closing the hand(s) and moving them along a di-rection the user can control the rotation of the modelaround its barycenter.

Z-axis rotation acting with both hands and dragging to-wards a given direction the user can rotate the modelaround an imaginary axis perpendicular to the surfaceof the screen.

Zoom operating with both hands, and imposing a move-ments along opposite directions, the user can change

Figure 3: Schema of the protocol infrastructure.

the size of the model. Moving the hands apart themodel is enlarged, while moving the hangs towardseach other the model is reduced.

Panning dragging with both hands the user can move theobject towards a given direction. Again, the abilityto interact with the 3D models just like if they werephysical objects, by means of both hands, lets the ex-ploration of the models an immediate and intuitive op-eration.

Being able to manipulate 3d models just as if they were phys-ical objects, by mimicking with both hands the real move-ments of grasping, rotating and moving in the real space,simplifies greatly such operations.

3.2 Description of the technologyA FTIR multitouch table [5][11]. has been used to support

the evaluation of the multitouch manipulation of 3d mod-els. The FTIR sensor was improved to allow further robust-ness to changing lighting conditions[6]. In order to supportfully unencumbered manipulation, we have implemented avision-based tracker based on the Kinect depth sensor. Weinitialize hand tracking with a skeleton tracking algorithm,which also detects the hand position in the depth image.We then incrementally track the hand position in the subse-quent images using local search. Hand open/closed shapesare recognized by estimating the local surface areas in thedepth image (figure 1). The pixels that in the depth mapbelong to the hand are considered; we compute the convex-hull of such region and its area is compared with the area ofthe shape of the hand. When the two silhouettes are nearlycoincident (closed hand) such ratio is close to 1. If, instead,the hand is open, the two silhouettes will differ consistently.The communication between the 3D viewer and the multi-touch sensor or the the depth sensor is supported by theTUIO protocol, according to the scheme in figure 3.

4. CONCLUSIONS AND FUTURE WORKPeople use their whole senses to gather and interpret the

physical world, and those tools that exploit such ability arethe most effective. From this point of view the user interfacemost common in HCI, based on different devices for sensingthe input of a command and representing the output of theeffect of that command, stands between the physical worldand its representation in a way that is not satisfactory formany practical cases. By contrast, a natural technique of ex-ploration based on direct manipulation positively builds onthe coincidence of the place of input and output. Multitouchand free-hand manipulation allow casual users to inspect 3Dobjects at various scales, integrating panning, rotating, andzooming controls into natural and intuitive operations. Fur-ther work will be aimed at exploring new applications forsuch technologies that suit both the way people interact withobjects and the way people interact with each other. The

social and collaborative dimensions will be explored in a sce-nario of interactive leisure/learning, such as the interactivemuseum. This will involve advancements on the recognitionof more abstract gestural languages[7] to control the system,in order to support more rich and exciting interaction.

5. REFERENCES[1] D. Bowman, E. Kruijff, J. LaViola, and I. Poupyrev.

3D User Interfaces: Theory and Practice.Addison-Wesley., Boston, 2005.

[2] D. A. Bowman, E. Kruijff, J. J. Laviola, andI. Poupyrev. An introduction to 3-d user interfacedesign. In Presence: Teleoperators and VirtualEnvironments, pages 96–108, 2001.

[3] N. Burtnyk, A. Khan, G. Fitzmaurice,R. Balakrishnan, and G. Kurtenbach. Stylecam:interactive stylized 3d navigation using integratedspatial & temporal controls. In Proceedings of the 15thannual ACM symposium on User interface softwareand technology, UIST ’02, pages 101–110, New York,NY, USA, 2002. ACM.

[4] J. M. Carroll. Human-Computer Interaction in theNew Millennium. ACM Press, New York, 2002.

[5] J. Y. Han. Low-cost multi-touch sensing throughfrustrated total internal reflection. In Proceedings ofthe 18th annual ACM symposium on User interfacesoftware and technology, UIST ’05, pages 115–118,New York, NY, USA, 2005. ACM.

[6] S. A. Iacolina, A. Soro, and R. Scateni. Improving ftirbased multi-touch sensors with ir shadow tracking. InProceedings of EICS 2011 (accepted), 2011.

[7] R. H. Jacoby, M. Ferneau, and J. Humphries. Gesturalinteraction in a virtual environment. In S. S. Fisher,J. O. Merritt, & M. T. Bolas, editor, Society ofPhoto-Optical Instrumentation Engineers (SPIE)Conference Series, volume 2177 of Presented at theSociety of Photo-Optical Instrumentation Engineers(SPIE) Conference, pages 355–364, Apr. 1994.

[8] A. Khan, B. Komalo, J. Stam, G. Fitzmaurice, andG. Kurtenbach. Hovercam: interactive 3d navigationfor proximal object inspection. In Proceedings of the2005 symposium on Interactive 3D graphics andgames, I3D ’05, pages 73–80, New York, NY, USA,2005. ACM.

[9] S. Lee, J. Seo, G. J. Kim, and C. mo Park. Evaluationof pointing techniques for ray casting selection invirtual environments. In In Third InternationalConference on Virtual Reality and Its Application inIndustry, pages 38–44, 2003.

[10] I. Poupyrev, M. Billinghurst, S. Weghorst, andT. Ichikawa. The go-go interaction technique:non-linear mapping for direct manipulation in vr. InProceedings of the 9th annual ACM symposium onUser interface software and technology, UIST ’96,pages 79–80, New York, NY, USA, 1996. ACM.

[11] J. Schoning, P. Brandl, F. Daiber, F. Echtler,O. Hilliges, J. Hook, M. Lochtefeld, N. Motamedi,L. Muller, P. Olivier, T. Roth, and U. von Zadow.Multi-touch surfaces: A technical guide. TechnicalReport TUM-I0833, University of Munster, 2008.