Embed Size (px)
Citation preview
i
Exploring gesture based interaction and
visualizations for supporting collaboration
Bachelor Thesis
Author: Andreas Simonsson Huck
Supervisors: Bahtijar Vogel & Oskar
Pettersson
Semester: Spring 2011
Level: C
Course code: 2ME10E
ii
Abstract
This thesis will introduce the concept of collaboratively using freehand gestures to interact
with visualizations. It could be problematic to work with data and visualizations together with
others in the traditional desktop setting because of the limited screen size and a single user
input device. Therefore this thesis suggests a solution by integrating computer vision and
gestures with interactive visualizations. This integration resulted in a prototype where
multiple users can interact with the same visualizations simultaneously. The prototype was
evaluated and tested on ten potential users. The results from the tests show that using gestures
have potential to support collaboration while working with interactive visualizations. It also
shows what components are needed in order to enable gestural interaction with visualizations.
Keywords
Gesture interaction, Vision based interaction, Interactive visualizations, Computer vision,
Collaborative interaction, Microsoft Kinect
iii
Contents 1 Introduction ......................................................................................................................... 1
1.1 Problem definition ....................................................................................................... 2
1.2 Purpose and goal .......................................................................................................... 3
1.3 Limitations ................................................................................................................... 3
1.4 Disposition ................................................................................................................... 3
2 Methods ............................................................................................................................... 5
2.1 Prototyping .................................................................................................................. 5
2.2 User tests ...................................................................................................................... 7
2.2.1 Usability test ......................................................................................................... 7
3 Theory ................................................................................................................................. 8
3.1 Interactive technologies ............................................................................................... 8
3.1.1 Technological ....................................................................................................... 8
3.1.2 Usability ............................................................................................................. 10
3.2 Data Visualization ..................................................................................................... 12
3.3 Collaboration using interactive technologies ............................................................ 13
3.4 Summary and features definition ............................................................................... 14
3.4.1 Interactive Technologies .................................................................................... 14
3.4.2 Visualizations ..................................................................................................... 16
3.4.3 Collaboration ...................................................................................................... 16
4 Design and development ................................................................................................... 17
4.1 Requirements ............................................................................................................. 17
4.2 Initial technological evaluation ................................................................................. 18
4.3 Development .............................................................................................................. 18
4.3.1 Design & implementation .................................................................................. 18
5 User tests ........................................................................................................................... 26
5.1 Users and settings ...................................................................................................... 26
5.2 Tasks .......................................................................................................................... 26
5.3 Questionnaires ........................................................................................................... 27
5.4 Data analysis and results ............................................................................................ 27
5.4.1 Calibration .......................................................................................................... 28
5.4.2 Click gesture ....................................................................................................... 29
5.4.3 Swipe gesture ..................................................................................................... 29
5.4.4 Swim gesture ...................................................................................................... 30
iv
5.4.5 Collaboration ...................................................................................................... 31
5.4.6 Summarizing impressions .................................................................................. 32
6 Conclusion ........................................................................................................................ 34
6.1 Discussion .................................................................................................................. 34
6.2 Future work ................................................................................................................ 37
6.3 Reflection ................................................................................................................... 38
References ................................................................................................................................ 39
Appendix A .............................................................................................................................. 43
Appendix B .............................................................................................................................. 45
Table of figures
Figure 2.1 Prototyping (CMS 2008) .......................................................................................... 6
Figure 3.1 OpenNI Framework (OpenNI User Guide, 2011) .................................................. 10
Figure 4.1 The gestures ............................................................................................................ 19
Figure 4.2 Basic interaction ..................................................................................................... 21
Figure 4.3 Google Maps ........................................................................................................... 22
Figure 4.4 Google Maps combined with charts and pictures ................................................... 23
Figure 4.5 Live demo ............................................................................................................... 24
Figure 4.6 System Components Overview ............................................................................... 25
Figure 4.7 Software overview .................................................................................................. 25
Figure 5.1 User tests ................................................................................................................. 26
Figure 5.2 Questionnaires/tasks results .................................................................................... 28
Figure 5.3 Calibration .............................................................................................................. 28
Figure 5.4 Click gesture ........................................................................................................... 29
Figure 5.5 Swipe gesture .......................................................................................................... 30
Figure 5.6 Swim gesture ........................................................................................................... 31
Figure 5.7 Collaboration .......................................................................................................... 32
List of Tables
Table 5.1 The Tasks ................................................................................................................. 27
1
1 Introduction Traditionally human computer interaction has consisted of different types of physical input
devices such as the mouse and keyboard. With the evolvement of ubiquitous computing and
touch technologies new types of interaction technologies have become more used and new
interfaces have been developed (Kela et al. 2005). New interaction technologies such as
computer vision and speech enable richer interaction with a computer (Hardenberg et al.
2001). However, these technologies have rarely been widely adopted due to cost and
complexity (Wang et al. 2009). This is often the case when interacting with large displays
where traditional input devices often restrict the user to be stationary while interacting with
the content. These devices also restrict the interaction to one user at the time. To enable
multiple users to interact with the same display, direct multi-touch approaches have been
suggested (Isenberg et al. 2007). Utilizing small and medium displays where the user can
physically reach all parts of the display has had some success. In order to create a more
natural and less restricted interaction, research into computer vision as an input device has
suggested gesture-based approaches (Nielsen et al. 2004).
Recent advances in the computer vision technology, such as the Microsoft Kinect (Microsoft,
2011), have enabled tracking of multiple users without any external input devices. This makes
it possible to interact with a computer without being restricted to for example a desktop
mouse or keyboard. Because a computer vision based system could allow the user to interact
from a distance it opens up for larger displays. The use of large displays is beneficial when for
example viewing visualized data because the user can use a wider field of vision. It also
allows multiple users to view and interact with the data (Zudilova-Seinstra et al. 2009).
The large amount of various data available today often has to be interpreted to be
comprehensible. One way this interpretation can be done is by creating visualizations.
―Visualization technologies empower users to perceive important patterns in a large amount
of data, identify areas that need further scrutiny and make sophisticated decisions‖ (Zudilova-
Seinstra et al. 2009:3). The writers also emphasize that the visualizations has to be interactive
and how that interaction is done affects the users understanding of the data. This is also the
case in visual analytics which Keim et al., defined as ―[v]isual analytics combines automated
analysis techniques with interactive visualizations for an effective understanding, reasoning
and decision making on the basis of very large and complex data sets.‖ (2008:157). Thomas et
al., (2005) suggests that visual analytics is a dialogue between the user and the visualizations
and through interaction the user asks the system for new views of the data.. Visualizations are
often used by people working in groups to more easily communicate and interpret the
information (Isenberg et al. 2007).
Collaboration occurs when people are working with information together and sharing a
common goal. Having people with different skills to work and interpret the information being
presented can enhance the understanding (Isenberg et al. 2010). However most visualization
systems only allow one user to interact with the content at the time and this is often done on a
small screen by using traditional input devices such as mouse and keyboard (Isenberg et al.
2
2007). Isenberg et al. stated that ―Attempting to collaborate under these conditions can be
awkward and unnatural‖ (2007:1232). Collaborative interaction of visualizations have often
been achieved by networked visualizations when people are physically distributed (Zudilova-
Seinstra et al. 2009).
The purpose of this thesis is to integrate interactive technologies with visualizations to support
collaboration. Integrating visualizations with multi-user computer vision technology will
enable collaboration between co-located users in the same workspace. Referring to the recent
advances in computer vision, the large amount of data that need to be displayed in an
interactive manner and the benefits of collaboration, these areas motivates this work.
1.1 Problem definition
While collaborating with other co-located users using interactive visualizations of data, the
interaction is restricted to one user at the time when using traditional desktop computers. The
relatively small display of a desktop computer also makes it more cumbersome for several
users to view the visualizations (Isenberg et al. 2007). Using a larger display makes it possible
to multiple users to interact with the visualizations (Zudilova-Seinstra et al. 2009). However
simultaneously multi-user interaction requires a system that supports multiple user input.
With advances of computer vision techniques, such as the Microsoft Kinect which is able to
recognize multiple users and gestures (Leyvand et al. 2011), this could be achieved. Using
freehand gestures removes the burden of having to physically interact with the content on a
large screen and it also supports natural interaction (Bellucci et. Al 2010). With the need for
easier means of multi-user interaction with visualized data on a large screen the main research
question in this thesis is formulated as follows.
How could the integration of interactive technologies and visualizations support
collaboration using gesture based controls utilizing large displays?
In order to tackle the main question the following sub-questions are formulated as well.
What are the components of an interactive gesture based visualization system?
Compared to traditional means of interacting with visualization systems, what are the
potentials of utilizing gesture based interaction?
These questions are answered by the development and evaluation of a prototype based on
existing research in the fields of computer vision, visualizations and collaborative interaction.
3
1.2 Purpose and goal
As stated in the introduction a vast amount of data is available and being used in different
kinds of visualizations. The interaction with these visualizations is often done in a traditional
setting with a desktop computer. With the benefit of using large displays while working with
visualizations of data combined with the less restricted interaction of a computer vision and
user recognition system. The purpose of this thesis is to design and develop a computer
vision-based interaction system to support collaboration while working with interactive
visualizations of data. By integrating existing visualizations of data with a computer vision
system that is able to track multiple users, this system supports collaboration between co-
located users.
In order to answer the research questions stated in the previous section the components of an
interactive gesture based visualization system has to be identified. The work includes
investigating potential technologies for computer vision and user gesture recognition then the
development of a prototype. The system is developed to support collaboration between two
co-located users while viewing visualized data in form of map with points of interests, charts
and pictures. The prototype is then tested by ten users to assess the usability and to see if it
could support collaboration.
1.3 Limitations
Both the fields of human computer interaction and visualization technologies are broad and
therefore a set of limitations are required. This thesis will focus on the interactive
technologies to support collaboration and thus not aim to create new visualizations or evaluate
the collaboration between the users.
The software efforts done as a part of this thesis were developed only to test the
gestural functionality and ideas.
This thesis does not aim to create new methods of visualization but instead makes use
of existing efforts.
This thesis does not evaluate the level of collaboration between the users; it merely
aims to support it by enabling gestures for multiple users.
1.4 Disposition
The second chapter of this thesis will introduce the methodical approaches that are used to
explore the problem definition and answer the research questions.
The third chapter will review current research in the fields of computer vision, visualizations
and collaborative interaction. Based on the literature review a number of features concerning
the different fields of research were identified.
The fourth chapter will describe the development of a prototype based on the features
identified in the literature review.
The fifth chapter will present the assessment of the prototype based on user tests.
4
The sixth and last chapter will lay out the results and discussion concerning the conclusions
from the previous chapters. In this chapter the research questions will be answered and
discussed. There will also be a discussion about future research that could be conducted to
further explore the area of this thesis.
5
2 Methods This chapter will discuss and argue for the methods used to answer the research question. The
methods will guide and structure the work in order to make it a scientific contribution. The
research consists of two main parts, prototyping and user testing. The prototype was
developed to investigate potential technologies that could be used for gestural interaction and
how it could support collaboration. The prototype was tested on ten users in order to evaluate
the system and the concept. The goal of using these methods combined was to answer the
research questions stated in chapter one by analyzing the data gathered from the tests.
2.1 Prototyping
A prototype is according to Sharp et al. ―[…] a limited representation of a design that allows
users to interact with it and to explore its suitability‖ (2007:530). The purposes of developing
a prototype was to be able to test it with users in order identify usability issues and assess the
general attitude towards such system. Sharp et al. states that a prototype can be used for
different purposes, for example ―[…] test out technical feasibility [and to] do some user
testing and evaluation‖ (2007:531). These statements cohere well with the choice of
prototyping as a method. Because this work makes use of a fairly new technology in a new
context a prototyping method was deemed most suitable. By developing a prototype the focus
could be on evaluating the concept and assessing the usability rather than developing flawless
code. The fast iterations of prototyping were another important reason for choosing it as a
development method due to the limited time frame of this project.
The prototype was developed as a high fidelity prototype which makes it fully functional.
This makes it possible to use it for exploration and tests (Sharp et al. 2007). Cooper et al.
argues that a product have to be fairly complete in order to conduct user tests with good
results (2007). Another aspect which argued for the use of a high fidelity prototype was the
relative completeness of the components which were to be implemented. Another reason was
to be able to test gestural interaction. This could be difficult if using a low fidelity prototype
such as a paper prototype and would not add much to the development. Therefore to be able
to test it with good results a fully functional prototype was deemed the most appropriate
approach.
To guide the prototyping efforts an iterative prototyping model was followed (CMS 2008). As
can be seen in Figure 2.1 it consists of a number of steps which are discussed below.
6
Figure 2.1 Prototyping (CMS 2008)
Initial investigation
Before any development could begin an initial investigation had to be done. The initial
investigation consisted of investigating potential computer vision technologies that
could be used for enabling gesture recognition. When appropriate hardware was
identified, different software approaches was evaluated. In this stage a basic concept
of the prototype was also constructed. This concept was refined throughout the whole
process.
Requirements definition
To get an understanding of what functionality the prototype was going to have,
requirements had to be identified. These requirements were created from identified
features in the literature survey. The initial and general features were identified during
the literature survey by looking at how earlier research have been approaching similar
problems. The features regarded the interactive technologies, usability, visualizations
and collaboration and are discussed more in chapter 3.
Additional requirements were identified during the iterations and were considered
while designing the prototype.
System Design
The design of the prototype consisted of three parts which in the end were combined
to cohere with the features identified in the literature survey. The first part was to
enable user tracking and gesture based interaction. The second part was to implement
a digital map and the third and final part was to implement the visualizations in
relation to the map.
Coding, testing…
The parts described in the System design were implemented by coding and testing.
Each part was first implemented separately and tested. The tests were conducted with
a couple of potential users which gave their opinions of the specific parts being tested.
These opinions guided the development throughout the iterations.
7
Implementation
When the parts were deemed complete they were combined and tested again. When
the prototype was fully functional a formal evaluation test was conducted. The test
will be described further in section 2.2.
Maintenance
This stage will not be considered during this work and the reason for this is that the
prototype will not reach a final working stage. The goal of using these steps was to be
able to test the acceptance of using gestures during collaboration and therefore this
final stage was considered to be redundant.
2.2 User tests
In order to assess the prototypes usefulness a series of user tests were conducted. The user
tests included both qualitative and quantitative methods using a number of tasks, observation
and questionnaires. The tasks can be viewed in chapter five, table 5.1, and the questionnaire
can be viewed in Appendix B. The data gathered from the tests was analyzed to assess the
prototype and identify usability issues. Sharp et al. suggests that evaluation of a product is
about checking if the users like and can use the product, especially if it is a new concept
(2007). Because the concept of using gestures to interact with visualizations is fairly new,
user testing was considered to be crucial to assess the usability prototype.
2.2.1 Usability test
To assess the usability of the prototype, a series of usability tests were conducted. Usability
tests include measuring the performance of a typical user on typical tasks according to Sharp
et al., (2007). They also claim that the usability test is used for evaluation of the prototypes
ease of use, effectiveness and user satisfaction.
The participants were given a set of tasks to complete. According to Lazar et al. a task list is
often needed when testing functional prototypes. They also emphasize that the tasks need to
be clear and should not need explanation. The tasks should include steps that are frequently
used in the prototype (2010). In addition to the tests qualitative data can be collected by
encouraging the participants to think aloud while interacting with the prototype. According to
Lazar et al. there is often very useful information to be gathered by letting the users express
their opinions during the tests (2010). The participants were observed while interacting with
the prototype in order to identify errors and problems of the interaction. Because of the multi-
user nature of the prototype the tests were conducted in pairs.
To capture the interaction between the users and the prototype the tests were recorded using a
video camera. Using multimedia content, such as video- and audio-recordings is a good way
to better understand the users‘ interaction with the system (Lazar et al. 2010). In order to
further complement the data from the user tests a questionnaire was given to the participants.
According to Sharp et al. questionaries‘ is a widely used approach for gather opinions from
the users and are used in addition to usability tests (2007). The questions in the questionnaire
regarded the tasks in the test and the overall attitude towards the prototype.
8
3 Theory The purpose of the literature survey was to get a greater understanding of similar systems and
how the interaction previously has been in computer vision based systems, visualization
systems and collaborative systems. To understand the scope of the current research in these
areas they were divided into sections. The information studied in the survey was broken down
into a list of features. These features were then used to support the development of the
prototype. Studying how existing systems work was argued by Sharp et al., as a viable
technique to help establishing requirements (2007).
The literature has been gathered from different sources. The main source was Google scholar
which led to a number of scientific databases such as ACM Digital Library (ACM, 2010)
IEEE Explorer (IEEE, 2010), Springerlink (SpringerLink, 2010). To be able to find relevant
articles in the databases a number of keywords was used. Keywords such as Gesture
interaction, Vision based interaction, Interactive visualizations, Computer vision,
Collaborative interaction, etc. Other sources of information have been in form of books.
3.1 Interactive technologies
This section will deal with interactive technologies such as computer vision and touch
surfaces. There has been much research in the field of gesture interaction using computer
vision. Some has used computer vision to enable touch on surfaces other have used computer
vision for mid-air interaction. There have been two prominent fields of research with two
different approaches related to computer vision, the technological and the usability
approaches. The technological approach has dealt with the technical aspects such as different
types of camera hardware and recognition algorithms. The usability approach has concerned
the usability and user experience of computer vision interaction.
3.1.1 Technological
The technological approach has focused on computer input devices such as different types of
cameras, laser and external handheld input devices to be able to accurately track movement of
the user.
There are different approaches to sense interaction using computer vision. First the computer
must be able to track the user. Then there has to be software that can recognize gestures from
the user. Nielsen et al. divided gestures in two categories, static and dynamic gestures (2004).
Static gestures are postures, for instance an open hand which is recognized by the system.
Dynamic gestures are movements that can be sensed by the system, one example is making a
circle motion or a push movement to interact.
An approach to track hands using external artifacts such as multi-colored gloves has been
purposed by Wang et al., (2009). Their technique was to use a color web-camera and a multi
colored glove in order to track one hand of the user in real time above a surface. They used
the tracking for interacting directly with 3d objects and hand poses for interacting indirectly.
They argue that their system can be extended to utilize two hands, however only if the hands
does not share the same space. The goal of their work was to create an inexpensive and robust
hand tracking input device.
9
Yin et al., (2010) made use of the technology created by Wang et al. in order to create a more
natural interaction for large displays in a case study with urban search and rescue (USAR).
They state that ―Interfaces that more closely conform to the way people naturally interact
would have the potential to lower the users‘ cognitive load, allowing them to concentrate on
the decision-making task‖ (2009:1). They used projectors and a camera mounted above a
surface table to track a hand of a user.
Vogel et al., (2005) used a motion-tracking system with reflective markers on each finger to
track one hand of the user. They investigated the possibility of pointing and clicking at a large
vertical display from a distance. They argued that the point and click metaphor is better than
using other gestures to interact.
Z-touch was a project that used (IR) laser to be able to sense touch above and on a surface
(Takeoka et al.2010). The authors stressed the importance of sensing depth of the users‘ hands
near a table-top surface because it allows 3d gestural interaction. Approaches using depth-
cameras have been investigated but the authors express that the cameras are still too
expensive and inaccurate.
One project using a depth-camera was DepthTouch by Benko et al. which combined direct
touch and mid-air interaction on a vertical display. The author stated that they wanted to
preserve the ―[…] ‗walk-up-and-use‘ simplicity of the touch-sensitive interactive surface‖
(2008:3) and thus not use any external controls. DepthTouch is using a depth camera to track
the user and hence does not make use of any external controls. Their approach was to track
the user‘s hands by calculating the nearest point to the camera and use them as ―blobs‖ for
interaction. They argued that their tracking procedure was robust but also suffers from
limitations. Their algorithm was not able to distinguish between hands that were close
together or close to the body.
Wilson (2010) explored the possibility to use a depth sensing camera as a touch sensor using
the Microsoft Kinect. He states that the Microsoft Kinect software development kit enables
skeletal models of the users viewed by the camera which is useful for animating in game
characters. His approach was to sense touch over a surface by mounting the Microsoft Kinect
sensor above the surface and hence not makes use of the skeleton tracking feature.
The Microsoft Kinect is a user tracking sensor for the Xbox 360. It combines an RGB camera
and depth sensors in order to track multiple users for gaming and entertainment purposes
(Microsoft 2010). Basically, users playing a game supported by the kinect can control it with
their body using gestures. However almost immediately after the release of the Microsoft
Kinect several ―hacks‖ were also made to it making it possible to connect the device to a PC (
Giles et al. 2010). This made it possible to use it for other purposes than entertainment and
gaming. In a couple of months several frameworks was developed to be able to make use of
the data from the sensors. One framework that supports the Microsoft kinect is OpenNI1
which enables communication between sensors, middleware and the application as seen in
figure 3.1.
1 OpenNI 2011 http://www.openni.org/
10
The OpenNI framework combined with the NITE middleware from Primesense2 made it
possible to use full body tracking in many different languages and applications. Open source
projects like AS3Kinecet3 made it possible to use the data and skeleton tracking from the
Microsoft Kinect via a server in Adobe Flash using ActionScript 3.
Another project that made use of the skeleton data from the device was as3osceleton4. The
author combined a server, OSCeleton5, with an actionscript3 multi-touch library, AS3TUIO
6.
The OSCeleton translates the data from the Microsoft Kinect to skeletal joints. These joints
are then used as touch points. By doing this the multi-touch features of the AS3TUIO library
can be used.
Figure 3.1 OpenNI Framework (OpenNI User Guide, 2011)
3.1.2 Usability
The usability approach has focused on the user friendliness of the different technological
solutions. But with the increase of touch and gesture enabled devices the usability has
sometimes become secondary objectives in the development. Norman et al., says that ‖[t]here
are several fundamental principles of interaction design that are completely independent of
technology‖ (2010:46) and that these often are overlooked when designing gestural
interfaces. These principles are:
• Visibility (also called perceived affordances or signifiers)
• Feedback
2 Primesense 2011 http://www.primesense.com/
3 AS3Kinect 2011 http://www.as3kinect.org/
4 as3osceleton 2011 http://blog.aboutme.be/2011/03/07/as3osceleton-using-kinect-as-a-multitouch-device-in-
adobe-air/ 5 OSCeleton 2011 https://github.com/Sensebloom/OSCeleton
6 AS3TUIO 2011 http://www.tuio.org/?flash
11
• Consistency (also known as standards)
• Non-destructive operations (hence the importance of undo)
• Discoverability: All operations can be discovered by systematic exploration of menus.
• Scalability: The operation should work on all screen sizes, small and large.
• Reliability: Operations should work. Period. And events should not happen randomly.
Norman (2010) also criticizes the use of gestures due to its lack of clues. He states that clues
of how the interaction should be done are essential for successful interaction with a computer.
Also gestures could be difficult to use due to cultural differences, one gesture that is natural
for one person is not necessarily natural for another. Nevertheless research in the area
describes different aspects of gestures and how they can be used.
The interaction in a vision based system could either use a single input such as controlling a
pointer with one hand or multiple inputs such as multi-touch but also combine with speech.
Systems like this are often referred to as multi-modal systems. Raisamo describes it like this
“Multimodal interaction is a way to make user interfaces natural and efficient with parallel
and synergistic use of two or more input or output modalities” (1999:vii). He also suggests
that many benefits of multimodal interaction also can be seen in two-handed interaction.
Forlines et al. compared uni-manual and bi-manual mouse interaction to direct touch
interaction on a horizontal display. They found that uni-manual works best with a traditional
mouse and bi-manual works better with direct touch on the surface. They also found that
direct touch might not lead to improved speed or performance but it could be better for ―[…]
fatigue, spatial memory, and awareness of other‘s actions in a multi-user setting[…]‖
(2007:655).
In the case of free hand interaction Cabral et al. found that gesture interaction was
significantly slower than mouse interaction and that it rapidly causes fatigue. However they
also found it easy to learn and could be useful in for instance collaborative work during short
periods. They argue for the benefits of using it in a multimodal interface due to its natural feel
of interaction (2005). The term natural interaction has been argued from different
perspectives. In the next section natural interaction is defined and discussed.
Natural interaction
Raisamo discusses the term Natural interaction and argues that it is not an exact expression
but merely a way to describe the ways to control an interface without any external devices
(1999). Others has also defined and discussed the naturalness of vision based interaction. Yin
et al. wrote ―By natural gestures we mean those encountered in spontaneous interaction, rather
than a set of artificial gestures chosen to simplify recognition‖ (2010:1) when describing a
multi-camera tabletop display system.
One way of achieving naturalness in interaction is to use multimodal or two handed
interaction (Raisamo 1999). He concluded that two handed interfaces were faster and easier to
interact with than traditional interfaces using mouse and a keyboard.
12
One thing that can affect the natural feel and user experience of a system is how fast it
responds to user input. In the next section the aspect of latency and responsiveness are
discussed.
Latency and responsiveness
The responsiveness in real time interaction is an important aspect. Wachs et Al. identified a
number of requirements for developing hand gesture interaction. One of these requirements
was responsiveness where they argued that ―The system should be able to perform real-time
gesture recognition. If slow, the system will be unacceptable for practical purposes.‖
(2011:62). They also argue that latency above 45ms between action and response is the
maximum value for the system to feel responsive.
Vogel et al., speaks about a classical problem in ―device-free interaction‖ where the user lacks
any physical buttons to click. One way to deal with this issue is to use a latency also known as
dwelling to make the pointer click after a set time. This could cause the interaction to suffer
from constant lag. To minimize any type of lag this technique should be avoided gestural
interaction (2005). However it might be a good way to give the user feedback and hints
concerning the systems state.
Lag could affect the ergonomics of a system by having to statically hold the hands over a
certain point. In the next section the ergonomics of gestural interfaces is discussed.
Ergonomics
While using one‘s body to interact with a computer system the aspect of fatigue has to be
taken into consideration. Standing and moving about while interacting with a screen could
cause the user to become tired and less focused. Cabral et al. experienced that even in short
periods of time gestural interaction could cause fatigue in their gesture based virtual reality
interface. They purposed that the hand should be used to control a cursor so the arms could be
closer to the body instead of having to keep the arm extended (2005). They considered
principles for creating good gestural interfaces established by Nielsen et al. (2004).
Avoid outer positions.
Relax muscles
Relaxed neutral position is in the middle between outer positions
Avoid repetition.
Avoid staying in static position
Avoid internal and external force on joints that may stop body fluids
3.2 Data Visualization
With the evolution of larger storage devices and new ways of collecting data the amount of
data being generated and saved today are enormous (Keim et al. 2008). The data itself are
seldom usable without ways to extract the information from it. Furthermore they describe an
―information overload problem‖ which refers to that the data might be interpreted incorrectly
13
due to the vast amount at ones disposal. One way to overview large amount of data is to make
graphical representations of it in form of visualizations.
Visualization of data is a powerful way to make sense of large amounts of data. However
without interactivity the visualizations are often used as communication and not a tool during
the workflow for better understanding of the data. How the interaction is done also affect the
users understanding of the data (Zudilova-Seinstra et al. 2009). Card et al. defines information
visualizations as ―The use of computer-supported, interactive, and dynamic visual
representations of data to amplify cognition‖ (2009:6). Thomas et al. states that ―Visual
representations invite the user to explore his or her data‖ and that it therefore has to be
interactive (2005:69). This highlights the need of interactivity in visualizations of data.
Depending on the nature of the data there are different ways of visualizing it. The data can for
example be one dimensional or two-dimensional (Keim et al. 2002). If a dataset consists of
two-dimensional GPS-coordinates the data can be represented on a digital map, for instance
Google maps. Other types of visualizations could be in from of one-dimensional charts.
One project that made use of a digital map and chars for visualization was the interactive
web-based visualization tool by Vogel (2011). The tool was developed to help students
understand environmental data collected during science based inquiry learning. They stated
that it is important to be able to, in an interactive manner, explore, analyze and reflect upon
the data. These data comprises from geo-tagged content and sensor data, and the overall
dataset is offered over Google spreadsheet (Vogel, 2011). The data was visualized using
Google maps and Google visualization API. The prototype in this work will make use of the
same data source in order to create visualizations.
3.3 Collaboration using interactive technologies
When people are working towards a common goal they often collaborate. Isenberg et al.,
states that people collaborating often use visualizations to interpret data. They also say that
having people with different skills working with the same information is beneficial especially
when the data is too complex for one user to handle or when the amount of data is too great
(2007).
Collaborative visualization is according to Zudilova-Seinstra et al. when more than one user
can interact with the same visualization at the time. They also say that this type of
collaboration often has been used over networks when users are physically distributed (2009).
Isenberg et al. presented a multi-user information visualization system using a large horizontal
multi-touch display. Their research concerned the collaboration between co-located users
while interacting with information visualization. They provided a set of guidelines for a ―co-
located collaborative information visualization systems‖ (2007:1233). In the guidelines of the
hardware they stressed the issue of display size. The display size is important in order to fit
the visualizations but also to enable multiple users to interact with it. Another aspect of the
hardware is the input from the users. In order to support collaboration multiple users have to
be able to interact with the content at the same time.
14
Another research project that enabled multiuser interaction was the DiamonTouch by Dietz et
al. which used a table-top touch screen to track multiple user interaction. In their research they
identified potential problems using a traditional mouse in a multiuser environment. The
problems consisted of keeping track of whose pointer was whose. They also meant that using
an external device hinders the users natural movement such as reaching, touching and
grasping (2001).
While evaluating a gestural interface for virtual reality Cabral et al. came to the conclusion
that despite the drawback of fatigue, their gestural interaction system could be used to support
collaboration by extend it to track multiple users (2005).
Stewart et al. found that children often gather around a computer screen and wanting to
interact with the content. They also found that the user experience increased when having
control over the content. Therefore they suggested a multi-user system that made use of
multiple input devices on a single display. They concluded that using multiple mice in one
screen was more fun while testing it with children (1999).
3.4 Summary and features definition
The purpose of this section is to identify possible features from the literature survey which
will serve as a basis and to be considered during the development of the prototype.
3.4.1 Interactive Technologies
The first section of the literature survey addressed interactive technologies, both technological
and usability aspects were described.
Technological
There have been many different approaches to use computer vision as input for interaction.
Some have used it to sense touch over a surface (Wang et al. 2009), (Takeoka et al. 2010)
(Wilson, 2010). Others have used it for freehand interaction (Benko et al. 2008). With the
release and hacks of the Microsoft Kinect a relatively low-cost camera with the ability to track
multiple users became available. One benefit of using the Microsoft Kinect could be that it
does not require any external devices for interaction. In the technological section the
Microsoft Kinect and some of its application were described. Because of the availability of
frameworks and the robustness of the user tracking this was considered to be the first feature.
A list of features related to the technological aspects is presented below.
Interactive Technologies
o Full body tracking
o Multi-user tracking
o Gesture recognition
o Device free – freehand
By looking at the Microsoft Kinect a number of sub-features were identified. The Microsoft
Kinect combined with the OpenNI framework and the NITE middleware enables full body
tracking of several users. This feature is essential to allow simultaneously interaction and thus
15
support collaborative interaction. Another aspect of gesture interaction discussed in the
literature survey was the benefit of device-free or freehand interaction. This is also enabled by
the Microsoft Kinect.
Usability
The second section of the interactive technologies was about the usability aspects of gesture
based interaction.
Norman et al. argued that principles and standards in usability often are overlooked while
designing gestural interfaces (2010). These principles are important to have in mind especially
when designing gestural interfaces which Norman claim are invisible and does not give the
user any clues of interaction (2010).
Raisamo concluded that interaction of a two handed interface is more natural and faster to use
if designed well (1999). Forlines et al. found that bimanual interaction works best with direct
touch on a surface (2007). Cabral et al., found device less two handed interaction is easy to
learn and could be useful for collaboration and that it feels natural (2005). Raisamo described
natural interaction as device-less interaction with a compute (1999). Norman (2010) does
however not think that most gestures are natural because of cultural differences. He does not
think that gestures are easy to learn and remember due to lack of clues in how the interaction
is supposed to be done.
Wachs et al. discussed that gestural interaction should be in real time thus not have high
latency (2011). Vogel et al. brought up the issue of clicking with device-free interaction
.Often the solution have been to use dwelling over an object to interact with it. This is not a
good way for interaction since it causes a constant lag (2005).
Cabral et al. found that gestural interaction could cause fatigue and is best used for short
periods of time. They followed a list of principles that would make gestural interaction more
ergonomically (2005). Below the features of the usability aspect are presented.
Usability aspects
o Principles of interaction design
o Clues of interaction
o Two-handed
o Low latency
o Principles of good ergonomics
To ensure good usability and user experience the principles of interaction design described by
Norman et al. (2010) would have to be considered. The principles of good ergonomics by
Nielsen et al. (2004) are also important to achieve a pleasant user experience. Because of
device-free interaction there has to be clues of how the interaction is done. With the argument
that two handed interaction is more natural and faster in a two handed interface this is also
considered a feature. Another important aspect related to the user experience would be that
the application should be responsive at all time. Therefor a direct click interaction should be
used instead of the alternative using dwelling.
16
3.4.2 Visualizations
The second section of the literature survey was about visualizations. Zudilova-Seinstra et al.,
highlighted the importance of visualization in order to understand data. They also states by
making visualizations interactive it becomes more of a tool in understanding the data than for
static presentation of it (2009).Vogel (2011) used a Google map, charts and pictures to present
environmental data. This was to let users explore data collected. The features of the
visualizations are presented below.
Interactive visualizations
o External data source
o Digital map
o Charts
o Pictures
Because of the importance of interactivity a number of approaches to visualize the data were
considered. The GPS data would need to be presented in a way that is simple and relates to
where the other data was gathered. This could be done by a digital map. To easily compare
the one dimensional data it could be presented in charts. To get a better view of the location
where the data was gathered, pictures would be needed.
3.4.3 Collaboration
The third and last section of the literature survey concerned collaboration using interactive
technologies. The benefit of having several people working together while trying to
understand data was discussed.
Isenberg et al. stated that visualizations often are used in collaboration in order to discuss and
understand data. Their multi-touch multi-user information visualization system made use of a
set of guidelines. Important aspects in the guidelines regarded the display size and input from
the users (2007). To enable interaction from multiple users a large display is needed.
Dietz et al. found that multiuser interaction using a traditional mouse was problematic. Instead
they used a multi-touch display to enable multi-user interaction (2001).
Stewart et al. found that the user experience could be better if one have control of the content
on the screen when collaborating (1999). Below features that enable collaboration is
presented.
Collaboration
o Large display
o Simultaneous interaction
A large display is important to physically allow numerous users to interact and view the data.
There has to be simultaneous interaction from several users to enable collaborative
interaction.
In the next chapter the features described above will be used to support the design and
development of the prototype.
17
4 Design and development This chapter will describe the design and development of the prototype. First the features
identified in the literature survey will be used to create requirements that will guide the
development process. Then a short technological evaluation was made. Third the development
efforts will be described in relation to the requirements.
4.1 Requirements The features identified in the literature survey are in this section used to create a set of
requirements that will support the development of the prototype.
Interactive technologies
To enable multiple users to interact using gestures without any external devices the system
should allow for:
Full body tracking: The prototype shall track the whole body of each user
independently.
Multi-user tracking: The prototype must allow multiple users to interact with the
content simultaneously.
Gestural interaction: The prototype shall have a set of gestures for interacting with
the content on the screen.
Device free interaction: The prototype shall not require any external devices in order
to interact with it.
Usability aspects
To increase the probability of a good user experience a number of usability requirements were
identified:
Principles of interaction design: The prototype shall follow the principles of
interaction design stated in the literature survey.
Clues of interaction: The prototype shall have visual clues of how to interact with it.
Two-handed: The prototype shall make use of both hands of each user.
Low latency: The interaction of the prototype shall be continuous and make use of
direct interaction instead of dwelling interaction where the user only moves the hand
over an object and the interaction is triggered by a set timer.
Principles of good ergonomics: The prototype shall conform to the principles of
good ergonomics stated in the literature survey.
Interactive visualizations
To create interactive visualizations the following requirements were identified:
External data source: The prototype shall make use of an external data source in
order to create visualizations.
Digital map: The GPS data shall be presented on an interactive digital map.
18
Charts: The one dimensional data shall be presented in charts.
Pictures: Pictures shall be displayed in relation to the charts.
Collaboration
Because the prototype should support collaboration, requirements regarding this aspect were
identified:
Large display: The prototype shall make use of a large display.
Simultaneously interaction: The prototype shall allow multiple users to interact
simultaneously.
4.2 Initial technological evaluation The development of the prototype started with testing out the feasibility of using the
Microsoft Kinect as an input device. Two Actionscript approaches were compared and
evaluated. The first framework was the AS3Kinect which had the skeleton tracking feature
but no good way of creating interaction. The other approach was as3osceleton combined with
the multi-touch library AS3Tuio which also had the skeleton tracking feature. Because of the
multi-touch features which made it possible to create gestures, the AS3Tuio was the given
choice for the development of the prototype.
4.3 Development
The development consisted of several iterations of high fidelity prototyping. The prototype
was developed with Adobe Flash Builder 47 with the Flex SDK
8. The Flex SDK is an open
source framework for cross platform web, mobile and desktop application development. The
SDK contains many components which are used to rapidly create rich interfaces (Adobe Flex,
2011). To support each step of the development the requirements were used to implement the
functionality of the prototype.
During the initial investigations a basic concept of the prototype was constructed. This was
done to get an overview of what was to be done. The concept of the prototype was to create a
digital map with geo-tagged data related to locations. The data was going to be visualized by
using charts and pictures. The map, charts and pictures was going to be interactive and
controlled solely by using gestures. The prototype was going to let multiple users to interact
with the visualizations simultaneously. This was going to be achieved by using the Microsoft
Kinect. The prototype was also going to be used on a large display in order to more easily
support collaboration.
4.3.1 Design & implementation
This section will describe the different parts of the prototype and how the requirements are
met during the development. The development consisted of three main components which
were iteratively implemented and combined. The parts are: the basic interaction using
7Flash Builder 2011 http://www.adobe.com/products/flash-builder.html
8Flex SDK 2011 http://www.adobe.com/products/flex/
19
gestures, a digital map and the charts and pictures. Below the development of each part is
described.
Basic interaction
The first part of the development regarded the implementation of the basic user tracking and
gestural interaction. The first step was to consider the requirements of the interactive
technologies. To enable full body tracking of multiple users a combination of the OpenNI
framework and the NITE middleware was used. The software combined registers the position
of the user and creates data of a skeleton which are later used in the prototype. When the
tracking of the users was functional, the requirements of gestural and device free interaction
was implemented.
To enable the device free interaction the skeletal data from the OpenNI and NITE software
was put through a server, OSCeletton, to translate it to skeletal joints. The data of the skeletal
joints are read by the prototype and then translated into touch points by the as3osceleton
library. By creating touch points from the skeletal joints that represent the hands of the user;
this lets the user to basically touch the screen from a distance.
During this stage some of the usability requirements also had to be considered. To conform to
the visibility principle in the principles of interaction design, graphical pointers in form of
hands were positioned at the touch points. The reason for making the pointers to look like
hands was to give the users clues of how to interact and thus meet the requirement of clues of
interaction. To strengthen this clue the hands are always visible, even if the user is resting his
arms along the sides. The hands and touch points are used to create the gestures in the
prototype.
There are four gestures that the prototype is making use of. Three of the gestures were
identified as a part of the AS3Tuio multi-touch library and the Flex SDK. The forth one is
required to calibrate the NITE middleware in order to track the users independently. The
gestures were chosen because of their simplicity and that they were supported by the multi-
touch library. The principles of ergonomics were used as guidelines for choosing the gestures
so they would be as usable as possible.
Figure 4.1 shows the gestures and below the gestures are described in relation to the
functionality.
Figure 4.1 The gestures
20
Calibration
Before any interaction can be done each user have to calibrate. This is done by
standing in front of the screen and raising the hands in level with the head. When the
calibration is done each user will be assigned a set of graphical pointers connected to
the hands.
Click
The most basic interaction of the prototype is the click. This is done by moving one
hand at a pace towards the screen. The click is a key component in the other gestures
and has to be performed as an initial gesture. The click alone is used to open markers
on the map and close charts and pictures.
Swipe
The swipe is done by performing a click gesture then while with the arm extended
moving it x or y direction. The swipe is used to pan the map and drag and drop the
charts and pictures.
Swim
The swim gestures are performed by first clicking with both hands and then make a
swim motion with the arms. Moving the hands further or closer to each other enables
the zoom out or in functionality. The swim gesture is used to zoom in and out the map
but also to scale the charts and pictures.
In this stage the requirement of two handed interaction was met. The click and swipe gesture
are can be performed by any hand while the swim gesture requires both hands. By using direct
interaction with objects the requirement of Low latency was also met. By using only three
gestures for all interaction, the prototype follow Consistency requirement of the basic
principles of interaction. The idea behind this was to use the same gestures for similar
operations in order to make it easier to recognize how to use. The other requirements of the
principles of interaction design were none-destructive operations, discoverability, scalability
and reliability. The prototype does not have any destructive functionality and therefore this
principle is not considered. Because of the simplicity of the prototype all the functionality is
shown to the user at all time. The scalability principle regarded that the operation should work
on all screen sizes. Even though the prototype is developed with large screens in mind it can
still be used with an ordinary desktop monitor. The reliability principle was considered
because it was going to be tested on potential users. Therefore it was important to have a
prototype that was stable and functional.
To conform to the requirement Principles of good ergonomics the gestures was meant to be as
easy and simple as possible. Below the principles are discussed in relation to the gestures.
Avoid outer positions.
The gestures utilized in the prototype do not require full extension of joints. This is an
important aspect to avoid causing fatigue while performing the gestures.
Relax muscles
To successfully complete a gesture it is unavoidable to tense the muscles. However the
interaction is designed to work best with relaxed motions and not to cause fatigue.
21
Relaxed neutral position is in the middle between outer positions
As stated earlier the gestures do not require the user to fully extend any joints. All the
interaction can be done by between the outer positions. The outer positions are a fully
extended arm or fully bent arm.
Avoid repetition.
Because the prototype makes use of three gestures for six different tasks, the gestures
will be repeated often. But because the gesture movement is a bit flexible the user can
vary how to perform the gestures. For instance the swipe motion can be done with
either hands, the hand can be closed, open or pointing with one finger at the screen.
Avoid staying in static position
There is no functionality of the prototype that requires the user to be in a static
position for a longer period of time. The gestures were used because of their
continuous movement which prevents static positions.
Avoid internal and external force on joints that may stop body fluids
By not using any handheld devices for interaction there is no external force that can be
discomforting. The gestures are simple and relaxed and are thus not introducing much
internal forces on the joints.
In this first part the gestures was used for creating interaction with simple objects. The
interaction consisted of scaling and moving objects. Figure 4.2 shows the basic concept with
the hands and a square which the user could move and scale.
Figure 4.2 Basic interaction
This was tested and demonstrated for a couple of users which gave their opinions. Some
opinions concerned the pointers which were connected to the hands. In the first tests, the
22
hands were static images with a fixed size which made it hard to perceive the depth of the
interaction. This was solved by making size of the pointers depend on how far from the screen
they are. Another concern stated by the participants was that it was hard to know when the
interaction occurred. A suggestion about using a glow around the hands was made and also
implemented. When the basic gestural interaction was deemed complete the implementation
of the digital map began.
Digital map
The second part of the prototype was the digital map which was created with the Google maps
API and can be seen in figure 4.3. In this section the requirement of using an external data
source and a digital map was met. The map was used to visualize data from an external data
source. The data was stored in a Google spreadsheet document which is published as an RSS
feed. The RSS feed was put through a PHP-proxy to be able to read it in the prototype. The
data comprises from geo-tagged content and sensor data. The geo-tagging consists of GPS
coordinates in latitude and longitude. These coordinates are used to position interactive
markers on the map in order to display the sensor data in relation to the locations.
The original mouse interaction of the Google maps consists of clicking markers, panning and
zooming of the map. This interaction was implemented by applying the gestures instead of
using a mouse. The functionality of the Google Maps API is rather restricted when it comes to
multi-touch interaction. Therefore the gestural interaction had to be adjusted to suit the map.
Figure 4.3 Google Maps
The map was also tested by a couple of users which gave their opinions regarding the gestural
interaction of the map. The first implementation made use of a satellite map which the
participants thought was slow and hard to navigate. Therefore a more simple map was used
instead. When the map was functional the implementation of the sensor data visualizations
began.
23
Charts & pictures
In this section the implementation of the charts and pictures are described and thus the
requirement of charts and pictures is met. The sensor data from the data source was
visualized by using the standard column chart component in Flex. The data source also
contained URL to pictures related to the locations. The charts and pictures were set to be
displayed when a marker was clicked. In order to be able to close the charts and pictures a red
square was added as a close button. The gestural interaction created in the first part was
implemented for the charts and pictures. This means that they can be scaled and moved.
Figure 4.4 shows the different parts combined.
Figure 4.4 Google Maps combined with charts and pictures
24
When the three parts were implemented they were tested again with a couple of users. The
participants wanted to be able view more than one chart and picture at the time. Therefore the
functionality of opening multiple markers was implemented. When the prototype was deemed
functional, an evaluation test was conducted. The evaluation test is described in detail in
chapter five. Figure 4.5 shows a user interacting with the prototype. Furthermore this is also
demonstrated in a short demo video which is available at: http://vimeo.com/24271754
Figure 4.5 Live demo
The implementation of the different parts resulted in a fully functional prototype where two
users can interact with a Google map, charts and pictures using gestures. During the
development stage a number of components were identified and integrated in order to create
the functionality.
Figure 4.6 presents an overview of the components used. These components can be divided
into two parts, user input and system output. The user input consists of the Microsoft Kinect
combined with tracking and gesture recognition software. The system output is the Google
map with visualizations displayed on a projector screen. These parts each is comprised of a
number of frameworks, libraries and APIs which were discussed during the implementation
and can be seen in figure 4.7.
25
Figure 4.6 System Components Overview
The OpenNI, NITE and OSCeleton software was used to track the users. The AS3osceleton,
AS3TUIO and native Actionscript gestures was implemented in a Flex application. These
parts combined enabled the user input from the Microsoft Kinect. The Google Maps and
Google Spreadsheet together with components from the Flex library were used to create the
visualizations and output to the users.
Figure 4.7 Software overview
26
5 User tests The user tests were conducted to assess the usability and identify usability issues of the
prototype. The participants were given a short user manual to familiarize with the concept
before the tests, see Appendix A. To further instruct the participants a short demonstration of
the basic functionality were made in connection to each test session. When the participants
had understood the basics they were given a set of tasks to complete. After the tasks had been
completed they were given a questionnaire with questions related to the tasks.
To gather as much data as possible a video camera was used to record the tests. The video
material was used to aid the analysis of the tests.
5.1 Users and settings
The user tests were conducted with ten participants; three of them were females and seven
were males. The participants age ranged from twenty-one to thirty-one. All of the participants
were used to work with computers. The test took place in a computer lab with a projector
screen. The size of the interaction space was 3x3 meters in front of the projector screen.
Figure 5.1 shows the setting in which the tests were conducted.
Figure 5.1 User tests
5.2 Tasks
The participants were given seven tasks to complete while interacting with the prototype, see
table 5.1. The tasks were composed to identify potential usability issues and drawbacks
concerning the interaction of the prototype. To assess the tasks they were related to the
gestures which are the core interaction of the prototype. To collect subjective opinions about
the interaction the participants were encouraged to think out loud while performing the task.
27
Table 5.1 The Tasks
Description
1 Stand in front of the screen, raise your hands as instructed and wait for
calibration(both, one at the time)
2 By using the swim gesture zoom the map.
3 Click one marker each to view the data (charts and pictures).
4 Move the charts and pictures by swiping so they do not overlap each
other.
5 Close the charts by clicking the red square.
6 Scale the pictures by using the swim gesture so they do not fit the
screen anymore, and then close them.
7 Open one marker each (as in Task 3) and switch objects with each other
(collaborate).
5.3 Questionnaires
After the test of the prototype the participants were given a questionnaire. The questionnaire
was comprised of eight close ended questions with a likert-scale ranging from 1 to 4 (1 =
Strongly disagree; 2 = Disagree; 3 = Agree; 4 = Strongly agree). The questions regarded the
participants opinions about the tasks performed in the test. In addition three open ended
general questions were included in the questionnaire. The general questions regarded the
overall impression of the prototype, collaboration and suggestions for improvement.
5.4 Data analysis and results
In this section the results from the questionnaire and the tasks are presented and analyzed. The
gestures identified during the development have served as a base for the test and analysis of
the results. The gestures have been used as categories and below the results for each gesture
are presented. The overall results presented in figure 5.2 shows the mean value for each task.
From these results usability issues, mainly related to the click and swim gesture, were
identified. The overall mean for the prototype was 2.9125 which indicate that it overall
worked rather well but clearly had some usability issues.
28
Figure 5.2 Questionnaires/tasks results
5.4.1 Calibration
The first task regarded the calibration of the users. Each participant had to make the
calibration pose in order use the prototype. The pose were described in the user manual and
also demonstrated to the users. In all the tests the initial calibration went fairly easy. Most of
the users agreed or strongly agreed that it was easy to understand according to figure 5.3.
However some drawbacks were observed during the tests.
Figure 5.3 Calibration
The users have to be in the Microsoft Kinects field of vision at all time otherwise the
calibration could be lost. If the calibration is lost the user has to recalibrate. This happened a
few times during the tests, and although the recalibration also went easy this is a tiring
interruption in the interaction. When this occurred discontent from the participants were
noted. Therefore the calibration should be easier and more robust.
0
0,5
1
1,5
2
2,5
3
3,5
4
Task1 Task3 Task5 Task 4 Task7q1 Task2 Task6 Task7q1 Task7q2
Calibration Click gesture Swipe Gesture Swim gesture Collaboration
Mean values of the tasks
0
1
2
3
4
5
6
Stronglydissagree
Disaagree Agree Strongly Agree
Callibration
Task1
29
5.4.2 Click gesture
Tasks three and five required the participant to use the Click gesture. Task three was to open a
marker on the map; task five was to close the charts and pictures displayed when a marker
was opened.
Figure 5.4 Click gesture
The results from the questionnaire were alike for both of the tasks. A majority of the
participants thought it was hard to do the click gesture. According to figure 5.4 the disagree
and strongly disagree results indicate that there are some usability issues. This was also
observed during the tests. One issue which became apparent was that it was hard to hold the
hand still while performing the click gesture. The result was that the participants missed the
markers or moved the map instead. Problems also occurred when one user wanted to click a
marker while the other paned the map. Some of the participants tried to reach objects by fully
extending their arms and click, this approach did not work well because it was hard to hit the
intended area. It worked better when they moved the object directly in front of them and
clicked.
The click gesture was however a bit easier to use when closing the charts and pictures. This
could be because the button used to close the charts and pictures was larger than the markers
on the map. The click gesture became easier to perform when the participants got used to it,
but they still thought that it was a bit cumbersome.
5.4.3 Swipe gesture
Task four and seven regarded the swipe gesture. The swipe gesture was used to pan the map
and move charts and pictures.
0
1
2
3
4
5
6
7
Stronglydissagree
Disaagree Agree Strongly Agree
Click gesture
Task3
Task5
30
Figure 5.5 Swipe gesture
According to the results from the questionnaire the swipe gesture worked well which also was
observed during the tests. However the disagree and strongly disagree results in figure 5.5
indicate that there might be some issues related to the usability. Also observations that suggest
this were made during the tests. Some of the participants had trouble moving the charts and
pictures in one relaxed swipe. Another issue that became apparent when one user wanted to
move a picture to the other side of the screen. He began the swipe motion and walked behind
the other user, which resulted in that the calibration was lost and he had to recalibrate. This
affected the experience of the swipe gesture.
Another drawback that was expressed by one participant was that he could not reach whole of
the screen from a stationary position. He could not either move to the other side because of
the screen without bumping in to or disturbing the other participant. He wanted to be able to
reach the whole screen without walking there.
Despite a few drawbacks the swipe motion worked well and was easy for most of the
participants to perform. However the negative answers from the questionnaire shows that it
needs improvement. Nevertheless this gesture became easier after getting used to it and was
also underlined by some participants.
5.4.4 Swim gesture
The swim gesture was tested in task two and six. In task two it was used to zoom the map; in
task six to scale charts and pictures.
0
1
2
3
4
5
6
7
8
Strongly dissagree Disaagree Agree Strongly Agree
Swipe Gesture
Task4
Task7q1
31
Figure 5.6 Swim gesture
Figure 5.6 clearly shows that there is some usability issues related to the swim gesture. The
participants thought that the swim gesture used to zoom the map was a bit easier than to scale
the charts and pictures. Zooming the map seemed easier when the participants both agreed
how and when to use it. But difficulties to accurately zoom a certain amount were observed.
Several participants zoomed too much unintentionally and had to zoom back out.
Nevertheless the scaling of charts and pictures actually seemed, by observation, to work better
than the zooming of the map. This could be because the scaling was smoother and more
continuous then the zooming. However it became difficult to scale the objects when they were
small. If the objects were scaled to big, they covered the other objects.
One issue observed was that when scaling charts and pictures it was easy to unintentionally
zoom the map instead. It was also hard to zoom the map while having charts and pictures in
the way. This could explain some of the negative answers from the questionnaire. One
participant requested a way to quickly hide the objects to access the map.
The swim gesture seemed to be the most difficult for the participants to get used to. Some
participants expressed that they thought it was hard to sync the movement with both hands.
Some also thought that it was easier to first ―grab‖ the object by selecting it with one hand
and then scale it with the other.
5.4.5 Collaboration
Task seven was to measure the collaboration between the participants. However in almost all
the other tasks collaboration was observed. Most of the participants agreed that it was easy to
interact with each other‘s charts and pictures and that it encourages collaboration. Still figure
5.7 indicate that there might be some issues related to the collaboration.
0
1
2
3
4
5
6
7
Strongly dissagree Disaagree Agree Strongly Agree
Swim Gesture
Task2
Task6
32
Figure 5.7 Collaboration
Collaboration was observed while zooming and panning the map, scaling and moving charts
and pictures, closing of charts and pictures and when finding places on the map. In some
cases one participant started to zoom then the other panned the map to the right location. Even
if this approach sometimes interfered with each other, it seemed to be a fast way to find places
on the map. However it also requires that the participants agree on what to do.
When there was more than one object on the screen, the participants often interacted with the
same objects. They discussed how to do certain things and were instructing each other. One
example of collaboration is when one participant wanted to give a picture to the other; he
swiped the picture towards the others hands and the other took it. However when they did this
a technical problem became prominent, when the second participant tried to move the picture
at the same time as the first they, instead of moving the picture, scaled it. This is because the
prototype does not distinguish between the hands of different users. This could be beneficial if
the users would like to scale an object together, otherwise frustrating.
When the participants were about to close a chart or picture they sometimes collaborated by
having one participant to move the picture so the other could close it. This approach was
effective when they had scaled the objects to a larger size because one of the users could
reach the close button before the other.
Even though the participants thought that this type of interaction encourages collaboration,
many of them complained about bumping in to each other while moving about. This could be
one reason to the negative answers from the questionnaire.
5.4.6 Summarizing impressions
The last three open ended questions in the questionnaire regarded the general impression of
the application, opinions about the collaboration and suggestions for improvement.
The general impression of the application was that it was ―really cool‖, ―interesting but
unpolished‖, ―a bit hard in the beginning but really fun and interesting‖. These statements
0
1
2
3
4
5
6
7
Stronglydissagree
Disaagree Agree Strongly Agree
Collaboration
Task7q1
Task7q2
33
were given in the questionnaire and showed an overall support. Other opinions from the think
out loud were noted during the test. These opinions often regarded that it was fun but also
quite difficult sometimes.
Overall the collaboration was according to the questionnaire fun and interesting. However one
participant stated that he did not find it very necessary but that it worked well. Some of the
participants stated that it they easily bumped into each other and that the gestures of one user
sometimes were interfering with the other users‘ gesture.
The last question gave the participants an opportunity to suggest improvement and to request
features. The conclusion was that the participants wanted more content to interact with, more
gestures, and gestures that were easier to perform. As the test result showed, the click and
swim gestures needs improvement. Several of the participants suggested that the click could
be performed by closing and opening the hand. This gesture could be more controlled and
give better accuracy. The same approach was suggested when moving objects or panning the
map.
One participant requested a functionality to hide all content to quickly view the map. She
thought that when having a lot of objects it became hard to access the map. The interaction of
the map was also a bit difficult when for instance one participant wanted to pan the map when
the other wanted to open a marker.
Throughout the tests several potentials of using freehand gestures could be observed by
looking at and listen to the participants while they were interacting with the prototype. For
example, the participants often worked together and helped each other with the tasks. How to
interact with the prototype seemed to be relatively easy to understand, even though actually
performing the interaction sometimes was not so easy.
The next and final chapter of this thesis will discuss these results in relation to the research
questions stated in chapter one.
34
6 Conclusion This chapter will present the results and conclusions from the other chapters in this thesis.
First there will be a discussion regarding the research questions, development of the prototype
and user tests. Secondly, suggestions for future research in the subject will be made. In the
final section of this chapter an overall reflection of the work will be presented.
6.1 Discussion
As stated in chapter one, this thesis aimed to research the possibility of using gestures to
support collaboration while interacting with visualized data. The main question asked: How
could the integration of interactive technologies and visualizations support collaboration
using gesture based controls utilizing large displays? To answer this question a set of features
and requirements were identified during a literature survey. These features and requirements
was the used to create a prototype. To assess the usability of the prototype and to identify
potentials in using gestural interaction but also see if it could support collaboration, five user
tests with ten participants were conducted.
During the user tests, the prototype itself was evaluated to in the future improve the user
experience and further develop the application. Also the collaboration and attitude towards
gestural interaction was observed and analyzed.
The participants were observed while interacting with the prototype and were given a
questionnaire with questions related to the interaction of the prototype. This combination of
methods gave valuable information about the prototypes interface and gestural interaction in
relation to collaboration. The overall mean value of 2.9125 out of 4 shows that the prototype
worked rather well but clearly had some issues. The conclusion was that the prototype alone
needs improvements in order to conform to good user experience. However the attitude
towards using such system in collaboration was in general good. Issues related to all of the
gestures were identified and these are discussed below.
As can be seen in figure 5.2 the participants had most trouble with the click and swim
gestures while the swipe and calibration gestures worked better. The click gesture did inflict
most problems for the participants. They often missed the intended hit area and thought it was
hard to perform the gesture accurately. The reason for this could be that it is hard to move the
hand in a perfectly straight line while extending it. Therefore another click approach could be
necessary which also was requested by some of the participants. One suggestion was to
perform the click gesture by hover the object and then closing the hand instead of extending
the arm.
The swim gesture also posed issues according to the test results. The gesture itself was
working rather well but the participants did not feel when the zooming or scaling began and
they often got surprised when map zoomed. The zooming of the map was less smooth and
continuous than the scaling of the objects which resulted in the participants not knowing when
the zoom began. Using the swim gesture to scale charts and pictures worked better but the
objects should have a minimum and maximum size. If the object gets scaled to a size smaller
than the touch points it could be hard to rescale it again.
35
The swipe gesture worked better than the other gestures. The reason for this was that it did not
require the same precision as the other gestures. However an alternate approach was
suggested by several participants. Instead of extending the arm one should grab it by hover
over the object then closing the hand. Another issue that was expressed by the participants
was that if they wanted to move an object to the other side of the screen, they had to walk
there and thus risk bumping in to each other. Therefore they suggested that the whole screen
should be reachable from a stationary position.
The gesture that posed least problem was the calibration gesture. This gesture should only be
needed initially and is required to get an accurate tracking with this current prototype.
Nevertheless the calibration is an interruption in the work and could make users not willing to
use the system. The calibration should be easier and more robust. Another aspect of the
calibration is that only users calibrated can interact with the content which means that
unintentional interaction is easily avoided.
To further answer the main question the collaborative aspects must be discussed. According
to the results from the user tests, gestural interaction on large screen could be beneficial when
collaborating with the same content. However this thesis does not measure the collaboration
between users, it merely tries to answer if this technology with gestures is suitable to support
collaboration. According to the results the participants were positive towards using the
prototype in collaboration even though the prototype itself was a bit limiting. Some
participants expressed however that they did not see the point of having multiple users
interacting with the same content when they could discuss it in front of the screen. The reason
for this could be that the participants did not relate to the tasks at hand and therefore was not
motivated to collaborate.
Even though the technology needs some improvement, almost all of the participants were
positive to use such a system to collaboratively interact with visualizations. Many of them
also saw potential in this kind of interaction in this setting as well as other applications. One
potential advantage is that the users are active together while working with the content, it also
could support peer teaching between the users. This potential was observed during the tests
where the participants often helped each other to learn the system. However using a projector
screen limits the interaction to two users at the time. Compared to a table-top screen which
could let more than two users interact at the same time using a vertical screen could be
limiting. One drawback with using a table-top screen could be that it is harder to track
individual users. The comparison between a table-top and vertical screen should be
investigated in relation to the collaboration between the users in the future work.
The second question asked: What are the components of an interactive gesture based
visualization system? This question was answered in chapter four and shows that there is a
combination of different technologies that are needed in order to enable collaborative gestural
interaction and visualizations. In large there are two main components. The first is the user
input in form of computer vision with user tracking and gesture recognition software. The
second is the system output in form of interactive visualizations that makes use of an external
data source. The combination of these components proved to be crucial in order to create the
36
prototype. These components could be used to further develop the prototype or other
applications. Even though other approaches could be used, these components met the
requirements of the prototype.
The third question regarded the potentials of using gestural interaction and was formulated as
follows: Compared to traditional means of interacting with visualization systems, what are
the potentials of using gesture based interaction? In the literature survey it was stated and
discussed that using gestures to interact could in fact be more natural. This is however argued
to not be the case by some practitioners who mean that gestures are related to context and
culture and does not conform to everyone. Nevertheless some argue that the naturalness of
device free interaction combined with gestures is beneficial due to the less restricted
interaction. The potential of not using external devices could be observed in the user tests.
The users did not need to learn a new external device such as a remote but instead interacted
more like in real life with gestures. Therefore it was fairly easy to learn how to interact with
the content. However when the complexity of the application increases the might be need for
more gestures. This could have a negative effect on the usability and learnability of the
system. Therefore a limited set of gestures should be considered and the reuse of gestures for
similar interaction.
Another potential noted was that freehand gestures makes it possible to interact with
visualizations in other contexts than the traditional desktop computer. The visualizations
could therefore be used in environments where the desktop computer is not suitable, in this
case in front of a large screen together with another user. A potential in not using external
devices could also be that it causes less fatigue. Even if an external remote such as a remote
control is fairly light weight it still introduces some external stress to the hand.
Another important aspect when discussing the potentials is the user experience which was
considered during the tests. Even though the prototype have some flaws most of the
participants thought it was fun and interesting. If the interaction between the user and
visualizations are considered to be fun then the user could be more enthusiastic to view the
visualization. This could be beneficial to use for instance in a learning environment where
several users could be motivated to discuss visualized data in a more fun way. Although the
fun-effect might have been the result of introducing the participants to a new technology and
should therefore be evaluated for a longer period of time to determine the long term attitude
towards using freehand gestures.
It was stated in the literature survey that traditional means of interacting with visualizations
have been restricted to one user at the time using a rather small screen. In this case with a
projector screen, using gestures lets several users to interact with the visualizations at the
same time. This is a potential in more easily understanding complex data which also was
argued in the literature survey. Although the prototype did not make use of very complex
data, this potential could be seen when the participants for instance discussed the location on
the map. However with the current gestures and visualizations used in the prototype it could
be difficult to work with very complex data and visualizations. Complex visualizations with
many variables could be cumbersome to interact with using gestures because of the limited
37
gesture vocabulary and that it is difficult to give visual clues of how the interaction could be
done. The observations from the user tests suggest that more simple visualizations or
overview of large amount of data could work better. Large objects seem to be easier to
interact than smaller, such as links and small icons. Using gestures for very precise interaction
could rapidly cause fatigue due to for example having to pose in static position for a long
period of time. Even though the prototype was designed to conform to the basic principles of
ergonomics there were some indication during the tests that the prototype could cause fatigue
if used for a longer period of time. Observations from the tests suggest that gesture based
systems might not be suitable to replace traditional desktop systems; it should perhaps rather
complement the existing ones. Nevertheless the use of a system like this could have great
potential where many different persons work together. The multiuser computer vision allows
for any user to walk up and interact together with others without having to switch place with
each other in front of a desktop computer. Therefore the potential of supporting collaboration
between co-located users is perhaps the most important aspect when contrasting it to
traditional visualization systems. Also simplicity of using gestures to interact could make it
easier to learn and therefore make it available to other than expert users.
To summarize, this thesis has explored interactive technologies to enable gestural interaction
with visualizations. The components of the system have been identified and the potentials of
using gestural interaction have been discussed. By the results and discussion there can be
concluded that freehand gestures to interact with a computer system has potential but also
some drawbacks. The drawbacks were mainly related to the particular gestures used in the
prototype which needs further improvement. The potentials are related to the collaboration,
simplicity and making visualizations usable and more available in other contexts than the
desktop environment.
It is important to identify in which contexts it might be suitable to use gestures and
consequently not use it as a substitute for traditional input devices. There is still no gesture
vocabulary that can be adapted to the same extent as for instance the mouse. Therefore the
need for further research in the field is important. The next section will introduce some
possible future work related to this work.
6.2 Future work
Even though there has been much research in the area of gestural interaction with computers
the technology used in this thesis is rather unexplored and require further research. More
precise computer vision and gestures should be investigated in further work. The user tests of
the prototype presented in this work have shown a number of potential future improvements.
The participants often requested more and different gestures while interacting with the
visualizations. Below some aspects are discussed.
Gestures
Some participants during the user tests requested different and more gestures to use for
interaction. One request was to instead of making a click motion a grab motion where
the user would close the hand to grab or click an object. This should be investigated to
see if this approach could work better. Different sets of gestures should perhaps be
38
evaluated against each other in different contexts in order to find a vocabulary that
could work for many purposes.
Users
The possibility of making the system more user dependent where perhaps different
users could perform different actions or have different roles. In relation to this it
should also be investigated if there are benefits to be drawn from having access the
whole screen from a stationary position. Will the interaction of the content be
smoother or will it make the users interfere with each other more?
Functionality of the prototype
Suggestions for additional features for the prototype were gathered from the
questionnaires and opinions during the tests. One specific request was to have either a
gesture or a button to hide all content and only display the map. Requests were also
made for more objects to interact with, filtering of data, lock map option, and more
ways to interact with the content.
Collaboration
The collaboration between users using a system like this should be further investigated
Perhaps the collaboration in relation to vertical and table-top displays and how the
users could be individually tracked while using a table-top. Which approach should be
used in which situation?
6.3 Reflection
In conclusion this thesis has been very interesting but also challenging. The work has revealed
a great potential of using gesture based interaction but it has also shown that there are much
work to be done in the combination of data visualization and gestural interaction from a
distance. The most challenging part of this work was the development of the prototype. The
Microsoft Kinect was released in November 2010, short after it was ―hacked‖. This work
began in January 2011 and yet as of May 2011 no official SDK has been released for the
Microsoft Kinect. Therefore the combination of different libraries and frameworks has
sometimes been difficult to work with due to lack of documentation and many different
versions. However the choice of making a high-fidelity prototype resulted in having the
benefits of testing it with potential users in a fairly early stage.
The user testing proved to be a good way to gather valuable information, both about the
prototype itself but also about the overall attitude towards gestural interaction while
collaborating. User tests should also be used in future for evaluation of the application.
Using gestures to interact with different types of computers will certainly be a more common
part of our everyday life in a not too distant future. With the availability fairly accurate
computer vision at a low cost and an active internet community, the field of gestural
interaction will without a doubt be very interesting in the time to come.
39
References
ACM (2011), http://portal.acm.org/dl.cfm [2011-05-13]
Adobe Flex (2011) Adobe Systems Incorporated. Available
at: http://www.adobe.com/products/flex/ [2011-05-13]
Bellucci, A., Malizia, A., Diaz, P., & Aedo, I. (2010). Donʼt touch me: multi-user annotations
on a map in large display environments. Proceedings of the International Conference on
Advanced Visual Interfaces, 391-392.
Benko, H., & Wilson, A. D. (2008). DepthTouch : Using Depth-Sensing Camera to Enable
Freehand Interactions On and Above the Interactive Surface. IEEE Workshop on
Tabletops and Interactive Surfaces.
Centers for Medicare and Medicaid Services (CMS) (2008). Selecting a development
approach.
Cabral, M. C., Morimoto, C. H., & Zuffo, M. K. (2005). On the usability of gesture interfaces
in virtual reality environments. Proceedings of the 2005 Latin American conference on
Human-computer interaction - CLIHC ’05, 100-108. New York, New York, USA: ACM
Press.
Card, S.K. and Mackinlay, J.D. and Shneiderman, B. (1999). Readings in information
visualization: using vision to think. Morgan Kaufmann.
Cooper, A. (2007). About Face 3: The Essentials of Interaction Design. Indianapolis: Wiley
Publishing, Inc.
Dietz, P., & Leigh, D. (2001). DiamondTouch: A Multi-User Touch Technology. Proceedings
of the 14th annual ACM symposium on User interface software and technology, 3(2),
219-226.
Forlines, C., Wigdor, D., Shen, C., & Balakrishnan, R. (2007). Direct-touch vs. mouse input
for tabletop displays. Proceedings of the SIGCHI conference on Human factors in
computing systems - CHI ’07, 647. New York, New York, USA: ACM Press.
Giles, J. (2010). Inside the race to hack the Kinect. The New Scientist. Retrieved from
http://www.newscientist.com/article/dn19762-inside-the-race-to-hack-the-
kinect.html?full=true.
Hardenberg, C. von, & Bérard, F. (2001). Bare-hand human-computer interaction.
Proceedings of the 2001 workshop on Percetive user interfaces - PUI ’01, 1. New York,
New York, USA: ACM Press.
40
Isenberg, P., & Carpendale, S. (2007). Interactive tree comparison for co-located collaborative
information visualization. IEEE transactions on visualization and computer graphics,
13(6), 1232-9.
Isenberg, P., Fisher, D., Ringel, M., Kori, M., & Mary, I. (2010). An Exploratory Study of
Co-located Collaborative Visual Analytics Around a Tabletop Display. Visual Analytics
Science and Technology (VAST), 2010 IEEE Symposium on, 179-186.
Keim, D., Andrienko, G., Fekete, J.-daniel, & G, C. (2008). Visual Analytics : Definition ,
Process , and Challenges. Information Visualization, 154-175.
Keim, D. a. (2002). Information visualization and visual data mining. IEEE Transactions on
Visualization and Computer Graphics, 8(1), 1-8.
Kela, J., Korpipää, P., Mäntyjärvi, J., Kallio, S., Savino, G., Jozzo, L., et al. (2005).
Accelerometer-based gesture control for a design environment. Personal and Ubiquitous
Computing, 10(5), 285-299. doi: 10.1007/s00779-005-0033-8.
Lazar J, Feng JH, Hochheiser H (2010) Research methods in human-computer interaction.
Wiley, New York
Leyvand, T. and Meekhof, C. and Y.-C. W. and J. S. and B. G. (2011). Kinect Identity :
Technology and Experience. Computer, 44, 94-96.
Microsoft (2011). Available
at: http://www.xbox.com/sv-SE/kinect (2011-02-13)
Microsoft (2010). Available
at: http://www.xbox.com/en-US/Press/PressKits/2010-Kinect-Launch (2011-06-13)
Nielsen, M. and Storring, M. and Moeslund, T.B. and Granum, E. (2004). A procedure for
developing intuitive and ergonomic gesture interfaces for HCI. Gesture-Based
Communication in Human-Computer Interaction, 105--106.
Norman, D. A., Nielsen, J., & Group, N. N. (2010). Gestural Interfaces : A Step Backward In
Usability. interactions, 17(5), 46-49.
Norman, D. A. (2010). Natural User Interfaces Are Not Natural. interactions, 17(3), 6-10.
OpenNI User Guide (2011). Available
at: http://www.adobe.com/products/flex/ [2011-05-13]
Pavlovic, V.I. and Sharma, R. and Huang, T. S. (1997). Visual Interpretation of Hand
Gestures. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 19, 677—
695.
41
Raisamo, R. (1999). Multimodal Human-Computer Interaction: a constructive and empirical
study. Tampere. Tampereen Yliopisto.
Sharp, H., Rogers, Y., and Preece, J. (2007). Interaction
design: Beyond Human Computer Interaction. 2nd edition. Wiley.
Stewart, J., Bederson, B. B., & Druin, A. (1999). Single Display Groupware : A Model for
Co-present Collaboration. Proceedings of the SIGCHI conference on Human factors in
computing systems: the CHI is the limit, (May), 286-293.
Takeoka, Y. (2010). Z-touch : An Infrastructure for 3D gesture interaction in the proximity of
tabletop surfaces. Image (Rochester, N.Y.), 3-6.
Thomas J.J., Cook K.A, (2005). Illuminating the Path: The Research and Development
Agenda for Visual Analytics, IEEE CS Press,
Verweirder, W. (2011). as3osceleton: Using Kinect as a multitouch device in adobe AIR.
Retrieved from http://blog.aboutme.be/2011/03/07/as3osceleton-using-kinect-as-a-
multitouch-device-in-adobe-air/.
Vogel, B. (2011). An Interactive Web-based Visualization Tool: Design and Development
Cycles. Proceedings of the IEEE 35th Annual International Computer Software and
Applications Conference Workshops (COMPSAC 2011) to be held in Munich, Germany,
July 18-22, 2011
Vogel, D., & Balakrishnan, R. (2005). Distant freehand pointing and clicking on very large,
high resolution displays. Proceedings of the 18th annual ACM symposium on User
interface software and technology - UIST ’05, 33. New York, New York, USA: ACM
Press.
Wachs, J. P., Kölsch, M., Stern, H., & Edan, Y. (2011). Vision-based hand-gesture
applications. Communications of the ACM, 54(2), 60.
Wang, R. Y., & Popović, J. (2009). Real-time hand-tracking with a color glove. ACM
Transactions on Graphics, 28(3), 1.
Wilson, A. D. (2010). Using a Depth Camera as a Touch Sensor. ACM International
Conference on Interactive Tabletops and Surfaces, 69-72.
Yin Y, Davis, R. (2010). Toward natural interaction in the real world: real-time gesture
recognition. In International Conference on Multimodal Interfaces and the Workshop on
Machine Learning for Multimodal Interaction (ICMI-MLMI '10). ACM, New York, NY,
USA, , Article 15 , 8 pages.
42
Zudilova-Seinstra, E. and Adriaansen, T. and Liere, R. van. (2009). Trends in interactive
visualization: State-of-the-art survey. Springer-Verlag New York Inc.
43
Appendix A
User manual This is a short description of how the application works and how you interact with it. The application is comprised of a map, charts and pictures and supports two users at the time.
Calibration The first thing before you can start using the application is a calibration. Both users must calibrate. How to:
Stand in front of the camera, raise your hands in level with your head and lean slightly backwards.
When calibration is done, a set of pointers(hands) for each user will be displayed on the screen.
The map The map can be interacted with in three ways: 1. Zoom
2. Pan
3. Click a marker
The charts and pictures The charts and pictures can be interacted with in three ways: 1. Move
2. Scale
3. Close
How to interact There are three basic gestures for interacting with the application: Click, Swipe and swim. Below these are described in relation to their functions. Click The click gesture is the most basic gesture and is used to open markers on the map and close charts and pictures. How to:
Hold your hands in front of you, close to the body. Move one of your hands closer to the screen in a “click motion”.
Swipe The swipe motion is used to pan the map and move charts and pictures. How to:
Hold your hands in front of you, close to the body. Move your hand closer to the screen. When the arm is extended towards the screen, move the arm in the direction you
want the object to go.
44
Swim The swim gesture is used for zooming the map and scaling of charts and pictures. How to:
Hold your hands in front of you, close to the body. Move both hands towards the screen. When arms are extended, move the hands closer or further from each other.
45
Appendix B
QUESTIONNAIRE RELATED TO TASKS
Usability Study
Dear participant,
Please read carefully the statements below before selecting your answer. The questions are related to
the tasks and will help me to assess the usefulness of the application. Thank you for your assistance.
Best regards.
Andreas Simonsson Huck
The values of your answers are based on the scale presented in the table below:
Strongly Disagree Disagree Agree Strongly Agree
1 2 4 5
My age is ____ years old and
I am
Female
Male
TASK 1
1. The calibration process was easy to understand.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 2
2. Performing the swim gesture to zoom the map was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 3
3. Performing the click gesture to open a marker was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 4
4. Performing the swipe gesture to move charts and pictures was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 5
46
5. Performing the click gesture to close charts and pictures was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 6
6. Scaling pictures by performing the swim gesture was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
TASK 7
7. Interacting with each other’s charts and pictures was easy.
Strongly Disagree
Disagree
Agree
Strongly Agree
8. Performing task 7 encourage collaboration.
Strongly Disagree
Disagree
Agree
Strongly Agree
GENERAL QUESTIONS
9. General impression of the application.
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
10. How did you find collaborating using this application?
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
11. Other comments and suggestions for improvement. (Such as added functionality, gestures etc.)
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
…………………………………………………………………..
