C hap te r 1
Getting Started with
Augmented reality (AR), in which virtual content is seamlessly
integrated with displays of real-world scenes, is an exciting area of interactive design.
With the rise of personal mobile devices capable of producing interesting AR environ-
ments, the vast potential of AR has begun to be explored. The goal of this chapter is to
help you become familiar with the terminology, tools, and technology necessary to begin
your own AR explorations and experiments.
In this chapter, you’ll learn about the following topics:
What is augmented reality?■■
Tools and technologies■■
036631c01.indd 1 9/1/11 5:33:25 PM
2■ ■ Chapter 1 : Getting Started with Augmented Reality
What Is Augmented Reality?
The term augmented reality (AR) is used to describe a combination of technologies
that enable real-time mixing of computer-generated content with live video displays.
Traditionally, it is distinguished from virtual reality (VR) in that VR involves creating
complete, immersive 3D environments, and AR uses various hardware technologies to
create an annotated, or “augmented,” composite based on the real world.
There are several ways that the virtual components and real content can be made to
interact. Techniques from image processing and computer vision can be used to make the
computer-generated elements interact with the content of the video in a convincing way.
Most current computer vision–based methods rely on predefined physical markers to
enable the computer vision system to get its
bearings in the visible 3D space. In Figure 1.1,
you can see an example of AR in which two
3D models are manipulated by use of printed
markers. AR systems that do not require
specially made markers, known as marker-
less systems, are also possible. Markerless
AR is a steadily progressing area of research,
and in the near future, robust visual AR sys-
tems that do not require markers will surely
become widely available.
Nonvision-based AR methods are gain-
ing in popularity on smartphone platforms.
These methods use a combination of the
device’s global positioning system (GPS)
or other location-tracking data and accelerometer data to determine where the device
is located and in what direction it is pointing. Based on this information, tags and
annotations are superimposed over the scene. These methods are the basis of several
geographical annotation services such as Layar, which is annotated for locations in the
Netherlands; mixare, an open source mix augmented reality engine, which currently
has data for Switzerland; and the Wikitude World Browser, which enables users from
around the world to contribute localized data.
This book is primarily concerned with computer vision–based AR. You’ll learn how
to incorporate computer-generated 3D content into live video using physical markers.
Getting into the details of location- and accelerometer-based AR is beyond the scope of
this book. However, I will mention relevant links and references to these technologies
where they are pertinent to the topic, such as in Chapter 10, “Setting Up NyARToolkit
036631c01.indd 2 9/1/11 5:33:25 PM
What Is Augmented Reality? ■ 3
A Brief History of AR
AR technology has its roots in computer interface research in the early days of computer
science. Many of the main concepts of AR have been familiar in movies and science fic-
tion at least as far back as movies like The Terminator (1984) and RoboCop (1987). Both
movies feature cyborg characters whose views of the world are augmented by a steady
stream of annotations and graphical overlays in their vision systems.
Practical systems using AR as it’s currently viewed began to be developed in the next
decade. The term augmented reality was coined in 1990 by Tom Caudell, a researcher at
The Boeing Company. Throughout the early and mid 1990s, Caudell and his colleagues
at Boeing worked on developing head-mounted display systems to enable engineers to
assemble complex wire bundles using digital, AR diagrams superimposed on a board over
which the wiring would be arranged. Because they made the wiring diagrams virtual, the
previous system of using numerous unwieldy printed boards was greatly simplified.
Throughout the 1990s, industrial and military AR applications continued to be devel-
oped. But the technical requirements for useful AR displays, such as bulky, expensive
head-mounted display devices, kept the technology out of reach for most users.
There were experiments incorporating AR technologies with the arts. Julie Martin,
wife and collaborator of art technology pioneer Billy Klüver, is credited with producing
the first AR theater production in 1994. The work, titled Dancing in Cyberspace, uses
dancers who interact with virtual content projected onto the stage.
At the end of the 1990s, another major development in AR came about when Hirokazu
Kato created ARToolKit, a powerful library of tools for creating AR applications.
ARToolKit helped to make AR accessible to a much wider audience of designers and
developers, and it provided the basis of much of the technology addressed in this book.
AR in Practice
In spite of the availability of ARToolKit, potential applications of AR have been slow to be
explored fully. One big reason for this has been the demanding hardware requirements
for achieving the benefits of an AR interface. The user must be looking at a display that
has access both to camera data and to data from a computer processor capable of running
the AR application. This setup was not standard even for consumer desktop environments
just a few years ago, and it was almost unheard of in consumer mobile environments until
the rise of smartphones, such as the iPhone and the Android family of phones. These
smartphones, and the burgeoning tablet computer market that has followed them, have
helped usher in a new era of interest in AR programming. With smartphones and tablets,
users have an integrated camera and computer at their fingertips at all times, opening up
many interesting new possibilities for AR applications.
036631c01.indd 3 9/1/11 5:33:25 PM
4■ ■ Chapter 1 : Getting Started with Augmented Reality
Whether by using mobile apps or through browser interfaces on computers with web-
cams, it has become easier than ever to give users an engaging AR experience on their
own devices, and the possibilities don’t end there. AR marketing campaigns have been
used by Hallmark, Adidas, and many other companies. In 2010, toymaker Lego created
a pioneering point-of-sale AR marketing campaign. Computer screens were set up at toy
stores carrying Lego toys and shoppers could hold up a box in front of the screen to see a
3D virtual representation of the completed Lego model in the video. In addition to being
an incredibly effective point-of-sale campaign, the campaign generated a great deal of
attention online, as shoppers posted their own videos of the experience. As discussed in
Chapter 7, “Interacting with the Physical World,” exploring alternate interfaces through
physical computing can expand the possibilities even further.
Several trends are leading to an explosion of interest in AR. One is the steady improve-
ment of computer vision technology, which will enable developers to work with more sub-
tle, less obtrusive, and more robust markers and even completely markerless AR. Another
trend is the rapid advance in display technologies for enabling AR. These include head-
mounted displays and projector-based displays, as well as handheld mobile devices.
Head-mounted displays are essentially a combination of a camera and display screens
that are worn like goggles or glasses. The viewing area may be a fully digital, opaque
video screen that displays video from the camera, or it may be transparent in a way that
allows the user to see the world directly with data laid over it. Head-mounted displays are
ideally suited to AR applications and are becoming lighter, less obtrusive, and less expen-
sive. In the long term, as wearable computers grow in popularity, head-mounted displays
are likely to become increasingly commonplace.
Perhaps even more interesting are recent experiments in projector-based AR displays.
Projector-based displays use projectors to project images or text directly onto surfaces
in the real world. This approach can be extremely versatile in suitable environments.
Projector-based displays can be used to present AR environments to large groups of
people at once in ways that head-mounted displays or mobile devices cannot. Small,
wearable projectors can also be used to create personal AR environments.
Other display methods may be available depending on specific applicati