TheKidsRoom - MIT Media Labintille/papers-files/kidsroom.pdfspaces began with Krueger’s Videoplace system (Krue-ger, 1993). Krueger designed installations that explored many different

Aaron F. BobickStephen S. IntilleJames W. DavisFreedom BairdClaudio S. PinhanezLee W. CampbellYuri A. IvanovArjan SchutteAndrew WilsonThe MIT Media Laboratory20 Ames StreetCambridge, MA [email protected]

The KidsRoom:A Perceptually-Based Interactive and

Immersive Story Environment

Abstract

The KidsRoom is a perceptually-based, interactive, narrative playspace for children.Images, music, narration, light, and sound effects are used to transform a normal child’sbedroom into a fantasy land where children are guided through a reactive adventurestory. The fully automated system was designed with the following goals: (1) to keepthe focus of user action and interaction in the physical and not virtual space; (2) topermit multiple, collaborating people to simultaneously engage in an interactive expe-rience combining both real and virtual objects; (3) to use computer-vision algorithmsto identify activity in the space without requiring the participants to wear any specialclothing or devices; (4) to use narrative to constrain the perceptual recognition, and touse perceptual recognition to allow participants to drive the narrative; and (5) to cre-ate a truly immersive and interactive room environment.

We believe the KidsRoom is the first multi-person, fully-automated, interactive,narrative environment ever constructed using non-encumbering sensors. This paperdescribes the KidsRoom, the technology that makes it work, and the issues that wereraised during the system’s development.1

1 Motivation and Background

1.1 Introduction

We are investigating the technologies that are required to build perceptu-ally-based interactive and immersive spaces that respond to people’s actions in areal, physical space by augmenting the environment with graphics, video,sound effects, light, music, and narration.

Using computer vision-based action recognition and other non-encumberingsensing technologies to interpret what people are doing in a space, a room canautomatically provide entertaining feedback in a natural way by manipulatingthe physical environment. For example, a kitchen might use audio and video toguide its occupants through the preparation of a recipe; a cafe might observehow people are interacting and change lighting, video, and music to enliven theatmosphere; and a child’s bedroom might stimulate a child’s imagination byusing images and sound to transform itself into a fantasy world.

In this paper, we detail our experience constructing and testing the Kids-Room, a fully-automated, interactive, narrative playspace for children. The

Presence, Vol. 8, No. 4, August 1999, 369–393

r 1999 by the Massachusetts Institute of Technology

1. A demonstration of the project, which complements the material presented here and includesvideos, images, and sounds from each part of the story is available athttp://vismod.www.media.mit.edu/vismod/demos/kidsroom.

Bobick et al. 369

space theatrically resembles a children’s bedroom, com-plete with furniture including a movable bed. Undercomputer control and in response to the children’s ac-tions, the room uses two large back-projected videoscreens, four speakers, theatrical lighting, three videocameras, and a microphone to carry the childrenthrough a story. The KidsRoom experience was designedprimarily for children six through ten years old and laststen to twelve minutes, depending upon how the partici-pants act in the room. Throughout the story, childreninteract with objects in the room, with one another, andwith virtual creatures projected onto the walls. The ac-tions and interactions of the children drive the narrativeaction forward. Most importantly, the children are awarethat the room is responsive.

The text is divided into three major sections, coveringthe motivation, implementation, and analysis of theproject. We have included details at all levels of designranging from broad considerations such as modeling thefocus of attention of the user, to technical issues such asvisual sensor integration, to the implication of imple-mentation details. Our goal is that the lessons we learnedfrom this project will be valuable to others constructingsuch perceptually-controlled, interactive spaces.

1.2 Project Goals

The initial goal of our group was the constructionof an environment that would demonstrate various com-puter-vision technologies for the automatic recognitionof action. As we developed the design criteria for thisproject, it became clear that this effort was going to bean exploration and experiment in the design of interac-tive spaces. The goals that shaped our choice of domainare described in the following sections.

1.2.1 Action in Physical Space. Because ourcomputer vision research focuses on the recognition ofactions performed by humans, we require that the usersbe engaged in activity taking place in the real physicalenvironment and not the virtual screen environment.Our goal was to augment a real space, stimulating theimagination using video, light, and sound, but not re-

placing the natural, real-world activity with which peopleare comfortable.

1.2.2 Vision-Based Remote Sensing. Remotesensing permits unencumbered activity in the physicalspace, not requiring users to wear sensors, head-mounted displays (HMDs), earphones, microphones, orspecially colored clothing. Also, computer-vision track-ing and action-recognition techniques allow a person toeasily and naturally enter and exit the room at any timewithout troublesome sensor requirements.2

1.2.3 Multiple People. ‘‘Interactive’’ entertain-ment spaces are more engaging, social, and fun whenone can play as part of a group. However, previous workin computer vision and fully automated interactivespaces has primarily considered environments containingonly one or at most two people. Our goal is to allowmultiple people in the environment, interacting not onlywith the environment but also with each other. If unen-cumbered by HMDs, people will naturally communicatewith each other about the experience as it takes place,and they will watch and mimic one another’s behavior.

1.2.4 Use of Context. One of our research top-ics is the use of context to increase the reliability of vi-sion-based sensing. Our goal is to build a system that isnot only aware of the context of the situation (e.g., thecurrent position in a storyline) but that also manipulatescontext by controlling much of the environment.

1.2.5 Presence, Engagement, andImagination. We want an environment that is trulyimmersive, is perceptually and cognitively engaging, anddoesn’t require participants to ask for outside help. Fur-thermore, we want to create a narrative experience inwhich the story and action of the people with respect to

2. Some of the ‘‘direct sensing’’ tasks could be implemented usingother devices such as microswitches to detect the presence of a personon a bed. Because our focus is on computer vision, we did not employsuch devices. However, even if a room is densely wired with sensingdevices, the difficult problem of understanding what is happening inthe space still needs to be addressed, and that is a fundamental compo-nent of our research on understanding action in the vision domain(Bobick, 1997).

370 PRESENCE: VOLUME 8, NUMBER 4

the story are the primary focus. The experience shouldengage each user’s imagination much like a children’sstory book; it is not necessarily required or desirable toprovide a complete fanciful rendition of a virtual world.The experience should be compelling in the sense thatthe users should be more concerned with their own ac-tions and behaviors than with how the interactive systemworks.

1.2.6 Children. We want the environment to betailored to children. Davenport and Friedlander (1995)and Druin and Perlin (1994) observed that adults visit-ing their interactive installations sometimes had diffi-culty immersing themselves in the narrative. Childrenalready like to play with each other in real spaces en-hanced by imaginary constructs (e.g., couches as caves)and can be easily motivated by supplementary imagery,sound, and lighting.3

The final design of the KidsRoom was intended toachieve each of these goals. The idea of a children’s play-space immediately addresses most of the concerns: alarge encompassing room with multiple children beingactive in an engaging activity. The goal of exploiting andcontrolling context was accommodated by embeddingthe experience in a narrative environment with a naturalstoryline to drive the situation.

1.3 Interaction in AugmentedEnvironments

Bederson and Druin (1995) classify work on com-puter interactive interface systems into those that focuson building interfaces in which information is superim-posed on the physical world and those that embed infor-mation into the physical world itself. The majority ofresearch in the human-computer interface and computergraphics fields has focused on systems of the first form,in which a user must wear gear such as glove sensors,specially colored clothing, or microphones. The Kids-

Room is an example of systems of the second form, inwhich the computer interface becomes unobtrusivelyembedded in the physical world itself. Such systems havebeen alternately termed ‘‘augmented environments,’’‘‘immersive environments,’’ ‘‘intelligent rooms’’ (Tor-rance, 1995), ‘‘smart rooms’’ (Pentland, 1996), and‘‘interactive spaces.’’

One early example of augmenting a physical space wasthe ‘‘media room’’ project of Bolt and Negroponte(Bolt, 1984). Their system allowed a user sitting in achair, ostensibly in his or her future living room, to in-teract with a screen by pointing and talking. Their goal,as is ours, was to augment spaces that we are comfort-able with, but the technology available at that time re-quired the use of body gear for sensing gesture andspeech. Even today, most work in augmented environ-ments requires cumbersome sensing and HMD gear(Azuma, 1997).

Research on physical, remotely sensed, interactivespaces began with Krueger’s Videoplace system (Krue-ger, 1993). Krueger designed installations that exploredmany different modes of interaction, most of which en-tailed large body gesture. In one example, the user inter-acted with his or her own silhouette on video screens.

The ALIVE project improved on Krueger’s system byreplacing special blue-screening hardware with com-puter-vision algorithms that can track the position andgestures of a single person moving in front of an arbi-trarily complex, static background (Maes, Pentland,Blumberg, Darrell, Brown, & Yoon, 1994). A singleuser can interact with virtual creatures by watching his orher own image superimposed with behavior-based crea-tures on a large video wall. The user must orient towardsthe video wall to observe the interesting action.

The ‘‘intelligent room’’ system consists of severalcameras and two large screens in a small room (Tor-rance, 1995). A single person is tracked using computervision, and simple pointing gestures are recovered. Userscan utter a lexicon of approximately 25 sentences into alapel microphone. The computer will understand in-structions such as ‘‘Computer, what is the weatherhere?’’ and will use the city that the person is pointing toon one of two large screens to retrieve weather forecast-ing information that is then displayed on the other

3. We also note that children are more forgiving of the smallglitches in timing, animations, and recognition certain to be present insuch an experimental facility. Our hope is that children will focus moreon having fun in the space than on figuring out how it works or how tobreak it.

Bobick et al. 371

screen. The room is controlled by a distributed agent-based architecture (Coen, 1997). The goal of the projectis to remove the computer from the human-computerinterface.

An alternative approach to the design of interactivespaces is to mediate computer interaction through themanipulation of real, physical objects. Druin, for in-stance, constructed a stuffed-animal doll named ‘‘Noo-bie’’ (Druin, 1988). Children interacted with Noobie bysqueezing the doll’s limbs and watching a display em-bedded in its belly. Instead of bringing children to thevirtual space on the screen and forcing interaction withspecial devices, the interface was brought into the worldof the children and embedded into devices with whichthey were already comfortable. (Also see Glos andUmaschi (1997).)

Druin and Perlin set out to construct immersive physi-cal environments for adults that responded to movementand touch in real physical spaces (Druin & Perlin,1994). They supplemented a real environment with asimple narrative and, in 1993, debuted an interactiveinstallation with three stories: one humorous story aboutbaby sitting, another more serious narrative aboutheaven and hell, and a final murder-mystery scenario.The system used computer control of lighting, sound,video, and physical props and sensors embedded in ob-jects. Interestingly, Druin and Perlin comment thatsome adults were confused with the whole idea of animmersive experience. One participant commented, ‘‘Ididn’t think I should touch anything. You know, Momalways said, ‘Do not touch!’ ’’

Narrative in interactive physical spaces was furtherexplored by Davenport and Friedlander (1995). Theywanted people to ‘‘feel as though they were walkingthrough a computer monitor into a magic landscape.’’They constructed a four-world, human-controlled instal-lation in which the narrative was ‘‘actualized by thetransformative actions of the visitor moving through it.’’The room used light, sound, video, and computer dis-plays. Each person in the space had a human guide, an-other ‘‘user’’ of the system, outside the space communi-cating with him or her via computers.

A variety of artistic experiments have been undertakeninvolving computerized spaces. A review of this work is

beyond the scope of this paper, but references and acritical analysis of some of the experiments are available(Lovejoy, 1989; Popper, 1993). A notable installation isMasaki Fujihata’s Beyond Pages, featuring a virtual bookwhose illustrations of objects such as a lamp and doorreact to the user’s gestures. Artists Christa Sommerer,Laurent Mignonneau, and Naoko Tosa have integratedcomputer vision, computer graphics, and emotion- andspeech-recognition techniques (Sommerer & Mignon-neau, 1997; Tosa, Hashimoto, Sezaki, Kunii, Yamada,Sabe, Nishino, Harashima, & Harashima, 1995).

Bederson and Druin believe, as do we, that the bestimmersive physical environments will have multimodalinputs and outputs (Bederson & Druin, 1995). Increas-ing computational speed is now making it possible toexplore domains like immersive office environments(Weiser, 1993; Ishii & Ullmer, 1997), living spaces, andtheater performances (Pinhanez & Bobick, 1998). Thetwo major obstacles to building fully automated reactivespaces are (1) finding practical, computationally feasiblesensing modalities that can be used to understand a vari-ety of different types of human action and interaction,and (2) developing a computationally feasible controlmechanism and intercommunication architecture forcoordinating perceptual input, narrative control, andperceptual output systems. Both these goals require thatwe further our understanding of how we represent ac-tions, interaction, time, and story.

1.4 Recognition of Action

Most virtual reality systems map a given configura-tion of the sensor outputs directly to some system re-sponse. For instance, in the ALIVE system, the positionof a person’s hands and head are estimated using com-puter vision, and the relative position of these objects isused to determine if a person is making a gesture (Wren,Azarbayejani, Darrell, & Pentland, 1997).

The KidsRoom moves beyond just measurement ofposition towards recognition of action using measure-ment and context. Although many of the mechanismsare simple, the KidsRoom combines the sensor outputswith contextual information provided by the story torecognize more than a dozen simple individual and


group actions in specific contexts. Examples includemoving through a forest in a group, rowing a boat, anddancing with a monster. These recognized actions drivethe story and control the narrative.

Throughout this paper, we will lump all types of ac-tion recognition together. However, within the com-puter-vision community, researchers are developing ataxonomy of action based on the computational repre-sentations and methods that are required to understandeach action type, e.g., a taxonomy of movement, activity,and action (Bobick, 1997). Many actions of interest re-quire that contextual knowledge be used for recognitionin addition to sensed motion and position information.In simple contexts, direct measurement of body positioncan sometimes be used to recognize activity. However,as the complexity of an environment increases, manydifferent measurements may correspond to the sameactions, or, depending on the context, the same mea-surement may correspond to different actions. Strongercontextual constraints are required to extract action la-bels from perceptual measurements. The environment ofthe KidsRoom is rich enough to begin to explore somecontext-sensitive recognition tasks.

2 Implementation and Experience

2.1 The Playspace

The KidsRoom theatrically re-creates a child’s bed-room. The space is 24 by 18 feet with a wire-grid ceiling27 feet high. Two of the bedroom walls resemble realwalls of a child’s room, complete with real furniture anddecoration. The other two walls are large video projec-tion screens, where images are back-projected from out-side of the room. Behind the screens is a computer clus-ter with six machines that automatically control theroom. Computer-controlled, theatrical colored lights onthe ceiling illuminate the space. Four speakers, one oneach wall, project sound effects and music into the space.Finally, four video cameras and one microphone are in-stalled. Figure 1 shows a view of the complete Kids-Room installation.

The room contains several pieces of real furniturewhich are used throughout the story and include a mov-

able bed. Because the other furniture is not explicitlytracked by the computer-vision system, they are sealedshut and fastened to the ground. Four colored rugs withanimal drawings and simulated stone markers on thefloor are used as reference points during the narrative.Colored cinder blocks on the floor prevent enthusiasticchildren from pushing the bed through the screens. Fig-ure 2 shows the layout of the room’s interior.

Figure 3 shows the four camera views. Camera one isthe top view, which is used for tracking people and forsome motion detection. Cameras two and three are usedto recognize body movements when children are stand-ing on the green and red rugs, respectively. Finally, cam-era four provides a spectator view of most of the roomand the two projection screens; this view is displayed tospectators outside the space and provides video docu-mentation. In addition to the visual input, a single mi-crophone is in the space that is used to detect the loud-ness of shouts.

The room has five types of output for motivating par-ticipants: video, music, recorded voice narration, soundeffects, and lighting. Still-frame video animation is pro-jected on the two walls. Voices of the narrator and mon-

Figure 1. The KidsRoom is a 24 ft. by 18 ft. space constructed in our

lab. Two walls resemble the walls in a real children’s room, complete

with posters and windows. The other two walls are large back-projection

screens. Computer-controlled lighting sits on a grid suspended above the

space. The door to the space, where all room participants enter and

exit, is pictured in the leftmost corner of the room.

Bobick et al. 373

sters, as well as other sound effects, are directionally con-trolled using the four speakers and so appear to comefrom particular regions of the room. Some sound effects,such as monster growls and boat crashes, are particularlyloud and can vibrate the floor, providing visceral input.As we discuss later, lighting must remain constant whenthe vision algorithms are operating; however, since thestory can be used to determine when vision is and is notrequired, it is possible to use lighting changes and col-ored lighting to mark important transitions.

Six computers power the KidsRoom. One SGI IndyR5000 workstation is used for tracking people and thebed, playing sound effects, and MIDI control of lightoutput. A second SGI Indy R5000 workstation is usedfor action recognition from cameras one and two, send-ing MIDI music commands to the music computer, andamplitude audio detection. A third SGI Indy worksta-tion is used for action recognition from camera three.Two DEC AlphaStations are used for displaying still-frame animations, one per screen. One of the AlphaSta-tions also runs the room’s control process. A Macintoshis used for running Studio Pro MIDI software con-

nected to a Korg 5R/W synthesizer.4 Finally, assortedvideo, lighting, and sound equipment are required tocomplete the installation.5

2.2 The Story

The KidsRoom guides children through an inter-active, imaginative adventure. Inspired by famous chil-dren’s stories in which children are transported fromtheir bedrooms to magical places (e.g., Barrie (1988);Walt Disney Productions (1971); and Sendak (1988)),the story begins in a child’s bedroom and progressesthrough three other worlds. We will describe the last

4. We used the hardware we had available, but current PCsequipped with video digitizers would suffice.

5. Includes two high-resolution video projectors and wall-sizedscreens, four Sony HandyCam color video cameras, four speakers and atwelve-channel, four-output mixer and amplifier, one microphone,fourteen lights (eleven white, three colored), a MIDI-based lightboard controller, and video distribution amplifiers.

Figure 2. The KidsRoom is furnished like a real children’s room,

complete with furniture, decorations, and a movable bed. Rugs and

stone-like markers are used throughout the narrative. Four speakers

project sound into the space. Colored cinder blocks at the base of the

large projection screens protect the screens (which comprise the right

and bottom walls in this image) from the movable bed. The square on

the floor in the bottom left marks the ‘‘door,’’ through which people

enter and exit the space.

Figure 3. Three cameras overlooking the KidsRoom are used for

computer vision analysis of the scene. Camera 1 is used for tracking the

people and the bed in the space and also for detecting motion during

particular parts of the story. Cameras 2 and 3 are used to recognize

actions performed by people standing on the red and green rugs.

Camera 4 is used to provide a view of most of the room and the screens

for spectators outside the space.


world in detail, to give the reader a feel for the story, itscharacters, and the interactive responsiveness of the en-tire system.

We note there that the primary story is a traditionallinear narrative, as opposed to the (typically weakly)nonlinear branching storylines found in many multime-dia presentations. Individual responses made by theroom are reactive and in that sense nonlinear. As we willdiscuss in the analysis section, we needed a strong narra-tive to motivate group behavior and to provide sufficientcontext. We will argue that this linear structure in noway reduced the interactivity since the pacing and indi-vidual reactions of the room are completely determinedby the participants.

The only instruction given to the children prior toentering the room was that this was a magic room, butthat to transform the room they needed to learn themagic password. To learn the password they should try‘‘asking the furniture.’’

2.2.1 The Bedroom World. Children enter theKidsRoom one at a time through the ‘‘door’’ in one cor-ner of the room. The tracking algorithm attends to thisregion, checking for people entering and exiting thespace. Whimsical, curious music plays softly, and the pro-jection walls display scenes from a bedroom. When atleast one child approaches some piece of furniture (e.g.,the blue desk or the green frog rug), each of which has adistinct personality, the furniture speaks and a scavengerhunt for the magic word commences, as shown in Figure4a. For example, a clothes trunk says (with suitable ac-cent), ‘‘Aye, matey, I’m the pirate chest. I don’t know themagic word, but the frog on the rug might know.’’ Thechildren then run to the rug with the frog painted on it,and the frog rug seemingly speaks, sending them to yetanother piece of furniture. The system randomly selectsthe ordering so that the interaction is slightly differenteach time the system is run.

Even a situation as simple as the bedroom requireshandling contingencies. If the children get confused anddo not go to the correct place, the system will eventuallyrespond by having some furniture character call the chil-dren over, ‘‘Hey, over here! It’s me, the yellow shelf !’’

This game continues for a few iterations, usually with

running, screaming, laughing children. Eventually, onepiece of furniture knows the randomly chosen magicword (e.g., ‘‘skullduggery’’) and reveals it to the chil-dren. As soon as the password is disclosed, all of the fur-niture start chanting the word loudly, ensuring the chil-dren hear and remember it. A mother’s voice soonbreaks in, silencing the furniture voices, and telling thekids to stop making noise and to go to bed. When theydo, the lights drop down, and a spot on one wall is high-lighted, revealing the image of a stuffed monster doll.The monster starts blinking and speaks, asking the chil-dren to loudly yell the magic word to go on a big adven-ture: ‘‘On the count of three, yell the magic word: One,two, three, . . .’’

2.2.2 The Forest World. After the kids yell themagic word loudly, the room darkens and the transfor-mation occurs.6 Images on the projection walls graduallyfade to images of a cartoon fantasy forest land, and col-ored, flashing lights combined with mysterious musicplay during the transformation, as shown in Figure 5.Simultaneously, a grandfatherly-voiced narrator—thevoice and personality of the room—says, ‘‘Welcome to theKidsRoom. It’s not what it seems. What you might see hereare things dreamt in your dreams.’’ The narrator alwaysspeaks in couplets, enhancing the experience of beingimmersed in a children’s story book.

6. There is no speech-recognition capability, with only the volumeof sound being measured. In no run of the KidsRoom did the childrenever yell anything but the magic word, illustrating how a compellingnarrative will constrain behavior.

Figure 4. (a) Children are told only to ‘‘ask the furniture for the magic

word.’’ (b) In the forest world, children must hide behind the bed to stop

the loud growling of distant monsters.

Bobick et al. 375

Figure 5.

Figure 6.

Figure 8.

As the lights come up, the narrator tells the childrenthat they are in the ‘‘Forest Deep,’’ that monsters arenear, and that they must follow the path to the river. Astone path is marked on the floor of the room and thechildren quickly realize it’s the path they should follow.The room provides encouraging narration if they do notdo so and instructs them to stay in a group and to re-main on that path. If they deviate from the path, ‘‘hints’’are given to induce the behavior. (Hints are loudly whis-pered suggestions made in a soft female voice to provideadditional instruction when needed; their power will bediscussed later in the analysis section.)

As they traverse the path (moving around the roomseveral times), monsters are heard growling from afar.The narrator warns, ‘‘The magic bed is now a tree. Hidebehind where monsters cannot see.’’ When the childrenmove behind the bed, the monsters stop growling, andthe children continue on the path. (See Figure 4b.) Ifthey do not hide, the loud growling intensifies, and adifferent narration encourages the kids to get behind thebed. After a short walk, the children reach the riverworld.

2.2.3 The River World. As the narrator an-nounces, ‘‘You’ve certainly managed a glorious act.You’ve arrived at the river and you’re still intact,’’ imagesof a river dissolve onto the two screens. One view showsthe river progressing forward (Figure 6), and the otherview shows the sideways-moving riverbank. In the riverworld, the children are told the ‘‘magic bed’’ is now aboat. They are encouraged to push the bed to the center

of the room and ‘‘jump inside’’ by climbing on top.When the children start making rowing motions, theriver images start moving. If someone gets off the boat,a splashing sound is heard. The narrator then shouts a‘‘passenger overboard’’ couplet, and encourages thechild to get back on the bed.

Soon, log and rock obstacles appear in the path of theboat, as shown in Figure 6a. As instructed by the narra-tor, the children must engage in collaborative rowing(making rowing motions on the correct side of the bed)to avoid the obstacles. If they successfully navigate theobstacle, a female voice softly whispers hints such as‘‘nice job.’’ When they do not, they hear a loud crashingsound, often motivating the kids to physically play-act acrash as in Figure 6b, and they receive some whisperedhints about how to avoid the obstacles.7

Eventually, the image of a shore appears, and the chil-dren are instructed to ‘‘land the boat’’ by pushing thebed towards the ‘‘shoreline’’ on the screen. As they do,the system produces loud grinding noises as if the bed isbeing pushed onto the sand. Suddenly, the mother’svoice is heard in the distance again telling the children tostop moving furniture and to put the bed back wherethey found it. If necessary, additional hints are provideduntil the bed is returned to approximately its originalposition. At this point, the children have reached themonster world.

7. We learned by experience that children prefer to crash and typi-cally paddle towards the obstacles!

Figure 5. Lighting effects are used to mark special transitions in the story. Here colored lights flash, transitional music plays, and the screens

gradually fade from bedroom to forest as the narrator welcomes the children to the forest world. Graphics are simple storybook animations.

However, combined with music, narration, and lighting effects, the transformation captures the attention of people in the room and gives the room a

somewhat magical quality.

Figure 6. (a) The speed of the boat is controlled by how vigorously people on the bed are rowing. If everyone stops making motions, the boat

imagery will stop moving forward. Obstacles such as the log in this series of images approach the boat. They can be avoided by rowing strongly on

the appropriate side of the bed (i.e., if the log is on the left as shown, then row on the left-hand side of the bed). (b) Audio feedback such as loud

crashes and narration signal when obstacles have been hit or avoided. Crashes tend to evoke expressive responses from children and subsequently

more enthusiastic rowing. (c) A child and mother row the boat together.

Figure 8. The dance moves are taught to the children by the monsters using still-frame animation. The first sequence shows one monster doing

the spin move: ‘‘Put your arms out and spin around like a top.’’ The second sequence shows another monster doing the ‘Y’ pose: ‘‘Throw your arms up

and make a ‘Y.’ The third sequence shows ‘‘Flap your arms like a bird,’’ and the last sequence shows ‘‘Crouch down and touch your toes.’’

Bobick et al. 377

2.2.4 The Monster World Forest images dis-played on the two screens, tense forest music playing inthe background, and jungle sounds like twigs crackingand owls hooting announce the arrival to the monsterland. To give a feel for the interaction, we present thisworld as an annotated dialog. The narrator speaks in acomforting, deep, somewhat mischievous voice.

Narrator: ‘‘Welcome to Monster Land. It’s a great spot.Time to have fun, ready or not.’’

Monsters are heard growling softly, but cannot yet beseen. Children are often huddled on the bed waitingexpectantly. Suddenly, there are loud roars and the mon-sters appear on the two screens. The monsters, shown inFigure 7a and 7b, are larger than the children and have afriendly, goofy, cartoon look. As they continue to growl,the room speaks and suggests that if the kids yell, themonsters might be quiet. The kids, in unison, yell. If theshout is loud, the story continues. If not, the room re-sponds with encouragement.

Narrator: ‘‘Get those monsters back in line. Try thatshout one more time!’’

If the kids still do not scream, the room responds:

Narrator: ‘‘Well, I can’t say that was a very loud shout,but perhaps the monsters will figure it out.’’

Either way, the monsters stop growling and show sur-prise for a moment. Energetic music starts to play.

Narrator: ‘‘The monsters invite you to shimmy anddance. Go stand on a rug and you’ll get your chance.’’

When the children are on rugs (sometimes promptedmultiple times in different ways), the room continues.The vision algorithm used in this world requires there beonly one child on each rug to get a clear view of eachparticipant. Therefore, if the system detects multiplepeople on a rug, one of the monsters in the story re-sponds in his raspy voice:

Monster 2: ‘‘Hey, only one kid per rug please, so’s wecan see what’s goin’ on.’’

Throughout this section of the story, the system detectswhen children get off a rug, and the characters in thestory respond accordingly.

Monster 2: ‘‘Hey, be sure to stay on your rug there,thanks a lot.’’

When each rug has a single child on it, the monsters be-gin to teach the children four dance moves:

Monster 1: ‘‘Hey, I’m going to do a crouch. You watchme first, then you try it. To do it right, crouch down andtouch your toes.’’

The monster, represented using still-frame animation asillustrated in Figure 8 does the move, and then says,‘‘Your turn!’’ The system watches the children on two of

Figure 7. (a) A child dancing with a monster. (b) Children spinning during the monster dance.


the rugs.8 When a child is observed having done thecrouch move, the monsters respond with positive com-ments:

Monster 1: ‘‘Yo! Kid on the red rug, you dance like apro!’’

The monsters continue, teaching the children threeother moves: throwing the arms to make a ‘Y,’ a flappingmove with arms extended, and a spinning move alsowith arms out.

Once the children know the moves, the music changesand becomes more energetic.

Narrator: ‘‘Now that you know the monster moves, see ifthey can catch your grooves! The monsters will be able tocopy if you do the moves, but don’t be sloppy.’’

The system will now respond to any of the four movesmade by the children on the front two rugs. The mon-sters will mimic a move that a child performs. If the sys-tem has a high confidence that it knows which move thechild is performing, then the monster also offers spokenconfirmation such as ‘‘Awesome flaps.’’ If the childrenstop doing recognizable moves, the monster characterschoose moves themselves, and a whispered hint like,‘‘Try a flap, spin, ‘Y,’ or crouch’’ is heard.

This dance phase continues for a few minutes (Figure7b), then the music grows louder and faster:

Narrator: ‘‘Now it’s time to do your own dance. Let’s seeyou dance and boogie all around the room!’’

The monsters, yelling kudos such as ‘‘Feel the groove!’’and ‘‘Shake that funky thing!’’, do a variety of monstermoves, including some new ones. The music’s character,tempo, and volume cause the children to run around theroom and do new dance moves. Suddenly, the mother’s

voice is heard again, and all music and sound abruptlyend:

Mom: ‘‘I told you kids to go to bed, and I mean busi-ness.’’

If the children all get on the bed, the story moves on.However, if not, the mother character continues to en-courage the desired action.

Mom: ‘‘I’m not fooling around. Get on that bed. All ofyou!’’

As soon as everyone’s back on the bed, the monstersrespond to the end of the scene:

Monster 1: ‘‘Hey y’all, let’s get quiet, and next time youcome back, we’ll have a rockin’ good time.’’

The lights drop down and colored transition lighting isused. Transformational music plays, as the monsters saygoodbye, ‘‘Take it easy! Bye bye!’’ The forest fades backto the room, and as the lights slowly come up:

Narrator: ‘‘Thanks for our Monster Land visit with you.We’ve enjoyed this wild dream, and we trust you didtoo.’’

Exit music plays as the children leave the space.

2.3 Perceptual Technology

As discussed in Section 1.4, in the KidsRoom wemeasure the position and movement of multiple, inter-acting people and then use that data and contextual in-formation to recognize action using vision-based per-ceptual algorithms. This section briefly describes theperceptual methods used by the KidsRoom. Discussionof the difficulties we encountered with each method aredeferred until the analysis section.

2.3.1 Object Tracking. Most immersive envi-ronments will need to keep track of the positions ofpeople and objects in the space. In the KidsRoom, wetrack the positions of up to four people and the movablebed. Some worlds, like the bedroom world, use posi-tional information to know whether people are near cer-

8. The children on the blue and yellow rugs near the back of theroom are tracked, but their actions are not analyzed by the system,since performing recognition on those rugs would require additionalcameras and computers not at our disposal. Children are only on theserugs when more than two people are in the room, and, in those cases,the children are clearly aware of the interaction between the monstersand their playmates.

Bobick et al. 379

tain objects: the pieces of furniture speak only when aperson is near. The positional information is also used tokeep track of whether people are in a group and whetherthey are moving or not. Most importantly, position in-formation is used to create known contexts for othervision processes by ensuring that people are in expectedregions of the room.

The KidsRoom tracking algorithm uses the overheadcamera view of the space, which minimizes the possibil-ity of one object occluding another. Further, lighting isassumed to remain constant during the time that thetracker is running. Our room lighting is designed tominimize brightness variation across the scene, but, inpractice, an object’s observed color and brightness cansignificantly change as it moves about.

Background subtraction is used to segment objectsfrom the background, and the foreground pixels areclustered into two-dimensional blob regions. The algo-rithm then maps each person known to be in the roomwith a blob in the incoming image frame. When blobsmerge due to the proximity of two or more children, thesystem maps more than one person to a given blob. Thesystem uses color, velocity estimation, and size informa-tion to disambiguate the match when the blobs laterseparate. It is important for the algorithm to keep trackof how many people are in the room, which is achievedby having everyone in the room enter and exit through a‘‘door’’ region.9 The context-sensitive, nonrigid object-tracking method is fully described and tested elsewhere(Intille, Davis, & Bobick, 1997).

Figure 9 shows the image view used for tracking andthe output of the tracking system. Each person is repre-sented by the rectangle bounding his or her blob. Thebox in the lower left corner represents the ‘‘door’’ re-gion of the room, where people can enter and exit. Thetracking algorithm is not limited to four people, but theKidsRoom narrative was designed for a maximum offour participants.

2.3.2 Movement Detection. Earlier we madethe distinction between measuring movement and rec-ognizing action. A strongly constraining context, how-ever, can allow inference of action directly from move-ment. For example, in the river world, measurements ofmotion energy used in conjunction with contextualknowledge are employed to recognize the participants’rowing actions. The amount of motion on each side ofthe bed is used by the control program to decide if theboat is moving (i.e., passengers are ‘‘rowing’’) and if thepeople have avoided obstacles in the river by rowing vig-orously on the correct side of the boat.

The rowing-detection algorithm presumes thateveryone is ‘‘inside the boat’’ (all on the bed). Thenarrative encourages participants to establish andmaintain this context (e.g., ‘‘Tuck your hands andfeet right in. The hungry sharks are eager to sin.’’),and a simple vision algorithm based upon the sizeof the bed is used to confirm that the context is in effectachieved. When the blob size is about right, everyone isassumed ‘‘in the boat’’ and the bed orientation iscomputed.

Once the system knows that everyone is on the bedand knows how the bed is oriented, it can use a simpletest to check if there is more rowing on the left or rightside. The algorithm computes the pixel-by-pixel differ-ence between consecutive video frames. If someone

9. During scene transitions, the lighting varies and the vision systemis disabled. The story and timing of the narrative are designed suchthat nobody would exit during a transition and nobody ever did; a hu-man gatekeeper prevented people from entering during those times.When lighting stabilized, therefore, the system knew the number ofpeople in the room on the bed and could initialize the tracker accord-ingly.

Figure 9. The left image shows a view with three people in the room

from the overhead camera used for tracking. The right image shows the

output of the tracking system, which is described and evaluated

elsewhere (Intille et al., 1997). All three people and the bed are being

tracked as they move about. The box in the lower left denotes the

room’s door region, where all objects must enter and exit. The KidsRoom

tracks up to four people and the bed.


moves quickly, a large difference between frames is de-tected. The difference over a region is the rowing en-ergy, which is measured on each side of the bed andscaled by the number of people in the boat. The controlprogram then uses these energy measures to detectwhether or not people are rowing and on which sidemost of the rowing is occurring. Figure 10 shows theoutput of the system when a person is rowing on theleft, right, and both sides of the bed.

2.3.3 Action Recognition in the MonsterWorld. More-sophisticated motion analysis is used dur-ing the dance segment of the monster world to recog-nize the four actions of ‘‘making a ‘Y,’ ’’ crouching, flap-ping, and spinning. We chose these moves for severalreasons: they are fun, natural gestures for children;they are easy to describe and animate using still-frameanimation; they are easy to repeat in about the sameway each time; and they allow us to demonstrate a fewdifferent recognition techniques using computervision.

Each of the real-time approaches for recognition de-scribed below are run in parallel as the kids performdance moves. The vision system reports which moves itthinks are being performed, as well as its confidence inthat assignment. All of the vision processes use back-ground-subtracted images that contain only a silhouetteof the person. They also require a training phase prior torun-time when the action models are constructed.

2.3.3.1 General Dynamics. The first and simplesttechnique for detecting crouching uses the size of thebackground-difference blob. Once in the monsterworld, the ‘‘standing’’ blob shape for a person is initial-ized as soon as the person moves onto the rug. The blobshape, which is modeled using an ellipse matched to theblob data, is compared at each time with the ‘‘standing’’model. If the elongation of the blob reduces signifi-cantly, the algorithm will signal that a crouch has takenplace. Figure 11b shows a person’s image blob andthe ellipse model for standing and crouchingpositions.10

2.3.3.2 Pose Recognition. The second recognitiontechnique uses the shape of the person’s background-subtracted blob to identify when the person’s arms areraised in a ‘Y.’ Here, we use a pattern-recognition ap-proach to classify the background-subtracted images ofthe person. Moment-based shape-features (Hu, 1962)are computed from the blob images like those shown inFigure 11d and are statistically compared to a databaseof training examples of people making a ‘Y.’

2.3.3.3 Movement Recognition. The last techniqueused to recognize monster moves uses recognition of

10. For all moves, the control system ignores the move reported bythe vision system if the tracking has indicated the person is not on therug.

Figure 10. These images show the motion energy that is detected from the overhead camera

when a person is ‘‘rowing’’ as they sit on top of the bed. The ellipse represents the position and

orientation of the bed, extracted by the system; the colored pixels indicate where the system

detected motion. The left, middle, and right images show rowing on the left, right, and both sides

of the boat, respectively. The amount of movement at any time is compared with the maximum

movement detected so far to compute how vigorously people are rowing and on which side of the

bed.

Bobick et al. 381

motion templates (Davis & Bobick, 1997). In thismethod, successive video frames of the background-sub-tracted images of the people are temporally integrated toyield a ‘‘temporal template’’ of the action. Templates forthe flap and spin moves are shown in Figure 12. Thesetemplate descriptions represent the movement oversome time interval with a single vector-valued image.The range of duration of integration is determined bytraining examples of the actions. A statistical moment-based description of the action template is then used tomatch to a database of examples of the moves.

2.3.4 Event Detection. In addition to recogniz-ing large body motions of individuals, most immersiveenvironments need to be able to detect other events ifthey are to provide interesting, reactive feedback. Forexample, the KidsRoom uses the output of the tracker toanswer questions such as ‘‘Is everyone in a group?’’, ‘‘Iseveryone on the bed?’’, ‘‘Is everyone on the path?’’, ‘‘Iseveryone moving around the path or standing still?’’,

and ‘‘Is someone near a particular object?’’ The Kids-Room uses straightforward methods to compute an-swers to these questions. The ‘‘in-a-group’’ detectorreceives the position of each person from the visiontracker and validates that every person is within somepredetermined distance of another person.

2.4 Story Control Technologies

In addition to the perceptual input technology, theKidsRoom has a narrative control program, a lightingcontrol program, MIDI music control programs, andnetworking protocols.

2.4.1 Narrative Control. The narrative controlprogram of the KidsRoom queries the sensor programsfor information about what is happening in the room ata given time and then determines how the room re-sponds so that participants are guided through the nar-rative. For example, when someone enters the room, thesystem must start tracking the person and the controlprogram must immediately learn of the person’s pres-ence. Similarly, if everyone leaves the room, the storymust freeze at its current point instead of continuing onas if there were still participants. The main control pro-gram is an event loop, much like those in the game in-dustry and in commercial software like Director orMAX. The event loop continuously monitors the stateof the room, checking all inputs as often as possible.

Dealing with real-time, physical interaction requirescontrol structures that are more complex than those re-quired in the typical keyboard-mouse situation, sinceactions take some amount of time during which the stateof the actuating devices may change. The KidsRoomcontrol structure partially handles those problems byusing the notion of timers, and associating a timer witheach event interval. Example uses of timers are ensuringthat noncomplimentary sounds do not play simulta-neously, that background sounds appear continuous,and that narrations are spaced appropriately. The timingproblems we encountered will be discussed in Section3.2.6.

Figure 11. (a) A person performing a crouch move. (b) A person’s

background difference blob. Overlayed on top is an ellipse model of the

blob. The first image shows a person in the standing position. The

second shows the same person crouching. The difference in elongation of

the ellipse model is used to detect crouching movement. (c) A person

performing a ‘Y’ move. (d) The blob image used to detect the ‘Y’ move.

This binary image is matched to a set of models of ‘Y’ moves using

moment-based shape features.


2.4.2 Music and Sound Control. The Kids-Room has an original score written for this interactiveinstallation. The music consists of fifty short MIDI seg-ments, many of which can be concatenated to form mu-sical phrases that gradually increase in complexity. Theselection of musical segments, tempo, and volume is un-der computer control and is changed based upon theaction in the room and the progression of the story.Computer control of the music is such that the controlprogram can interrupt music abruptly or at the end of amusical phrase, depending upon the situation.

When the control program calls for a particular soundeffect, the sound file is streamed to a process that adjuststhe volume of the sound in the four speakers to localizethe sound in a region of the room specified by the con-trol program.

2.4.3 Lighting Control. The computer-visiontracking and recognition algorithms require that theroom be well lighted and that the lighting settings canbe reliably set prior to each run. Consequently, speciallighting is used only in transition segments during which

time the vision algorithms are disabled. Even this mod-est use of lighting effects enhances the ambience of theKidsRoom. The lighting is fully computer controlledusing a MIDI light board. Some of our recent efforts inusing automated vision systems in theater (Pinhanez andBobick, 1998) use a multicamera segmentation methodthat is invariant to lighting changes (Ivanov, Bobick, &Liu, 1998).

2.4.4 Animation Control. To capture the imagi-native flavor of a storybook and to prevent the videoeffects from dominating the attention of the children,the KidsRoom uses layered, still-frame, cartoon-like ani-mation sequences. The control program requests an ani-mation, like ‘‘blue-monster-crouch’’ for a particularscreen at a certain frame rate (usually about two framesper second), and several frames are streamed to the dis-play. A benefit of such storybook animation is that we donot need to tightly synchronize the motion of the ani-mated characters to that of the children, but, like a sto-rybook, the still-frame cartoons can convey rich charac-ter activity.

Figure 12. Two of the dance move actions are recognized using a motion-template

matching method (Davis & Bobick, 1997). The top left images show a person doing a

flap move. The system detects the flap move by matching motion models (which have

been computing using a database of example flap moves) to the motion template

shown. Similarly, the bottom images show a person doing the spin move and the

corresponding motion template. The top part of the blob is generated by the moving

arms. The bottom part is generated by shadows from the arms. In the KidsRoom,

shadows were incorporated into the models of the moves.

Bobick et al. 383

2.4.5 Process Control. The KidsRoom controlarchitecture is based on a client-server model. The con-trol program is the client that communicates with tenservers to receive information about the state of theroom and to control the output. The sensor servers arethe object-tracker server, the motion-detector server, thetwo action-recognition servers, and the scream-detectorserver. The output servers are the directional soundserver, the music server, the lights server, and the twodisplay servers.

Communication is achieved using the RPC protocol.The server architecture has proven effective for allowingdifferent individuals to work on different components ofthe system using the computer system most appropriatefor the particular task. As noted by Coen (1997), it iscritical for any large, distributed, room-control mecha-nism that individual components can be stopped andstarted without requiring a reboot of the entire system.

3 Evaluation and Analysis

In the remainder of this paper, we evaluate theKidsRoom with respect to our initial project goals anddescribe issues raised that would impact the constructionof any similar environment.

3.1 Achieving Project Goals

We review the goals of Section 1.2 considering notonly how well the goals were achieved but also the influ-ence those goals had on the development of vision algo-rithms and on the overall success of the project.

3.1.1 Real Action, Real Objects. One of ourprimary goals was to construct an environment in whichaction and attention were focused primarily in the roomand not on the screens. We wanted a rich environmentthat would watch what the children do and respond innatural ways.

We believe the KidsRoom achieves this goal. Childrenare typically active when they are in the space, runningfrom place to place, dancing, and acting out rowing andexploring fantasies. They interact with each other as

much as they do with the virtual objects, and their ex-ploration of the real space and the transformation of realobjects (e.g., the bed) adds to the excitement of the play.

We were only partially successful in the use of real ob-jects to enhance the experience. The only manipulatedobject in the KidsRoom is the bed, which is rolledaround, jumped on, and hid behind, and thus becomes acritical part of the narrative. However, two major ob-stacles—tracking and narrative control—prevented usfrom incorporating more objects into the room.

The KidsRoom tracking algorithm sets a limit of fourpeople and one bed in the space because more partici-pants or larger objects makes the space visually cluttered,debilitating the tracking algorithm and interfering withperceptual routines. The second and perhaps more seri-ous problem is that, as more objects are added to aspace, the behavior of the people in the space will be-come less predictable, because the number of ways inwhich objects may be used (or misused) increases. Forthe room to adequately model and respond to all ofthese scenarios, it will require both especially clever storydesign and tremendous amounts of narration and con-trol code. The more person-object interactions that thesystem fails to handle in a natural way, the less engagingand sentient the entire system feels.

To further achieve the goal of keeping the action onthe participants’ side of the screens, we designed thevisual and audio feedback to only minimally focus theattention of the children. Typically, virtual reality sys-tems use semi-realistic, three-dimensional renderedscenes and video as the primary form of system feedback.We decided, however, that, in order to give the room amagical, theatrical feel and to keep the emphasis of thespace in the room and not on the screens, images wouldhave a two-dimensional storybook look and video wouldconsist of simple, still-frame animations of those images.During much of the KidsRoom experience, the videoscreens are employed as mood-setting backdrops andnot as the center of the participants’ attention.

Audio is the main form of feedback in the room assound does not require participants to focus on any par-ticular part of the room. Children are free to listen tomusic, sound effects, and narration as they play, runabout the space, and talk to one another. During the


scenes in which sound is the primary output mechanism,such as the bedroom and forest worlds, the children arefocused on their own activity in the space. Combiningambient sound effects with appropriate music can set atone for the entire space. Audio feedback can be furtherenhanced by using spatial localization. Even with justfour speakers, the KidsRoom monster growls sound as ifthey are coming from the forest side of the room, and,when the furniture speaks, the sound originates fromapproximately the correct part of the room.

3.1.2 Remote Visual Sensing. In the Kids-Room, the only encumbrance on or requirement ofpeople who enter the space is that they must enter one ata time. Further, occupants in the room (particularlyyoung children) are typically unaware of how the roomis sensing their behavior. There are no obvious sensors inthe room embedded in any objects or the floor. Thecameras are positioned high above the space, well out ofthe line of sight and visible only if someone is lookingfor them. This enhances the magical nature of the roomfor all visitors, especially for children. They are not push-ing buttons or sensors, they are just being themselves,and the room is responding.

3.1.3 Multiple, Collaborating People. Anotherdesign goal was to create a system that could respond tothe interaction of multiple people. Since selfconscious-ness seems to decrease as group size increases, the kindof role-playing encouraged by the KidsRoom is mostnatural and fun with a group. Also, when unencumberedby HMDs, people will naturally communicate with eachother about the experience as it takes place, and they willwatch and mimic one another’s behavior. For instance,during the rowing scene, children shout to one anotherabout what to do, how fast to row, and where to rowand play-act together as they hit virtual obstacles.Groups of friends and parent-child pairs have an espe-cially good time.

3.1.4 Exploiting and Controlling Context.Given the difficulty of designing robust perceptual sys-

tems for recognizing action in complex environments,we strove to use narrative to provide context for the vi-

sion algorithms. Most of the vision algorithms dependupon the story to provide constraint. The boat-rowingexample described earlier typifies such a situation. It iscurrently well beyond the state of the art of computervision to robustly recognize a group of closely situatedpeople rowing a boat. In the KidsRoom, context makesit almost trivial.

Another example is the monster-dance scene in whichthe story was constructed in such a way to ensure thateach camera has a clear view of a child performing thedance moves. Potentially interfering children arecajoled by the monsters to stand in locations that do notinterfere with the sensing. The advantage of an activesystem over that of a monitoring situation is the oppor-tunity to not only know the context but to control it aswell.

3.1.5 Presence, Engagement, andImagination. The power of a compelling storyline can-not be overstated when constructing a space like theKidsRoom that integrates technology and narrative. Theexistence of a story seems to make people, particularlychildren, more likely to cooperate with the room thanresist it and test its limits. In the KidsRoom, a well-crafted story was used to make participants more likelyto suspend disbelief and more curious and less apprehen-sive about what will happen next. The story ties thephysical space, the participants’ actions, and the differentoutput media together into a coherent, rich, and there-fore immersive experience.

Some existing work has studied the criteria that leadto the feeling of ‘‘immersion’’ or ‘‘presence’’ in virtualenvironments (Sheridan, 1992). Here we just note thatour system meets eight of the ten criteria commonlyidentified as important for creating a feeling of presencein a virtual space (Slater & Usoh, 1993). One of the twocriteria the KidsRoom does not meet—‘‘a similarity invisual appearance of the subjects and their representationin the virtual environment’’—does not apply to a systemin which people are interacting in the real world. Criteriafor presence that are met include high-resolution infor-mation being presented to the appropriate senses, free-dom from sensing devices, easily perceived effects of ac-tions, an ability to change the environment, and

Bobick et al. 385

‘‘virtual’’ objects that respond spontaneously to theuser. The remaining unmet criteria—that the systemshould adapt over time—is not met explicitly, but, asdiscussed later, the KidsRoom does allow the user tocontinuously and naturally control the pace of the expe-rience.

An immersive space is most engaging when partici-pants believe their actions are having an effect upon theenvironment by influencing the story. The KidsRoomuses computer vision to achieve this goal by making theroom responsive to the position, movements, and ac-tions of the children. Immediately upon their first inter-action with the room, the children realize that whatthey do makes a difference in how the room responds.This perceptual sensing enhances and energizes the nar-rative.

A goal that was critical to obtaining a sense of pres-ence in the KidsRoom was to naturally embed the per-ceptual constraints into the storyline. For example, inthe monster dance scene, the vision systems require thatthere is only one child per rug and that all children areon some rug. One way to impose this constraint wouldhave been for the narrator to say, ‘‘Only one kid per rug.Everyone must be on a rug.’’ Instead, the monsters tellthe children to stay on the rugs and that there can beonly one per rug ‘‘so’s we can see what’s goin’ on.’’That the monsters have some visual constraints seemsperfectly natural and makes children less likely to feelrestrained or to question why they need to engage insome particular behavior.

No matter how well an interactive storyline is de-signed, participants, especially children, will do the un-expected, especially when there are up to four of theminteracting together. This unpredictable behavior cancause the perceptual system to perform poorly. There-fore, we designed the story so that such errors wouldnot destroy the suspension of disbelief. When perceptualalgorithms fail, the behavior of the entire room degradesgracefully.

One example of this principle in the KidsRoom is inthe way the vision system provides feedback during themonster dance. If a child is ignoring the instructions ofthe characters and is too close to another child on a rug,

the recognition of the movements of the child on therug will be poor. Therefore, when actions are not recog-nized with high confidence, the monsters on the screenwill animate, doing the low-confidence action, but themonster will not say anything. To the child, this just ap-pears as though the monster is doing its own thing; itdoes not appear that the monster is any way confused.This choice was preferred over the possibility that themonster says ‘‘Great crouch!’’ while the child is actuallyspinning.

Similarly, we tried to minimize the number of storysegments that required a particular single action on thepart of all participants. For instance, to get a particularpiece of furniture to speak, only one child needed to bein its proximity; if all children needed to be close to it,they might never discover how to make the furnituretalk.

Finally, to give the KidsRoom narrative a cohesive,immersive feel, thematic threads run throughout thestory. For example, the careful observer will note thatthe stuffed animals on the walls in the children’s bed-room (shown in Figure 5) are similar to the monstersthat appear later in the story (Figure 8). Some of thefurniture characters in the first world have the samevoices as the monsters in the monster world. The art-work has the same storybook motif in all four scenes,and several objects on the shelves in the room becomepart of the forest world backdrop during the transforma-tion.

3.1.6 Children as Subjects. Building a space forchildren was both wonderful and problematic. The posi-tives include the tremendous enthusiasm with which thechildren participate, their willingness to play with peersthey do not know, the delight they experience when be-ing complimented by virtual characters, and their com-plete disregard of minor technical embarrassments thatarose during development.

Children provided unique challenges as well. The be-havior of children, particularly their group behavior, isdifficult to predict. Further, children have short atten-tion spans and often move about with explosive energy,leaving the longer-playing narrations behind. Young


children are small compared to adults, which can createproblems when developing vision algorithms.

In balance, having children as the primary user groupnot only inspired us to think imaginatively but also,quite frankly, made the project all the more fun to con-struct.

3.2 Observations and Failures

We failed to consider some issues in the designphase of the KidsRoom that are important for develop-ing other interactive, immersive spaces, particularlythose for children. In the next sections, we present sev-eral of these in an effort to prevent others from repeat-ing our mistakes.

3.2.1 Group Versus Individual Activity. Theinteraction in the KidsRoom changes significantly de-pending upon the number of people in the space. First,as mentioned previously, all system timings differ de-pending upon the number of people in the room. It isonly a small window of time outside of which each unitof the experience becomes too short or too long, andthe ideal timing changes based on the number of peoplearound. Since automatically sensing when people aregetting bored is well beyond our current perception ca-pability, the KidsRoom uses an ad hoc procedure to ad-just the duration of many activities depending upon thenumber of people in the space.

In general, the more children there are in the space,the more fast-paced the room appears to be, because assoon as one child figures out the cause-and-effect rela-tionship between some activity and response, the otherchildren will follow. A single child is more hesitant andtherefore needs more time to explore before the roominterjects. Also, a lone child often requires more inter-vention from the system to guide him or her throughthe experience.

A final consideration when developing for group ac-tivity is the importance of participants being able to un-derstand cause-and-effect relationships. If too much ishappening in the room and there is not a reasonable ex-pectation within the child’s mind of strong correlationbetween some action and a reasonable response, the

child will not understand that he or she has caused theaction to happen.

3.2.2 Exploratory Versus Scripted NarrativeSpaces. In our initial design of the KidsRoom, weplanned to create a primarily exploratory space, modeledsomewhat on popular nonlinear computer games likeMyst (Broderbund Software, 1994). We designed andbuilt prototypes for the first and second worlds usingthis model. This first world had no talking furniture.Instead, when children walked near objects the objectsmade distinctive sounds: moving near the shelves with amirror would make a crashing sound, stepping on therugs with animal pictures elicited the corresponding ani-mal noises.

Our hope was that children would enter, figure outthat they could make such sounds, and then explore theroom, gradually creating a frenzy of sounds and activity.It didn’t work. When we brought in some children fortesting, we found that they did not understand that theywere causing the sounds; there was simply too muchgoing on as each child explored independently. Thesame was true for some adults. Even a child alone in theroom had trouble identifying cause-and-effect relation-ships. We had also planned to develop an exploratorysecond world using forest sounds, forest images, andcreepy, exploratory music. Again, testing proved theconcept too weak.

The most significant problem was that the exploration‘‘plot’’ did not encourage particular actions, nor did itcause people to act in a group fashion. Users did notshare any common goals. Other authors have observedthat exploratory, puzzle-solving spaces can sometimesmake it difficult for adults to immerse themselves in aninteractive world (Druin & Perlin, 1994; Davenport &Friedlander, 1995). When a story is added to the physi-cal environment, a theatrical-like experience is created.Once the theatrical nature of the system is apparent, it iseasier for people to imagine their roles and, if they arenot too selfconscious, act them out. Furthermore, rec-ognizing action is simpler in a story-based environmentbecause the number of action possibilities at any mo-ment is more constrained.

Bobick et al. 387

A prima facie criticism of a linear storyline is that thesystem loses its interactive nature. This is perhaps truefor a mundane interface such as a mouse or keyboard, asthere are no dynamics to the actions performed. For amultiperson, room-sized environment, however, theinteraction comes from making physical exertion, con-trolling the pace11 of the adventure, recognizing that theroom understands what is happening and is responding,and interacting with fellow users.

Finally, although the scenes and the plot are simpleand linear, the actions within each scene are not. Thesystem responds appropriately to user actions dependingupon the context. Only the changes in context are linear.

3.2.3 Anticipating Children’s Behavior. Fromtesting with children, we learned that there are threeaspects of children’s behavior in the space that we hadnot adequately considered during narrative develop-ment.

First, the story must take into account the children’s‘‘behavioral momentum.’’ The KidsRoom is capable ofmaking children exceedingly active. By the end of thefirst bedroom world when the furniture all start loudlychanting the magic word, the children are often runningenergetically around the room. Next, the children endup on the bed, shout the magic word loudly, and thetransformation occurs. The location is now the forestworld, and the children are instructed and expected toexplore. However, the transformation typically calmsthem down and their tendency is often to remain on thebed. We found through testing that a fairly direct in-struction (e.g., ‘‘Follow the path’’), sometimes repeatedseveral times, is required to get them to start movingagain. When designing for a space where physical actionis the focus, behavioral momentum needs to be consid-ered.

A related problem was the need for attention-grab-bing cues. Particularly when kids are in a state of highphysical activity, they almost never hear the first thingthat the room says to them. Since we did not anticipatethis problem, sometimes children missed important in-structions. We modified the narrative so that it repeats

some critical instructions more than once. Ideally, weshould have built attention-grabbing narrative into thestoryline for every critical narration.

Finally, children need to clearly understand the cur-rent task. The less certain they are of what to do, themore unpredictable their behavior becomes. Perhapsbecause they had never experienced a room such as thisbefore, the children seem more inclined to wait forthings to happen than to explore and try to make thingshappen. The children enjoyed the experience more oncethe system was modified so that there was always a cleartask and when those tasks changed quickly.

3.2.4 Avoiding Repetitiveness and Hints. Oneway to break the suspension of disbelief of the experi-ence is for the system to exactly repeat a single narrationas it tries to encourage some behavior. Unfortunately, ina space like the KidsRoom which is built to encouragechildren to physically move around, instructions do needbe repeated. For example, in the dance segment of themonster world, the control program continually checksif someone has stepped off their rug or if two people areon the same rug. If someone drifts off a rug more thanonce, narration is needed, but repeating a narration justplayed moments before imparts a mechanical sense tothe responses and causes the entire experience to feel lessalive.

One solution we developed was to use two differentnarrators. The main narrator has a deep, male voice andspeaks in rhymes like a grandfather reading a storybook.The second narrator, with a soft, whispered, femalevoice, delivers ‘‘hints.’’ The first time someone gets off arug, the monsters will tell the person to get back on.After that, however, a voice whispers a hint, ‘‘Stay onyour rug.’’ This type of feedback is easily understood byroom participants but does not break the flow of thestory and primary narration. The delivery of hints by adifferent voice and typically from a different sound di-rection than the narrator made them perceptually sa-lient, increasing their effectiveness. Because the hintswere not long rhyming couplets, it was easy to have mul-tiple phrases to encourage a single behavior, reducingthe repetition problem.11. We intend a more psychologically loaded term than speed.


3.2.5 Perceptual Limitations. Some perceptual-sensing difficulties and related issues follow.

3.2.5.1 Perceptually Based EnvironmentalConstraints. A major challenge when designing the Kids-Room was to minimize the impact of our sensors andoutput modalities on the development of an interestingstory environment. Some constraints are listed below.

• Vision algorithms generally require bright lighting,but large projection displays appear dim whenplaced in bright spaces.

• Large video screens displaying video violate the as-sumption of static background used by the visionalgorithms. We were forced, therefore, to choosecamera and rug positions so that people in the roomwould never appear in front of a screen in the imageviews—a serious limitation for a space like the Kids-Room or (even worse) the Cave (Cruz-Neira, San-din, & DeFanti, 1993). Recent work motivated bythis problem may alleviate this constraint (Ivanov etal., 1998; Davis & Bobick, 1998).

• Even in a space as large as 24 feet by 18 feet with a20-foot-high ceiling, camera placement was severelyconstrained. For example, due to viewpoint, occlu-sion, and story constraints, there is no flexibility inthe positioning of the red and green rugs and theircorresponding cameras.

• Every space-related decision required careful consid-eration of imaging requirements. For instance, rugsand carpet had to be short-haired, not shaggy, toprevent the background from changing as peoplemoved around, and objects were painted in a flatpaint to minimize specularity.

• The four speakers in the KidsRoom provide reason-able localization when listeners are near the centerof the room. However, when a participant is near aloudspeaker playing a sound, that single speakertends to dominate the positional percept and thespatial illusion breaks down. Sound localization isimportant if real objects in a space are to be given‘‘personalities’’ using sound effects (e.g., as in thebedroom world), because the effect is destroyed ifthe sound is not perceived to come from the object.

3.2.5.2 Object-Tracking Difficulties. The Kids-Room tracking algorithm does an excellent job of keep-ing track of where people are but occasionally makes er-rors when keeping track of who people are (Intille et al.,1997). In other words, the tracker sometimes swaps twopeople, thinking that one is the other, but does not losea person altogether in normal operation.

The KidsRoom, therefore, was designed with the ex-pectation that perfect tracking of identity might beproblematic. The room uses information about wherepeople are, but, when referring to individuals, it usesenvironmental indicators, not absolute labels. For in-stance, instead of saying, ‘‘Great job kid number one,’’ itwill say, ‘‘Great job, kid on the red rug.’’

While there are some improvements that could bemade to the tracking algorithm, perfect tracking of iden-tity is unlikely. However, unlike conventional surveil-lance tasks, an immersive environment that must keeptrack of identity can manipulate people in its environ-ment so that, when it becomes uncertain of identity, itcan ‘‘bootstrap’’ itself automatically. For instance, a sys-tem controlling an environment with a telephone mightphysically call the room, ask to speak to a particular per-son and, when that person comes to the phone, reinitial-ize tracking. Integrating such ‘‘bootstrapping’’ devicesinto a narrative requires careful story development andwill restrict the designer’s flexibility.

3.2.5.3 Monster World Action-RecognitionDifficulties. All of the monster world action-recognitionstrategies make assumptions about the viewing situation.First, images of the child cannot be occluded by otherchildren. Careful camera and rug placement minimizedthis problem, although occasionally when large adultsenter the space the system’s recognition performancewill suffer slightly due to small occlusions. Second, sinceit is difficult to remove shadows accurately, shadows(and therefore light positions) were incorporated intothe motion models. This turned out to yield a more ro-bust representation but requires that the lighting setupdoesn’t change between the training phase and run time.Third, the motion-template recognition algorithms usedin the KidsRoom are limited to recognizing actions ofindividual people, which prevents the interactive narra-

Bobick et al. 389

tive from explicitly recognizing some multiperson ac-tion, such as people shaking hands.12

3.2.5.4 Event Detection and Noncooperation. Oneproblem we encountered when designing the KidsRoomwas that ‘‘simple’’ events are strongly context sensitiveand memory-less. For example, our ‘‘in-a-group’’ detec-tor will signal false continuously if one mischievous childrefuses to cooperate with the remaining children. In thiscase, a more robust detector might ignore the renegadegiven that the child hasn’t been following the rules for awhile. Similarly, if one child is scared and remains on thebed while other children explore the forest on the path,the ‘‘in-a-group’’ detector should ignore this child aswell. We accommodated such possibilities not by fixingthe detectors (which remains interesting future work incontext-sensitive action recognition), but by ensuringthat nowhere did the story stall endlessly if some re-quested or expected behavior was not observed.

3.2.6 Sensitivity to Timing. Multiple people in aspace increases the number of possible situations andresponses that are required, thereby making narrativetiming control difficult.

Our lack of any systematic approach to checking forinconsistent timings was most painfully apparent as wetested the nearly completed system. Since advancing thestory forward manually often creates timing problems,the only way to really test the room is to put people in itand run the narrative repeatedly. The problem is that theroom provides an experience of ten to twelve minutes,and thorough testing of every timing scenario is out ofthe question due to the large number of possible timingsituations. Further, once the room is tuned, any smallchange to any timing-related code requires having sev-eral people around to interact in the space. Our onlymethod of addressing this problem is to create modularstory fragments such that the timers in one fragment donot affect those in another.

We note that an alternative mechanism to timers is touse the AI concept of planning to model the change in

the states of the world (Bates, Loyall, & Reilly, 1992).Recently, Pinhanez, Mase, and Bobick (1997) have pro-posed the use of interval scripts, where all the sensingand actuating activities are associated with temporal in-tervals, whose activation is determined by the result of aconstraint-satisfaction algorithm based on the inputfrom the sensors and the past interaction. Such a repre-sentation may facilitate automatically checking offlinefor temporally-related narrative control problems.

3.2.7 Communication Model. The KidsRoomlacks a rich model of what information has been commu-nicated to the users at any given time and how sequencesof instructions can be presented to the users in a naturalway. For example, in the river world, a room narrationsometimes presents information to the children on theboat. In the middle of that narration, a child ‘‘jumpsoverboard’’ by getting off the bed. The system detectsthis event immediately, but has no way of promptly indi-cating this to the children. The solution is not as simpleas cutting off the main narrator midphrase. First,abruptly cutting narrations destroys the sentient feel ofthe characters. Second, even if a narration can be natu-rally cutoff (e.g., using cue phrases like ‘‘oh!’’), the sys-tem then needs a model of what partial information hasbeen conveyed to the children and how to naturally pickup the narration when the overboard activity has ended.Significant time may have elapsed during the overboardactivity, for example, which requires a modification ofthe original narration. Exactly how such a communica-tion model should be represented and used for reason-ing is an open research question.

3.2.8 Wasting Sensing Knowledge. Sometimesthe system has the capability to detect some situationbut no way of informing the participants. Given the im-poverished sensing technology, no knowledge should bewasted; all information should be used to enhance thefeeling of responsiveness.

For example, we encountered this problem when thechildren shout. Suppose the room asks for the kids toshout the magic word; the kids shout but not loudly.The room then responds with, ‘‘Try that shout onemore time.’’ Initially, we felt that children like to shout,

12. Group activity in the KidsRoom is always recognized using in-put from the person tracker, not using motion templates.


so we would encourage them do it twice. The problemwas that the room responded with ambiguous narration.Are they doing it again because it wasn’t good enoughor for some other reason? Worse, if nobody shouted any-thing, the narration mildly suggested that the room ac-tually heard a shout. As we improved the system, weadded hints and new narration to try and ensure thatwhen the system knows something (e.g., how loud ascream is) it lets the participants know that it knows. Theeffect is a room that feels more responsive.

Similarly, there are times when the system is dealingwith uncooperative participants and, despite several at-tempts, has not elicited the desired activity. Another(shouting) example is when the narrator requests thechildren shout ‘‘Be quiet!’’ to the monsters. If after tworequests the children do not shout, the story continues,ignoring their lack of cooperation. However, it is impor-tant to explicitly acknowledge that the system under-stands that something is wrong but is ignoring the prob-lem. In this example, the narrator says ‘‘Well, I can’t saythat was a very loud shout, but perhaps the monsters willfigure it out.’’

3.2.9 Perceptual Expectation. Given the tech-nological limitations of unencumbered sensing, the in-teraction must be designed as to not establish any per-ceptual expectations on the part of the user that cannotbe satisfied. For instance, use of some speech recogni-tion in the KidsRoom might prove problematic. If achild or adult sees that the room can respond to one sen-tence, the expectation may be established that the char-acters can understand any utterance. Any immersive en-vironment that encourages or requires people to test thelimits of the perception system is more likely to feelmore mechanical than natural.

The KidsRoom is not entirely immune to this prob-lem, but we believe we have minimized the ‘‘responsive-ness testing’’ that people do by making the system flex-ible to the type of input it receives (e.g., in the boatscene, most any large body movement will be inter-preted as rowing) and by having characters in the storyessentially teach the participants what they can and can-not recognize (i.e., the allowed dance moves in themonster land).

3.3 Summary and Contributions

The KidsRoom went from whiteboard sketches toa fully-operational installation in eight weeks. This paperhas described the story, technology, and design decisionsthat went into building the system. We believe the Kids-Room is the first perceptually-based, multi-person, fully-automated, interactive, narrative playspace ever con-structed, and the experience we acquired designing andbuilding the space has allowed us to identify some majorquestions and to propose a few solutions that shouldsimplify construction of complex spaces in the future.

We believe the KidsRoom provides several fundamen-tal contributions. First, unlike most previous interactivesystems, the KidsRoom does not require the user towear special clothing, gloves, or vests; does not requireembedding sensors in objects; and has been explicitlydesigned to allow for multiple simultaneous users.

Second, it demonstrates that non-encumbering sen-sors can be used for the measurement and recognition ofindividual and group action in a rich, interactive, story-based experience. Relatively simple perceptual routinesintegrated carefully into a strong story context are ad-equate for recognizing many types of actions in interac-tive spaces.

Finally, we believe the KidsRoom is a unique and funchildren’s environment that merges the mystery and fan-tasy of children’s stories and theater with the spontaneityand collaborative nature of real-world physical play.

Acknowledgments

The KidsRoom owes much of its success to the fact that itsconcept and design was developed, collaboratively, by all of theauthors. Aaron Bobick acted as advisor and producer. StephenIntille (interaction control, production issues) and Jim Davis(vision systems) were the chief architects of the system. Free-dom Baird, with help from Arjan Schutte, wrote the originalscript, handled most narrative recording, and provided videodocumentation. Claudio Pinhanez was in charge of music andlight design and control. Lee Campbell worked on sound con-trol and related issues. Yuri Ivanov and Andrew Wilson wrotethe animation control code. Composer John Klein wrote the

Bobick et al. 391

original KidsRoom score, and artist Alex Weissman created thecomputer illustrations.

The authors would like to thank Glorianna Davenport for hersupport; Ed McCluney, John UnderKoffler, Laurie Ward, andMichelle Kemper for character narrations; Hugo Barra forwork on MIDI light control; Greg Tucker; and Laurie Wardfor administrative support. We also owe thanks to OpCode andMark of the Unicorn for software donations. This work wasfunded by the MIT Media Laboratory’s Digital Life Consor-tium.

References

Azuma, R. (1997). A survey of augmented reality. Presence:Teleoperators and Virtual Environments, 6 (4), 355–385.

Barrie, J. (1988). Peter Pan. EP Dutton.Bates, J., Loyall, A. B., & Reilly, W. S. (1992). An architecture

for action, emotion, and social behavior. Proceedings of theFourth European Workshop on Modeling Autonomous Agentsin a Multi-Agent World. S. Martino al Cimino, Italy.

Bederson, B., & Druin, A. (1995). Computer augmented en-vironments: New places to learn, work, and play. In Ad-vances in Human Computer Interaction (volume 5, chapter2). Ablex.

Bobick, A. F. (1997). Movement, activity, and action: The roleof knowledge in the perception of motion. Phil. Trans.Royal Society London B, 352, 1257–1265.

Bolt, R. (1984). The Human Interface. Wadsworth, Inc. Bel-mont, California.

Broderbund Software (1994). Myst. An interactive CD-ROM.Coen, M. (1997). Building brains for rooms: Designing dis-

tributed software agents. In Proc. of the Conf. on InnovativeApplications of Artificial Intelligence (pp. 971–977). AAAIPress.

Cruz-Neira, C., Sandin, D., & DeFanti, T. (1993). Surround-screen projection-based virtual reality: The design andimplementation of the CAVE. In Proc. of SIGGRAPH Com-puter Graphics Conference (pp. 135–142). ACMSIGGRAPH.

Davenport, G., & Friedlander, G. (1995). Interactive transfor-mational environments: Wheel of life. In E. Barrett & M.Redmond (Eds.), Contextual media: Multimedia and inter-pretation (ch. 1, pp. 1–25). Cambridge: MIT Press.

Davis, J. W., & Bobick, A. F. (1997). The representation andrecognition of action using temporal templates. In Proc.Computer Vision and Pattern Recognition (pp. 928–934).IEEE Computer Society Press.

———. (1998). A robust human-silhouette extraction tech-nique for interactive virtual environments. In Lecture Notesin Artificial Intelligence (vol. 1537, pp. 12-25). Springer-Verlag.

Druin, A. (1988). Noobie: The animal design playstation. InProc. of Human Factors in Computing Systems (CHI) (vol.20, pp. 45–53).

Druin, A., & Perlin, K. (1994). Immersive environments: Aphysical approach to the computer interface. In Proc. of Hu-man Factors in Computing Systems (CHI) (pp. 325–326).

Glos, J., & Umaschi, M. (1997). Once upon an object. . .:computationally-augmented toys for storytelling. In Proc. ofthe Int’l Conf. on Computational Intelligence and Multime-dia Applications (ICCIMA) (pp. 245–249).

Hu, M. (1962). Visual pattern recognition by moment invari-ants. IRE Transactions on Information Theory, IT-8(2).

Intille, S. S., Davis, J. W., & Bobick, A. F. (1997). Real-timeclosed-world tracking. In Proc. Computer Vision and PatternRecognition (pp. 697–703). IEEE Computer Society Press.

Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seam-less interfaces between people, bits, and atoms. In Proc. ofHuman Factors in Computing Systems (CHI) (pp.234–241).

Ivanov, Y. A., Bobick, A. F. , & Liu, J. (1998). Fast lightingindependent background subtraction. In IEEE Workshop onVisual Surveillance—VS’98 (pp. 49–55). Also appears asMIT Media Lab Perceptual Computing Group TR#437.

Krueger, M. (1993). Environmental technology: Making thereal world virtual. In Communications of the ACM (vol. 36,pp. 36–37).

Lovejoy, M. (1989). Postmodern Currents: Art and Artists inthe Age of Electronic Media. Ann Arbour: London, England.

Maes, P., Pentland, A., Blumberg, B., Darrell, T., Brown, J., &Yoon, J. (1994). ALIVE: Artificial life interactive video envi-ronment. Intercommunication, 7, 48–49.

Pentland, A. (1996). Smart rooms. Scientific American, 274(4), 68–76.

Pinhanez, C., & Bobick, A. (1998). It/I: Theatre with an au-tomatic and reactive computer graphics character. In Proc. ofSIGGRAPH’98 Sketches.

Pinhanez, C. S., Mase, K., & Bobick, A. F. (1997). Intervalscripts: A design paradigm for story-based interactive sys-


tems. In Proc. of Human Factors in Computing Systems(CHI) (pp. 287–294).

Popper, F. (1993). Art of the Electronic Age. Thames and Hud-son: London, England.

Sendak, M. (1963). Where the Wild Things Are. HarperCollinsJuvenile Books.

Sheridan, T. (1992). Musings on telepresence and virtual pres-ence. Presence: Teleoperators and Virtual Environments, 1(1), 120–125.

Slater, M., & Usoh, M. (1993). Presence in immersive virtualenvironments. In IEEE Virtual Reality Annual Interna-tional Symposium (pp. 90–96).

Sommerer, C., & Mignonneau, L. (1997). Art as a living sys-tem. Leonardo, 30 (5).

Torrance, M. C. (1995). Advances in human-computer inter-

action: The intelligent room. In Working Notes of the CHI95 Research Symposium.

Tosa, N., Hashimoto, H., Sezaki, K., Kunii, Y., Yamada, T.,Sabe, K., Nishino, R., Harashima, H., & Harashima, F.(1995). Network-based neuro-baby with robotic hand. InProc. of IJCAI’95 Workshop on Entertainment and AI/Alife.

Walt Disney Productions (1971). Bedknobs and Broomsticks.Movie.

Weiser, M. (1993). Some computer science issues in ubiqui-tous computing. In Communications of the ACM (vol. 36,pp. 74–84).

Wren, C., Azarbayejani, A., Darrell, T., & Pentland, A. (1997).Pfinder: Real-time tracking of the human body. IEEE Trans.Pattern Analysis and Machine Intelligence, 19 (7), 780–785.

Bobick et al. 393

Documents

TheKidsRoom - MIT Media Labintille/papers-files/kidsroom.pdfspaces began with Krueger’s Videoplace system (Krue-ger, 1993). Krueger designed installations that explored many different