Kevin L Deankldchemucsdedu

Xylar S Asay-DavisEvan M FinnTim FoleyJeremy A FriesnerYo ImaiBret J NaylorSarah R WustnerUniversity of California San DiegoLa Jolla CA 92093-0339

Scott S FisherTelepresence Media

San Francisco CA USA

Kent R WilsonDepartment of Chemistry andBiochemistryUniversity of California San Diego

Presence Vol 9 No 6 December 2000 505ndash523

copy2001 by the Massachusetts Institute of Technology

Virtual ExplorerInteractive Virtual Environment for

Education

Abstract

The Virtual Explorer project of the Senses Bureau at the University of CaliforniaSan Diego focuses on creating immersive highly interactive environments for edu-

cation and scientic visualization which are designed to be educationalmdashand excit-ing playful and enjoyable as well We have created an integrated model system onhuman immunology to demonstrate the application of virtual reality to education

and wersquove also developed a modular software framework to facilitate the furtherextension of the Virtual Explorer model to other elds The system has been in-stalled internationally in numerous science museums and more than 7000 individu-

als have participated in demonstrations The complete source codemdashwhich runs ona variety of Silicon Graphics computersmdashis available on CD-ROM from the authors

1 Overview and Purpose

The Senses Bureau is an undergraduate research group with a thirty-yearhistory of innovation in computer graphics and multimedia technology for ed-ucation and scientic visualization We at the Bureau believe that virtual reality(VR) has excellent potential as an educational medium to supplement conven-tional techniques because it provides both greater interactivity as well as theability to create a convincing sense of immersion in the computer-generatedenvironment that is beyond what is possible with conventional textbook- andblackboard-based educational approaches

Many topics in science education involve processes that occur simulta-neously on multiple time and length scales that are difcult to accurately repre-sent perceive and visualize with traditional static media Examples can befound in complex elds such as immunology astronomy relativistic dynamicsquantum mechanics and rainforest ecology We wanted to create a system thatwould be suitable for a diverse target audience that includes several types ofeducational venues such as high school college and university institutionsmuseums and other public places and independent student use Although wedo not feel that 3-D graphics technology can entirely replace conventionalclassroom teaching techniques we are convinced that properly implementedvirtual environments can serve as valuable supplemental teaching and learningresources to augment and reinforce traditional methods

The Virtual Explorer project employs a two-tiered approach to demonstrat-ing VRrsquos potential for scientic visualization as well as to creating interactivevirtual environments for education First wersquove developed a proof-of-concept

Dean et al 505

Virtual Explorer system to demonstrate and study thepotential applications and benets of an integrated VRinstallation in an educational arena This prototype in-stallation currently runs our example module whichfocuses on human immunology (Figure 1) Second wehave created a modular software framework and toolkitfor the further development of virtual reality for educa-tion based on the Virtual Explorer model We envisionnumerous applications for the Virtual Explorer as a visu-alization tool in diverse scientic elds and hope thatthis toolkit (which is available from the authors in fullsource code version for a wide variety of Silicon Graph-ics computers) will provide others with the means toexpand upon our work

2 Background

In the past thirty years many research and com-mercial efforts have investigated the application of newmedia technologies to education In particular the de-velopment of computer-based interaction with educa-tional material has enabled the development of learningenvironments that can be personalized to better matchindividual vocabularies styles and specic needs Morerecently advances in interactive computer graphics have

enabled the development of user-interface technologiesthat can immerse a student in these interactive learningenvironments It seems that the capabilites of these newtechnologies facilitate learning through a process of self-paced exploration and discovery in contrast to the moretraditional approach of instruction and memorizationThrough the interactive exploration of immersive envi-ronments a student can engage in a curriculum that isbased on learning by doing as well as encounteringsubject matter in contexts that are more meaningful

Several attempts to develop immersive learning envi-ronments predate the use of computational technolo-gies and two of the most memorable specically relateto the human body A surviving example is the walk-through scale model of the human heart at the FranklinInstitute science museum in Philadelphia PennsylvaniaSince the 1950s visitors can explore the giant chambersof the heart surrounded by a soundtrack of boomingheartbeats Later in the 1970s neurosurgeon DavidBogen and artist David Macaulay developed a detailedproposal for a thirty-story replica of the human brain asa new museum for San Jose California Bogen intendedthe structure as an important learning environment formedical students studying neuroanatomy as well as forthe general public (Bogen 1972)

For many decades interactive real-time graphics havebeen used for training applications that require the ac-quisition of specic skill sets for unique missions or pur-poses (such as the control of a variety of aircraft auto-mobiles or ships) But its use for more-generaleducational applications hasnrsquot been explored until therecent development of lower-cost hardware platformsand powerful software tools Recent research efforts thatexamine the use of virtual environment technology ineducation include

c Science SpacemdashThis joint research effort betweenGeorge Mason University and NASArsquos JohnsonSpace Center is developing a series of ldquovirtual real-ity microworldsrdquo for teaching science concepts andskills through the use of an interactive virtual labo-ratory conguration Current modules includeNewtonWorld MaxwellWorld and PaulingWorld(Dede 1996 Salzman 1999)

Figure 1 Concept view of the Virtual Explorer theater in the

development lab

506 PRESENCE VOLUME 9 NUMBER 6

c Zengo SayumdashThis immersive interactive virtualenvironment is designed to teach Japanese preposi-tions to students who have no prior knowledge ofthe Japanese language In one conguration stu-dents can hear digitized speech samples represent-ing the Japanese name of many virtual objects andtheir relative spatial location when touched by theuser in the virtual environment The system wasdeveloped at the Human Interface Laboratory atthe University of Washington (Rose 1996)

c Anatomic VirtualizermdashThis interactive immersivevirtual environment for teaching anatomy at theuniversity level was developed at the Learning Re-sources Center in the School of Medicine Univer-sity of California San Diego (Hoffman 1999)

3 The Mission

The Virtual Explorer allows students to interac-tively explore the immune system at both the cellularand molecular scales and at more familiar time andlength scales while still retaining a sense of overall sys-temic scale Students are free to explore realistic virtualenvironments that include blood vessels cell surfacesand lymph nodes while carrying out detailed missionsand several series of assigned tasks (Figure 2) We seekto provide students with a means not only to explorethe structure appearance and function of various com-ponents of the immune system but also with a tool forgaining an understanding of the interactions amongthese components

We present immunology in a rich game-like environ-

ment that features compelling visual and interactive quali-ties and that has been designed to be attractive to studentswho have been raised in an age of computer games andmusic videos An entertaining background story whoseplot is set on an isolated spacecraft captures the userrsquosimagination with a fantastic setting and expands the mis-sion beyond its immunological content (Figure 3)

After selecting the immunology module (Figure 4)the student is presented with a brief computer-animatedmovie that describes an ill-fated mission into deepspace Returning with samples of a dangerous off-worldbacteria the transport ship USS Archon suffers an ex-plosion caused by an unnoticed fuel leak in the propul-sion system This explosion allows the bacteria to escapeand to contaminate the shiprsquos air supply resulting in theinfection of the pilot the shiprsquos sole crew member Hepossesses only minimal medical knowledge and theshiprsquos supply of antibiotics has proven useless againstthis foreign pathogen Being an accomplished engineerhowever the pilot has been able to modify the remote-controlled nanobots that are normally used for repairingthe shiprsquos computers for operation within his own body(Figure 5) Online references a helpful ldquoshiprsquos com-puterrdquo character and virtual tools are available to assistthe student-pilot in completing the mission For exam-ple the nanobot is equipped with several tools that aidthe pilot in carrying out this unique mission includingmonoclonal antibody-based protein dye jets for identify-ing different types of white blood cells a remote probethat allows the pilot to explore cell surfaces at the mo-lecular scale a vacuum for collecting bacterial samplesand protein dye jets (Figure 6)

Figure 2 Students can interact with the immune system in multiple environments (left to right blood vessel cell surface lymph node)

Dean et al 507

Figure 3 A solo space mission gone terribly wrong Background story for the Virtual Explorerrsquos immunology module

508 PRESENCE VOLUME 9 NUMBER 6

Additionally the nanobotrsquos outer hull can be dynami-cally modied so that it can emulate cell surfaces and func-tionality Fortunately in addition to its quirky personalitythe shiprsquos computer is equipped with an extensive databaseon human immunology thus allowing it to offer guidanceduring the mission and to recommend a course of actionto the pilot The pilot must use the nanobot to identifyand explore the site of infection emulate the function ofthe damaged component of the immune system and initi-ate a successful immune response The missionrsquos level ofdifculty the overall sense of urgency and the video

game-like appeal is all heightened by challenges such asnite resources (for example the number of times the pro-tein dye jets can be red) damage incurred by the nano-bot ship (from collisions bacterial toxin and phagocyticcells) and the amount of time allowed to complete eachtask (Figure 7)

Although the ldquoshiprsquos computerrdquo character functionsin an advisory capacity offering verbal and textual sup-port to guide student-pilots through the various mis-sions the ultimate course remains under the studentrsquoscontrol Help screens which appear in the plane of the

Figure 4 The immunology module Diverse scientic disciplines ranging from astronomy to quantum mechanics are also candidates for the

Virtual Explorer

Figure 5 Remote-controlled nanobots These nanobots provide a

vehicle allowing students to interact with the immune system at

microscopic scales

Figure 6 Virtual tools Such tools including a bacterial sample

collection vacuum shown here assist students in performing assigned

tasks

Dean et al 509

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

Virtual Explorer system to demonstrate and study thepotential applications and benets of an integrated VRinstallation in an educational arena This prototype in-stallation currently runs our example module whichfocuses on human immunology (Figure 1) Second wehave created a modular software framework and toolkitfor the further development of virtual reality for educa-tion based on the Virtual Explorer model We envisionnumerous applications for the Virtual Explorer as a visu-alization tool in diverse scientic elds and hope thatthis toolkit (which is available from the authors in fullsource code version for a wide variety of Silicon Graph-ics computers) will provide others with the means toexpand upon our work

2 Background

In the past thirty years many research and com-mercial efforts have investigated the application of newmedia technologies to education In particular the de-velopment of computer-based interaction with educa-tional material has enabled the development of learningenvironments that can be personalized to better matchindividual vocabularies styles and specic needs Morerecently advances in interactive computer graphics have

enabled the development of user-interface technologiesthat can immerse a student in these interactive learningenvironments It seems that the capabilites of these newtechnologies facilitate learning through a process of self-paced exploration and discovery in contrast to the moretraditional approach of instruction and memorizationThrough the interactive exploration of immersive envi-ronments a student can engage in a curriculum that isbased on learning by doing as well as encounteringsubject matter in contexts that are more meaningful

Several attempts to develop immersive learning envi-ronments predate the use of computational technolo-gies and two of the most memorable specically relateto the human body A surviving example is the walk-through scale model of the human heart at the FranklinInstitute science museum in Philadelphia PennsylvaniaSince the 1950s visitors can explore the giant chambersof the heart surrounded by a soundtrack of boomingheartbeats Later in the 1970s neurosurgeon DavidBogen and artist David Macaulay developed a detailedproposal for a thirty-story replica of the human brain asa new museum for San Jose California Bogen intendedthe structure as an important learning environment formedical students studying neuroanatomy as well as forthe general public (Bogen 1972)

For many decades interactive real-time graphics havebeen used for training applications that require the ac-quisition of specic skill sets for unique missions or pur-poses (such as the control of a variety of aircraft auto-mobiles or ships) But its use for more-generaleducational applications hasnrsquot been explored until therecent development of lower-cost hardware platformsand powerful software tools Recent research efforts thatexamine the use of virtual environment technology ineducation include

c Science SpacemdashThis joint research effort betweenGeorge Mason University and NASArsquos JohnsonSpace Center is developing a series of ldquovirtual real-ity microworldsrdquo for teaching science concepts andskills through the use of an interactive virtual labo-ratory conguration Current modules includeNewtonWorld MaxwellWorld and PaulingWorld(Dede 1996 Salzman 1999)

Figure 1 Concept view of the Virtual Explorer theater in the

development lab

506 PRESENCE VOLUME 9 NUMBER 6

c Zengo SayumdashThis immersive interactive virtualenvironment is designed to teach Japanese preposi-tions to students who have no prior knowledge ofthe Japanese language In one conguration stu-dents can hear digitized speech samples represent-ing the Japanese name of many virtual objects andtheir relative spatial location when touched by theuser in the virtual environment The system wasdeveloped at the Human Interface Laboratory atthe University of Washington (Rose 1996)

c Anatomic VirtualizermdashThis interactive immersivevirtual environment for teaching anatomy at theuniversity level was developed at the Learning Re-sources Center in the School of Medicine Univer-sity of California San Diego (Hoffman 1999)

3 The Mission

The Virtual Explorer allows students to interac-tively explore the immune system at both the cellularand molecular scales and at more familiar time andlength scales while still retaining a sense of overall sys-temic scale Students are free to explore realistic virtualenvironments that include blood vessels cell surfacesand lymph nodes while carrying out detailed missionsand several series of assigned tasks (Figure 2) We seekto provide students with a means not only to explorethe structure appearance and function of various com-ponents of the immune system but also with a tool forgaining an understanding of the interactions amongthese components

We present immunology in a rich game-like environ-

ment that features compelling visual and interactive quali-ties and that has been designed to be attractive to studentswho have been raised in an age of computer games andmusic videos An entertaining background story whoseplot is set on an isolated spacecraft captures the userrsquosimagination with a fantastic setting and expands the mis-sion beyond its immunological content (Figure 3)

After selecting the immunology module (Figure 4)the student is presented with a brief computer-animatedmovie that describes an ill-fated mission into deepspace Returning with samples of a dangerous off-worldbacteria the transport ship USS Archon suffers an ex-plosion caused by an unnoticed fuel leak in the propul-sion system This explosion allows the bacteria to escapeand to contaminate the shiprsquos air supply resulting in theinfection of the pilot the shiprsquos sole crew member Hepossesses only minimal medical knowledge and theshiprsquos supply of antibiotics has proven useless againstthis foreign pathogen Being an accomplished engineerhowever the pilot has been able to modify the remote-controlled nanobots that are normally used for repairingthe shiprsquos computers for operation within his own body(Figure 5) Online references a helpful ldquoshiprsquos com-puterrdquo character and virtual tools are available to assistthe student-pilot in completing the mission For exam-ple the nanobot is equipped with several tools that aidthe pilot in carrying out this unique mission includingmonoclonal antibody-based protein dye jets for identify-ing different types of white blood cells a remote probethat allows the pilot to explore cell surfaces at the mo-lecular scale a vacuum for collecting bacterial samplesand protein dye jets (Figure 6)

Figure 2 Students can interact with the immune system in multiple environments (left to right blood vessel cell surface lymph node)

Dean et al 507

Figure 3 A solo space mission gone terribly wrong Background story for the Virtual Explorerrsquos immunology module

508 PRESENCE VOLUME 9 NUMBER 6

Additionally the nanobotrsquos outer hull can be dynami-cally modied so that it can emulate cell surfaces and func-tionality Fortunately in addition to its quirky personalitythe shiprsquos computer is equipped with an extensive databaseon human immunology thus allowing it to offer guidanceduring the mission and to recommend a course of actionto the pilot The pilot must use the nanobot to identifyand explore the site of infection emulate the function ofthe damaged component of the immune system and initi-ate a successful immune response The missionrsquos level ofdifculty the overall sense of urgency and the video

game-like appeal is all heightened by challenges such asnite resources (for example the number of times the pro-tein dye jets can be red) damage incurred by the nano-bot ship (from collisions bacterial toxin and phagocyticcells) and the amount of time allowed to complete eachtask (Figure 7)

Although the ldquoshiprsquos computerrdquo character functionsin an advisory capacity offering verbal and textual sup-port to guide student-pilots through the various mis-sions the ultimate course remains under the studentrsquoscontrol Help screens which appear in the plane of the

Figure 4 The immunology module Diverse scientic disciplines ranging from astronomy to quantum mechanics are also candidates for the

Virtual Explorer

Figure 5 Remote-controlled nanobots These nanobots provide a

vehicle allowing students to interact with the immune system at

microscopic scales

Figure 6 Virtual tools Such tools including a bacterial sample

collection vacuum shown here assist students in performing assigned

tasks

Dean et al 509

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

c Zengo SayumdashThis immersive interactive virtualenvironment is designed to teach Japanese preposi-tions to students who have no prior knowledge ofthe Japanese language In one conguration stu-dents can hear digitized speech samples represent-ing the Japanese name of many virtual objects andtheir relative spatial location when touched by theuser in the virtual environment The system wasdeveloped at the Human Interface Laboratory atthe University of Washington (Rose 1996)

c Anatomic VirtualizermdashThis interactive immersivevirtual environment for teaching anatomy at theuniversity level was developed at the Learning Re-sources Center in the School of Medicine Univer-sity of California San Diego (Hoffman 1999)

3 The Mission

The Virtual Explorer allows students to interac-tively explore the immune system at both the cellularand molecular scales and at more familiar time andlength scales while still retaining a sense of overall sys-temic scale Students are free to explore realistic virtualenvironments that include blood vessels cell surfacesand lymph nodes while carrying out detailed missionsand several series of assigned tasks (Figure 2) We seekto provide students with a means not only to explorethe structure appearance and function of various com-ponents of the immune system but also with a tool forgaining an understanding of the interactions amongthese components

We present immunology in a rich game-like environ-

ment that features compelling visual and interactive quali-ties and that has been designed to be attractive to studentswho have been raised in an age of computer games andmusic videos An entertaining background story whoseplot is set on an isolated spacecraft captures the userrsquosimagination with a fantastic setting and expands the mis-sion beyond its immunological content (Figure 3)

After selecting the immunology module (Figure 4)the student is presented with a brief computer-animatedmovie that describes an ill-fated mission into deepspace Returning with samples of a dangerous off-worldbacteria the transport ship USS Archon suffers an ex-plosion caused by an unnoticed fuel leak in the propul-sion system This explosion allows the bacteria to escapeand to contaminate the shiprsquos air supply resulting in theinfection of the pilot the shiprsquos sole crew member Hepossesses only minimal medical knowledge and theshiprsquos supply of antibiotics has proven useless againstthis foreign pathogen Being an accomplished engineerhowever the pilot has been able to modify the remote-controlled nanobots that are normally used for repairingthe shiprsquos computers for operation within his own body(Figure 5) Online references a helpful ldquoshiprsquos com-puterrdquo character and virtual tools are available to assistthe student-pilot in completing the mission For exam-ple the nanobot is equipped with several tools that aidthe pilot in carrying out this unique mission includingmonoclonal antibody-based protein dye jets for identify-ing different types of white blood cells a remote probethat allows the pilot to explore cell surfaces at the mo-lecular scale a vacuum for collecting bacterial samplesand protein dye jets (Figure 6)

Figure 2 Students can interact with the immune system in multiple environments (left to right blood vessel cell surface lymph node)

Dean et al 507

Figure 3 A solo space mission gone terribly wrong Background story for the Virtual Explorerrsquos immunology module

508 PRESENCE VOLUME 9 NUMBER 6

Additionally the nanobotrsquos outer hull can be dynami-cally modied so that it can emulate cell surfaces and func-tionality Fortunately in addition to its quirky personalitythe shiprsquos computer is equipped with an extensive databaseon human immunology thus allowing it to offer guidanceduring the mission and to recommend a course of actionto the pilot The pilot must use the nanobot to identifyand explore the site of infection emulate the function ofthe damaged component of the immune system and initi-ate a successful immune response The missionrsquos level ofdifculty the overall sense of urgency and the video

game-like appeal is all heightened by challenges such asnite resources (for example the number of times the pro-tein dye jets can be red) damage incurred by the nano-bot ship (from collisions bacterial toxin and phagocyticcells) and the amount of time allowed to complete eachtask (Figure 7)

Although the ldquoshiprsquos computerrdquo character functionsin an advisory capacity offering verbal and textual sup-port to guide student-pilots through the various mis-sions the ultimate course remains under the studentrsquoscontrol Help screens which appear in the plane of the

Figure 4 The immunology module Diverse scientic disciplines ranging from astronomy to quantum mechanics are also candidates for the

Virtual Explorer

Figure 5 Remote-controlled nanobots These nanobots provide a

vehicle allowing students to interact with the immune system at

microscopic scales

Figure 6 Virtual tools Such tools including a bacterial sample

collection vacuum shown here assist students in performing assigned

tasks

Dean et al 509

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

Figure 3 A solo space mission gone terribly wrong Background story for the Virtual Explorerrsquos immunology module

508 PRESENCE VOLUME 9 NUMBER 6

Additionally the nanobotrsquos outer hull can be dynami-cally modied so that it can emulate cell surfaces and func-tionality Fortunately in addition to its quirky personalitythe shiprsquos computer is equipped with an extensive databaseon human immunology thus allowing it to offer guidanceduring the mission and to recommend a course of actionto the pilot The pilot must use the nanobot to identifyand explore the site of infection emulate the function ofthe damaged component of the immune system and initi-ate a successful immune response The missionrsquos level ofdifculty the overall sense of urgency and the video

game-like appeal is all heightened by challenges such asnite resources (for example the number of times the pro-tein dye jets can be red) damage incurred by the nano-bot ship (from collisions bacterial toxin and phagocyticcells) and the amount of time allowed to complete eachtask (Figure 7)

Although the ldquoshiprsquos computerrdquo character functionsin an advisory capacity offering verbal and textual sup-port to guide student-pilots through the various mis-sions the ultimate course remains under the studentrsquoscontrol Help screens which appear in the plane of the

Figure 4 The immunology module Diverse scientic disciplines ranging from astronomy to quantum mechanics are also candidates for the

Virtual Explorer

Figure 5 Remote-controlled nanobots These nanobots provide a

vehicle allowing students to interact with the immune system at

microscopic scales

Figure 6 Virtual tools Such tools including a bacterial sample

collection vacuum shown here assist students in performing assigned

tasks

Dean et al 509

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

Additionally the nanobotrsquos outer hull can be dynami-cally modied so that it can emulate cell surfaces and func-tionality Fortunately in addition to its quirky personalitythe shiprsquos computer is equipped with an extensive databaseon human immunology thus allowing it to offer guidanceduring the mission and to recommend a course of actionto the pilot The pilot must use the nanobot to identifyand explore the site of infection emulate the function ofthe damaged component of the immune system and initi-ate a successful immune response The missionrsquos level ofdifculty the overall sense of urgency and the video

game-like appeal is all heightened by challenges such asnite resources (for example the number of times the pro-tein dye jets can be red) damage incurred by the nano-bot ship (from collisions bacterial toxin and phagocyticcells) and the amount of time allowed to complete eachtask (Figure 7)

Although the ldquoshiprsquos computerrdquo character functionsin an advisory capacity offering verbal and textual sup-port to guide student-pilots through the various mis-sions the ultimate course remains under the studentrsquoscontrol Help screens which appear in the plane of the

Figure 4 The immunology module Diverse scientic disciplines ranging from astronomy to quantum mechanics are also candidates for the

Virtual Explorer

Figure 5 Remote-controlled nanobots These nanobots provide a

vehicle allowing students to interact with the immune system at

microscopic scales

Figure 6 Virtual tools Such tools including a bacterial sample

collection vacuum shown here assist students in performing assigned

tasks

Dean et al 509

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

screen upon user command contain information that isessential to understanding the tasks to be performedincluding visual simulations as well as informationabout cells and proteins encountered in the simulation(Figure 8)

Full-motion video animation provides outlines bothof the relevant immunology and of the specic tasksfrom a third-person perspective providing crucial sup-port for students in understanding their intended roles(Figure 9)

Additionally students can pause the simulation at anytime to access database information and simulation con-trols through a simple pop-up menu system (Figure 10)

In this manner mission outlines help screens andanimated mission briengs can be reviewed throughoutthe simulation Added text and spoken support serves toaugment the visual cues that are provided in missionbriengs and help screens For those students who con-tinue to have difculty a ldquohintrdquo functionality is alsoavailable which provides explicit instructions for thetask at hand and becomes increasingly specic as thestudent continues to have difculty and requests addi-tional help It can be reviewed as needed for assistance

in completing the mission Overall this multifacetedhelp system has played a key role in making the simula-tion accessible and relevant to a broad target audienceIt provides students with sufcient information to makethe Virtual Explorer accessible to inexperienced users

Figure 7 An optional display keeping the user updated about the nanobot status (left to right hull structural integrity protein dye jets

remaining current viewing scale and time remaining for current task)

Figure 8 An example of the help screens providing students with more-detailed information about each cell or protein they encounter

Figure 9 Full-motion video animation complementing audio and

textual instructions in introducing students to assigned tasks

510 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

yet without sacricing the challenge that retains the in-terest of more-advanced users

The Virtual Explorerrsquos immunology module currentlycontains two interactive missions (Figure 11) Followingthe brief introductory movie the user is given a trainingmission in which the user can explore and observe thesite of a bacterial infection and must collect a bacterialspecimen for analysis (Figures 12 and 13)

This rst mission introduces the user to the look andfeel of the virtual environment and also allows familiar-ization with the controls Students are also challengedwith phagocytic components of the innate immune sys-tem (such as neutrophils) and must master appropriatepiloting skills to complete this mission Upon complet-ing this mission the student can decide to emulate oneof several white blood cells (currently only the helper Tcell is available) and he or she must use the nanobot tofulll this characterrsquos role in an immune response In theldquoHelper T Cell Missionrdquo we present a compromisedimmune system that the student can ldquorepairrdquo by pilot-ing a small nanobot ship in such a way so as to fulll therole of a helper T cell in a humoral immune responseThe inherent complexity of the immune system how-

ever makes it impossible for one mission to touch uponthe entire range of material and issues that are presentedto students in an immunology course Eventually wehope that others will go beyond this work and add mis-sions that detail the involvement of other componentsof the immune system which can be explored throughthe individual viewpoints of those components Ideallysuch future missions (such as ldquokiller T cellrdquo or ldquoneutro-philrdquo missions) would expand upon the helper T cellmissionrsquos focus and include additional facets of immu-nology such as the innate and cell-mediated immuneresponses

Mission outlines were scripted to maximize user inter-action and freedom while still providing sufcient sup-port to guide even those users with no immunologybackground Missions are divided into individual tasksthus establishing a series of mini-goals which are pre-sented to the user in a scavenger-hunt fashion

Preliminary user feedback revealed that clear missionoutlines must not only be presented before each task (toprovide clear instructions for that task) but must also becontinually available for review during task executionAlthough the mission outlines and help screens havebeen made clear and simple the virtual environmentshave also been carefully constructed to show as muchrelevant detail as possible Although much of the simu-lationrsquos visual detail is not referenced in the mission out-lines (Figure 14) we have found that providing visualaccuracy is essential to avoid misleading users who havelimited immunology backgrounds and to maintain thesimulationrsquos relevance for more-experienced users Adetailed Website provides additional scientic informa-tion about each of the models in a glossary format

4 Educational Content

We chose immunologymdash one of the most complexsubjects studied by students of biology and medi-cinemdashas the subject for the rst module because it pre-sents unique visualization challenges Its processes occursimultaneously in diverse locations of the body and of-ten on time and length scales that although too smallto be directly perceptible still vary over several orders ofmagnitude Consequently the study of basic immu-

Figure 10 The familiar pop-up menu system providing easy access

to nanobot functions for novice users

Dean et al 511

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

nology presents several common conceptual pitfallswhich we feel can be claried with properly imple-mented interactive virtual environments The compart-mentalization of instructional material that is requiredfor the efcient organization of a textbook makes it dif-cult for students to gain an overall ldquoroad maprdquo of theimmune response while still retaining a sense of the de-tails of each microenvironment Thus processes andmicroenvironments are usually studied individually sothat each can be explored in detail but the systemic re-lationship among these details often remains difcult toconceptualize

One common misunderstanding that interactive 3-Dgraphics are particularly well suited to clarify is the con-cept of relative scale Textbooks and other static teach-ing materials are inherently limited in their abilities tosimultaneously show microscopic details and the largermacroscopic systems within which they operate

Consequently textbooks and the like are often unableto clearly represent the vast scale differences that are key toimmunology (Figure 15) For instance immunology textsoften utilize schematic diagrams that depict cell surfaceproteins that are oversized and underpopulated by severalorders of magnitude Although these diagrams are useful

Figure 11 The immunology module allowing the student to select from missions that emulate the roles of key players in the immune system

as well as an introductory training mission

Figure 12 Detail from the training mission A shard of glass creates an opportunity for bacteria to enter the body

512 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

for conveying cell-protein identity and for suggesting themediation of cell-to-cell interactions through these pro-teins students are unable to gain a sense of how muchsmaller surface proteins are than typical cells Additionallythe implications in many diagrams that cell-to-cell interac-tions can be mediated by single surface proteins are inher-ently misleading (Figure 16)

The concept of relative concentration provides addi-tional conceptual challenges that are similar to thoseencountered in the exploration of relative scale Stu-dents are often required to memorize lists of averageconcentrations but without a visual representation ofthese numbers it is very difcult to understand the im-plications of ratios which also can vary by several ordersof magnitude (Figure 17)

For example in healthy individuals red blood cells

outnumber white blood cells by a ratio of almost 700 to1 Similarly IgM and IgD surface receptors are typicallyseveral times as abundant as MHC Class I and Class IIproteins on the surfaces of mature B cells

Interactive 3-D graphics can provide students with avisual model that helps them gain a basic understandingof the relative frequency of occurrence of different com-ponents Certain components however are so rare thatwe are required to exaggerate measured concentrationsin our VR presentation simply to include even a fewspecimens For example the relative concentrations ofmonocytes and granulocytes in the bloodstream are solow that they could appear to be virtually nonexistentamong the many red blood cells The representation ofimportant constituents with vanishingly small concen-trations requires that we include a few specimens in the

Figure 13 Results summaries concluding each task with an update on the current status of the immune system and providing an overview

of the next task Additionally students can pause the simulation at any time to access database information and simulation controls through a

simple pop-up menu system (Figure 10)

Figure 14 Text outlines of each task augmented by full-motion video and available to students for review throughout each mission

Dean et al 513

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

simulation to remind the student of their essential rolesAlthough we would have preferred to have shown exactconcentrations we were limited by available computa-tional power

Another area that is particularly enhanced by interac-tive 3-D graphics is the description of shape and struc-ture The characteristic shapes of cells proteins andreceptors have critical implications for binding func-tion and identication Structural differences betweenMHC Class I and Class II for example are critical indetermining the nature of the immune response Also

lymphocytes are very difcult to distinguish visuallyalthough such discrimination is often critical to the un-derstanding of an immune response

ldquoVirtual dyesrdquomdashwhich simulate the binding of mono-clonal antibody dyes to the surface proteins of thesecellsmdashallow the students to quickly identify subsets of Band T cells in their native environment (Figure 18) Ad-ditionally static teaching materials such as textbooksoften fail to remind students of the dynamics of the sys-tems being studied Cell surfaces for example arehighly uid and dynamic in nature and surface proteinsare often free to migrate and diffuse across the surface

A complete immune response involves a complex se-ries of steps and interactions (Figure 19) For examplethe immune response to a bacterial infection might in-volve immediate inammation at the site of infectionand lymphocyte activation in some subset of the lymphnodes or spleen which is then followed by an antibodyand complement response and so on One commonmisconception involves the locations of the immuneresponse the primary adaptive immune response is actu-ally mediated in the lymph node rather than at the siteof infection (Figure 20) Because the processes in animmune response occur at several different locations inthe body and involve important processes at several dif-ferent length scales the interactive visual simulation ofthese processes is a potentially unique aid to under-standing We therefore believe that immunologyrsquos visu-

Figure 15 Differing scales Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics

(left to right blood vessel at 2000 3 magnication cell surface at 1000000 3 magnication)

Figure 16 Surface proteins These proteins allow for recognition

and signaling between cells and are often misrepresented by

immunology textbooks in both scale and population

514 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

alization challenges make it especially well suited todemonstrate the benets of interactive 3-D graphics foreducation

5 Hardware Con guration

The Virtual Explorer is currently running on afour-processor Silicon Graphics Power Onyx Thislevel of performance allows us to render in real time

six independent video signals which are split by anMCO board to drive three contiguous displays in ste-reo while still supporting well-populated virtual envi-ronments and fast frame rates Rapid advancement incomputer hardware leads us to believe that this levelof computer graphics performance will be available atthe educational and consumer levels in the near fu-ture In parallel we have developed a version of oursystem for the Silicon Graphics O2 workstation (a

Figure 17 Virtual Explorerrsquos depiction of the bloodstream helping to clarify issues of relative cell size and population

Figure 18 Protein dye jets allowing students to visually identify different types of white blood cells based on their surface protein

characteristics

Dean et al 515

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

$5000-$10000 platform) as well as for variousother Silicon Graphics workstations The exibility ofthe software framework has allowed us to easily adaptthe Virtual Explorer for most Silicon Graphics IRIX-based hardware systems and their supported user in-put devices (See Figure 21)

The Virtual Explorer installation in our lab is en-

closed in a small soundproof theater (approximately 4 mby 6 m) and employs three 52 in rear-projection con-sumer-grade television screens arranged at 120 deg an-gles creating a large window into the virtual environ-ment (See Figure 22)

The graphics are driven by a four-processor SiliconGraphics Power Onyx with RealityEngine2 graphics

Figure 19 Full-motion video animation supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to

give students a more comprehensive look at an immune response (left to right the nanobot facilitates an immune response by emulating a

Helper T cell shown here docking with a B cell a complement cascade helps to carry out the nal stage of an immune response)

Figure 20 Lymph nodes Although often misunderstood or unfamiliar to students lymph nodes take center stage as the foci of adaptive

immune responses

516 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

and two RM4 raster managers The Onyx uses an MCOboard to split the video signal into six independentchannels and stereoscopic multiplexers combine thesechannels into the three eld-sequential stereo channelsthat are displayed on the three large TV screens De-pending upon the available graphics hardware and thelevel of processor performance the software can alsosupport several other combinations of stereo and monovideo channels (See Figure 23)

Field-sequential stereo LCD shutter glasses (Figure24) which are synchronized to the video eld frequencywith two infrared transmitters allow multiple studentsto experience the virtual environment simultaneouslyAlthough we experimented with several stereo videosystems we ultimately selected the VRex Mux-1 multi-plexer system because of its support of the NTSC videostandard and its relatively low cost Initially we alsoconsidered using a head-mounted display but preferredthe greater versatility comfort and ability to handlelarge numbers of users that our current large-screen sys-tem provides It presently accommodates approximatelyfteen observers and this capacity is theoretically lim-ited only by the range of the infrared transmitters (ap-

proximately 10 ft to 12 ft) and the size of the viewingroom

6 User Interface

Depending upon the requirements of the physicalinstallation the Virtual Explorer system can accommo-date multiple user input devices To be effective theinterface paradigm must be easily understandable espe-cially by nontechnical users We believe that acceptableuser input devices must provide a familiar interface thatis relatively simple and easily recognized so that studentscan focus on interacting with the simulation and not onmastering the controls (Figure 25)

We are currently using a CH Products force-feedbackightstick and throttle whichmdashin addition to providingan interface that is already found in many computervideo gamesmdashalso provides the level of control neces-sary to successfully navigate in a dynamic three-dimen-sional environment (Figure 26) Force-feedback capabil-ities allow properties of the environment (such asviscosity) to be tactually communicated to the user andenhance the userrsquos experience of immersion in the vir-tual environment by reecting ship collisions speedand acceleration Although joystick control is not veryprocessor intensive the scarcity of joystick-type input

Figure 22 Three large-screen rear-projection monitors creating a

wraparound viewport into the virtual world

Figure 21 The Virtual Explorer software in our most expansive

version running on a four-processor Silicon Graphics Power Onyx

which controls the interactive 3-D graphics and coordinates the

simulation Six-channel video output from the Power Onyx drives three

large-screen displays that form a wraparound viewport into the virtual

world (Figure 22) Four-channel spatialized sound is generated by a

sound server running on an SGI Indigo2 Extreme which communicates

with the Onyx through TCPIP User input from a force-feedback

joystick is processed through a Windows PC which also communicates

with the Onyx via TCPIP (See Figure 27) Another version runs on an

individual single-processor SGI computer

Dean et al 517

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

devices for SGI computers led us to choose this systemwhich is driven by a Windows NT PC communicatingwith the Onyx via TCPIP (Figure 27) AdditionallyVirtual Explorer also supports the Nintendo 64 control-ler (connected directly to an SGI serial port with anadapter box) and Microsoftrsquos Sidewinder ForceFeedbackPro Joystick

Navigating the nanobots has proven to be the most

challenging issue for users with limited computer gamingexperience Although wersquove found that a certain degree ofdifculty in navigation is essential in maintaining excite-ment for experienced users it was also clear that inexperi-enced users must also be able to control the most basicfunctions of the craft simply to complete the assigned mis-sions Mechanisms for obtaining additional help and in-

Figure 23 The Onyx generating six-channel video (RGBS) which is processed through RGBS to composite video encoders (CV-233)

Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens which are displayed on large rear-

projection displays Infrared transmitters which are connected to each of the outside monitors synchronize stereo shutter glasses to the 60Hz

video eld frequency

Figure 24 Field-sequential stereo shutter glasses providing a full

three-dimensional experience Figure 25 Stereo shutter glasses and large screen displays combine

with a familiar force-feedback joystick and throttle to provide an

interactive and immersive learning experience

518 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

structions had to be made easily understandable andreadily identiable Creating a simple hardware-softwareinterface that was easy to learn and operatemdashyet that stillprovided access to the many controls required by the userduring the simulationmdashproved to be one of the more per-sistent design challenges that we encountered Many usersnd it difcult to remember the functions of many rela-tively nondescript buttons (such as may exist when eachbutton controls a separate function)

In an early attempt to deal with this problem weadded a speaker-independent speech-recognition fea-ture to the software This feature was supposed to as-sume the burden of controlling many nanobot auxiliaryfunctions Based upon commercially available speech-recognition software the software listens for verbalcommands such as ldquocomputer start enginesrdquo and relaysthe appropriate signal to the simulation We quickly dis-covered several problems however which convinced usto pursue other solutions The main problem was thenoisy environment within which Virtual Explorer typi-cally runs the system we tested requires that the envi-ronment be virtually free of ambient background noiseVirtual Explorer however generates substantial back-ground audio (engine hum blood-ow pulse and thelike) which made the speech recognition substantiallyless accurate and essentially incompatible

Ultimately a much more modest solution provedmost successful in providing students with the option ofa simplied user interface while still maintaining thesame level of user control The Virtual Explorer soft-ware contains a menu-based control system (similar tofamiliar PC GUIs) that can be used in place of the joy-stick buttons to access online help and to control nano-bot auxiliary functions Users who are more comfortablewith this interface can use it instead of the joystick but-tons although the joystick is still used for navigation

Audio in Virtual Explorer is carefully designed to en-hance the userrsquos sense of immersion as well as to allowstudents to better orient themselves within the virtual envi-ronment Background music (based on the ProTrackerstandard) aids students in distinguishing among differentscales and environments Students can also identify spatialrelationships between the ldquoshiprdquo and the objects in thevirtual environment by 3-D sound and thereby benetfrom a heightened sense of immersion and overall en-hanced awareness of the dynamics of the environmentOur audio system supports multiple sound le formats andmultiple independent audio channels (based on hardwarecapabilities) which allow for both global (mono) and lo-calized sound effects We have created our own spatializedaudio algorithm which allows us to successfully mimic 3-Daudio including simple panning localization and Dopplershift effects The audio system can be controlled either bythe same computer as the main simulation or a secondaryIRIX-based system that is connected to the graphics hard-ware via TCPIP Currently the audio server is runningon a Silicon Graphics Indigo2 because our Onyx lackssound output Four independent audio channels providequadraphonic sound and drive four high- and midrangespeaker systems two directly driven bass speaker systemsand two powered long-excursion subwoofers for visceraleffects

7Software Design

The Virtual Explorer software is written in C++based upon the IRIS Performer toolkit Although weconsidered other development options such asOpenGL Open Inventor VRML and proprietary pack-ages such as World ToolKit we ultimately chose Per-

Figure 26 ForceFX force-feedback joystick and throttle from CH

Products provide a ightstick-style navigation interface

Dean et al 519

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

former for several reasons it allows us to freely redistrib-ute the generated code it provides a high-level graphicsAPI while still allowing direct access to GL and lower-level rendering details and it supports multiprocessingWe constructed the immunology module within the

Virtual Explorer software framework which is con-structed on top of Performer This should facilitate eas-ier and quicker development of additional missionsmodules and educational worlds

The basic graphics-rendering pipeline for Virtual Ex-

Figure 27 User input from a Windows PC and audio output to an SGI Indigo2 Extreme linked to the Onyx by Ethernet and communicating

with the Virtual Explorer software through TCPIP

Figure 28 Four-channel audio generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the

Onyx through TCPIP over an Ethernet connection Front and rear audio signals are processed through separate ampliers (AVR-10) resulting in

effective spatialized sound Four satellite speakers two passive subwoofers and two powered subwoofers provide a wide dynamic range

520 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

plorer is subdivided into six threads of execution basedupon Performerrsquos multiprocessing framework applica-tion cull draw database intersection (object collisiondetection) and user IO The six threads can run onone to four of the available processors depending uponmachine conguration The application thread controlsthe high-level simulation including mission progressobject motions and simple dynamics calculation (suchas the translational and angular momentum of the shipand other objects) The database user IO and inter-section threads run asynchronously from the applicationthread to maintain a constant and acceptable frame rate

Virtual Explorer contains three basic scene typesblood vessel (which is essentially linear) cell surface (es-sentially planar) and lymph node (volume-oriented)(See Figure 2) Variables such as clip-plane depth fogeffect global lighting characteristics database pagingparameters and motion models for the ship can be ad-justed to differentiate between individual scenes Scenesare created based on a specied combination of xedgeometry and procedural scene generation

Each scene has specic information about xed ge-ometry such as the shell of the lymph node the nano-bot extraction needle or the shape and position of theblood vessel Additional scenery is created quasi-ran-domly and cached when the application is launchedbased on variables such as cell population and averageconcentrations This cached scenery can be dynamicallyrearranged during the simulation Earlier versions of thesoftware included actual dynamic generation of sceneryduring the simulation but that technique proved to betoo processor intensive to maintain a sufcient level ofgraphics performance A voxel-based paging schemedynamically recongures and pages cached geometry asneeded during the simulation allowing large sceneswith large amounts of geometry to be simulated with-out sacricing graphics performance and frame rate Al-though the overall complexity varies signicantly be-tween scenes most scenes contain between 3000 and8000 textured polygons per frame The RealityEngine2allows us to maintain steady six-channel video with aframe rate of approximately 20 Hz

The simulation contains biologically accurate scalemodels of over thirty different cells and proteins that are

important to the study of immunology Cells have beenmodeled at the scale of 12000 and proteins at11000000 which is consistent with the two viewingscales available to the user We have created these mod-els and dened their interactions based upon availablemicroscopy images x-ray crystallography and NMRstructures as well as other structural data Each modeltypically contains ve geometric levels of detail and hasan associated information le with the dening charac-teristics that are used by the simulation Additionallyeach model is accompanied by a help screen containinginformation of interest to the student (Figure 8) Tech-niques such as object sequences (which allow for mor-phing models) and dynamic texture shifting (which al-lows for protein ldquodyeingrdquo) show biologicalcharacteristics and improve the interaction between theuser and the individual objects in the simulation

8 Conclusions

The response from the educational scientic andcomputer graphics communities has been very positiveMore than 7000 people have already participated indemonstrations (Figure 29) We are distributing thecomplete source code and installer scripts for a variety ofSilicon Graphics computers with illustrated instructionmanuals included as a CD-ROM Several science andtechnology museums have licensed Virtual Explorer for

Figure 29 Electric Garden at SIGGRAPH rsquo97

Dean et al 521

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

permanent exhibits and it has already been installed inthe Heinz Nixdorf MuseumsForum (Figure 30) in Pad-erborn Germany (for which we wrote a German versionof the text and audio track) and the Tech Museum ofInnovation (Figure 31) in San Jose California Otherinstallations are in the planning stages Future directionsfor study may include characterization of the educa-tional benets of interactive three-dimensional virtualenvironments like Virtual Explorer over interactive yetnon-immersive two-dimensional systems

Further information on the system and how to obtaina video demonstration of Virtual Explorer (as well as theCD-ROMs of the source code and instruction manuals)can be obtained from the Virtual Explorer Website atwww-wilsonucsdeduve

Acknowledgments

We would like to thank the following individuals for their in-valuable contributions to the Virtual Explorer project AprilApperson (adviser for immunology) School of Medicine Uni-versity of California San Diego (La Jolla CA) Jon Chris-tensen (former project director) Painted Word Inc (Cam-bridge MA) Glen D Fraser (adviser for interactive 3-Dgraphics) Montreal Quebec Canada David Goodsell (advis-er for cellular and molecular visualization) Scripps ResearchInstitute (La Jolla CA) Mizuko Ito (adviser for educational

interface) Institute for Research on Learning (Menlo ParkCA) and Stanford University (Stanford CA) Teresa Larsen(adviser for biology and computer animation) Scripps Re-search Institute (La Jolla CA) Barbara Sawrey (adviser formultimedia education and visualization) Department ofChemistry and Biochemistry UCSD (La Jolla CA) GabrieleWienhausen (adviser for multimedia education and visualiza-tion) Department of Biology University of California SanDiego (La Jolla CA) and Michael Zyda (adviser for interac-tive 3-D graphics) Department of Computer Science NavalPostgraduate School (Monterey CA)

References

Bogen J E (1972) A giant walk-through brain Bulletin ofthe Los Angeles Neurological Society 37(3)

Dean KL Asay-Davis X S Finn E M Friesner J ANaylor B J Wustner S R Fisher S S amp Wilson K R(1998) Virtual Explorer Creating interactive 3D virtualenvironments for education In M T Bolas S S Fisherand J O Merritt (Eds) Stereoscopic Displays and VirtualReality Systems V Proceedings of SPIEmdashthe InternationalSociety for Optical Engineering 3295 (p 429) BellinghamWA

Dean K Asay-Davis X Finn E Friesner J Naylor BWustner S Fisher S amp Wilson K (1997) Electric gar-den The Virtual Explorer Computer Graphics 31(4) 16-17 81

Figure 31 Life Tech Theater at the Tech Museum of Innovation in

San Jose California

Figure 30 SoftwareTheater at HeinzNixdorf Museumsforum in

Paderborn Germany

522 PRESENCE VOLUME 9 NUMBER 6

Dean K L Finn E M Friesner J A Naylor B J Wust-ner S R Wilson K R amp Fisher S S (1997) Electricgarden Virtual Explorer In R Hopkins (Ed) Visual Pro-ceedings The Art and Interdisciplinary Programs ofSIGGRAPH 97 (p 110) New York Association for Com-puting Machinery

Dede C Salzman M C amp Loften B (1996) Sciencespace Virtual realities for learning complex and abstractscientic concepts In Proc IEEE Virtual Reality AnnualInternational Symposium (pp 246-253)

Hoffman H M amp Murray M (1999) Anatomic Visual-izeR Realizing the vision of a VR-based learning environ-ment In Medicine Meets Virtual Reality The Convergence of

Physical and Informational Technologies Options for a NewEra in Healthcare (pp 134-140) IOS Press

Kuby J (1997) Immunology (3rd ed) New York W HFreeman and Company

Rose H amp Billinghurst M (1996) Zengo Sayu An immer-sive educational environment for learning Japanese (Techni-cal report) Seattle University of Washington HumanInterface Laboratory of the Washington TechnologyCenter

Salzman M C Dede C Loftin R B amp Chen J (1999)A model for understanding how virtual reality aids complexconceptual learning Presence Teleoperators and Virtual En-vironments 8(3) 293-316

Dean et al 523