MUltimo3-D: a Testbed for MUltimo3-D: a Testbed for Multimodel 3-D PC Multimodel 3-D PC

Presenter: Yi Shi & Saul RodriguezPresenter: Yi Shi & Saul Rodriguez

March 14, 2008March 14, 2008

OutlineOutline

► Motivations and conceptsMotivations and concepts

► Operating SystemOperating System

► 3-D Display3-D Display

► Input DevicesInput Devices

► ApplicationsApplications

► DiscussionDiscussion

Motivations and concepts in 3D PCMotivations and concepts in 3D PC

► 3D display3D display

► Contact free on human bodyContact free on human body

► Higher bandwidth: hands, head, eyes, etc.Higher bandwidth: hands, head, eyes, etc.

► Intelligent intention interpretationIntelligent intention interpretation

► Less human strainLess human strain

Operating System of mUltimo3-DOperating System of mUltimo3-D

► Developing toolDeveloping tool

Based on a C++ class library, which is based on Based on a C++ class library, which is based on

and extended the virtual reality development and extended the virtual reality development

environment dVS/DIVISION from PTC Ltd.environment dVS/DIVISION from PTC Ltd.

Operating System of mUltimo3-D Operating System of mUltimo3-D (cont.)(cont.)

► Interface agencyInterface agency Collector agentCollector agent

► Collects and interprets data on the users and makes this Collects and interprets data on the users and makes this

information available to other agents.information available to other agents.

► All the data collected is stored in the form of user model for All the data collected is stored in the form of user model for

later uselater use

Placer agentPlacer agent► Arranges the media objects in the interaction space.Arranges the media objects in the interaction space.

► It avoids overlapping and collision of objects.It avoids overlapping and collision of objects.

► Gradually move away and remove obsolete objectsGradually move away and remove obsolete objects

Operating System of mUltimo3-D Operating System of mUltimo3-D (cont.)(cont.)

► Interface agency (cont.)Interface agency (cont.) Starter agentStarter agent

► Automatic starts applications.Automatic starts applications.

Visualization agentVisualization agent► Critical action confirmation, undo/redo.Critical action confirmation, undo/redo.

► Status display: planning, working, satisfied, surprised, Status display: planning, working, satisfied, surprised,

confusedconfused

Figure 1. Communication states of the avatar: idle state, confirmation request and inquiry.

3-D Display of mUltimo3-D3-D Display of mUltimo3-D

► Autostereoscopic displaysAutostereoscopic displays

Lenticular lens based 3D display.Lenticular lens based 3D display.

Figure 2. 3D display using an array of lenticular lenses to channel the left and right eye images to the respective eye (L and R refer to image stripes of the left and right images of a stereo pair).

3-D Display of mUltimo3-D (cont.)3-D Display of mUltimo3-D (cont.)

► Face to FaceFace to Face

The display screen is placed on a platform that turns The display screen is placed on a platform that turns

around to continuously orient the screen to the user’s around to continuously orient the screen to the user’s

eyes.eyes.

The L/R picture changes when the screen moves which The L/R picture changes when the screen moves which

avoids the spatial and optical distortions and makes the avoids the spatial and optical distortions and makes the

view angle wider.view angle wider.

3-D Display of mUltimo3-D 3-D Display of mUltimo3-D (cont.)(cont.)

► Full focus problemsFull focus problems Physical and virtual positions are different, thus undesirable Physical and virtual positions are different, thus undesirable

when supporting direct virtual object manipulation with for when supporting direct virtual object manipulation with for

example, hands.example, hands.

Figure 3. With conventional 3D displays, interaction between virtual and real objects in the grasp area is affected by blurred effects, either the virtual object is sharp (left) or the real object is sharp (middle), depending on what object the user is looking at. On the accommodation display both objects are seen in full focus (right).

3-D Display of mUltimo3-D 3-D Display of mUltimo3-D (cont.)(cont.)

► Possible solutionPossible solution

Accommodation 3-D displayAccommodation 3-D display

Figure 4. The user accommodates and perceives object on the aerial image plane. The Fresnel lens projects the exit pupils of the stereo projector to the left and right eye, respectively (field-lens principle) and, hence, completely separates the constituent stereo images (in the prototype system there is virtually no crosstalk).

Input Devices of mUltimo3-DInput Devices of mUltimo3-D

► Multiple input Multiple input signalssignals

Figure 5. The overall system diagram of the mUltimo3D testbed

Input Devices of mUltimo3-DInput Devices of mUltimo3-D

► Multiple input signalsMultiple input signals

Figure 6.1. The input devices of mUltimo3-D

Input Devices of mUltimo3-DInput Devices of mUltimo3-D

► The mUltimo3-D systemThe mUltimo3-D system

Figure 6.2. The mUltimo3-D system

Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)

►Voice inputVoice input IBM Via VoiceIBM Via Voice is integrated in the mUltimo3-D. is integrated in the mUltimo3-D.

Logox3 speech synthesis software from G-DATA is used as Logox3 speech synthesis software from G-DATA is used as

speech outputspeech output

Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)

► Video head trackerVideo head tracker Skin color is used as tracking featureSkin color is used as tracking feature

Eye blinking is used to locate the positions of eyes on the Eye blinking is used to locate the positions of eyes on the

headhead

Able to cope with changes in size, orientation and Able to cope with changes in size, orientation and

illumination.illumination.

(demo)(demo)

Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)

► Video hand trackerVideo hand tracker Two cameras and infrared are used to capture and Two cameras and infrared are used to capture and

segment handssegment hands

Figure 7. Video hand tracker (the device attached to the keyboard) at the mUltimo3D user interface and the segmented hand.

Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)

► Video hand trackerVideo hand tracker Hand center is determined and Hand center is determined and radial lines are radial lines are

derived from the centerderived from the center

Different line features are used to recognize the hand Different line features are used to recognize the hand

shape pattern as well as movementshape pattern as well as movement

(demo)(demo)

Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)

► Video gaze trackerVideo gaze tracker Used cornea-reflex methodUsed cornea-reflex method

Low-intensity infrared lightLow-intensity infrared light

Center of the pupil and the reflection of the light from the Center of the pupil and the reflection of the light from the

cornea are found to determine the direction of the eyes gazing cornea are found to determine the direction of the eyes gazing

atat

(demo)(demo)

Figure 8. The eye image captured by the gaze tracker (left) and an enlarged part of the eye showing the pupil and light reflections on the cornea (right).

Applications in mUltimo3-DApplications in mUltimo3-D

Figure 9. The placer agent arranges the display of the media objects in the interaction space. The tool objects (cubes) show icons for gaze-controlled selection of applications. The cubes turn overproportionally when the head moves, so that all sides become visible with slight head movements.

Applications in mUltimo3-D (cont.)Applications in mUltimo3-D (cont.)

Figure 10. Screenshots of the Internet browser (left) and the multimodal 3D CAD application (right). By changing the viewing perspective (head movement), hidden objects can be viewed and addressed by eye gaze. The 3D CAD object shelf covering parts of the video windows has currently been moved to the front by gaze interaction.

DiscussionDiscussion

► Combining and optimizing multimodel information Combining and optimizing multimodel information in the best wayin the best way

► Real-time database updatingReal-time database updating

► More user testing and feedbackMore user testing and feedback

Thank you!Thank you!