Augmented and Mixed Reality
Uma Mudenagudi
Dept. of Computer Science and Engineering,
Indian Institute of Technology Delhi
Phd seminar series - Uma Mudenagudi - 14.10.2004
Outline
Introduction to Augmented Reality(AR) and MixedReality(MR)
A Typical AR System
Issues in AR
Different methods of Augmenting object
Algorithms in AR
Abstract Model of AR/MR
Initial Results Obtained
Summary
Phd seminar series - Uma Mudenagudi - 14.10.2004
Introduction to AR and MR
AR combines real and virtual objects in real environment
Virtual objects(3D model/images/video) are merged withreal environment
Milgram et al(1994) described the relation between AR, MRand VR
AR AV
REAL ENV VRMixed Reality
Figure 1: Continuum of real and virtual environment
MR spectrum lies between the extremes of real life andVirtual reality(VR) . Views of the real world are combined insome proportion with views of a virtual environment
Augmented Virtuality- Enhances the virtual experience byadding elements of the real environment
Phd seminar series - Uma Mudenagudi - 14.10.2004
Typical AR system
AR SYS = Computer vision + Computer graphics + Userinterfaces
� � �� � �� � �� � �� � �
� � �� � �� � �� � �� � �
PZT
ALIGN GRAPHICS CAMERA TO REAL
GRAPHICSRENDERING
REAL SCENE
VIR OBJ
GRAPHICS IMAGE
GRAPHICSIMAGE
WORLD CO−ORD
REAL IMAGE CO−ORD
CO−ORD
SCENE CO−ORD
VIR OBJCO−ORD
GRAPHICSCO−ORD
VIDEO IMAGE
CAMERA POSITION
Figure 2: Typical AR system
Phd seminar series - Uma Mudenagudi - 14.10.2004
Issues in AR:Registration
Process of estimating an optimal transformation betweentwo images(also known as spatial Normalization)
To align the virtual object to real objects in 3D
Phd seminar series - Uma Mudenagudi - 14.10.2004
Issues in AR:Registration contd..
Most critical requirement of AR system :Since human visualsystem is very good at detecting even small mis-registration
Methods: No generalized method of registration for all thetype of augmentations
Static errors:Optical distortion, mechanical misalignmentand incorrect viewing parameters
Dynamic errors:System delaysPhd seminar series - Uma Mudenagudi - 14.10.2004
Issues in AR: Tracking
Tracking : View point tracking as the view point moves
Tracked viewing pose defines the AR alignment andregistration
Issues: Foreshortening, Scaling, Occlusions
Phd seminar series - Uma Mudenagudi - 14.10.2004
Issues in AR: Modeling
Modeling : Modeling of the destination with Texture andextraction of the 3D model from the source environment
Single view : Single view reconstruction
Multiple view reconstruction: two view
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Source:3D Model, Destination:Single Image
Destination Image Augmented image
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Source:Multiple Images, Destination:Single
Image
Source Images(2/7)
Augmented image
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Source:Multiple Images, Destination:Multiple
Images
Source Images(2/20)
Augmented image
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Source:3D model, Destination:Video
Source1,
Augmented1
Source2,
Augmented2,
Augmented2,
Augmented2 and
Augmented2
Augmented3
Phd seminar series - Uma Mudenagudi - 14.10.2004
Possible ways of augmenting
Source
3D Model: Modelis given
Single Image:Extract 3D/2DModel of theobject and Texture
Multiple Images:Extract 3D Modelof the object andTexture
Video : Extract 3DModel of the ob-ject and Texture
Destination
3D Model:Modelof destination
Single Image:Single viewReconstructionwith Texture
Multiple Images:Multiple viewreconstructionwith Texture
Video: Trackingand Registrationof destination
Phd seminar series - Uma Mudenagudi - 14.10.2004
Mixed Reality
Mixed Reality Example
Microsoft research lab: SIGGRAPH 2004
Video view interpolation using layered approach
Color Segmentation based stereo algorithm
Mattes near discontinuities
Two layer compressed representation to handle matting
Major disadvantages:Synchronizing many cameras, andacquiring and storing of images
Phd seminar series - Uma Mudenagudi - 14.10.2004
Move matching method
Source:3D Model, Destination:Video
Given by Zisserman et.al
Initialization:Manually indicate the planar region in image-0. (cornerdetection and matching are restricted to this region)Detect interest points in image 0Initialize camera calibration K
Steady state: computing H from frame i to i
�
1
Detect interest points in two images say
�
X j
� Nj �1 and
�
X1k
� N1
k � 1Match interest points which maximizes the crosscorrelation in 7x7 mask: x j
�� x1k
Phd seminar series - Uma Mudenagudi - 14.10.2004
Move matching method contd..
Randomly sample subset of four matched pairs andcompute homography. Each candidate H is testedagainst all the correspondence by computing distancebetween x1 and Hx. Choose H for which most pairs arewithin the thresholdCompute pose from H i
�
1i
Result set 1:Source, Tracking and Augmented
Result set 2:Source Tracking and Augmented
Phd seminar series - Uma Mudenagudi - 14.10.2004
Method-2,G. Simon et.al
Results Augmented
the first image
computation of initial pose +set of visible model features in
+extraction of key points in thefirst frame
Initialization stage
image k−−>k+1
Tracking of set of visible modelfeatures in the current imageUpdating the set of tracked pts
scene. Trajectory ofbe added in the
Model of the obj to
the object in the scene
Camera parameters3D/2D corr of 4 pts.
Key points are extracted in frame k+1. They are matchedwith the points extracted in k
The view point is computedusing mixing method
The computer generatedobject is added in scene
STEP−1
STEP−2
STEP−3
STEP−4
Figure 3: Reg Method
Phd seminar series - Uma Mudenagudi - 14.10.2004
Augmented views:z-keying method
z-keying method: Given by T. Kanade et.alUses dense depth map as a switchFor each pixel, the z-key switch compares the pixeldepth values of two images, and routes the color valueof the foreground image that is nearer to the camera forthe merged output imageReal and virtual objects will occlude correctlyUses real time stereo-machine:Specifications are
Number of cameras: 2 to 6Frame rate: max 30frames/secDepth image size: up to 256 � 240Disparity search range:60 pixels
Results: Augmented
Phd seminar series - Uma Mudenagudi - 14.10.2004
Abstract Model of AR/MR
Render Image
4D data: Space and Time
3D model of space
Dynamic PartsTemporal Info(real+virtual)
Static Part
QueryF(V,t)
Figure 4: Problem of 4D Fly-through
Phd seminar series - Uma Mudenagudi - 14.10.2004
Problem of 4D Fly-through
Static 3D Model:Reconstructed from set ofimages
Temporal Information: ExtractTemporal information for eachview
4D data:3D space and time
Query:F
�
V � t
�
?
Generate view from the staticand temporal information andanswer the query
Phd seminar series - Uma Mudenagudi - 14.10.2004
Results
Source:3D model and Destination: Still image
Method: Single view reconstruction method
Destination Image
3D model augmented into still image
Augmented movie-1,Augmented movie-2 andAugmented movie-3
Phd seminar series - Uma Mudenagudi - 14.10.2004
Results contd..
Source:Object extracted from video and Destination:Stillimage
Method:
Source: Object is extracted as 2D plane
Destination: Sparse 3D modeling of possible occluders
Registration: Object plane to ground(virtual) planeHomography
Rendering: Modified ray tracing algorithmsource movieAugmented movie
Phd seminar series - Uma Mudenagudi - 14.10.2004
summary
Typical AR system and Issues in AR
Possible augmentation methods and review results
Abstract model of AR and MR
4D fly through problem
Preliminary results to wards the 4D fly through problem
Phd seminar series - Uma Mudenagudi - 14.10.2004
Thank you
THANK YOU
Phd seminar series - Uma Mudenagudi - 14.10.2004