Augmented and Mixed Reality - ERNETuma/phdSem141004.pdf · REAL ENV VR Mixed Reality Figure 1: Continuum of real and virtual environment MR spectrum lies between the extremes of real

Augmented and Mixed Reality

Uma Mudenagudi

Dept. of Computer Science and Engineering,

Indian Institute of Technology Delhi

Phd seminar series - Uma Mudenagudi - 14.10.2004

Outline

Introduction to Augmented Reality(AR) and MixedReality(MR)

A Typical AR System

Issues in AR

Different methods of Augmenting object

Algorithms in AR

Abstract Model of AR/MR

Initial Results Obtained

Summary


Introduction to AR and MR

AR combines real and virtual objects in real environment

Virtual objects(3D model/images/video) are merged withreal environment

Milgram et al(1994) described the relation between AR, MRand VR

AR AV

REAL ENV VRMixed Reality

Figure 1: Continuum of real and virtual environment

MR spectrum lies between the extremes of real life andVirtual reality(VR) . Views of the real world are combined insome proportion with views of a virtual environment

Augmented Virtuality- Enhances the virtual experience byadding elements of the real environment


Typical AR system

AR SYS = Computer vision + Computer graphics + Userinterfaces

� � ��

� � ��

PZT

ALIGN GRAPHICS CAMERA TO REAL

GRAPHICSRENDERING

REAL SCENE

VIR OBJ

GRAPHICS IMAGE

GRAPHICSIMAGE

WORLD CO−ORD

REAL IMAGE CO−ORD

CO−ORD

SCENE CO−ORD

VIR OBJCO−ORD

GRAPHICSCO−ORD

VIDEO IMAGE

CAMERA POSITION

Figure 2: Typical AR system


Issues in AR:Registration

Process of estimating an optimal transformation betweentwo images(also known as spatial Normalization)

To align the virtual object to real objects in 3D


Issues in AR:Registration contd..

Most critical requirement of AR system :Since human visualsystem is very good at detecting even small mis-registration

Methods: No generalized method of registration for all thetype of augmentations

Static errors:Optical distortion, mechanical misalignmentand incorrect viewing parameters

Dynamic errors:System delaysPhd seminar series - Uma Mudenagudi - 14.10.2004

Issues in AR: Tracking

Tracking : View point tracking as the view point moves

Tracked viewing pose defines the AR alignment andregistration

Issues: Foreshortening, Scaling, Occlusions


Issues in AR: Modeling

Modeling : Modeling of the destination with Texture andextraction of the 3D model from the source environment

Single view : Single view reconstruction

Multiple view reconstruction: two view


Possible ways of augmenting

Source

3D Model: Modelis given

Single Image:Extract 3D/2DModel of theobject and Texture

Multiple Images:Extract 3D Modelof the object andTexture

Video : Extract 3DModel of the ob-ject and Texture

Destination

3D Model:Modelof destination

Single Image:Single viewReconstructionwith Texture

Multiple Images:Multiple viewreconstructionwith Texture

Video: Trackingand Registrationof destination



Source





Destination







Source





Destination






Source:3D Model, Destination:Single Image

Destination Image Augmented image



Source





Destination






Source:Multiple Images, Destination:Single

Image

Source Images(2/7)

Augmented image



Source





Destination






Source:Multiple Images, Destination:Multiple

Images

Source Images(2/20)

Augmented image



Source





Destination






Source:3D model, Destination:Video

Source1,

Augmented1

Source2,

Augmented2,

Augmented2,

Augmented2 and

Augmented2

Augmented3



Source





Destination






Mixed Reality

Mixed Reality Example

Microsoft research lab: SIGGRAPH 2004

Video view interpolation using layered approach

Color Segmentation based stereo algorithm

Mattes near discontinuities

Two layer compressed representation to handle matting

Major disadvantages:Synchronizing many cameras, andacquiring and storing of images


Move matching method

Source:3D Model, Destination:Video

Given by Zisserman et.al

Initialization:Manually indicate the planar region in image-0. (cornerdetection and matching are restricted to this region)Detect interest points in image 0Initialize camera calibration K

Steady state: computing H from frame i to i

�

1

Detect interest points in two images say

�

X j

� Nj �1 and

�

X1k

� N1

k � 1Match interest points which maximizes the crosscorrelation in 7x7 mask: x j

�� x1k


Move matching method contd..

Randomly sample subset of four matched pairs andcompute homography. Each candidate H is testedagainst all the correspondence by computing distancebetween x1 and Hx. Choose H for which most pairs arewithin the thresholdCompute pose from H i

�

1i

Result set 1:Source, Tracking and Augmented

Result set 2:Source Tracking and Augmented


Method-2,G. Simon et.al

Results Augmented

the first image

computation of initial pose +set of visible model features in

+extraction of key points in thefirst frame

Initialization stage

image k−−>k+1

Tracking of set of visible modelfeatures in the current imageUpdating the set of tracked pts

scene. Trajectory ofbe added in the

Model of the obj to

the object in the scene

Camera parameters3D/2D corr of 4 pts.

Key points are extracted in frame k+1. They are matchedwith the points extracted in k

The view point is computedusing mixing method

The computer generatedobject is added in scene

STEP−1

STEP−2

STEP−3

STEP−4

Figure 3: Reg Method


Augmented views:z-keying method

z-keying method: Given by T. Kanade et.alUses dense depth map as a switchFor each pixel, the z-key switch compares the pixeldepth values of two images, and routes the color valueof the foreground image that is nearer to the camera forthe merged output imageReal and virtual objects will occlude correctlyUses real time stereo-machine:Specifications are

Number of cameras: 2 to 6Frame rate: max 30frames/secDepth image size: up to 256 � 240Disparity search range:60 pixels

Results: Augmented


Abstract Model of AR/MR

Render Image

4D data: Space and Time

3D model of space

Dynamic PartsTemporal Info(real+virtual)

Static Part

QueryF(V,t)

Figure 4: Problem of 4D Fly-through


Problem of 4D Fly-through

Static 3D Model:Reconstructed from set ofimages

Temporal Information: ExtractTemporal information for eachview

4D data:3D space and time

Query:F

�

V � t

�

?

Generate view from the staticand temporal information andanswer the query


Results

Source:3D model and Destination: Still image

Method: Single view reconstruction method

Destination Image

3D model augmented into still image

Augmented movie-1,Augmented movie-2 andAugmented movie-3


Results contd..

Source:Object extracted from video and Destination:Stillimage

Method:

Source: Object is extracted as 2D plane

Destination: Sparse 3D modeling of possible occluders

Registration: Object plane to ground(virtual) planeHomography

Rendering: Modified ray tracing algorithmsource movieAugmented movie


summary

Typical AR system and Issues in AR

Possible augmentation methods and review results

Abstract model of AR and MR

4D fly through problem

Preliminary results to wards the 4D fly through problem


Thank you

THANK YOU


Documents

Augmented and Mixed Reality - ERNETuma/phdSem141004.pdf · REAL ENV VR Mixed Reality Figure 1: Continuum of real and virtual environment MR spectrum lies between the extremes of real