Augmented and Mixed Reality - uma/ آ  REAL ENV VR Mixed Reality Figure 1: Continuum of

  • View
    0

  • Download
    0

Embed Size (px)

Text of Augmented and Mixed Reality - uma/ آ  REAL ENV VR Mixed Reality Figure 1: Continuum of

  • Augmented and Mixed Reality

    Uma Mudenagudi

    Dept. of Computer Science and Engineering,

    Indian Institute of Technology Delhi

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Outline

    Introduction to Augmented Reality(AR) and Mixed Reality(MR)

    A Typical AR System

    Issues in AR

    Different methods of Augmenting object

    Algorithms in AR

    Abstract Model of AR/MR

    Initial Results Obtained

    Summary

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Introduction to AR and MR

    AR combines real and virtual objects in real environment

    Virtual objects(3D model/images/video) are merged with real environment

    Milgram et al(1994) described the relation between AR, MR and VR

    AR AV

    REAL ENV VR Mixed Reality

    Figure 1: Continuum of real and virtual environment

    MR spectrum lies between the extremes of real life and Virtual reality(VR) . Views of the real world are combined in some proportion with views of a virtual environment

    Augmented Virtuality- Enhances the virtual experience by adding elements of the real environment

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Typical AR system

    AR SYS = Computer vision + Computer graphics + User interfaces

    � � �� � �

    � � �� � �

    � � �

    � � �� � �

    � � �� � �

    � � �

    PZT

    ALIGN GRAPHICS CAMERA TO REAL

    GRAPHICS RENDERING

    REAL SCENE

    VIR OBJ

    GRAPHICS IMAGE

    GRAPHICS IMAGE

    WORLD CO−ORD

    REAL IMAGE CO−ORD

    CO−ORD

    SCENE CO−ORD

    VIR OBJ CO−ORD

    GRAPHICS CO−ORD

    VIDEO IMAGE

    CAMERA POSITION

    Figure 2: Typical AR system

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Issues in AR:Registration

    Process of estimating an optimal transformation between two images(also known as spatial Normalization)

    To align the virtual object to real objects in 3D

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Issues in AR:Registration contd..

    Most critical requirement of AR system :Since human visual system is very good at detecting even small mis-registration

    Methods: No generalized method of registration for all the type of augmentations

    Static errors:Optical distortion, mechanical misalignment and incorrect viewing parameters

    Dynamic errors:System delays Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Issues in AR: Tracking

    Tracking : View point tracking as the view point moves

    Tracked viewing pose defines the AR alignment and registration

    Issues: Foreshortening, Scaling, Occlusions

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Issues in AR: Modeling

    Modeling : Modeling of the destination with Texture and extraction of the 3D model from the source environment

    Single view : Single view reconstruction

    Multiple view reconstruction: two view

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Source:3D Model, Destination:Single Image

    Destination Image Augmented image

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Source:Multiple Images, Destination:Single

    Image

    Source Images(2/7)

    Augmented image

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Source:Multiple Images, Destination:Multiple

    Images

    Source Images(2/20)

    Augmented image

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Source:3D model, Destination:Video

    Source1,

    Augmented1

    Source2,

    Augmented2,

    Augmented2,

    Augmented2 and

    Augmented2

    Augmented3

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Possible ways of augmenting

    Source

    3D Model: Model is given

    Single Image: Extract 3D/2D Model of the object and Texture

    Multiple Images: Extract 3D Model of the object and Texture

    Video : Extract 3D Model of the ob- ject and Texture

    Destination

    3D Model:Model of destination

    Single Image: Single view Reconstruction with Texture

    Multiple Images: Multiple view reconstruction with Texture

    Video: Tracking and Registration of destination

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Mixed Reality

    Mixed Reality Example

    Microsoft research lab: SIGGRAPH 2004

    Video view interpolation using layered approach

    Color Segmentation based stereo algorithm

    Mattes near discontinuities

    Two layer compressed representation to handle matting

    Major disadvantages:Synchronizing many cameras, and acquiring and storing of images

    Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Move matching method

    Source:3D Model, Destination:Video

    Given by Zisserman et.al

    Initialization: Manually indicate the planar region in image-0. (corner detection and matching are restricted to this region) Detect interest points in image 0 Initialize camera calibration K

    Steady state: computing H from frame i to i

    1

    Detect interest points in two images say

    X j

    � N j �1 and�

    X1k

    � N1 k � 1

    Match interest points which maximizes the cross correlation in 7x7 mask: x j

    �� x1k Phd seminar series - Uma Mudenagudi - 14.10.2004

  • Move matching method contd..

    Randomly sample subset of four matched pairs and compute homography. Each candidate H is tested against all the correspondence by computing distance between x1 and Hx. Choose H for which most pairs are within the threshold Compute pose from H i

    1 i

    Result set 1: Source, Tracking and Augmented

    Result set 2: Source Tracking and Augmented

    Phd seminar ser