Presented by : ……..Presented by : ……..
Ramesh Raskar, Greg Welch and Ramesh Raskar, Greg Welch and Henry FuchsHenry Fuchs
University of North Carolina at Chapel University of North Carolina at Chapel HillHill
Presented by : ……..Presented by : ……..
Ramesh Raskar, Greg Welch and Ramesh Raskar, Greg Welch and Henry FuchsHenry Fuchs
University of North Carolina at Chapel University of North Carolina at Chapel HillHill
Seamless Projection Overlaps using Image Warping and Intensity
Blending
Acknowledgements
• Office of the Future group at UNC Chapel HillOffice of the Future group at UNC Chapel Hill
• Andrei State (sketches)Andrei State (sketches)
• SupportSupport
– NSF Science and Technology Center for Graphics NSF Science and Technology Center for Graphics and Visualization, USAand Visualization, USA
– Advanced Networks and Services,National Tele-Advanced Networks and Services,National Tele-Immersion InitiativeImmersion Initiative
– Defense Advanced Research Projects Agency, USA Defense Advanced Research Projects Agency, USA
– Intel CorporationIntel Corporation
• Office of the Future group at UNC Chapel HillOffice of the Future group at UNC Chapel Hill
• Andrei State (sketches)Andrei State (sketches)
• SupportSupport
– NSF Science and Technology Center for Graphics NSF Science and Technology Center for Graphics and Visualization, USAand Visualization, USA
– Advanced Networks and Services,National Tele-Advanced Networks and Services,National Tele-Immersion InitiativeImmersion Initiative
– Defense Advanced Research Projects Agency, USA Defense Advanced Research Projects Agency, USA
– Intel CorporationIntel Corporation
Seamless Projection Overlaps
P1
P2
P3 P1
P2P3
Display Surface
Projectors
Well Defined Display Configuration
Irregular display and projector configuration
Traditional Flexible Setup
Goal• High Resolution, Wide Field of View High Resolution, Wide Field of View
DisplayDisplay
• Overlapping front projection systemOverlapping front projection system
• Seamless displaySeamless display
– Geometric Registration of Geometric Registration of overlapsoverlaps
– Intensity Blending near edgesIntensity Blending near edges
• Irregular display surface Irregular display surface configurationconfiguration
• High Resolution, Wide Field of View High Resolution, Wide Field of View DisplayDisplay
• Overlapping front projection systemOverlapping front projection system
• Seamless displaySeamless display
– Geometric Registration of Geometric Registration of overlapsoverlaps
– Intensity Blending near edgesIntensity Blending near edges
• Irregular display surface Irregular display surface configurationconfiguration
P1
P2P3
Motivation :Office of the Future (UNC ‘98)
• Irregular Irregular surfacessurfaces
• No specific No specific projector projector configuratioconfigurationn
• Irregular Irregular surfacessurfaces
• No specific No specific projector projector configuratioconfigurationn
Other Spatially Immersive Displays
1. CAVE (UIC 1. CAVE (UIC EVL)EVL)
• Non-overlapping Non-overlapping projectionsprojections
• Flat displays wallsFlat displays walls
• Well-defined Well-defined projection projection configurationconfiguration
1. CAVE (UIC 1. CAVE (UIC EVL)EVL)
• Non-overlapping Non-overlapping projectionsprojections
• Flat displays wallsFlat displays walls
• Well-defined Well-defined projection projection configurationconfiguration
Other Spatially Immersive Displays (Cont.)
2. VisionDome 2. VisionDome (Alternate Realities)(Alternate Realities)
• Single Projector with Single Projector with expensive opticsexpensive optics
• Well defined display Well defined display surfacessurfaces
2. VisionDome 2. VisionDome (Alternate Realities)(Alternate Realities)
• Single Projector with Single Projector with expensive opticsexpensive optics
• Well defined display Well defined display surfacessurfaces
Other Spatially Immersive Displays (Cont.)
3. Cylindrical Screens (Trimension, 3. Cylindrical Screens (Trimension, Panoram)Panoram)
• 3 Projectors with side-by-side rectangular 3 Projectors with side-by-side rectangular overlapoverlap
• Well defined display surfaceWell defined display surface
• Precise electro-mechanical setupPrecise electro-mechanical setup
3. Cylindrical Screens (Trimension, 3. Cylindrical Screens (Trimension, Panoram)Panoram)
• 3 Projectors with side-by-side rectangular 3 Projectors with side-by-side rectangular overlapoverlap
• Well defined display surfaceWell defined display surface
• Precise electro-mechanical setupPrecise electro-mechanical setup
Office of the Future (UNC ‘98)
• Irregular surfacesIrregular surfaces
• No specific projector configurationNo specific projector configuration
• Irregular surfacesIrregular surfaces
• No specific projector configurationNo specific projector configuration
Office of the Future Configuration
Projectors
Single User
Irregular Surfaces
User
Top View
Our Approach
• Calibration of display configuration using a Calibration of display configuration using a video cameravideo camera
• Rendering using texture mapping hardwareRendering using texture mapping hardware
• Pair-wise registration and blending of Pair-wise registration and blending of projection overlapsprojection overlaps
• Calibration of display configuration using a Calibration of display configuration using a video cameravideo camera
• Rendering using texture mapping hardwareRendering using texture mapping hardware
• Pair-wise registration and blending of Pair-wise registration and blending of projection overlapsprojection overlaps
Key Features
+ Display surfaces may not be flat wallsDisplay surfaces may not be flat walls
+ Projection axis not orthogonal to displaysProjection axis not orthogonal to displays
+ Flexible projection configurationFlexible projection configuration
+ Projector’s intrinsic or extrinsic parameters Projector’s intrinsic or extrinsic parameters not needed not needed
– Generated image correct for a single ideal Generated image correct for a single ideal viewer’s locationviewer’s location
+ Display surfaces may not be flat wallsDisplay surfaces may not be flat walls
+ Projection axis not orthogonal to displaysProjection axis not orthogonal to displays
+ Flexible projection configurationFlexible projection configuration
+ Projector’s intrinsic or extrinsic parameters Projector’s intrinsic or extrinsic parameters not needed not needed
– Generated image correct for a single ideal Generated image correct for a single ideal viewer’s locationviewer’s location
Calibration
• Step 1 : Geometric RegistrationStep 1 : Geometric Registration
– Pre-distort images so that when projected Pre-distort images so that when projected they appearthey appear
◊ Perspectively correctPerspectively correct
◊ Aligned with neighboring projectionAligned with neighboring projection
• Step 2 : Intensity NormalizationStep 2 : Intensity Normalization
– Blend images from overlapping projectionsBlend images from overlapping projections
• Step 1 : Geometric RegistrationStep 1 : Geometric Registration
– Pre-distort images so that when projected Pre-distort images so that when projected they appearthey appear
◊ Perspectively correctPerspectively correct
◊ Aligned with neighboring projectionAligned with neighboring projection
• Step 2 : Intensity NormalizationStep 2 : Intensity Normalization
– Blend images from overlapping projectionsBlend images from overlapping projections
Calibration Step 1 : Geometric Registration
Warping Images Warping Images
• For irregular display surfacesFor irregular display surfaces
– Desired image is pre-warpedDesired image is pre-warped
– Warping function is found using a wide field Warping function is found using a wide field of view camera (WFOV)of view camera (WFOV)
• For flat display surfacesFor flat display surfaces
– Projected images related by a 3x2 Projected images related by a 3x2 transformationtransformation
Warping Images Warping Images
• For irregular display surfacesFor irregular display surfaces
– Desired image is pre-warpedDesired image is pre-warped
– Warping function is found using a wide field Warping function is found using a wide field of view camera (WFOV)of view camera (WFOV)
• For flat display surfacesFor flat display surfaces
– Projected images related by a 3x2 Projected images related by a 3x2 transformationtransformation
Calibration Step 1 : Geometric Registration
Warping Images for Irregular Warping Images for Irregular Display SurfacesDisplay Surfaces
• Keep the WFOV camera at ideal viewer’s Keep the WFOV camera at ideal viewer’s locationlocation
• Project dots with the projector one by oneProject dots with the projector one by one
• Find mapping from projector image to camera Find mapping from projector image to camera imageimage
• Invert mapping to find the warping functionInvert mapping to find the warping function
Warping Images for Irregular Warping Images for Irregular Display SurfacesDisplay Surfaces
• Keep the WFOV camera at ideal viewer’s Keep the WFOV camera at ideal viewer’s locationlocation
• Project dots with the projector one by oneProject dots with the projector one by one
• Find mapping from projector image to camera Find mapping from projector image to camera imageimage
• Invert mapping to find the warping functionInvert mapping to find the warping function
Camera to Projector warping function
Display Surface
WFOV Camera
Icamera(u,v)
Observing a point on display surface
Icamera(u,v) = Which projector pixel ?
Camera to Projector warping function
P1
Display Surface Projector
P2
Iproj2(x,y)
Iproj1(x,y)
Projector
Which projector pixel illuminated the same point on display surface ?
Icamera(u,v) = Iproj1(x,y), Iproj2(x,y)
Calibration Step 2 : Intensity Normalization
Blending Projected Images Blending Projected Images
• For irregular display surfacesFor irregular display surfaces
• Projection overlap observed in camera image Projection overlap observed in camera image spacespace
• Assign intensity weights for projector pixelsAssign intensity weights for projector pixels
Blending Projected Images Blending Projected Images
• For irregular display surfacesFor irregular display surfaces
• Projection overlap observed in camera image Projection overlap observed in camera image spacespace
• Assign intensity weights for projector pixelsAssign intensity weights for projector pixels
Intensity in Projector Overlap
P1
Display Surface
Projector
P2
Iproj2(x,y)
Iproj1(x,y)
Projector
How to make intensity in overlap region same as everywhere else ?
WFOV Camera
Icamera(u,v)
Intensity Normalization
Camera Scanline
Camera Scanline
Projected Intensity
P1 P2
P1 P2
Resultant Intensity
Resultant Intensity
Before Normalization
After Normalization
Projected Intensity
Rendering
• Two pass rendering methodTwo pass rendering method
• First pass : Compute the desired imageFirst pass : Compute the desired image
• Second pass :Second pass :
– Warping with standard OpenGL texture Warping with standard OpenGL texture mappingmapping
– Intensity blending using Alpha channel of Intensity blending using Alpha channel of graphics hardwaregraphics hardware
• Two pass rendering methodTwo pass rendering method
• First pass : Compute the desired imageFirst pass : Compute the desired image
• Second pass :Second pass :
– Warping with standard OpenGL texture Warping with standard OpenGL texture mappingmapping
– Intensity blending using Alpha channel of Intensity blending using Alpha channel of graphics hardwaregraphics hardware
Results: correct view for one user
Live Panoramic Video Image captured using a WFOV camera cluster
Summary
• Robust general purpose method to create Robust general purpose method to create seamless imagesseamless images
• General configuration of projectors and display General configuration of projectors and display surfacessurfaces
• High-resolution wide-field of view display using High-resolution wide-field of view display using cluster of projectorscluster of projectors
• No expensive optics or electro-mechanical No expensive optics or electro-mechanical setupsetup
• One-time calibration procedureOne-time calibration procedure
• Robust general purpose method to create Robust general purpose method to create seamless imagesseamless images
• General configuration of projectors and display General configuration of projectors and display surfacessurfaces
• High-resolution wide-field of view display using High-resolution wide-field of view display using cluster of projectorscluster of projectors
• No expensive optics or electro-mechanical No expensive optics or electro-mechanical setupsetup
• One-time calibration procedureOne-time calibration procedure
Applications
• Office Scenarios, flexible setupOffice Scenarios, flexible setup
• Dome shaped displays using cluster of Dome shaped displays using cluster of projectors instead of expensive opticsprojectors instead of expensive optics
• Ease of setup for cylindrical projection Ease of setup for cylindrical projection screens using multiple projectorsscreens using multiple projectors
• Synthetic images or Video images on large Synthetic images or Video images on large display environmentsdisplay environments
• Office Scenarios, flexible setupOffice Scenarios, flexible setup
• Dome shaped displays using cluster of Dome shaped displays using cluster of projectors instead of expensive opticsprojectors instead of expensive optics
• Ease of setup for cylindrical projection Ease of setup for cylindrical projection screens using multiple projectorsscreens using multiple projectors
• Synthetic images or Video images on large Synthetic images or Video images on large display environmentsdisplay environments
Ten Years from now .. • Light bulbs replaced by Light bulbs replaced by
cheap projectors cheap projectors
• Sufficient number of Sufficient number of inexpensive camerasinexpensive cameras
• Widely available texture Widely available texture mapping graphics hardwaremapping graphics hardware
• Projectors, LCD panels to Projectors, LCD panels to create ‘desktop’ instead of create ‘desktop’ instead of looking at a 21’ monitorlooking at a 21’ monitor
• Light bulbs replaced by Light bulbs replaced by cheap projectors cheap projectors
• Sufficient number of Sufficient number of inexpensive camerasinexpensive cameras
• Widely available texture Widely available texture mapping graphics hardwaremapping graphics hardware
• Projectors, LCD panels to Projectors, LCD panels to create ‘desktop’ instead of create ‘desktop’ instead of looking at a 21’ monitorlooking at a 21’ monitor