Capture and Capture and Displays Displays

CS 211ACS 211A

HDR ImageHDR Image

Bilateral FilterBilateral Filter

Color GamutColor Gamut

Natural Colors

Camera Gamut

Traditional DisplaysTraditional Displays• LCD panels

–The gamut is the result of the filters used

• In projectors

Recent high gamut displaysRecent high gamut displays• Comes from use of LEDs

–LEDs are much saturated primaries–1.5 times HDTV gamut

• Projectors (MERL)

Recent high gamut displaysRecent high gamut displays• Comes from use of LEDs

–LEDs are much saturated primaries–1.5 times HDTV gamut

• LCD panels (HP Dream Color)–Use LED backlighting

• Laser multi-primary ones are coming up

Color GamutColor Gamut

Natural Colors

Camera Gamut

What about capture?What about capture?• No HG capture

–Not in the required spatial and temporal resolution

• Closest is hyperspectral imagery–Captures multiple narrow spectral bands – Low spatial resolution (512x512)–Low temporal resolution (10 fps)–Cost is $50,000–Used for scientific application

Gap between Capture and Gap between Capture and DisplayDisplay

• Hollywood have defined a digital cinema standard gamut –What they want?

• Close to the current high gamut displays

• No capture device• Sophisticated gamut extrapolation

methods

High Resolution ImageryHigh Resolution Imagery• Capture

–Panoramic Image generation

• Display–Tiled displays

Panoramic ImagePanoramic Image

Basic AlgorithmBasic Algorithm• Assume distant scene, locally planar• Detect features across adjacent pics• Relate two adjacent pictures by a

homography• Stitch them

Problems?Problems?• What if circular?

–Can have inconsistencies

• What is more than one row?–Same issue inconsistencies

• Use some global optimization techniques–Optimize across all images together–Bundle adjustment

Another problemAnother problem• Spatial intensity

fall off in cameras–Primarily due to

the lens–Vignetting

• Radial Fall off• May not be

centered

Inaccurate feature detectionInaccurate feature detection

Radiometric Camera Radiometric Camera CalibrationCalibration

• Finding transfer function and vignetting

• Debevec’s method–Recovers transfer function–Assumes no vignetting effect–Assures by setting aperture of a narrow

FOV camera to a very low value

How to calibrate the How to calibrate the camera?camera?

• Debevec–Need a few pixels to recover transfer

function–Can be the center 10x10 of a camera

where negligible vignetting–Find g

Goldman et al (2005)Goldman et al (2005)• Assumes known transfer function• Takes panoramic images with very large

overlaps• Same irradiance imaged at different pixels

– Have different intensity due to different vignetting or exposure

• Basic idea– For corresponding feature f in image i and j

• g(Zi) = ln(E) + ln (Vi) + ln(ti)• g(Zj) = ln(E) +ln(Vj) +ln(tj)

• Recover both vignetting and exposure

ResultsResults

Radiometric camera Radiometric camera calibrationcalibration

• Underconstrained problem• Cannot get both transfer function

and vignetting effect–Transfer function upto an exponential

ambiguity

• Both upto scale factor– If transfer function is known

JuangJuang and and MajumderMajumder 20072007• Use a projector in the loop• Constrain the problem by using

known inputs to the projector

24

fc( )fc( )

ModelModel

Projector ITFfp

Spatial VariationDue to Projector

P(x,y)Spatial Variation

Due to ScreenS(x,y)

Exposuretj

Spatial VariationDue to Camera

C(x,y)

Camera ITFfc

tj

tj

fp( )fp( ) P(x,y)P(x,y) S(x,y)S(x,y) C(x,y)C(x,y)iZout =Zout =

L(x,y)L(x,y)fc( )fc( )

Calibration AlgorithmCalibration Algorithm

tj

tj

fp( )fp( ) P(x,y)P(x,y) S(x,y)S(x,y) C(x,y)C(x,y)iZout =Zout =

• Since fc( ) is monotonic, inverse exists.

fc( )fc( )L(x,y)L(x,y) tjtjfp( )fp( )iZout =Zout =fc-1( )fc-1( )

Calibration AlgorithmCalibration Algorithm

L(x,y)L(x,y) tjtjfp( )fp( )iZout =Zout =fc-1( )fc-1( )

ln fc-1( )ln fc-1( )Zout =Zout = ln L(x,y)ln L(x,y) ln (tj )ln (tj )ln fp( )ln fp( )i ++ ++

Calibration AlgorithmCalibration Algorithm• Take the log of both sides

Calibration AlgorithmCalibration Algorithm• Images of multiple projector inputs

at multiple camera exposures• Setup system of equations and solve

using least-squares

ln fc-1( )ln fc-1( )Zout =Zout = ln L(x,y)ln L(x,y) ln (tj )ln (tj )ln fp( )ln fp( )i ++ ++

Separating ParametersSeparating Parameters• Recall model

fc( )fc( )L(x,y)L(x,y) tjtjfp( )fp( )iZout =Zout =

P(x,y)P(x,y) S(x,y)S(x,y) C(x,y)C(x,y)

Camera vignettingCamera vignettingProjector vignettingProjector vignetting

Separating ParametersSeparating Parameters• Recall model

• Different camera aperture => different L

fc( )fc( )L(x,y)L(x,y) tjtjfp( )fp( )iZout =Zout =

P(x,y)P(x,y) S(x,y)S(x,y) C(x,y)C(x,y)

Camera vignettingCamera vignettingProjector vignettingProjector vignetting

kk

Separating ParametersSeparating Parameters• At narrow camera apertures, camera

vignetting is negligible, almost constant

Lf/32(x,y) =Lf/32(x,y) = P(x,y)P(x,y) S(x,y)S(x,y) Cf/32(x,y)Cf/32(x,y)

Separating ParametersSeparating Parameters• At wider apertures

• Normalized vignetting effect

Lf/32(x,y) =Lf/32(x,y) = P(x,y)P(x,y) S(x,y)S(x,y)La(x,y) =La(x,y) = P(x,y)P(x,y) S(x,y)S(x,y) Ca(x,y)Ca(x,y)

==kk

==Ca(x,y)Ca(x,y)

kk

High Resolution DisplaysHigh Resolution Displays• Tile Projectors or LCD panels• Seamless or with seams

Building Tiled DisplaysBuilding Tiled Displays

Client

Network

ServersProjectorsDisplay Screen

Viewer

Geometric/Photometric Geometric/Photometric MismatchMismatch

Camera Based RegistrationCamera Based Registration• Camera feedback detects

misregistration• Encoded in a mathematical function

–Both geometric and photometric

• Change the projected image digitally–Apply the inverse function– In real-time via GPU

Different spacesDifferent spacesDisplay D(s, t)

Simple Geometric AlignmentSimple Geometric Alignment

Projector(xi, yi)

Display(s, t)

G

Simple Geometric Simple Geometric AlignmentAlignment

Projector(xi, yi)

Display(s, t)

G = F ● H

Camera(u, v)

H F

Apply G-1 for registration

Planar DisplayPlanar Display

Projector(xi, yi)

Display(s, t)

G = F ● H

Camera(u, v)

H F

• Calibrated camera (no radial distortion)• F is linear (3x3 matrix called homography)

R. Raskar, Immersive Planar Display using Roughly Aligned Projectors, IEEE VR, 2000.

• G = F x H• G-1 is just a matrix inversion

Perfect ProjectorsPerfect Projectors

Projector(xi, yi)

Display(s, t)

G = F ● H

Camera(u, v)

H F

R. Raskar, Immersive Planar Display using Roughly Aligned Projectors, IEEE VR, 2000.

Homography

Linear Linear GG

Corrected using Corrected using GG--11

Intensity VariationIntensity Variation

• If the projectors are good and similar– No vignetting

• Take care of overlaps

• If not, measure accurately

1) A. Majumder, Properties of Color Variation in Multi Projector Displays, SID Eurodisplay, 2002.2) A. Majumder and R. Stevens, Color Non-Uniformity in Multi Projector Displays: Analysis and

Solutions, IEEE Transactions on Visualization and Computer Graphics, Vol. 10, No. 2, 2003.

Use some edge blendingUse some edge blending

Software Blending

Before

1) Lyon Paul, Edge-blending Multiple Projection Displays On A Dome Surface To Form Continuous Wide Angle Fields-of-View, Proceedings of 7th I/ITEC, 203-209, 1985.

2) R. Raskar et al, Seamless Camera-Registered Multi-Projector Displays Over Irregular Surfaces,Proceedings of IEEE Visualization, 161-168, 1999.

3) K. Li et.al, Early experiences and challenges in building and using a scalable display wall system,IEEE Computer Graphics and Applications 20(4), 671-680, 2000.

Handle all variationsHandle all variations• Measure each projector’s vignetting• Measure each projector’s transfer

function

Add them upAdd them up

Strict Luminance UniformityStrict Luminance Uniformity

u

L

Strict Luminance UniformityStrict Luminance Uniformity

u

L

Before

After Strict Luminance Uniformity

A. Majumder and R. Stevens, Color Non-Uniformity in Multi Projector Displays: Analysis and Solutions, IEEE Transactions on Visualization and Computer Graphics, Vol. 10, No. 2, 2003.

ResultsResults

Strict Luminance UniformityStrict Luminance UniformityL

u

L

Suboptimal use of system resources

Significant Contrast/ Dynamic Range Compression

Smooth the Luminance Smooth the Luminance functionfunction

Optimization ProblemOptimization Problem

u

L

u

L

Strict luminance uniformity is a special case.

Before

After Strict Luminance Uniformity

ResultsResults

Before

After Luminance Smoothing

ResultsResults

1) A. Majumder, R. Stevens, Perceptual Photometric Seamlessness in Tiled Projection Based Displays, ACM Transactions on Graphics, Vol. 24, No. 1, 2005.

2) A. Majumder, Improving Contrast of Multi-Displays Using Human Contrast Sensitivity,IEEE CVPR 2005.

Smooth

Original

Flat

Different smoothing parametersDifferent smoothing parameters(2x2 array of four projectors)(2x2 array of four projectors)

Smoother

Smooth

OriginalSmoother

Flat

Different smoothing parametersDifferent smoothing parameters(3x5 array of fifteen projectors)(3x5 array of fifteen projectors)