Embed Size (px)
Citation preview
Matting and Compositing
Digital Visual Effects, Spring 2006Yung-Yu Chuang2006/5/10
Traditional matting and composting
Photomontage
The Two Ways of Life, 1857, Oscar Gustav RejlanderPrinted from the original 32 wet collodion negatives.
Photographic compositions
Lang Ching-shan
Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Optical compositing
King Kong (1933) Stop-motion + optical compositing
Digital matting and compositing
The lost world (1925) The lost world (1997)
Miniature, stop-motion Computer-generated images
Digital matting and composting
King Kong (1933) Jurassic Park III (2001)
Optical compositingBlue-screen matting, digital composition,
digital matte painting
Digital matting: bluescreen matting
Forrest Gump (1994)
• The most common approach for films.• Expensive, studio setup.• Not a simple one-step process.
Titanic
Matting and compositing Matting and Compositing
backgroundreplacement
backgroundediting
Color difference method (Ultimatte)
Blue-screenphotograph
C=F+αB
Spill suppressionif B>G then B=G
F
Matte creationα=B-max(G,R)
α
Chroma-keying (Primatte)
Compositing
BFC )α(α −+= 1
αF B
C
foreground color alpha matte background plate
composite
compositingequationB
F
C
α=0
Compositing
BFC )α(α −+= 1
αF B
Ccomposite
compositingequationB
F
Cα=1
Compositing
BFC )α(α −+= 1
αF B
Ccomposite
compositingequationB
FC
α=0.6
Matting
CobservationBFC )α(α −+= 1
compositingequation
αF B
Matting
CBFC )α(α −+= 1
compositingequation
αF B
Three approaches:
1 reduce #unknowns
2 add observations
3 add priors
Matting (reduce #unknowns)
C
F BB
BFC )α(α −+= 1
differencematting
α
Matting (reduce #unknowns)
C
F
BFC )α(α −+= 1B
blue screenmatting
α
Matting (add observations)
F
BFC )α(α −+= 1
triangulation
α
BFC )α(α −+= 1
BC
Natural image matting
BC
Matting (add priors)
F
BFC )α(α −+= 1
α B
rotoscopingRuzon-Tomasi
FG
BG
unknown
Bayesian image matting
Bayesian framework
posterior probability
likelihood priors
Priors
Optimization
repeat
until converge
1. fix alpha
2. fix F and B
inputtrimapalpha
Results
input composite
Results
Comparisons
input imagetrimap
Comparisons
Bayesian Ruzon-Tomasi
Comparisons
Bayesian Ruzon-Tomasi
Comparisons
input image
Comparisons
Bayesian Mishima
Comparisons
Bayesian Mishima
Video matting
inputvideo
Video matting
inputkeytrimaps
inputvideo
Video matting
interpo-latedtrimaps
inputvideo
Video matting
interpo-latedtrimaps
inputvideo
outputalpha
Video matting
Compo-site
interpo-latedtrimaps
inputvideo
outputalpha
Video matting
optical flow
optical flow
Sample composite
Garbage mattes Garbage mattes
Background estimation Background estimation
Alpha matte Comparison
without background
withbackground
C
B
P(F)
More on matting
Recent progresses on matting• Poisson matting• LazySnapping/GrabCut• BP matting• Two-camera matting methods• Defocus matting• Flash matting
Poisson matting Poisson matting
LazySnapping
LazySnapping LazySnapping
LazySnapping Interactive matting
Interactive matting Interactive matting
• Use belief propagation to solve the optimization problem
Two-camera matting methods• Invisible lights
– Polarized lights– Infrared
• Thermo-key• Depth Keying (ZCam)
Invisible lights (Infared)
Invisible lights (Infared) Invisible lights (Infared)
Invisible lights (Infared) Invisible lights (Infared)
Invisible lights (Infared) Invisible lights (Polarized)
Invisible lights (Polarized) Thermo-Key
Thermo-Key ZCam
ZCam Defocus matting
Defocus Composite
BFBFCIP )1(),,( ααα −+==
Pinhole Composite
))(1()( ),,,,(
gBhhFhgBFCI
⊗⊗−+⊗=≈
ααα
Defocus (Lens) Composite
g and h are aperture masks of varying radius
))(1()( ),,,,(
gBhhFhgBFCI
⊗⊗−+⊗=≈
ααα
Defocus Composite
• Matting is underconstrained for a single composite• 3 constraints x 3 composites = 9 constraints; solvable
3 constraints 7 unknowns (Lens Parameters)
Defocus matting
Flash matting
flash no flash matte
Flash matting
Background is much further than foreground and receives almost no flash light
Flash matting
Foreground flash matting equation
Generate a trimap and directly apply Bayesian matting.
Foreground flash matting
Joint Bayesian flash matting Joint Bayesian flash matting
Comparison
flash no flash
Comparison
foreground flash matting
ioint Bayesian flash matting
Flash matting
Shadow mattingand composting
target backgroundsource scene target backgroundblue screen image
target backgroundblue screen composite photographblue screen composite
photographblue screen composite
Geometric errors
photographblue screen composite
Photometric errors
S L
C
β S L
C
β
shadowcompositingequation
Shadow compositing equation
C=βL+(1-β)S
C
β
shadowcompositingequation
Shadow matting
C=βL+(1-β)S
S L
C
β
shadowcompositingequation
Shadow matting
C=βL+(1-β)S
S L
C
β
shadowcompositingequation
Shadow matting
β
C=βL+(1-β)S
C
S Lβ
shadowcompositingequation
Shadow compositing
C=βL+(1-β)S
Geometric errors
target backgroundsource scene
target backgroundsource scene
Requirement #1
target backgroundsource scene
Requirement #2
Environment matting
photogrphblue screen matting
F
foreground
Ccomposite
Tbackground
1-αB
traditional compositing equation
F
foreground
Ccomposite
Tbackground
1-αB
AR
environment compositing equation [Zongker’99]
Environment matting [Zongker’99]
O(k) images
Problem: color dispersion
photographZongker et al.
Problem: glossy surface
photographZongker et al.
Problem: multiple mappings
photographZongker et al.
C Tweightingfunction
W
backgroundcomposite
Arbitrary weighting functionMultimodal oriented Gaussian
Problem: color dispersion
photographZongker et al.high accuracyalgorithm
Glossy surface
photographZongker et al.high accuracyalgorithm
Oriented Gaussian
photograph
withoutorientation
withoutorientation
withorientation
Problem: multiple mappings
photographZongker et al.high accuracyalgorithm
mattingmethod
compositingmodel
BFC )α(α −+= 1color
belnding
shadow
refractionreflection
blue-screenBayesian
Shadowmattingβ)L(βSC −+= 1
High-accuracyenv. matting∫+= WBFC
