Matting and Compositing - 國立臺灣大學cyy/courses/vfx/17spring/lectures/... · Digital matting and composting King Kong (1933) Jurassic Park III ... Chroma-keying (Primatte)

Matting and Compositing

Digital Visual Effectsg ffYung-Yu Chuang

OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs

Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions



Photomontage Photomontage

The Two Ways of Life, 1857, Oscar Gustav Rejlander y jPrinted from the original 32 wet collodion negatives.

Photographic compositionsPhotographic compositions

Lang Ching-shan

Use of mattes for compositingUse of mattes for compositing

The Great Train Robbery (1903) matte shot

Use of mattes for compositingUse of mattes for compositing

The Great Train Robbery (1903) matte shot

Optical compositingOptical compositing

King Kong (1933) Stop-motion + optical compositing

Digital matting and compositingDigital matting and compositing

The lost world (1925) The lost world (1997)

Miniature, stop-motion Computer-generated images

Digital matting and compostingDigital matting and composting

King Kong (1933) Jurassic Park III (2001)

Optical compositingBlue-screen matting, digital composition, Optical compositing digital composition,

digital matte painting

Smith Duff Catmull Porter

O d 1996Oscar award, 1996

Titanic

M tti d C itiMatting and Compositing

backgroundreplacement

background

M tti d C iti

editing

Matting and Compositing

Digital matting: bluescreen mattingDigital matting: bluescreen matting

Forrest Gump (1994)

• The most common approach for films.• Expensive, studio setup.p p• Not a simple one-step process.

Color difference method (Ultimatte)Color difference method (Ultimatte)C=F+B F

Blue-screenphotograph

Spill suppressionif B>G then B=G

Matte creation=B-max(G,R)p g p (G, )

demo with Paint Shop Pro (B=min(B,G))

Problems with color differenceProblems with color difference

Background color is usually not perfect! (lighting, shadowing…)

Chroma-keying (Primatte)Chroma keying (Primatte) Chroma-keying (Primatte)Chroma keying (Primatte)

demo



F BF B

foreground color alpha matte background plate

BFC )α(α 1CcompositeF BFC )α(α 1

compositingC

C iti

p gequationB =0

Compositing

F BF B

BFC )α(α 1CcompositeF BFC )α(α 1

compositingC

C iti

p gequationB

C=1

Compositing

F BF B

BFC )α(α 1CcompositeFC

BFC )α(α 1

compositing

C iti

p gequationB =0.6

Compositing

F BF B

CobservationBFC )α(α 1 BFC )α(α 1

compositing

M tti

p gequation

Matting

F BF B

CBFC )α(α 1

Three approaches:

1 d # k BFC )α(α 1

compositing

1 reduce #unknowns

2 add observations

M tti

p gequation3 add priors

Matting

F BBF BB

CBFC )α(α 1 BFC )α(α 1

difference

M tti ( d # k )

differencematting

Matting (reduce #unknowns)

F F

CBFC )α(α 1

BBFC )α(α 1

blue screen

M tti ( d # k )

blue screenmatting

Matting (reduce #unknowns)

F F

BFC )α(α 1BC

BFC )α(α 1BFC )α(α 1

M tti ( dd b ti )triangulation

Matting (add observations)

F BF B

BCBFC )α(α 1BG BFC )α(α 1

FGunknown

Natural image mattingM tti ( dd i )rotoscopingRuzon-Tomasi

unknown

Natural image mattingMatting (add priors)



para observedf(z)+z ypara-

metersobserved

signal

)|(max* yzPz Example: super-resolution)|(max yzPz

z)()|(max zPzyP

super-resolutionde-blurringde-blocking

)(max

yPz de-blocking

…

B i f k

)()|(max zLzyLz

Bayesian framework

para observedf(z)+z ypara-

metersobserved

signal

)()|(max* zLzyLz )()|(max zLzyLzz

2)(zfy data a-priori

B i f k

2evidence knowledge

Bayesian framework

posterior probability

likelihood priors

B i f kBayesian framework

P iPriors B i ttiBayesian matting

repeat1. fix alpha

2. fix F and B

O ti i tiuntil converge

Optimization

DDemo

inputtrimapalphainputtrimapalpha

R ltResults

input compositeinput composite

R ltResults

input imagetrimap

C iComparisons

Bayesian Ruzon-Tomasi

C iComparisons

Bayesian Ruzon-Tomasi

C iComparisons

MishimaMishima

C iComparisons

BayesianBayesian

C iComparisons

input image

C iComparisons

Bayesian Mishima

C iComparisons

Bayesian Mishima

C iComparisons

i tinputvideo

Vid ttiVideo matting

i tinputvideo

inputkeytrimaps


i tinputvideo

interpo-latedtrimaps


inp tinputvideo

interpo-t tlated

trimapsoutputalpha


Compo-inp t Compo-site

inputvideo

interpo-t tlated

trimapsoutputalpha


optical flow

optical flow

S l itSample composite

G b ttGarbage mattes G b ttGarbage mattes

B k d ti tiBackground estimation B k d ti tiBackground estimation

Al h ttAlpha matte

without b k d

withb k d

C i

background background

Comparison

C

P(F)

B

Problems with Bayesian mattingProblems with Bayesian matting

• It requires fine trimaps for good results• It requires fine trimaps for good results• It is tedious to generate fine trimaps• Its performance rapidly degrades when

foreground and background patterns foreground and background patterns become complex Th i di t d l l t l t th • There is no direct and local control to the resulting mattes



Scribble based inputScribble-based input

trimap scribblep

MotivationMotivation

LazySnappingLazySnapping

L S in-th mean foreground color

LazySnapping

L S iLazySnapping L S iLazySnapping

Matting approachesMatting approaches• Sampling approaches: solve for each • Sampling approaches: solve for each

alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian mattingKnockout and Bayesian matting.

• Propagation approaches: solve the whole matte together by optimizing, e g Poisson BP random walker e.g. Poisson, BP, random walker, closed-form and robust matting.

Poisson mattingPoisson matting

Poisson mattingPoisson matting Robust mattingRobust matting• Jue Wang and Michael Cohen CVPR • Jue Wang and Michael Cohen, CVPR

2007

Robust mattingRobust matting• Instead of fitting models a non-• Instead of fitting models, a non-

parametric approach is used

B i R bBayesian Robust

Robust mattingRobust matting• We must evaluate hypothesized • We must evaluate hypothesized

foreground/background pairs

Bj C

Fidistance ratio

F

Robust mattingRobust matting• To encourage pure fg/bg pixels add • To encourage pure fg/bg pixels, add

weights

B F1C

FF2

Robust mattingRobust matting• Combine them together Pick up the • Combine them together. Pick up the

best 3 pairs and average themconfidence

Robust mattingRobust matting Robust mattingRobust matting

matte confidencematte confidence

Matte optimizationMatte optimization

Solved by Random Walk Algorithm

Matte optimizationMatte optimizationdata constraintsdata constraints

neighborhood constraints

Demo (EZ Mask)Demo (EZ Mask) EvaluationEvaluation• 8 images collected in 3 different • 8 images collected in 3 different

ways• Each has a “ground truth” matte

EvaluationEvaluation• Mean square error is used as the • Mean square error is used as the

accuracy metric• Try 8 trimaps with different accuracy

for testing robustness for testing robustness • 7 methods are tested: Bayesian, y ,

Belief propagation, Poisson, Random Walk KnockOut2 Closed Form and Walk, KnockOut2, Closed-Form and Robust mattingg

Quantitative evaluationQuantitative evaluation Subjective evaluationSubjective evaluation

Subjective evaluationSubjective evaluation Ranks of these algorithmsRanks of these algorithms

Poissonaccuracy

6 9robustness

6 8PoissonRandom walk

6.96.0

6.84.4

Knockout2Bayesian

4.53 9

4.56 0Bayesian

Belief PropagationClose form

3.93.32 6

6.03.12 0Close-form

Robust matting2.61.0

2.01.3

SummarySummary• Propagation-based methods are more • Propagation-based methods are more

robust• Sampling-based methods often

generate more accurate mattes than generate more accurate mattes than propagation-based ones with fine trimaps Robust matting combines strengths • Robust matting combines strengths of both

New evaluation (CVPR 2009)New evaluation (CVPR 2009)• http://www alphamatting com/• http://www.alphamatting.com/

Soft scissorSoft scissor• Jue Wang et al SIGGRAPH 2007• Jue Wang et. al., SIGGRAPH 2007• Users interact in a similar way to y

intelligent scissors

FlowchartFlowchart

FlowchartFlowchart Soft scissorSoft scissor

Demo (Power Mask)Demo (Power Mask) OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs


Matting with multiple observationsMatting with multiple observations• Invisible lightsg

– Polarized lightsInfrared– Infrared

• Thermo-key• Depth Keying (ZCam)• Flash matting• Flash matting

I i ibl li ht (I f d)Invisible lights (Infared)

I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)

I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)

I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (P l i d)Invisible lights (Polarized)

I i ibl li ht (P l i d)Invisible lights (Polarized) Th KThermo-Key

Th KThermo-Key ZCZCam

D f ttiDefocus mattingvideo

M tti ith video

Matting with camera arrays

flash no flash matteflash no flash matte

Fl h ttiFlash matting

Background is much further than foreground and receives almost no flash light


Foreground flash matting equationg g q

Generate a trimap and directly apply Bayesian matting.


F d fl h ttiForeground flash matting J i t B i fl h ttiJoint Bayesian flash matting

J i t B i fl h ttiJoint Bayesian flash matting

flash no flash

C iComparison

foreground flash matting

ioint Bayesian flash matting

C iComparison Fl h ttiFlash matting



ConclusionsConclusions• Matting algorithms improves a lot in • Matting algorithms improves a lot in

these 10 years d ll l • In production, it is still always

preferable to shoot against uniform p gbackgroundsAlgorithms for more complex • Algorithms for more complex backgrounds

• Devices or algorithms for automatic mattingmatting

Thanks for your attention!Thanks for your attention! Shadow mattingand composting

target backgroundsource scene target backgroundblue screen image

target backgroundblue screen composite photographblue screen composite

photographblue screen composite

G t i Geometric errors

photographblue screen composite

Ph t t i Photometric errors

S LS L

C

S LS L

C=L+(1-)Sshadowcompositing

Cp g

equation

Sh d iti tiShadow compositing equation


Cp g

equation

Sh d ttiShadow matting

S LS L


Cp g

equation


S LS L


Cp g

equation


S LS L


Cp g

equation

Sh d itiShadow compositing

G t i Geometric errors

target backgroundsource scene


R i t #1Requirement #1


R i t #2Requirement #2

E i t ttiEnvironment matting

photographblue screen matting

traditional compositing equation

BFC T

BFC T

backgroundforegroundcomposite

g

environment compositing equation [Zongker’99]

RA

BFC T

BFC T

backgroundforegroundcomposite

g

O(k) images

E i t tti [Z k ’99]Environment matting [Zongker’99]

photographZongker et al.

P bl l di iProblem: color dispersion


P bl l fProblem: glossy surface


P bl lti l iProblem: multiple mappings

WW

C TC Tweighting backgroundweightingfunction

gcomposite

A bit i hti f tiM lti d l i t d G iArbitrary weighting functionMultimodal oriented Gaussian

high accuracy photographZongker et al.high accuracyalgorithm

P bl l di iProblem: color dispersion


Gl fGlossy surface

withoutwithoutwith photographwithoutorientation

withoutorientation

withorientation

withoutorientationorientation

O i t d G iOriented Gaussian


P bl lti l iProblem: multiple mappings

WR

A

BFC T

BFC T

backgroundweightingforegroundcomposite

bac g ou dweightingfunction

3 3 1 433

3 observations

3 1 43

3 observations11 variables

A RA, R F

433 433

3 observations7 variables3 observations

A RA, R F

413 413

3 observations5 variables3 observations

A R A A, R

A, colorless

F

T(c c )13T(cx , cy)

1113

3 observations

11

3 variables3 observations

A R A cx cy A, R

A, colorless

cx , cy , specularly

F refractive

Stimulus functionStimulus functionx

Stimulus functionStimulus functionx

yA

y

T( )M(T , A) T(cx, cy)

(cx, cy)

T

Ideal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cube

x

y

T(x y)T(x,y)

Calibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cube

x (cx ,cy)C (cx ,cy)C

T(x y)y

T(x,y)

Estimate c c and Estimate c c and

W

Estimate cx ,cy and Estimate cx ,cy and

backgroundif ld

W

Tmanifold

C (cx ,cy)

C = KC

C

KC

KK

Problem: noisy matteProblem: noisy matte Edge-preserving filteringEdge-preserving filteringEdge preserving filteringEdge preserving filtering

with filteringwithout filtering gg

Input imageInput image Difference thresholdingDifference thresholding

Morphological operationMorphological operation FeatheringFeathering

Heuristics for specular highlightsHeuristics for specular highlights

Wbackgroundmanifold

W

Tmanifold

CC >1+ C >1+

K

Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights

cx cy

C=T(c c )C=T(cx, cy)

Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights

==

input estimation foregroundinput estimation foreground(highlights)

Composite with highlightsComposite with highlights

mattingmethod

compositingmodel methodmodel

color blue-screenBFC )α(α 1

color blending

blue-screenBayesian

shadowShadowmattingβ)L(βSC 1

refraction

matting

High accuracyrefractionreflection

High-accuracyenv. matting WBFC

Documents

Matting and Compositing - 國立臺灣大學cyy/courses/vfx/17spring/lectures/... · Digital matting and composting King Kong (1933) Jurassic Park III ... Chroma-keying (Primatte)