Poisson Image Editing

SIGGRAPH 2003Patric Perez Michel GangnetAndrew Black

Most Frequently Used PDE

Wave Equation

Heat Equation

Most Frequently Used PDE

Poisson Equation, Steady State of Wave Equation and Heat Equation

Laplace’s Equation

Boundary Conditions

Dirichlet Boundary ConditionsSpecify the value of the function on a surface

Neumann Boundary ConditionSpecify the normal derivative of the function on a surface

Guided Interpolation

f: to be solved, f*: known regionv: guided field, g: v is prob. gradient of g

Simple Interpolation

Maximize the Smoothness

Solution: Laplace Equation with DirichletBoundary Conditions

Guided Interpolation

Interpolation-> minimization

Solution: Poisson Equation with DirichletBoundary Conditions

Relationship with Laplace case?

Or from vector field decomposition

Helmholtz-Hodge decomp.

Discrete Poisson SolverDiscretize the Minimization Directly

Partial Derivative

Partial Derivative for Interior Points

Discrete Poisson Solver

Linear System of EquationsGauss-Seidel Method with Successive OverrelaxationV-cycle MultigridDiscretize Laplacian with Discrete Laplacian of Gaussianhttp://www.tau.ac.il/~stoledo/taucs/ Taucs

Seamless Cloning :Importing Gradients

Importing Gradients from a Source Image

Discretize

Seamless Cloning Results

Seamless Cloning Results

Texture

Alignment

Transfer intensity only

Seamless Cloning: Mixing Gradients

Two ProposalsDefine v as Linear Combination of Source and Destination GradientsSelect Stronger one from Source and Destination Gradients (not conservative!)

Discretization

Mixing Gradients Results

Mixing Gradients Results

Mixing Gradients Results

Just reduce terrible things ! Another solution

Texture Flattening

Remain Only Salient Gradients

Discretization

Texture Flattening

Edge mask

Local Illumination Changes

Fattal Transformation

Local Color Changes

Mix two different colored version of original image

One provide f* outsideOne provide g inside

Local Color Changes

Monochrome of backgroundWhite in flower

Seamless Tiling

Select original image as gBoundary condition:

f*north=f*

south=0.5(gnorth+gsouth)Similarly for the east and west

Seamless Tiling

Discussion

Discussion

Discussion

Discussion

Drag-and-Drop Pasting

Leo Jiaya Jia Leo Jiaya Jia Jian Sun Jian Sun ChiChi--Keung TangKeung TangHeungHeung--Yeung ShumYeung Shum

The Chinese University of Hong KongMicrosoft Research AsiaThe H.K. University of Sci. & Tech.Microsoft Research Asia

SIGGRAPH 2006

Slides by the authors

Introduction to our method

Our method improves the Poisson image editing with

a new boundary optimization algorithm,an easier user interface,and an integration of alpha values.

Poisson equations in images

A case study

+

sftf

Poisson equations in imagesA case study

+

sftf

Poisson equations in imagesThe optimization problem in image blending [Perez et al. 2003]

Taking , we have

What does it imply?

0 00

2

pmin | | with | |f s tf f dp f f∂Ω ∂Ω∈Ω

∇ −∇ =∫ 0 00

2

pmin | | with | |tsf dp ff ff ∂Ω ∂Ω∈Ω

=∇ −∇∫ 0 00

2

pmin | | with | | f s tf f fdp f∂Ω Ω ∂Ω∈

∇ −∇ =∫sf∇

tf'

sf f f= −

0 00

2' p

min | ' | with ' | |f t sf dp f f f∂Ω ∂Ω∈Ω∇ = −∫ 0

t sf f−

Poisson equations in images

The minimization problem equals to solving the Laplace equation:

Image blending should take both the source and the target images into consideration.Property of solving the Laplace equation:

The variational energy will approach zero if and only if all boundary pixels satisfy , where k is a constant value.

0 0' 0 with ' | |t sf f f f∂Ω ∂ΩΔ = = −

0

2| ' |fΩ

∇∫0

( ) |t sf f k∂Ω− =

Poisson equations in images

Where is the optimal boundary ?Inside the user drawn regionOutside the object of interest

How is the object extracted?Lazy snapping or Grabcut [Rother et al. 2004] (September 23)

How to optimize it?Minimum color variance

∂Ω

2min (( ( ) ( )) ) , s.t. bluet sp

f p f p k∈∂Ω

− − ∂Ω∈∑

Boundary optimization

∂Ω and k are all unknownsAn iterative optimization

Initialize ∂Ω as the user drawn boundary.Given new ∂Ω, the optimal k is computed:

Given new k, optimize the boundary ∂Ω.Repeat the previous two steps until convergence.

2( , ) (( ( ) ( )) ) , s.t. bluet sp

E k f p f p k∈∂Ω

∂Ω = − − ∂Ω∈∑

( , ) 0E kk

∂ ∂Ω=

∂Shortest path problem

Boundary optimizationIn 2D graph, computing the shortest path between any two points: Dynamic ProgrammingOur problem is to compute a closed path

Boundary optimizationA shortest closed-path algorithm

Breaking closed boundary

Boundary optimizationA shortest closed-path algorithm

Boundary optimizationA shortest closed-path algorithm

Computation complexity O(N)

Boundary optimizationA shortest closed-path algorithm

Total computation complexity O(NM)

Boundary optimization discussion

OptimalityAvoiding that the path twists around the cut by selecting the initial cut position.

How to select the initial cut?Making it short to reduce O(MN)Passing smooth region

One example

Integrating fractional boundary

Fractional boundary is important in image composting: (transparency)

MattingComposition Image I is generated by foreground and background with alpha matteI = alpha*F + (1-alpha)*BMatting is a problem to get alpha,F,B from a given image I.User have to devide the image into three region: Foreground, Background and Unknown area. In foreground area, F = I, alpha = 1, B = 0; In background area, F = 0, alpha = 0, B = I. Our task is to get F,B,alphain unknown area.

Integrating fractional boundary

Where to use the fractional values?only the pixels where the optimized boundary is near the blue ribbon

∂Ω

Integrating fractional boundary

Where to use the fractional values?only the pixels where the optimized boundary is near the blue ribbon

fractional integration: the green region

otherwise: the yellow region

'

( , ) ( , ), ( 1, )( , ) ( (1 ) ) ( , ), ( 1, )

0 otherwise

x s

x x s t

f x y x y x y yellowv x y f f x y x y greenα α

∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩

Integrating fractional boundary

How to integrate the fractional values in Poisson blending?

A blended guidance field

'

( , ) ( , ), ( 1, )( , ) ( (1 ) ) ( , ), ( 1, )

0 otherwise

s

s

x

x x t

f x y x y x y yellowv x y f f x y x y greenα α

∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩

'

( , ) ( , ), (( , ) ( (1 ) ) ( , ), ( 1, )

0 otherwi

1, )

sex x s

x s

tv x y f f x y x y greef x y x y x y

nyellow

α α⎧⎪= ∇ + −

∇ ∈+ ∈⎨

+

⎪⎩

( , ) ( 1, ) ( , )x f x y f x y f x y∇ = + −

' ( (1 ) ) ( , ), ( 1, )( , ) ( , ), ( 1, )

( , )0 otherwise

x s t

x s

x

f x y x yf f x y x y gr

x y yellowv y nx eeα α

∇ +∇ + − +

∈= ∈⎧⎪⎨⎪⎩

Integrating fractional boundaryHow to integrate the fractional values in Poisson blending?

A blended guidance field

'

0 o

( , ) ( , ), (

therw

1, )( , ) ( (1 ) ) ( , ), ( 1,

s)i e

x s

x x s t

f x y x y x y yellowv x y f f x y x y greenα α

∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩

Integrating fractional boundaryHow to integrate the fractional values in Poisson blending?

A blended guidance field

Integrating fractional boundary

Final minimization:

New boundary: Solving the corresponding Poisson equation.

2* *p *

min | ' | with | |f tf v dp f f∂Ω ∂Ω∈Ω∇ − =∫

Results and comparison

Alpha blending

Results and comparison

Poisson blendingOur method

Results and comparison

Alpha blendingOur method

Results and comparison

Poisson blendingOur method

Poisson Image Editing Extended

SIGGRAPH 2006 sketch

Daniel Leventhal Brown Univ.Bernard Gordon Brown Univ.Peter G. Sibley Brown Univ.

Problem caused by texture

Alpha control

Alpha in [0, 1], background 0, and foreground 1.Obtain unselected area’s Alpha by blurring.

Implemented in YUV rather than RGB

Result

Luminance rescaling

Far from solved…

Only pure texture, how about mixture of texture and non-textureFar from perfect…

Parallel Method on Mesh

Poisson based mesh editingOptical Boundaries for mesh mergingExtend from image space to mesh manifold

Mesh Editing with Poisson-Based Gradient Field Manipulation

SIGGRAPH 2004Yizhou Yu UIUCKun Zhou MSRADong Xu Zhejiang Univ, MSRAXiaohan Shi Zhejiang Univ, MSRAHujun Bao Zhejiang Univ.Baining Guo MSRAHeung-Yueng Shum MSRA

Mesh merging

Deformation

Basic idea

Quite similar to the case of imageVector field decomposition can be extended to manifold

Discretization

NOT real gradient and divergence, but it works

Sparse linear system

Mesh Merging

Some ‘ugly’ detailsBoundary interactionBoundary correspondenceRe-parameterization

Mesh deformation

Interactively change guided vector fieldChange normal on a curvePropagate to other areas

Smooth normal field – smooth

Optimal Boundaries for Poisson Mesh Merging

SPM 2007Xiaohuang Huang Zhejiang Univ. & HUSTHongbu Fu HUSTOscar Kin-Chung Au HUSTChiew-Lan Tai HUST

Basic idea

Best boundary for Poisson MergingSimilar to ‘Drag-and-Drop Pasting’.

Optimal boundary

2( , ) (( ( ) ( )) ) , s.t. bluet sp

E k f p f p k∈∂Ω

∂Ω = − − ∂Ω∈∑

Mesh ‘ugly’ limitation

Correspondence between source and target images are trivialMeaningful correspondence between source and target meshes are VERY difficult.

Results

Failure?