Image RenaissanceUsing Discrete Optimization

Cdric AllneNikos ParagiosENPC CERTIS

ESIEE ASIECP - MASFrance

Image Renaissance

OutlineTheory: positions and patches

Finding positions with good similarities

Determining the patch sizes

Results and discussions

Image Renaissance

- 1 -Theory:positions and patches

Image Renaissance

1 - Theory: positions and patchesMain idea: inpainting as a jigsawInpainting: Reconstructing a destroyed part of an image (specified by a mask)The process should be undetectable

Our method:Considering the reconstruction as a jigsaw which pieces are patches taken from the good part of the image

Image Renaissance

Two stepsFinding positions/offsets in the good image having strong similarities with the content of the image at the boundary of the area to rebuild

For each offset found, finding which pixels should be copied to reconstruct the destroyed area1 - Theory: positions and patches

Image Renaissance

- 2 -Finding positionswith good similarities

Image Renaissance

Positions and offsets

Such a problem consists in finding, given an origin point at the boundary of the region to inpaint, a number of candidate positions in the image domain that have similar content with the one at the boundary of the area to rebuild

A few remarks:Missing content is not necessarily in the neighborhood of the destroyed regionTesting each offset in the image for each position at the boundary of the destroyed area would be too expensive in computation time

Our solution: using a probabilistic method, the particle filters2 Finding positions with good similarities

Image Renaissance

Particle filters theoryConsidering a fixed number of samples, we have to repeat this steps:Weight the samples with a Probability Density Function (pdf)Generate a new set of samples (of equal number) from the existing ones with the biggest weights applying small perturbations (importance of perturbation decreases with the iterations to obtain a convergence)

2 Finding positions with good similarities

Image Renaissance

Particle filters in positions researchA particle has an origin point (at the boundary of the area to inpaint, which is invariant), an offset and a size (width and height)

The perturbation is a displacement of the offset and a modification of the size

The weight is evaluated by a Sum of Square Differences (SSD) between the sized neighborhood around the origin point and the offset point balanced with the size

2 Finding positions with good similarities

Image Renaissance

- 3 -Determiningthe patch sizes

Image Renaissance

How to find the optimal partitionFor each pixel in the area to inpaint we have multiple offsets (and, as a consequence, multiple pixel values possible), so we have to find the best label partition to rebuild the image:

In that aim we use a combinatorial optimization: the -expansion algorithm (using graph cuts)3 Determining the patch sizes

Image Renaissance

Term 1 to minimize:constraints to data

: region of the image to be reconstructed

: region of the image to be reconstructed and its close neighborhood (which contains data)

: distance (similarity) between the label and the data of the image at position 3 Determining the patch sizes

Image Renaissance

Term 2 to minimize:constraints between patches

: region of the image to be reconstructed

: 4-neighborhood of

: distance (similarity) between the label L and the label at position

: Euclidian distance from to the boundary of the area to rebuild (to reconstruct firstly the borders of the region)3 Determining the patch sizes

Image Renaissance

Term 3 to minimize:smoothing term

: region of the image to be reconstructed

: 4-neighborhood of

:

Potts model: penalty for neighbors with different3 Determining the patch sizes

Image Renaissance

Global energy function on partitionto minimize

Minimized through successive -expansion (using graph cuts)3 Determining the patch sizes

Image Renaissance

Alpha-expansion principle

With a given partition, the algorithm expands the label over all the other labels minimizing the energy function

3 Determining the patch sizes"green"-expansion

Image Renaissance

Alpha-expansion algorithm

For each offset selected (explaining the interest to reduce their number), we:Construct a graph representing the energy on the area to inpaintExecute the graph-cut algorithm on the graphModify the partition

This is repeated until stability (but in practice, one iteration on each offset gives a result near of the final result)

Each iteration will so enhance the result by minimizing the global energy

Y. Boykov, O. Veksler and R. Zabih. Fast Approximate Energy Minimization via Graph Cuts.PAMI 2001.

3 Determining the patch sizes

Image Renaissance

- 4 -Results and discussions

Image Renaissance

Results4 Results and discussion

Image Renaissance

Results4 Results and discussion

Image Renaissance

Conclusion

Graph-based combinatorial approach for inpainting

Advantages:Efficient for textured imagesQuite fast: in general a couple of minutes (but the final result is nearly reached after the first series of -expansion)

Drawbacks:Polynomial cost depending on the number of offsets selected (so particular attention must be paid on this step)

4 Results and discussion

Image Renaissance

Thank you for your attention!

Any question?