An Introduction to Texture SynthesisAn Introduction to Texture Synthesis

2008/05/29

Desirable PropertiesDesirable Properties

Result looks like the Result looks like the inputinput

EfficientEfficient

GeneralGeneral

Easy to useEasy to use

ExtensibleExtensible

The ChallengeThe Challenge

Need to model the whole Need to model the whole spectrum: from spectrum: from repeated to stochastic repeated to stochastic texturetexture

repeated

stochastic

Both?

[Shannon,’48] proposed a way to [Shannon,’48] proposed a way to generate English-looking text using generate English-looking text using N-grams:N-grams:•Assume a generalized Markov model

•Use a large text to compute prob. distributions of each letter given N-1 previous letters

•Starting from a seed repeatedly sample this Markov chain to generate new letters

•Also works for whole words

Assuming Markov property, compute P(Assuming Markov property, compute P(pp|N(|N(pp))))• Building explicit probability tables infeasible

• Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p

To synthesize To synthesize pp, just pick one match at random, just pick one match at random

pp

non-parametricsampling

Input image Synthesizing a pixel

Basic AlgorithmBasic Algorithm

Exhaustively search neighborhoodsExhaustively search neighborhoods

NeighborhoodNeighborhood

Neighborhood size determines the quality & costNeighborhood size determines the quality & cost

33 55 77

99 1111 4141

423 s 528 s 739 s

1020 s 1445 s 24350 s

Efros & Leung ’99Efros & Leung ’99

The algorithmThe algorithm•Very simple

•Surprisingly good results

•Synthesis is easier than analysis!

•…but very slow

Optimizations and ImprovementsOptimizations and Improvements•Multi-resolution Pyramid

• Jump Map

•Chaos Mosaic

Multi-resolution PyramidMulti-resolution Pyramid

High resolution Low resolution

Multi-resolution AlgorithmMulti-resolution Algorithm

Multi-resolution PyramidMulti-resolution Pyramid

Better image quality & faster computation Better image quality & faster computation

1 level55

3 levels55

1 level1111

What is a Jump Map?What is a Jump Map?

Same size as inputSame size as input

What is a Jump Map?What is a Jump Map?

Same size as inputSame size as input

Set of jumps per Set of jumps per pixelpixel

What is a Jump Map?What is a Jump Map?

Same size as inputSame size as input

Set of jumps per Set of jumps per pixelpixel

Jumps are weighted Jumps are weighted according to according to similaritysimilarity• Need not sum to 1

0.4

0.3

0.2

Output

Synthesis with Jump MapsSynthesis with Jump Maps

Input

Chaos Mosaic [Xu, Guo & Shum, ‘00]Chaos Mosaic [Xu, Guo & Shum, ‘00]

Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges

input

idea result

Used in Lapped Textures [Praun et.al,’00]

Chaos Mosaic [Xu, Guo & Shum, ‘00]Chaos Mosaic [Xu, Guo & Shum, ‘00]

Of course, doesn’t work for structured textures

input

result

Image QuiltingImage Quilting

Idea:Idea:• let’s combine random block placement of

Chaos Mosaic with spatial constraints of Efros & Leung

pp

Efros & Leung ’99 extendedEfros & Leung ’99 extended

Observation:Observation: neighbor pixels are highly correlated neighbor pixels are highly correlated

Input image

non-parametricsampling

BB

Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))

• Much faster: synthesize all pixels in a block at once

• Not the same as multi-scale!

Synthesizing a block

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

min. error boundary

Minimal error boundaryMinimal error boundary

overlapping blocks vertical boundary

__ ==22

overlap error

),,min( 1,1,11,1 jijijiijij EEEeE

AlgorithmAlgorithm

•Pick size of block and size of overlap

•Synthesize blocks in raster order

•Search input texture for block that satisfies overlap constraints (above and left)

•Easy to optimize using NN search

•Paste new block into resulting texture

•use dynamic programming to compute minimal error boundary cut

Failures

(Chernobyl

Harvest)

Image Quilting for Texture Transfer

Image Quilting for Texture Transfer

++ ==

Texture TransferTexture Transfer

Take the texture from Take the texture from one object and “paint” it one object and “paint” it onto another objectonto another object

• This requires separating texture and shape

• That’s HARD, but we can cheat

• Assume we can capture shape by boundary and rough shading

Then, just add another constraint when Then, just add another constraint when sampling: similarity to underlying image sampling: similarity to underlying image at that spotat that spot

++ ==

++ ==

parmesan

rice

++ ==

Sourcetexture

Target image

Sourcecorrespondenc

eimage

Targetcorrespondence image

++ ==

input image

Portilla & Simoncelli

Wei & Levoy Image Quilting

Xu, Guo & Shum

Portilla & Simoncelli

Wei & Levoy Image Quilting

Xu, Guo & Shum

input image

Portilla & Simoncelli

Wei & Levoy Image Quilting

input image

Homage to Shannon!

Xu, Guo & Shum