Upload
rhonda-cook
View
53
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Texture Optimization for Example-based Synthesis. Vivek Kwatra 1,2 Irfan Essa 1 Aaron Bobick 1 Nipun Kwatra 1. 1 GVU Center/College of Computing, Georgia Tech 2 Computer Science Dept., UNC Chapel Hill. Motivation. +. =. Input Texture. Input Flow Field. Output Sequence. - PowerPoint PPT Presentation
Citation preview
Texture Optimization for Example-based Synthesis
Vivek Kwatra1,2
Irfan Essa1
Aaron Bobick1
Nipun Kwatra1
1GVU Center/College of Computing, Georgia Tech2Computer Science Dept., UNC Chapel Hill
Motivation
+ =
InputTexture
InputFlow Field
OutputSequence
Example-based Texture Synthesis
Example-based Texture Synthesis
1. Dynamic texturing (animation/video)
2. Greater control: external knobs, e.g., flow, shape, illumination
3. Explicit texture quality evaluation and refinement
Contributions I
• Global Optimization for texture synthesis– Global texture quality metric
– Iterative multi-scale optimization
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
Contributions II
• Flow-guided texture animation– Flow incorporated into optimization
– Applications: fluid rendering, flow visualization
+ =
InputTexture
InputFlow Field
OutputSequence
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Graphcut Textures[Kwatra’03]
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Better Quality
Graphcut Textures[Kwatra’03]
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Better Quality
Greater Controllability
Graphcut Textures[Kwatra’03]
Related Work
• Flow-guided Synthesis– Advected Textures [Neyret’03]
– Flow-based Video Editing [Bhat’04]
• Optimization– Image-based Priors [Fitzgibbon’03]
– Space-time Video Completion [Wexler’04]
• Flow consistency– Perceived motion similar to flow
• Texture similarity– Shape, size, orientation of texture
elements similar to source
Flow-guided Texture Animation
Target FlowFlowing Target
Source Texture
TextureSimilarity
FlowConsistency
Naive Approach
Target Flow
• Ignore Texture Similarity
• Warp each frame via flow field
X1 Xn
TargetFrames X0
Warp
Flowing Target
FlowConsistency
Source Texture
TextureSimilarity
Source Texture
TextureSimilarity
Naive Approach
Texture structure not maintained
Target FlowFlowing Target
FlowConsistency
Source Texture
TextureSimilarity
Warp
Instead of…
WarpXnX0 X1
TargetFrames
Warp
Instead of…
WarpXnX0 X1
TargetFrames
Warp + Correct
Our Approach
TargetFrames XX–
Warp
W
Correct
WarpedFrame
SynthesizedFrame
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Warp
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Flow ConsistencyWarp
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
Energy = Flow Energy + Texture Energy
Optimization of Energy / Cost
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
Flow Energy
XTargetFrames X–
Warp
W
Flow Energy
Texture Energy
• Tricky!– Pixel-by-pixel comparison of source and target
not possible
• Compare texture “elements”– Local pixel neighborhoods
• Want each target neighborhood to be similar to some source neighborhood
Texture Energy
X(target frame)
Z(source)
Energy for Single Neighborhood
p (pixel)Z(source)
X(target frame)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
(nearest neighbor) Zp
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
X
Z
(nearest neighbor) Zp
Xp (neighborhood)
Texture Energy(single neighborhood)
Texture Energy for Entire Image
X
Z
Texture Energy for Entire Image
X
Z
individual neighborhood energy
• Optimize Total Energy of target frame:
• Initialize: X W(target frame warped frame)
• Iteratively improve target frame
Optimization
Optimization: Intuition
Warped Frame
Source
Target Frame
Flow Force
Texture ForceZp
Xp
Wp
Optimization: Intuition
Source
Zp
Xp
Wp
Blend
Warped Frame
Target Frame
Overlapping Neighborhoods
Source
Wp
BlendXqZq
Xp
Warped Frame
Target Frame
Zp
We are not done yet!
Source
Xp
Wp
ZpNearest neighbor may change
Warped Frame
Target Frame
Iterative Algorithm
• Step 1 – Find Nearest Source Neighborhoods
• Step 2– Blend Source & Warped Neighborhoods
• Repeat until convergence
Iterative Algorithm
• Step 1 – Find Nearest Source Neighborhoods
• Minimize E( X ) w.r.t {Zp}
• Approximate nearest-neighbors– Hierarchical Clustering [Johnson’67] – Tree-Structured Vector Quantization [Wei’00]
Iterative Algorithm
• Step 2– Blend Source & Warped Neighborhoods
• Minimize E( X ) w.r.t X
• Set Solve Linear System
Robust Optimization
• Quadratic Texture Energy Et sensitive to outlier neighborhoods
• Non-quadratic Et more robust
– Et( X ) = p kXp – Zpkr, 0 < r < 2
Robust Optimization
• Quadratic Texture Energy Et sensitive to outlier neighborhoods
• Non-quadratic Et more robust
– Et( X ) = p kXp – Zpkr, 0 < r < 2
• Solve weighted quadratic at each iteration Et( X ) = p kXp – Zpkr-2kXp – Zpk2
= p p kXp – Zpk2
Minimize Texture Energy Alone
• Novel Static Texture Synthesis Algorithm
CorrectXX–
W
Texture Similarity
Flow Consistency
Source
• Random Initialization
• Multiple– Resolution Levels– Neighborhood Sizes
• Progressively refined output
Texture Energy Graph
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
• Random Initialization
• Multiple– Resolution Levels– Neighborhood Sizes
• Progressively refined output
Texture Energy Graph
Results: Texture Synthesis
Results: Texture Synthesis
Input Wei-Levoy[Wei’00]
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization
[Kwatra’05]
Results: Texture Synthesis
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization[Kwatra’05]
Results: Flow-guided Synthesis
Results: Timing
• Static texture synthesis– 5-10 iterations per resolution level
• Flowing texture synthesis– 20-60 seconds per frame
Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
Summary
• Global Optimization for controllable texture synthesis
• Explicit evaluation of texture quality
• Iterative progressive refinement of texture
• Flow-guided texture animation within optimization framework
+ =Input
TextureInput
Flow FieldOutput
Sequence
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
Limitations and Future Work
• Optimization finds local minima– Blurry / misaligned texture elements
• Texture scale should be compatible with flow field
• Extensions– Controllable video textures
– Other control criteria like shape, illumination
Acknowledgements
• Eugene Zhang for flow design code
• Greg Turk, Gabriel Brostow, Delphine Nain, Ravi Ruddarraju, Gaurav Chanda, Stephanie Brubaker, Mitch Parry, Pei Yen
• Computational Perception Lab, Geometry Group @ Georgia Tech
• CMU Graphics Lab
Web page:
http://www.cc.gatech.edu/cpl/projects/textureoptimization/