ICCV 2019STRUCT @ PKUSpatial and Temporal Restoration, Understanding and Compression

Controllable Artistic Text Style Transfer via Shape-Matching GAN

Shuai Yang1,2, Zhangyang Wang2, Zhaowen Wang3, Ning Xu3, Jiaying Liu1 and Zongming Guo1

1 Institute of Computer Science and Technology, Peking University 2 Texas A&M University 3 Adobe Research

ICCV 201902 Aim and Challenge

Problem: Controllable Text Style Transfer

Input: style image, target text, deformation degree ℓ

Output: artistic text

Large ℓ more artistry; less legibility: balance?

Input Adjust the stylistic degree of glyph Controllable output

Parameter ℓ0 1

ICCV 201903 Aim and Challenge

Problem: Controllable Text Style Transfer

Bidirectional shape matching Backward structure transfer: prepare training data

Forward structure transfer: learn shape deformationImage matting backward structure transfer

Stage I

forward structure transfer

Stage II

forward texture transfer

Stage II

TRAINING

TESTING

ICCV 201904 Aim and Challenge

Problem: Controllable Text Style Transfer

Challenge Limited Data: one style image to train the network?

Controllable: one network for fast forward multiple scales

CHALLENGE I

Limited Data

CHALLENGE II

Fast Multi-Scale Transfer

ICCV 201905

matting

Input Y Structure X

cropping

{x y

}x~

SketchModule

GB

Sketchy structure X~

Framework

Proposed Method

Stage I: Input preprocessing (Backward Structure Transfer) Structure map of Y: Photoshop or image matting

Train Sketch Module to obtain a sketchy version of X

CHALLENGE I: Limited Data

Generate training data: random cropping X, X, Y~

ICCV 201906

matting SketchModule

GB

Input Y

cropping

{x yStructure X

}x~Sketchy structure X

~

Backward Structure Transfer (GB)

Proposed Method

Gaussian blur to maps T and X into a smooth domain

Train CNN to map the smoothed image back to the text domain

Source domain Target domain

DBTest

Train

Sketch ModuleSmooth domain

Smooth

ness block

Transformation block

ICCV 201907

Backward Structure Transfer (GB)

Proposed Method

CHALLENGE II: Fast Multi-Scale Transfer The standard deviation of Gaussian kernel is controlled by ℓ

matting SketchModule

GB

Input Y

cropping

{x yStructure X

}xℓ~

Sketchy structure Xℓ~

Source domain Target domain

DBTest

Train

Sketch ModuleSmooth domain

Smooth

ness block

Transformation block

ℓℓ

ℓℓ

ICCV 201908

Backward Structure Transfer (GB)

Proposed Method

CHALLENGE II: Fast Multi-Scale Transfer Multi-scale training data generation

More blurry More sketchy Higher deformation degree

Style image Structure map Sketchy Structure map

ICCV 201909

Framework

Proposed Method

Stage II: Forward Structure Transfer Conditional image-to-image translation framework

Training: learn to map 𝑥ℓ with different deformation degrees back to x

GlyphGS

DSℓ

𝑥ℓ xLS

rec

LSadv

ICCV 2019010

Forward Structure Transfer (GS)

Proposed Method

CHALLENGE II: Fast Multi-Scale Transfer Controllable Resblock: linear combination of 2 ResBlocks weighted by ℓ

ℓ = 0/1: solely deal with greatest / tiniest structure deformation

ℓ ∈ (0,1): compromise between the two extremes

GlyphGS

DSℓ

Controllable ResBlock

𝑥ℓ xLS

rec

LSadv

ICCV 2019011

Forward Structure Transfer (GS)

Proposed Method

GlyphGS

DSℓ

LSrec

LSadv

Controllable ResBlock

GlyphGS

LS

ℓ

gly

𝑥ℓ x

T 𝑇ℓ𝑋

Stage II: Forward Structure Transfer Conditional image-to-image translation framework

Test: transfer the shape style of x onto T, producing 𝑇ℓ𝑋

Glyph loss: text legibility preservation

ICCV 2019012

Forward Texture Transfer (GT)

Proposed Method

Stage II: Forward Texture Transfer Standard image-to-image translation framework

Train: learn to map x to y

GlyphGS

LS

ℓ

gly

GlyphGS

TextureGT

DS DTℓ

LSrec

LSadv

𝑥ℓ yx

T 𝑇ℓ𝑋

LTadv

recLT

ICCV 2019013

Forward Texture Transfer (GT)

Proposed Method

Standard image2image translation framework Test: render the texture in Y onto 𝑇ℓ

𝑋 to yield the final artistic text 𝑇ℓ𝑌

Style loss: enhance texture details

GlyphGS

LS

ℓ

gly

LT

advLTGlyphGS

TextureGT

DS DTℓ

LSrec

LSadv

rec

TextureGT

LTstyle

𝑇ℓ𝑋 𝑇ℓ

𝑌

yx

T

𝑥ℓ

ICCV 2019014

Framework

Proposed Method

GlyphGS

LS

ℓ

gly

LT

advLTGlyphGS

TextureGT

DS DTℓ

LSrec

LSadv

rec

TextureGT

LTstyle

𝑇ℓ𝑋 𝑇ℓ

𝑌

yx

T

𝑥ℓ

matting SketchModule

GB

Input Y

cropping

{x yStructure X

}xℓ~

Sketchy structure Xℓ~

ℓℓ

Stage I: Input Preprocessing (Backward Structure Transfer)

Stage II: Forward Style (Structure and Texture) Transfer

ICCV 2019015

1A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. SIGGRAPH. 20012 L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using Convolutional neural networks. CVPR. 2016

3 A. J. Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. Arxiv. 20164 S. Yang, J. Liu, Z. Lian, and Z. Guo. Awesome typography: statistics-based text effects transfer. CVPR. 2017

5 S. Yang, J. Liu, W. Yang, and Z. Guo. Context-aware text-based binary image stylization and synthesis. TIP. 2019

Input style Target text Image Analogy1 NST2 Doodle3 T-Effect4 UT-Effect5 Ours

Comparison with Other Methods

Experimental Results

ICCV 2019016

Adjusting glyph deformation degreelegible stylish

MAPLEReference style Target text

Experimental Results

Scale-Controllable Style Transfer

ICCV 2019017

Adjusting glyph deformation degreelegible stylish

SNOWReference style Target text

Scale-Controllable Style Transfer

Experimental Results

ICCV 2019018

Applications

Experimental Results

diverse structure/texture mixture stroke-based art design

By adding random noises By adding interpolated noise

dynamic text generation

ICCV 2019019 Conclusion

Bidirectional Shape Matching

Training data generation

Backward structure transfer

Image cropping

Fast forward multi-scale structure transfer

Smoothness-based sketch module

Controllable Resblock

Experimental Results

Impressive results compared with other state-of-the-arts

Applications

ICCV 2019020

Shuai Yang: http://williamyang1991.github.io

Project Poster Info:

Poster # 02

Poster 3.1 (Hall B)

Thursday, 10:30 – 13:00