Separating Style and Content with Bilinear Models

Joshua B. Tenenbaum, William T. Freeman

Computer Examples

Barun Singh25 Feb, 2002

PHILOSOPHY & REPRESENTATIONData contains two components: style and contentWant to represent them separately

1 1

T

I J

ij i ji j

a b

y w

a Wb

Symmetric Bilinear Model:y : observed dataa : style vectorb : content vectorI, j : components of style and contentW : matrix of basis vectors (e.g., “eigenfaces”)

Y : (SK) x C

A : (SK) x J

b : J x C

Ab

Asymmetric Bilinear Model:A : matrix of style-specific basis vectors

More flexible modelEasier to deal with

PROBLEMS TO BE SOLVED

Given a labeled training set of observations in multiple styles and content classes,

Fit asymmetric model (find A and b for known styles and contents) using SVD

1

1

2 2

Csc s c s OLC

c

SOLC s

ss

E

y A b A A

A A

Find style matrix that best explains data for incomplete style (i.e., minimizes E given below)

Extrapolate using the estimated style matrix

OLC used to solve overfitting problem Parameters involved:

= 0 : Purely asymmetric model = : Purely symmetric model

extrapolate a new style to unobserved content classes

PROBLEMS TO BE SOLVEDGiven a labeled training set of observations in multiple styles and

content classes,

Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style

Parameters: model dimensionality J, model variance 2, max number of EM iterations tmax

2

,

2Pr( | , ) exp / 2σ

Pr( ) Pr( | , ) Pr( , )

Pr( , | ) Pr( | , ) Pr( , ) / Pr( )

s c

c s

s c

s c s c

s c s c s c

y y A b

y y

y y y

*

1

1 1

choose to maximize log-likelihood

L = log Pr( )

Pr( , | )

Pr( , | )

s

C CT Ts sc c sc c c

c c

sc

sc

n

s c

n s c

y

y

y

A

y

A m b b b

m y y

y

classify content observed in a new style Fit asymmetric model

Select content class c that maximizes Pr(s’,c|y)

PROBLEMS TO BE SOLVED

Given a labeled training set of observations in multiple styles and content classes,

translate from new content observed only in new styles into known styles or content classes

Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure

1VTs c

a Wb y

1VTc VT s

b W a y

Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):

TOY EXAMPLE - introImage made of 4 pixels, each of which are either white or red. Style represents if the top or bottom rows are red or whiteContent represents if the left or right columns are red or white.

SYMMETRIC MODEL

Basis Images ( W )

Content Vectors ( b )

Sty

le V

ecto

rs (

a ) O

utp

ut Im

ag

es ( y

)

TOY EXAMPLE - intro

ASYMMETRIC MODEL

*Note: Images drawn as blocks, but represented as vectors, not matrices

Content Vectors ( b )

Sty

le-s

pecifi

c B

asis

Im

ag

es

( A

)

Ou

tpu

t Imag

es ( y

)

TOY EXAMPLE - extrapolation

?

Fitting the asymmetric model

Content Vectors ( b )

Sty

le-s

pecifi

c B

asis

Im

ag

es

( A

)

Extrapolate

FONTS EXAMPLE - extrapolation

Training Set Incomplete Style

model dim= 60

Con

ten

t (L

ett

er)

Style (Font)No prior, OLC only, best result, true letters.

FONTS EXAMPLE - extrapolationNo prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.Asymmetric ModelNo prior, OLC only, best result, true letters.

10 20 30 40 50 60

model dimension

No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.

10 20 30 40 50 60

model dimension

Symmetric Model

No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.

Sym. W/ Asym. Prior

(dim = 60) vs. Actual

TOY EXAMPLE - classification

1: Fit asymmetric model to training set

Content Vectors ( b )

Sty

le-s

pecifi

c

Basis

Im

ag

es

( A

)

TOY EXAMPLE - classification

2: Use Separable Mixture Model w/ EM to classify

2 = 0.5

2 = 0.6

2 = 0.35

Content Vectors ( b )

Sty

le-s

pecifi

c

Basis

Im

ag

es

( A

)

Actual

Resu

lting

Im

ag

es

FACES EXAMPLE - translation

Content : faces

Style: ligting

finitofinito