Learning Jigsaws for clustering appearance and shape John Winn, Anitha Kannan and Carsten Rother...

Learning Jigsawsfor clustering appearance and shape

John Winn, Anitha Kannan and Carsten Rother

NIPS 2006

Learning jigsawsAim: Cluster regions in images with similar appearance and shape.

Examples of clusters (jigsaw pieces)

EyeNoses Cheek Eyebrows

Road map

Clustering image patches

The Jigsaw model

Results on toy and real images

Learning jigsaw pieces

Discussion and conclusions

Clustering image patches

Patches

Clusters

[Leibe & Schiele, BMVC 2003]

Clustering image patches

Cluster?

Patch wrong shape

Clustering image patches

Cluster?

Patch wrong shape

Clustering image patches

Cluster?

Part is occluded

Clustering image patches

Cluster?

Need to adapt the patch shape depending on the image.

Road map

Clustering image patches

The Jigsaw model

Results on toy and real images

Learning jigsaw pieces

Discussion and conclusions

Aims of jigsaw model

Learn clusters (jigsaw pieces) so that:

1. Clustered patches have similar shape and appearance

2. Patches are as large as possible

3. Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)

The Jigsaw model

Jigsaw J

Image I1

...Image I2 Image INOffset map L2 Offset map LNOffset map L1

Region of constant offset

The Jigsaw model

Jigsaw J

Offset map prior (Potts model)

Appearance model

JigsawMean μ(z) and inverse variance λ(z) for each jigsaw pixel z.

Image I Offset map L

offset at pixel i

cost of patch boundary

Road map

Clustering image patches

The Jigsaw model

Results on toy and real images

Learning jigsaw pieces

Discussion and conclusions

Toy example

Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps

(see paper for details)

Image with segmentation Jigsaw

Mean Variance

Comparison: Mixture of Gaussians

fixed patch shape

Cluster centres

Comparison: Epitome

[Jojic et al., ICCV 2003]

fixed patch shape translation invariant

Epitome

Comparison: Jigsaw

learned patch shape translation invariant non-overlapping patches

Jigsaw

Comparison: all methodsOriginal

JigsawEpitome

Error = 0.054Error = 0.071

MoG

Error = 0.103

Faces example

Source: Olivetti face database

Face images with segmentations Jigsaw

128128 mean

Road map

Clustering image patches

The Jigsaw model

Results on toy and real images

Learning jigsaw pieces

Discussion and conclusions

Learning the jigsaw pieces

Jigsaw J

...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1

Learning the jigsaw pieces

Jigsaw J

...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1

Learning the jigsaw pieces

Jigsaw J

...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1

Shape clustering on faces

Jigsaw showing piecesCommonly used pieces

Road map

Clustering image patches

The Jigsaw model

Results on toy and real images

Learning jigsaw pieces

Discussion and conclusions

Jigsaw applications

Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems.

Applications include: Object recognition/detection Object segmentation Stereo matching Texture synthesis Super-resolution Motion segmentation Image/video compression

Future work

Allow rotation/scaling/deformationof the patches.

Incorporate shape clustering into the probabilistic model

Incorporate additional invariances e.g. to illumination

Apply to other domains: audio, biology

Conclusions

Jigsaw model allows learning the shape and appearance of recurring regions in images.

Jigsaw performs unsupervised discovery of object parts.

Thank you

Jigsaw paper (compressed)

http://johnwinn.org

Comparison: Epitome

[Jojic et al., ICCV 2003]

fixed patch shape translation invariant overlapping patches

Epitome

Patch averaging

Error = 0.071 Error = 0.054

EpitomeMoG