Classifying Visual Objects Regardless of Depictive Style

Qi Wu, Peter HallDepartment of Computer Science

University of Bath

Summary

• Conventional Comp.Vis. classifiers do not generalise well across depictive styles.

• We propose new visual class model,one invariant to depictive style.

• Experiments validate our model.

People can see objects in a wide variety of depictive styles.

Photos Artwork

Literature Gap: BoW does not generalise across depictive styles

Photos Artwork47% (Dense SIFT)

Photos Artwork51% (Dense SIFT)

Our solution: A new Visual Class Model that does generalise across styles.

Photos Artwork47% (Dense SIFT)

Photos Artwork51% (Dense SIFT)

64%

67%

A New Visual Class Model• We assume an object class is characterised by:

– the qualitative shape of object parts,– the structural arrangement of those parts.

• A hierarchical graph model per image:– coarse-to-fine representation (layered),– nodes labelled by primitive shapes,

• abstracting region shape brings greater robustness.– arcs labelled with displacement vectors

• Median graph models:– aggregates models from several instances,– single class model.

Making a VCM

(a): An input collection. (b): Probability maps for each input image, and graph models for each map. (c): The median graph model for the whole class. (d): The refined median graph as the final class model

A schematic VCM• A hierarchical description

• Berkeley segmentation• Filtering process using cLge

• A graph• Arcs at same level denote

touching neighbours.• Arcs between layers link

parent – children.

• Nodes label• A 6-elements

probability vector.• The probability that a

region belongs to a given prime shape class.

Prime Shapes,BMVC 2012

Prime Shapes• Does a set of elementary planar shapes

appear commonly in the world ?• Art provides strong anecdotal evidence “yes”– 20th century Western Art --- Cubism

Determine Prime Shapes• A fully unsupervised framework

Determine Prime Shapes

Back to our model…

Build graphs, one for each image

Left: graph model. Right: Object broken in primitive shapes

Compute an initial Visual Class Model• Median Graph

• First compute the graph distance between each pair.

• Using the distance matrix to embed graph into a vector space

• Compute the Euclidean Median of all the data points.

• Transfer the median vector back to graph using a state-of-art method proposed in [Ferrer and Valveny, 2008]

Refine the Visual Class ModelThe initial model contains nodes and arcs that derive from visual clutter in back ground of images in the training set

• Refine the model• Match the median back

into each training image.

• Count the number of times a given node in the model appears in the training data.

• Delete all nodes below a frequency threshold., which is computed via maximising matching score.

Some Examples

Experiments

• Compare with other two methods• PHOW features (Dense SIFT) [Bosch and Zisserman, ICCV

2007]• Local PAS features [Ferrari and Jurie, IJCV 2010]• Structure Only [Bai and Song, CVIU 2011]

Results

Conclusions

• It’s possible to learn models of objects classes that generalise across depictive styles.

• Many applications are promised.

• Just a first step– Simplify the model, still too much nodes and arcs.– Time consuming.– Additional labelling– Move to object localisation.

Questions?

One application of Prime shapes