Models for Scene Understanding – Global Energy models and a Style-Parameterized

boosting algorithm (StyP-Boost)

Jonathan Warrell,1

Simon Prince,2 Philip Torr,1

Lubor Ladicky,1 Chris Russell1

1Oxford Brookes University, 2University College London

Overview

• CRF-based semantic segmentation• Recent models

• Detectors• Stereo• Co-occurence• Hierarchical Energies

• Style parameterized boosting (StyP-Boost)• Open questions / problems

CRF-based semantic segmentation

• Semantic segmentation = dense labeling using fixed object set

CRF-based semantic segmentation • Conditional Random Field model (pairwise)

Observed Variables

Hidden Variables

Unary Pairwise

Example: -expansion

Sky

House

Tree

GroundInitialize with Tree

Status:

Expand GroundExpand HouseExpand Sky

Courtesy: Pushmeet Kohli

Move Making Algorithms

Search Neighbourhood

Current Solution

Optimal Move

Solution Space

En

erg

y

Higher order CRF models • Higher order models

Unary Pairwise Higher-order

Segment-based Potentials

No. of pixels not taking l in c

Detector-based Potentials

Strength of detector response

Lubor Ladicky, Paul Sturgess, Karteek Alahari, Chris Russell, Philip H.S. Torr, What,Where & How Many? Combining Object Detectors and CRFs, ECCV 2010

Co-occurrence Potentials

Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Graph Cut based Inference with Co-occurrence Statistics, ECCV, 2010

Global image label set

Joint Stereo + Segmentation

Joint potentials

Lubor Ladicky, Paul Sturgess, Chris Russell, Sunando Sengupta, Philip H.S. Torr, Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction, BMVC 2010

Object only potentials

Hierarchical Energies

Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Associative Hierarchical CRFs for Object Class Image Segmentation, ICCV, 2009.

Energy between levels 1 and 0

Style-based Potentials

Jonathan Warrell, Simon Prince, Philip H.S. Torr, StyP-Boost: A Bilinear Boosting Algorithm for Learning Style-Parameterized Classifiers, BMVC, 2010

Style-based unary potential

Style 1: Style 2:

TextonBoost (Shotton et al ’09)

• Image first convolved with 17-d filter bank• Vectors are clustered, and assigned to ~150

texton indices

TextonBoost (Shotton et al ’09)

• Texture-layout features derived from textons

• Boosted classifier predicts semantic class

DenseBoost (Ladicky et al ’09)

• DenseBoost extends TextonBoost to include• HOG• ColourHOG• Structure / Motion features

• State of the art performance on• MSRC (Ladicky et al ’09)• CamVid (Sturgess et al ’09)

Paul Sturgess, Karteek Alahari, Lubor Ladicky, Philip H.S. Torr, Combining Appearance and Structure from Motion Features for Road Scene Understanding, BMVC, 2009

Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Associative Hierarchical CRFs for Object Class Image Segmentation, ICCV, 2009.

StyP-Boost Framework (Training)

• Training Set

• Objective

• Classifier form

Local features Style Parameters Target vectors

StyP-Boost Framework (Training)

• Training Set

• Objective

• Classifier formLoss for class k Strong learner

for class k

StyP-Boost Framework (Training)

• Training Set

• Objective

• Classifier formWeak learner m Style s

Corel: Styles through clustering

• Styles found in Corel through clustering

2-styles (98%)

3-styles(96%)

4-styles (89%)

Corel: Styles through clustering

• Cluster images based on label histograms during training (2-4 clusters)

• Train classifier to predict cluster from image

• Use smoothed classifier posteriors as style parameters (training and testing)

cluster

label label label

Corel: Qualitative results

• StyP-Boost reduces noise from classes which don’t co-occur

Corel: Qualitative results

• StyP-Boost provides better discrimination of

co-occuring classes

Corel: Quantitative results

Training set Test set

Open questions / Problems

• Learning from sparsely labeled data

Lamp-post

Sign

Open questions / Problems

• Incorporating 3D and Video

Image CRF

Ground-plan CRF

Volumetric CRF

Open questions / Problems

• Using temporal information• Extend detector potentials to include

tracking• Use global scene variables for times of day,

seasons etc.

Further Questions

• Further Questions?