SEMANTIC SEGMENTATION USING REGIONS AND PARTS

Pablo Arbel´aez1, Bharath Hariharan1, Chunhui Gu1,2, Saurabh Gupta1, Lubomir Bourdev1,3,† and Jitendra Malik1

1University of California, Berkeley - Berkeley, CA 947202Google Inc., 1600 Amphitheatre Pkwy, Mountain View, CA 940433Facebook, 1601 Willow Rd, Menlo Park, CA 94025

OUTLINE

Introduction Related Work Region Generation Region Representation Region Scoring Pixel Classification Experiments

INTRODUCTION

Bottom-up region cues and top-down part detectors provide complementary information for recognizing articulated objects.

INTRODUCTION

RELATED WORK

CRF Approaches Refining top-down detections Scoring bottom-up region

hypotheses

REGION GENERATION

Uses bottom-up regions as object candidates

Generate object candidates building on the segmentation method of [4]

Compute UCMs at three resolutions of the input image

[4] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE Trans. on PAMI, 2011.

REGION REPRESENTATION

Part Compatibility Features Part Activations

use the poselet framework introduced in [8, 7] use pre-trained models and masks from [9]

Part-Based Region Ranking

|I | : the total area of the imageα = (α1, ..., α6) ∈ N6

[7] L. Bourdev, S. Maji, T. Brox, and J. Malik. Detecting people using mutually consistent poselet activations. In Proc. ECCV, 2010.[8] L. Bourdev and J. Malik. Poselets: Body part detectors trained using 3d human pose annotations. In Proc. ICCV, 2009.[9] T. Brox, L. Bourdev, S. Maji, and J. Malik. Object segmentation by alignment of poselet activations to image contours. In Proc. CVPR, 2011

REGION REPRESENTATION

Part Compatibility Features Part-Based Region Ranking

P = {P1, ..., PA}

REGION REPRESENTATION

Global Appearance Features a set of first-order appearance cues defined

on the region support Shape, Color, Texture

Semantic Contours Features 4 region features per semantic contour map

Generic geometrical properties 16 generic geometric properties for each

region

REGION REPRESENTATION

Multi-Class Features the three high-level descriptor types are

category-specific and the low-level geometric properties are shared

REGION SCORING

Predict the probability of belonging to each category of interest for each object candidate

After classification, each region is assigned a score for all the categories of interest

PIXEL CLASSIFICATION

Train a final set of classifiers that operate on pixels rather than on regions average maximum non-max suppression

EXPERIMENTS

Control Experiments

Calibration of multiple detectors through pixel classification

EXPERIMENTS

Test set performance

EXPERIMENTS