Visual Recognition With Humans in the Loop

1

Visual Recognition With Humans in the Loop

Steve Branson

Catherine Wah

Florian Schroff

Boris Babenko

Serge Belongie

Peter WelinderPietro Perona

ECCV 2010, Crete, Greece

2

What type of bird is this?


3…?

Field Guide

4


Computer Vision

?

5


Bird?

Computer Vision

6


Chair?Bottle?

Computer Vision

7

Parakeet Auklet

• Field guides difficult for average users

• Computer vision doesn’t work perfectly (yet)

• Research mostly on basic-level categories

8

Visual Recognition With Humans in the Loop

Parakeet Auklet

What kind of bird is this?

9

Levels of Categorization

Airplane? Chair? Bottle? …

Basic-Level Categories

[Griffin et al. ‘07, Lazebnik et al. ‘06, Grauman et al. ‘06, Everingham et al. ‘06, Felzenzwalb et al. ‘08, Viola et al. ‘01, … ]

10


American Goldfinch? Indigo Bunting? …

Subordinate Categories

[Belhumeur et al. ‘08 , Nilsback et al. ’08, …]

11


Yellow Belly? Blue Belly?…

Parts and Attributes

[Farhadi et al. ‘09, Lampert et al. ’09, Kumar et al. ‘09]

12

Visual 20 Questions GameBlue Belly? no

Cone-shaped Beak? yes

Striped Wing? yes

American Goldfinch? yes

Hard classification problems can be turned into a sequence of easy ones

13

Recognition With Humans in the Loop

Computer Vision

Cone-shaped Beak? yes

American Goldfinch? yes

Computer Vision

• Computers: reduce number of required questions• Humans: drive up accuracy of vision algorithms

14

Research Agenda

Heavy Reliance on Human Assistance

More Automated

Computer Vision

Improves

Blue belly? noCone-shaped beak? yesStriped Wing? yesAmerican Goldfinch? yes

Striped Wing? yesAmerican Goldfinch? yes

Fully AutomaticAmerican Goldfinch? yes

2010 2015

2025

15

Field Guides

www.whatbird.com

16

Field Guides

www.whatbird.com

17

Example Questions

18

Example Questions

19

Example Questions

20

Example Questions

21

Example Questions

22

Example Questions

23

Basic Algorithm

Input Image ( )xQuestion 1:

Is the belly black?

Question 2:Is the bill hooked?

Computer Vision

A: NO

A: YES

)|( xcp

),|( 1uxcp

),,|( 21 uuxcp

1u

2u

Max Expected Information Gain


…

24

Without Computer Vision

Input Image ( )xQuestion 1:

Is the belly black?

Question 2:Is the bill hooked?

Class Prior

A: NO

A: YES

)(cp

)|( 1ucp

),|( 21 uucp

1u

2u



…

25

Basic Algorithm

Select the next question that maximizes expected information gain:• Easy to compute if we can to estimate probabilities of

the form:)...,,|( 21 tuuuxcp

Object Class

Image Sequence of user responses

26

Basic Algorithm

Zxcpcuuup t )|()|...,( 21

Model of user responses

Computer vision

estimate

)...,,|( 21 tuuuxcp

Normalization factor

27

Basic Algorithm

Zxcpcuuup t )|()|...,( 21

Model of user responses

Computer vision

estimate

)...,,|( 21 tuuuxcp

Normalization factor

28

Modeling User Responses

ti it cupcuuup...121 )|()|...,(• Assume:

• Estimate using Mechanical Turk)|( cup i

grey red

blac k

whi

tebr

own

blue

grey red

blac k

whi

tebr

own

blue

grey red

blac k

whi

tebr

own

blue

Definitely

Probably Guessing

What is the color of the belly?

Pine Grosbeak

29

Incorporating Computer Vision

• Use any recognition algorithm that can estimate: p(c|x)

• We experimented with two simple methods:

)}(exp{)|( xmxcp 1-vs-all SVM

i i capxcp )|()|(

Attribute-based classification

[Lampert et al. ’09, Farhadi et al. ‘09]

30

Incorporating Computer Vision

[Vedaldi et al. ’08, Vedaldi et al. ’09]

Self SimilarityColor Histograms Color Layout

Bag of WordsSpatial Pyramid

Geometric Blur Color SIFT, SIFT

Multiple Kernels

•Used VLFeat and MKL code + color features

31

Birds 200 Dataset•200 classes, 6000+ images, 288 binary attributes

•Why birds?

Black-footed Albatross

Groove-Billed Ani

Parakeet Auklet Field Sparrow Vesper Sparrow

Arctic Tern Forster’s Tern Common Tern Baird’s Sparrow Henslow’s Sparrow

32


•Why birds?


Groove-Billed Ani



33


•Why birds?


Groove-Billed Ani



34

Results: Without Computer Vision

Comparing Different User Models

35

Results: Without Computer Vision

if users answers agree with field guides…Perfect Users: 100% accuracy in 8≈log2(200) questions

36

Results: Without Computer VisionReal users answer questions

MTurkers don’t always agree with field guides…

37

Results: Without Computer VisionReal users answer questions

MTurkers don’t always agree with field guides…

38

Results: Without Computer VisionProbabilistic User Model: tolerate imperfect user responses

39

Results: With Computer Vision

40


Users drive performance: 19% 68%

Just Computer Vision19%

41


Computer Vision Reduces Manual Labor: 11.1 6.5 questions

42

Examples

Without computer vision: Q #1: Is the shape perching-like? no (Def.)With computer vision: Q #1: Is the throat white? yes (Def.)

Western Grebe

Different Questions Asked w/ and w/out Computer Vision

perching-like

43

Examples

computer vision

Magnolia Warbler

User Input Helps Correct Computer Vision

Is the breast pattern solid? no (definitely)

Common Yellowthroat

Magnolia Warbler

Common Yellowthroat

44

Recognition is Not Always SuccessfulAcadian Flycatcher

Least Flycatcher

Parakeet Auklet

Least Auklet

Is the belly multi-colored? yes (Def.)

Unlimited questions

Summary

11.1 6.5 questions

Computer vision reduces manual labor

Users drive up performance

19%

45

Recognition of fine-grained categories

More reliable than field guides

Summary

11.1 6.5 questions



19%

46



Summary

11.1 6.5 questions



19%

47



Summary

11.1 6.5 questions



19%

48



49

Future Work• Extend to domains other than birds• Methodologies for generating questions• Improve computer vision

50

Questions?

Project page and datasets available at:http://vision.caltech.edu/visipedia/

http://vision.ucsd.edu/project/visipedia/

http://vision.caltech.edu/visipedia/

http://www.vision.caltech.edu/visipedia/

Documents

Visual Recognition With Humans in the Loop