Where computer vision needs help from computer science (and machine learning)

Bill FreemanElectrical Engineering and Computer Science Dept.

Massachusetts Institute of TechnologySeptember 9, 2009

Outline

• My background• Computer vision applications• Computer vision techniques and problems:

– Low-level vision: underdetermined problems– High-level vision: combinatorial problems– Miscellaneous problems

At Photokina, in Cologne, Germany

Me (Foreign Expert) and my wife (English teacher), riding from the Foreigners’ Cafeteria at the Taiyuan University of Technology,

Shanxi, China

While in China, I read this book (to be re-issued by MIT Press this year), and got very excited about computer vision. Studied for PhD at MIT.

Worked for 9 years at Mitsubishi Electric Research Labs, an

industrial research lab doing fundamental research across the

street from MIT.

2001 – present, MIT

Infinite images

Joint work with:Biliana KanevaJosef SivicShai AvidanAntonio Torralba

A computer graphics application of belief propagation for optimal seam finding

The image database

•We have collected ~6 million images from Flickr based on keyword and group searches

– typical image size is 500x375 pixels– 720GB of disk space (jpeg compressed)

Image representation

Color layout

GIST [Oliva and Torralba’01]

Original image

Obtaining semantically coherent themesWe further break-up the collection into themes of semantically coherent scenes:

Train SVM-based classifiers from 1-2k training images [Oliva and Torralba, 2001]

Basic camera motions

Forward motion Camera rotation Camera pan

Starting from a single image, find a sequence of images to simulate a camera motion:

3. Find a match to fill the missing pixels

Scene matching with camera view transformations: Translation

1. Move camera

2. View from the virtual camera

4. Locally align images

5. Find a seam

6. Blend in the gradient domain

4. Stitched rotation

Scene matching with camera view transformations: Camera rotation

1. Rotate camera

2. View from the virtual camera

3. Find a match to fill-in the missing pixels

5. Display on a cylinder

More “infinite” images – camera translation

Virtual space as an image graph

ForwardRotate (left/right)

Pan (left/right)

• Nodes represent Images

• Edges represent particular motions:

• Edge cost is given by the cost of the image match under the particular transformation

Image graph

Kaneva, Sivic, Torralba, Avidan, and Freeman, Infinite Images, to appear in Proceedings of IEEE.

Virtual image space laid out in 3D

Kaneva, Sivic, Torralba, Avidan, and Freeman, Infinite Images, to appear in Proceedings of IEEE.