Efficient Spatiotemporal Grouping Using the Nyström

Method

Charless Fowlkes, U.C. Berkeley

Serge Belongie, U.C. San Diego

Jitendra Malik, U.C. Berkeley

Grouping With Pairwise Affinities

Sarkar and Boyer (1996), Shi and Malik (1997), Perona and Freeman (1998), Gdalyahu, Weinshall, and Werman (2000), .....

• Compute the similarities between pairs of points in the image

• Find groups of points which have high similarity with each other and low similarity with the rest of the image.

Normalized Cuts

2. Find the leading eigenvectors of the Normalized Laplacian

3. Segment the image using the leading eigenvectors

1. Compute matrix K which contains the pairwise similarities

Computational Complexity: Need to find eigenvectors of anNxN matrix where N is the number of pixels.

Other spectral partitioning techniques have same complexity.

Spatiotemporal grouping

Adelson and Bergen (1985), Bolles, Baker and Marimont (1987), Shi and Malik (1998)

Computational Problem

• Hard to exploit pairwise clustering techniques since 256x384x30 frames entails ~1013 pairwise similarities.

• How can we overcome this problem?

Coping with the computational burden

1 Zero out small entries in the affinity matrix Shi and Malik (97,98)

2 Exploit redundancy between rows of the affinity matrix (this talk)

Outline

• Exploiting Redundancy• The Nyström approximation• Application to segmenting video sequences

Exploiting Redundancy

Exploiting Redundancy

Compute Affinity Matrix

Exploiting Redundancy

Compute LeadingEigenvectors of Normalized Laplacian

Exploiting Redundancy

Choose sample points

Exploiting Redundancy

Compute strip of KCompute strip of K

Exploiting Redundancy

Exploiting Redundancy

Interpolate complete eigenvectors

Outline

• Exploiting Redundancy• The Nyström approximation• Application to segmenting video sequences

Approximating eigenfunctions

E. J. Nyström (1929) Baker (1977) Williams and Seeger (2001)

Interpolate eigenfunctions using The Nyström Extension:

We would like to find numerical solutions to:

Matrix Completion

Affinity Matrix:

Approximate it with:

Approximation Error:

Approximation Error

Extrapolating Eigenvectors

Just matrix notation for the Nyström extension

Diagonalize approximate K to get complete eigenvectors

Nyström-NCuts Algorithm

1 Choose sample points in image

2 Compute similarities for A and B blocks of K

3 Compute row sums to estimate degree

4 Normalize A and B blocks by degree

5 Compute approximate eigenvectors and orthogonalize

6 Cluster the embedded points using k-means

Outline

• Exploiting Redundancy• The Nyström approximation• Application to segmenting video sequences

Affinity Function for VideoPairwise affinity function between pixels in a video sequence makes of three cues

-Similarity in color-Proximity in time and space-Common Fate (similarity in optical flow)

We use squared-exponential kernel with diagonal weighting

Performance

• Segmenting a 5 frame video sequence at 120x150 resolution (~100,000 pixels) takes less than 1 minute in MATLAB on a PC

Conclusions

• Applied the Nystrom approximation to Normalized Cuts

• Exploited redundancy in image sequences in order to perform efficient spatiotemporal grouping

K-Way Normalized Cuts

Find the leading eigenvectors of Normalized Laplacian

Embed data and cluster

jth pixel

ith vector

V

Sampling the image