Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, Michael Felsberg

Discriminative Correlation Filters (DCF)

Applications

• Object recognition

• Object detection

• Object tracking

– Among state-of-the-art since 2014

– KCF, DSST, HCF, SRDCF, Staple …

2Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Discriminative Correlation Filters (DCF)

3Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Single-resolution feature map

Limitations:Coarse output

scores

DCF Limitations:1. Single-resolution feature map

• Why a problem?

– Combine convolutional layers of a CNN

• Shallow layers: low invariance – high resolution

• Deep layers: high invariance – low resolution

• How to solve?

– Explicit resampling?

• Artefacts, information loss, redundant data

– Independent DCFs with late fusion?

• Sub-optimal, correlations between layers

4Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

DCF Limitations:2. Coarse output scores

• Why a problem?

– Accurate localization

• Sub-grid (e.g. HOG grid) or sub-pixel accuracy

• More accurate annotations=> less drift

• How to solve?

– Interpolation?

• Which interpolation strategy?

– Interweaving?

• Costly

5Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

DCF Limitations:3. Coarse labels

• Why a problem?

– Accurate learning

• Sub-grid or sub-pixel supervision

• How to solve?

– Interweaving?

• Costly

– Explicit interpolation of features?

• Artefacts

6Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Our Approach: Overview

7Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Continuous filters

Continuous output

Multi-resolution features

Multiresolution Features

8Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Interpolation Operator

9Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Interpolation Operator

10Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Convolution Operator

11Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Training Loss

12Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

[Danelljan et al., ICCV 2015]

Training Loss – Fourier Domain

13Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Training Loss – Fourier Domain

14Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Localization

15Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Localization

16Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

How to set and ?

• Use periodic summation of functions :

• Gaussian function for

• Cubic spline kernel for

• Fourier coefficients with Poisson’s summation formula:

17Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Object Tracking Framework: Features

• VGG network

– Pre-trained on ImageNet

– No fine-tuning on application specific data

18Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

C-COT: Conjugate Gradient

Only need to evaluate

No sparse matrix handling

Finite memory

Warm starting: non-trivial

Tuning of pre-conditioners

SRDCF: Gauss-Seidel

Explicit computation of

Sparse matrix handling

“Infinite” memory

Warm starting: trivial

Object Tracking Framework: Optimization

Solving

19Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Object Tracking Framework: Pipeline

• Simple: … – Track – Train – Track – Train – …

• No thresholds

• No hidden “tricks”

20Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Experiments: Object Tracking

• 3 datasets: OTB-100, TempleColor, VOT2015

• Layer fusion on OTB:

• Compared to explicit resampling in DCF

– Performance gain: AUC

– Efficiency gain: data size

21Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Experiments: OTB (100 videos)

22Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Experiments: Temple-Color (128 videos)

23Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Experiments: VOT2016

24Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

[Matej et al., ECCV VOT workshop 2016]

Object Tracking: Speed

• With CNN features: slow ~1 FPS (no GPU)

• With HOG features: ~ real time at SRDCF performance

25Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Feature Point Tracking Framework

• Grayscale pixel features,

• Uniform regularization,

26Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Experiments: Feature Point Tracking

• The Sintel dataset

27Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Feature Point Tracking: KITTI

28Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

C-COT

KLT

Future Work

• Features

– Fine tuning

– Unsupervised learning

• Optimization

– Warm start in CG (theory and heuristics)

– Preconditioners

– Implementation aspects

– Alternative strategies or update rules

• Further explore of the continuous formulation

29Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Conclusions

• Continuous domain learning formulation

– Multi-resolution deep feature maps

– Sub-pixel accurate localization

– Sub-pixel supervision

• Superior results for two applications

– Object tracking

– Feature point tracking

30Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

Oral and poster: O-4B-03

Friday afternoon(last session)

31Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

www.liu.se

Martin Danelljan