Efficient Classification for Additive Kernel SVMsacberg.com/papers/mbm2012pami.pdf1 Efficient Classification for Additive Kernel SVMs Subhransu Maji 1, Alexander C. Berg2, and

1

Efficient Classification for Additive Kernel SVMsSubhransu Maji1, Alexander C. Berg2, and Jitendra Malik1

[email protected], [email protected], [email protected] Science Division, EECS 2Computer Science Department,

University of California, Berkeley Stony Brook University

Abstract—We show that a class of non-linear kernel SVMs admit approximate classifiers with run-time and memory complexity thatis independent of the number of support vectors. This class of kernels which we refer to as additive kernels, include the widely usedkernels for histogram based image comparison like intersection and chi-squared kernels. Additive kernel SVMs can offer significantimprovements in accuracy over linear SVMs on a wide variety of tasks while having the same run-time, making them practical forlarge scale recognition or real-time detection tasks. We present experiments on a variety of datasets including the INRIA person,Daimler-Chrysler pedestrians, UIUC Cars, Caltech-101, MNIST and USPS digits, to demonstrate the effectiveness of our method forefficient evaluation of SVMs with additive kernels. Since its introduction, our method has become integral to various state of the artsystems for PASCAL VOC object detection/image classification, ImageNet Challenge, TRECVID, etc. The techniques we propose canalso be applied to settings where evaluation of weighted additive kernels is required, which include kernelized versions of PCA, LDA,regression, k-means, as well as speeding up the inner loop of SVM classifier training algorithms.

Index Terms—Image Classification, Support Vector Machines, Efficient Classifiers, Additive Kernels

✦

1 INTRODUCTION

Consider sliding window detection, one of the leadingapproaches for detecting objects in images like faces [42],[61], pedestrians [42], [10], [16] and cars [44]. In thisapproach, first, a classifier is trained to recognize anobject at a fixed “pose” - for example, as shown inFigure 1, one may train a classifier to classify 64×96 pixelpedestrians which are all centered and scaled to the samesize, from background. In order to detect pedestrians atarbitrary location and scale in an image, the classifieris evaluated by varying the location and scale of theclassification window. Finally, detections are obtained byfinding peaks of the classification score over scales andlocations, a step commonly referred to as non-maximumsuppression. Although this approach is simple – theclassifier does not have to deal with invariance – a keydrawback of this approach is computational complexity.On typical images these classifiers can be evaluatedseveral tens of thousands of times. One may also want tosearch over aspect ratios, viewpoints, etc., compoundingthe problem. Therefore efficient classifiers are crucial foreffective detectors.

Discriminative classifiers based on Support Vector Ma-chines (SVMs) and variants of boosted decision treesare two of the leading techniques used in vision tasksranging from object detection [42], [61], [10], [16], multi-category object recognition in Caltech-101 [20], [32], totexture discrimination [67]. Classifiers based on boosteddecision trees such as [61], have faster classificationspeed, but are significantly slower to train. Furthermore,the complexity of training can grow exponentially withthe number of classes [55]. On the other hand, given theright feature space, SVMs can be more efficient during

Fig. 1. A typical “sliding window” detection pipeline.

training. Part of the appeal of SVMs is that, non-lineardecision boundaries can be learnt using the “kerneltrick” [50]. However, the run-time complexity of a non-linear SVM classifier can be significantly higher thana linear SVM. Thus, linear kernel SVMs have becomepopular for real-time applications as they enjoy bothfaster training and faster classification, with significantlyless memory requirements than non-linear kernels.

Although linear SVMs are popular for efficiency rea-sons, several non-linear kernels are used in computervision as they provide better accuracy. Some of the mostpopular ones are based on comparing histograms oflow level features like color and texture computed overthe image and using a kernel derived from histogramintersection or chi squared distance to train a SVMclassifier. In order to evaluate the classification function,a test histogram is compared to a histogram for eachof the support vectors. The number of support vectorscan often be a significant fraction of the training data,

Digital Object Indentifier 10.1109/TPAMI.2012.62 0162-8828/12/$31.00 © 2012 IEEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCEThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.

2

so this step is computationally very expensive as thetest time scales linearly with the number of supportvectors. This paper presents and analyzes a technique togreatly speed up that process for histogram comparisonfunctions that are additive - where the comparison is alinear combination of functions of each coordinate ofthe histogram. In particular we show it is possible toevaluate the classifier to arbitrary precision in timeindependent of the number of support vectors – similarto that of a linear SVM.

This more efficient approach makes SVMs with addi-tive kernels – used in many of the current most success-ful object detection/recognition algorithms – efficientenough to apply much more broadly, even possibly toreal-time applications. The class of kernels includes thepyramid matching or intersection kernels used in Grau-man & Darell [20] ; and Lazebnik, Schmid & Ponce [32],and the chi squared kernel used by Varma & Ray [57];and Chum & Zisserman [8], which together representsome of the best results in image and object recognitionon the Caltech [14] and PASCAL VOC [12] datasets.

Although the results in this paper apply to any addi-tive kernel, we begin by analyzing the histogram inter-section kernel, Kmin(ha, hb) =

∑imin (ha(i), hb(i)), that

is often used as a measurement of similarity betweenhistograms ha and hb. Because it is positive definite [54]for non-negative features and conditionally positive def-inite for arbitrary features [34], it can be used as akernel for discriminative classification using SVMs. Re-cently, intersection kernel SVMs (henceforth referred toas IKSVMs), have become popular with the introductionof pyramid match kernel [20] and spatial pyramid matchkernel [32] for object detection and image classification.Unfortunately this success typically comes at great com-putational expense compared to simpler linear SVMs,because non-linear kernels require memory and compu-tation linearly proportional to the number of supportvectors for classification.

In this paper we show the following:• SVMs using the histogram intersection kernel can

be exactly evaluated exponentially faster than thestraight forward implementation used in the pre-vious state of the art, as has been previously shownin [25], and independently in our own work in [35](Section 3).

• A generalization allows arbitrary additive kernelSVMs to be evaluated with the same “big O”computational cost, as linear SVMs (Section 4), aswell as significantly reducing the memory overhead,making them practical for detection and real timeapplications.

• We show that additive kernels arise naturally inmany computer vision applications (Section 5), andare already being used in many state of the artrecognition systems.

• Additive kernels, such as histogram intersectionare sufficiently general, i.e., the corresponding ker-nel SVM classifier, can represent arbitrary additive

classifiers. The difference between additive kernelscan be analyzed mainly in terms of the impliedregularization for a particular kernel. This helpsus to understand both the potential benefit andthe inherent limitations of any additive classifier,in addition to shedding some light on the trade-offs between choices of additive kernels for SVMs(Section 6).

• Our approach can be computationally more efficientcompared to some of the recently proposed methodsand the previous state of the art in kernel classifierevaluation (Section 7).

• Combining these efficient additive classifiers with anovel descriptor provides an improvement over thestate of the art linear classifiers for pedestrian de-tection, as well for many other datasets (Section 8).

• These techniques can be applied generally to set-tings where evaluation of weighted additive kernelsis required, including kernel PCA, kernel LDA, andkernelized regression, kernelized k-means as well asefficient training (Section 9).

2 SUPPORT VECTOR MACHINES

We begin with a review of support vector machines forclassification. Given labeled training data of the form{(yi,xi)}

Ni=1, with yi ∈ {−1,+1}, xi ∈ R

n, we use a C-SVM formulation [9]. For the linear case, the algorithmfinds a hyperplane which best separates the data byminimizing :

τ(w, ξ) =1

2||w||2 + C

N∑i=i

ξi (1)

subject to yi(w·xi+b) ≥ 1−ξi and ξi ≥ 0, where C > 0, isthe tradeoff between regularization and constraint viola-tion. For a kernel on data points, K(x, z) : Rn×R

n → R,that is the inner product, Φ(x) · Φ(z), in an unrealized,possibly high dimensional, feature space, one can obtainthe same by maximizing the dual formulation :

W (α) =

N∑i=i

αi −1

2

∑ij

αiαjyiyjK(xi,xj) (2)

subject to: 0 ≤ αi ≤ C and∑

αiyi = 0 (3)

The decision function is sign (h(x)), where:

h(x) =

m∑l=1

αlylK(x,xl) + b (4)

Notice that the dual formulation only requires accessto the kernel function and not the features Φ(.), allowingone to solve the formulation in very high dimensionalfeature spaces efficiently – also called the kernel trick. Forclarity, in a slight abuse of notation, the features, xl :l ∈ {1, 2, . . . ,m}, will be referred to as support vectors.Thus in general, m kernel computations are needed toclassify a point with a kernelized SVM and all m supportvector must be stored. Assuming these kernels can be


3

computed in O(n) time, the overall complexity of theclassifier is O(mn). For linear kernels we can do betterbecause, K(x, z) = x ·z, so h(x) can be written as h(x) =w ·x+ b, where w =

∑ml=1 αlylxl. As a result, classifying

with a linear SVM only requires O(n) operations, andO(n) memory.

3 FAST EXACT IKSVMS

We motivate our discussion using the histogram intersec-tion or the min kernel. Often similarity between imagesis obtained by comparing their distribution over lowlevel features like edge orientations, pixel color values,codebook entries, etc. These distributions could be rep-resented as histograms and a similarity measure likethe histogram intersection can be used. The histogramintersection kernel is known to be positive definite [54]for histogram based features and hence can be usedwith the standard SVM machinery. This representation ispopular for the “bag-of-words” approaches which haveled to state of the art results in many object detectionand classification tasks.

We first show that it is possible to speed up classifica-tion for intersection kernel SVMs (IKSVMs). This analy-sis was first presented in [25] and later independently inour own work [35]. For histogram based feature vectorsx, z ∈ R

n+, the intersection kernel Kmin(x, z) is defined

as:

Kmin(x, z) =

n∑i=1

min (xi, zi) (5)

and classification is based on evaluating:

h(z) =

m∑l=1

αlylKmin(z,xl) + b (6)

=

m∑l=1

αlyl

(n∑

i=1

min (zi, xl,i)

)+ b (7)

The non linearity of min prevents us from “collapsing”the weight vector in a similar manner for linear kernels.Thus the complexity of evaluating h(x) in the standardway is O(mn). The key property of intersection kernelsis that we can exchange the summations in equation 7to obtain:

h(z) =

m∑l=1

αlyl

(n∑

i=1

min (zi, xl,i)

)+ b (8)

=n∑

i=1

(m∑l=1

αlyl min (zi, xl,i)

)+ b (9)

=

n∑i=1

hi(zi) + b (10)

Thus the overall function h(·) can be rewritten as thesum of one dimensional functions hi(·), where :

hi(s) =

m∑l=1

αlyl min (s, xl,i) (11)

The complexity of computing each hi(s) in the naive wayis still O(m) with an overall complexity of computingh(x) still O(mn). We now show how to compute each hi

in O(logm) time.Consider the functions hi(s) for a fixed value of i. Let

xl,i denote the sorted values of xl,i in increasing orderwith corresponding α’s and labels as αl and yl. If s < x1,i

then hi(s) = s∑

l αl = 0, since∑

l αl = 0. Otherwise letr be the largest integer such that xr,i ≤ s. Then we have,

hi(s) =

m∑l=1

αlyl min (s, xl,i) (12)

=∑

1≤l≤r

αlylxl,i + s∑

r<l≤m

αlyl (13)

= Ai(r) + sBi(r) (14)

Where we have defined,

Ai(r) =∑

1≤l≤r

αlylxl,i, (15)

Bi(r) =∑

r<l≤m

αlyl (16)

Equation 14 shows that hi is piecewise linear. Further-more hi is continuous because:

hi(xr+1) = Ai(r) + xr+1Bi(r)

= Ai(r + 1) + xr+1Bi(r + 1).

Notice that the functions Ai and Bi are independent ofthe input data and depend only on the support vectorsand α. Thus, if we precompute them, then hi(s) canbe computed by first finding r, the position of s in thesorted list xl,i using binary search and linearly interpo-lating between hi(xr) and hi(xr+1). This requires storingthe xl as well as the hi(xl) or twice the storage of thestandard implementation. Thus the runtime complexityof computing h(x) is O(n logm) as opposed to O(nm),a speed up of O(m/ logm). This can be significant if thenumber of support vectors is large.

4 APPROXIMATE ADDITIVE KERNEL SVMS

It is possible to compute approximate versions of theclassifier even faster. Traditional function approximationquickly breaks down as the number of dimension in-crease. However for the intersection kernel SVMs wehave shown that the final classifier can be representedas a sum of one dimensional functions. As long as thekernel is “additive”, i.e., the overall kernel K(x,y) canbe written as,

K(x,y) =

n∑i=1

Ki(xi, yi) (17)

the resulting kernel SVM classifier is also additive, i.e.,h(s) can be written as,

h(s) =

n∑i=1

hi(si) + b (18)


4

where,

hi(si) =m∑l=1

αlylKi(si, xl,i) (19)

and xl,i denotes the i’th dimension of the l’th supportvector.

This decomposition allows us to approximate the finalclassifier by approximating each dimension indepen-dently. The simplest of these is a piecewise polynomialapproximation in which we represent the function ineach dimension as a piecewise polynomial function us-ing b sections, each of degree k. This requires b× (k+1)floating points per dimension. Classification requirestable lookup followed by the evaluation of a k degreepolynomial, which requires 2(k+1) floating point oper-ations using Euler’s method. Two special cases are thepiecewise constant and piecewise linear approximationscorresponding to degree k = 0 and k = 1 respectively. Inour experiments we restrict ourselves to these cases, asone can approximate any function arbitrary well. Thefinal classifier corresponds to a lookup table of sizem × (b + 1). The overall complexity of the classifierthen is O (2(k + 1)n) – essentially the same as that ofa linear SVM classifier.

In our MATLAB/C++ implementation the speed of thepiecewise linear approximations is about 5.5× slowerthan the linear classifier. The more expensive tablelookups can be avoided by rewriting the piecewise linearinterpolation as a dot product of a dense vector offunction values and a sparse vector indicating the binindices, for e.g. see [34] or [45]. These implementationsare essentially as fast as the linear classification method,especially, when there are a large number of classes andthe encoding time can be amortized over the number ofclasses.

Although these one dimensional functions can be pre-computed once for each classifier – this could becomea bottleneck if the number of classes are large, or ifthe classifier needs to be updated often for exampleduring training. To approximate these one dimensionalfunctions using a piecewise linear approximation, onehas to sample these functions at a fixed set of points.The complexity of evaluating these one dimensionalfunctions hi(si) =

∑m

l=1 αlylKi(si, xl,i) at b locationsis O(bm). When b is large, i.e. b >> logm, for theintersection kernel one can sample these functions fasterusing the exact IKSVM evaluation presented in Section 3in O((m+b) logm) time, making it the approach of choicefor certain applications.

5 ADDITIVE KERNELS IN COMPUTER VISION

We identify several naturally arising additive kernelsin computer vision applications, though we note thatvariants of these kernels also arise in natural languageprocessing, such as text classification, etc. There aretwo important classes of additive kernels used in thecomputer vision, which we describe next.

5.1 Comparing Histograms

Often similarity between images is obtained by compar-ing their distribution over low level features like edgeorientations, pixel color values, codebook entries, tex-tures, etc. These distributions are typically represented ashistograms and a similarity measure like the histogramintersection or the negative χ2 or l2 distance is used. Boththe histogram intersection kernel [54], and the χ2 kernelsare known to be positive definite for histogram basedfeatures and hence can be used with the standard SVMmachinery. See [3], [41] for a proof that the histogramintersection kernel and its variants are positive definiteand [2] for a proof for the χ2 kernel.

The histogram intersection kernel, Kmin, and the χ2

kernel, Kχ2 , for normalized histograms are defined asfollows:

Kmin(x, z) =

n∑i=1

min (xi, zi) , Kχ2(x, z) =

n∑i=1

2xizixi + zi

(20)Figure 2 visualizes these additive kernels. We also

note that the intersection kernel is conditionally positivedefinite for all features and hence can be used with SVMseven when the features are not histograms, i.e. theyneed not be positive or normalized. For proof see paper[34]. A special case worth mentioning is the generalizedhistogram intersection kernel [3] defined by :

K(x, z) =

n∑i=1

min(|xi|

β , |zi|β)

(21)

This is known to be positive definite for all β > 0.Chappelle et al. [7] observe that this remapping of thehistogram bin values by x → xβ , improves the per-formance of linear kernel SVMs to become comparableto RBF kernels on an image classification task over theCorel Stock Photo Collection. Simply square-rooting thefeatures with linear kernel, which is also called the Bhat-tacharyya kernel, has also shown to provide significantimprovements, when used with ”bag-of-words” stylefeatures, for various image classification and detectiontasks [59], [45]. This representation also arises in text clas-sification setting where the histograms represent countsof words in a document.

5.2 Approximate Correspondences

Another class of additive kernels are based on thematching sets of features between images. Two popularvariants are the pyramid match and the spatial pyramidmatch kernels. We describe each of them briefly.

Pyramid Match Kernel: Introduced by Graumanand Darell [20], [22], who proposed a way to mea-sure similarity between sets of features using partialcorrespondences between the elements in the sets. Thesimilarity measure reduces to a weighted histogram in-tersection of features computed in a multi-resolution his-togram pyramid, hence the name. This approach buildson Indyk and Thaper’s [27] approximation to matching


5

Klinear

(x,y) = xy

0

0.5

1

0

0.5

10

0.5

1

Kmin

(x,y) = min(x,y)

0

0.5

1

0

0.5

10

0.5

1

Kχ2 = 2xy/(x+y)

0

0.5

1

0

0.5

10

0.5

1

Fig. 2. Visualization of linear, intersection and χ2 kernelsfor one dimensional features. One can see that the χ2

kernel is a smoother version of the intersection kernel andis twice differentiable on the interior.

costs using l1 embeddings. An attractive feature of thismethod is that the matching has linear time complexityin the feature dimension, and naturally forms a Mercerkernel which enables it to be used with discriminativelearning frameworks like kernel SVMs. This kernel hasbeen used in various vision tasks like content-based-image-retrieval, pose estimation, unsupervised categorydiscovery [21] and image classification. This kernel isadditive because the overall kernel is simply a weightedhistogram intersection.

Spatial Pyramid Match Kernel: Lazebnik, Schmidand Ponce [32] introduced a similarity based on approx-imate global geometric correspondence of local featuresof images. Instead of a global histogram of features onecreates histograms of features over increasingly fine sub-regions of the image in a “spatial pyramid” representa-tion. Like the pyramid match kernel, the spatial match-ing is now approximated by the weighted histogramintersection of the multi-resolution spatial pyramid. Thisremarkably simple and computationally efficient exten-sion of an orderless bag-of-features has proved to beextremely useful, and has become a standard baselinefor various tasks which require image to image sim-ilarity like object detection, image classification, poseestimation, action recognition, etc. Many state of theart object detection and image classification results onPASCAL Visual Object Challenge [12], ImageNet [28]and TRECVID [53] challenge are based on variants ofthe kernel where the underlying features change. Nev-ertheless this kernel is also additive as the overall kernelis once again a weighted histogram intersection of theunderlying features.

6 LEARNING ADDITIVE CLASSIFIERS

Additive classifiers are based on functions of the form:

f(x) =∑i=1

fi(xi) (22)

i.e., the overall function f is a sum of one dimen-sional functions. Additive functions were popularized by

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.1

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.2

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.3

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.4

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.5

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.6

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.7

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.8

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 0.9

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

y

K(x

,y)

x = 1.0

Klinear

(x,y)

Kmin

(x,y)

Kχ2(x,y)

y > x

Fig. 3. Visualization of the basis functions of linear,intersection, χ2 kernels and decision stumps for onedimensional features.

Hastie and Tibshirani [23], for fitting statistics of data.Linear classifiers are the simplest additive classifierswhere each fi(xi) = wixi. By allowing arbitrary fi, ad-ditive models can provide better fits to the training datathan linear models. Our key insight in Section 4, was toobserve that if the kernel K is additive, then the learnedSVM classifier is also additive. Thus the standard SVMtraining machinery provides an efficient way to trainadditive classifiers compared to the traditional backfittingalgorithm [18]. Additive classifiers also arise in boostingwhen the weak-learners are functions of one dimension,for example, decision stumps, (xi > c). Hence thestandard AdaBoost algorithm [48], is yet another wayof training additive classifiers.

We now show that the additive classifiers based onhistogram intersection kernel are general, i.e., can rep-resent any additive function on the input features as alinear combination of intersection kernel of the featuresas shown by the next theorem.

Theorem 6.1. Let x1,x2, . . . ,xn be points in Rd ≥ 0 and

f(xi) = f1(xi,1) + f2(xi,2) + . . . + fd(xi,d), be an additivefunction, where xi,j denotes the value of j’th dimension of thei’th point. Then there exists α1, α2, . . . , αn such that f(xi) =∑

j αjKmin(xi,xj), ∀i = 1, 2 . . . , n.

Proof: We prove this by showing that there existsa weight vector w, in the Reproducing Kernel HilbertSpace of the intersection kernel, Kmin, such that w ·φ(xi) = f(xi). First we show that there is a weight vectorwk, for each fk, such that wk · φ(xj,k) = fk(xj,k). Thisfollows immediately from the fact that the gram matrixGk, consisting of entries Gk

ij = min(xi,k, xj,k) is full rankfor unique xi,k, and the system of equations, αGk = fk,has a solution (if the values are not unique, one canremove the repeated entries). Since the overall functionis additive, we can obtain the weight vector w with therequired property by stacking the weight vectors, wk,from each dimension. Thus by representer theorem, thereexists α such that w ·φ(xi) =

∑j αjKmin(xi,xj) = f(xi).

Note that the α is shared across dimensions and thisproof may be applied to any additive kernel whichsatisfies the property that the kernel in each dimension


6

0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5si

n(2π

y)

y0 0.2 0.4 0.6 0.8 1

0.75

0.8

0.85

0.9

0.95

1

1 −

(y

− 0

.5)2

y

0 0.2 0.4 0.6 0.8 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

0.25

< y

< 0

.75

y0 0.2 0.4 0.6 0.8 1

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

exp(

−10

0(y

− 0

.3)2 )

+ e

xp(−

100(

y −

0.6

)2 )

y

TrueK

linK

min

Kχ2

y > x

Fig. 4. Approximations of various one dimensional func-tions by linear, intersection, χ2 kernels and decisionstumps as the basis functions.

is full rank, for example the χ2 kernel.Thus the SVM classifier represents the overall function

as a linear combination of kernel functions in eachdimension. The one dimensional functions Ki(si, xl,i)for a fixed value of xl,i can be thought of as a basisfunction for each dimension of the classifier. Figure 3shows these basis functions for the intersection and χ2

kernels. Figure 4 shows several one dimensional func-tions approximated by a linear combination of 10 basisfunctions centered at 0.1, 0.2, . . .1.0 and a constant. Thelinear combination coefficients were found using linearleast-squares regression. The decision stumps (xi > c),gives us a piecewise constant approximation while thehistogram intersection gives a piecewise linear approx-imation and χ2 kernel gives smoother polynomial likeapproximation. Compared to linear case, kernels like theintersection, χ2 kernel and decision stumps are able toapproximate these functions much better.

7 PREVIOUS WORK

There are several approaches for speeding up classifi-cation using kernel SVM classifiers, which we discussbriefly next.

Approximate Kernel SVMs: For the histogram in-tersection kernel, Herbster [25] first proposed the fastevaluation algorithm we presented in Section 3. In ourearlier work [35] we independently proposed the samemethod for exact classification along with the approxi-mate method described in Section 4, which is more gen-eral and applies to arbitrary additive kernels. Recently,Rahimi and Recht [46], propose embeddings that approx-imate shift-invariant kernels, i.e., K(x,y) = f(|x − y|),using a feature map Φ, such that K(x,y) ∼ Φ(x) · Φ(y).Based on this analysis and our own work [35], [34],Vedaldi and Zisserman [59] propose embeddings whichapproximate a class of additive kernels that are ”homo-geneous”. This allows one to use the explicit form of

the classifier f(x) = w · Φ(x), instead of the kernelizedversion, which can be more efficient in some settings.We discuss some of these methods in Section 9.

However during classification for additive kernels,the piecewise linear approximation we proposed canbe much faster. To see this observe that the piecewiselinear approximation can be written as a dot product of aweight vector corresponding to the values of the functionsampled at uniformly spaced points, with a sparse vectorcorresponding to the projection of the data on to auniformly spaced linear B-Spline basis centered at thesepoints (also see [34]). In this representation, evaluatingthe classifier requires only two multiplications and oneaddition per dimension, which can be much smallercompared to the approximate embeddings of [59].

Another line of approach applicable to Gaussian ker-nels is the work of Yang et al. [66] who use the fastGauss transform to build efficient classifiers – howeverthis is applicable when the feature dimension is verysmall, typically less than ten.

Reduced Set Methods: For general kernels, a classof methods, known as “reduced set methods”, approxi-mate the classifier by constructing representations usinga small subset of data points, typically much smallerthan the number of support vectors. These set of pointscan be the set of input points themselves as in thework of [6], [43], where the most representative supportvectors are kept as a post processing step. Instead ofhaving a single approximation, one can have a seriesof approximations with more and more points to obtaina cascade of classifiers, an idea which has been usedin [47] to build fast face detectors. Another class ofmethods build classifiers by having a regularizer in theoptimization function which encourages sparseness, (e.g.l1-norm on the alphas) or pick support vectors in agreedy manner till a stopping criteria is met [30]. Thesemethods may be able to reduce the number of supportvectors by a order of magnitude, but are still significantlyslower than a linear SVM. Often this come at the expenseof classification accuracy. Thus, these approaches are notcompetitive when the kernel is additive compared to ourapproach.

Coarse to Fine methods: The coarse to fine ap-proach for speeding up the classification is popular inmany realtime vision applications. Simpler features andclassifiers are used to reject easy examples quickly ina cascade. This idea has been applied to face [24], [4]and pedestrian detection [64] to achieve an order ofmagnitude speedup in the overall detection time. Meth-ods like branch and bound [31], context [26], bottom-up regions [56], Hough transformation [37], [33], etc.,improve efficiency by reducing the number of classi-fier evaluations. This paper improves the efficiency ofthe underlying discriminative classifier, allowing morepowerful classifiers to be evaluated exponentially faster– in practice up to several thousand times faster thannaive implementations and entirely complementary tothe techniques mentioned for reducing the number of


7

classifier evaluations.

8 EXPERIMENTAL RESULTS

Since its introduction, our ideas for efficiently computingweighted combination for additive kernels has beenapplied to many applications like image-classification onCaltech-101 [15], PASCAL Visual Object Challenge [12],handwritten digits [36], video retrieval (TRECVID [53]),near-duplicate image detection [52], pedestrian detectionframeworks combining static image features and opticalflow [62], efficient classifiers for training large scaledata [34], [59], [63], etc. We summarize some of theseapplications in Section 10.

We present experiments on several image classificationand detection datasets and compare the performance oflinear, intersection as well as a non-linear kernel, suchas radial basis, or polynomial kernel. We also report thespeedup obtained by the piecewise linear approximationcompared to the naive method of evaluating the clas-sifier. Table 1 contains a summary of our results. Thepiecewise linear approximations are as accurate as theexact additive classifier classifier with about 100 pieceson all datasets. On various datasets the intersectionkernel SVM is significantly better than the linearSVM and often comparable to rbf-kernel SVM, whileoffering up to three orders of magnitude speedup. Thedetails of each dataset and the features are presentedbelow.

8.1 Toy Example : Learning a circle

We illustrate the additive kernel approximation using atoy example. The data is generated by sampling pointsfrom a two dimensional Gaussian and all points withina certain radius of the center belong to one class andthe points outside belong to the other class as seen inFigure 5 (top-left).

A linear classifier works poorly in this case as no twodimensional line can separate the points well. However,the intersection kernel SVM is able to achieve an accu-racy of 99.10% on this data. This is because it is ableto approximate the circle which is an additive function(x2 + y2 ≤ r), using two one-dimensional curves, x2 andy2. Figure 5 shows the learned classifier represented withvarying number of bins using a piecewise linear approxi-mation as well as the classification accuracy as a functionof the number of approximation bins. The accuracy satu-rates with 10 bins. On more realistic datasets the numberof bins required for a good approximation depends on ofsmoothness of the underlying function, but empirically100 bins were sufficient in all our experiments.

8.2 MNIST and USPS Digits

The MNIST dataset1 was introduced by Yann LeCun andCorinna Cortes and contains 60, 000 examples of digits

1. http://yann.lecun.com/exdb/mnist/

−5 0 5−5

−4

−3

−2

−1

0

1

2

3

4

5Training Data

x

y

−1 0 1 20

0.5

1

1.5

2

2.5

3

3.5

4

4.5

x value

f x(x)

learned function: x−axis

−2 −1 0 1 20

0.5

1

1.5

2

2.5

3

3.5

4

4.5

y value

f y(y)

learned function: y−axis

1 2 4 8 25 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

clas

sific

atio

n ac

cura

cy

number of bins

approximation accuracy

1

2

4

8

25

100

Fig. 5. Toy example. (Top Left) The training data (BottomLeft) Accuracy of the learned classifier approximated bya piecewise linear function of varying number of bins.(Top Right and Top Left) Learned functions on x and ydimensions respectively as well as the piecewise linearapproximations using a varying number of bins.

0 − 9 for training and 10, 000 examples for testing. Asbefore we construct features based on histograms overoriented responses computed by convolving the imagewith a Gaussian derivative filter with σ = 2 and bin theresponse in 12 orientations. The images in this datasetare 28× 28 pixels and we collect histograms over blocksof sizes 28× 28, 14× 14 , 7× 7 and 4× 4 pixels. We alsofound that adding overlapping blocks which overlap byhalf the block size improves performance at the expenseof increasing the feature vector dimension by a factor ofabout four. This is similar in spirit of the overlappingblocks in the HOG descriptor in the pedestrian detectorof [10]. These features with an IKSVM classifier achievesan error rate of 0.79%, compared to an error rate of1.44% using linear and 0.56% using polynomial kernel.Similar features and IKSVM achieves an error rate of3.4% on the much harder USPS dataset. We refer thereaders to [36], for a complete set of experiments for thetask of handwritten digit classification. Figure 6 showsthe errors made by our digit recognition system on theMNIST dataset.

A key advantage is that the resulting IKSVM clas-sifier is very fast. The estimated number of multiply-add operations required by the linear SVM is about40K while the intersection kernel requires about 125Koperations including the time to compute the features.This is significantly less than about 14 million operationsrequired by a polynomial kernel SVM reported in thework of [11]. The reduced set methods [5](1.0% error)requires approximately 650K operations, while the neu-


8

Dataset Features Measure Linear SVM IKSVM Kernel SVM Kernel Type

Toy Dataset Raw Accuracy 51.9% 99.10% (22×) 99.50% rbf, γ = 0.5

MNIST Digits OV-SPHOG Error Rate 1.44% 0.77% (1200×) 0.56% poly, d = 5

USPS Digits Raw Pixels Error Rate 11.3% 8.7% (24×) 4.0% poly, d = 3OV-SPHOG Error Rate 3.4% 3.4% (26×) 3.2% poly, d = 5

INRIA Pedestrian SPHOG Recall at 2 FPPI 43.12% 86.59% (2594×) -DC Pedestrians SPHOG Accuracy 72.19± 4.40% 89.03± 1.39% (2253×) 88.13± 1.43% rbf, γ = 175

Caltech 101, 15 examples SPHOG Accuracy 38.79± 0.94% 50.10 ± 0.65% (37×) 44.27± 1.45% rbf, γ = 25030 examples SPHOG Accuracy 44.33± 1.33% 56.59 ± 0.77% (62×) 50.13± 1.19% rbf, γ = 250

UIUC Cars (Single Scale) SPHOG Precision at EER 89.8% 98.5% (65×) 93.0% rbf, γ = 2.0

TABLE 1Summary of our results. We show the performance using a linear, intersection and non-linear kernel as well as thespeedup obtained by a piecewise linear approximation of the intersection kernel classifier on each dataset. The rbf

kernel is defined as K(x,y) = exp(−γ(x− y)2

)and the poly kernel of degree d, is defined as

K(x,y) = (1 + γ(x · y))d. All the kernel hyper-parameters were set using cross-validation.

4 → 9

116

2 → 7

322

5 → 3

341

6 → 0

446

3 → 5

450

8 → 2

583

4 → 9

741

3 → 5

939

8 → 9

948

1 → 3

957

6 → 0

1015

4 → 6

1113

7 → 2

1227

9 → 4

1233

9 → 5

1248

7 → 1

1261

8 → 2

1365

5 → 3

1394

8 → 7

1531

9 → 5

1710

5 → 2

1738

8 → 3

1879

9 → 4

1902

5 → 3

2036

4 → 9

2054

2 → 0

2099

6 → 0

2119

4 → 9

2131

9 → 8

2294

9 → 1

2388

9 → 4

2407

2 → 0

2463

2 → 4

2489

5 → 3

2598

6 → 1

2655

8 → 0

2897

9 → 5

2940

3 → 5

2954

6 → 0

3031

8 → 9

3290

6 → 0

3423

6 → 4

3521

5 → 0

3559

8 → 9

3728

4 → 6

3781

7 → 8

3809

1 → 3

3907

7 → 1

4028

9 → 8

4079

1 → 7

4202

2 → 7

4206

9 → 7

4225

7 → 3

4239

2 → 8

4249

9 → 5

4285

3 → 2

4444

6 → 8

4572

3 → 5

4741

9 → 8

4762

8 → 0

4808

6 → 0

4815

8 → 6

4880

8 → 2

5750

5 → 3

5938

3 → 8

5956

3 → 0

6012

8 → 9

6556

1 → 8

6573

7 → 1

6577

8 → 3

6626

2 → 1

8060

8 → 5

8409

7 → 2

9010

7 → 2

9016

6 → 4

9680

9 → 7

9693

5 → 6

9730

4 → 9

9793

2 → 7

9840

Fig. 6. All the errors made by the classifier on the MNISTdataset. On each image a → b means that the digit a wasmisclassified as b.

ral network methods like LeNet5 (0.9% error) requires350K and the boosted LeNet4 (0.7% error) requires 450Koperations. For a small cost for computing features weare able to achieve competitive performance while at thesame time are faster at both training and test time.

8.3 INRIA Pedestrians

The INRIA pedestrian dataset [10] was introducedas an alternate to the existing pedestrian datasets

Classification Method Detection Rate (2 FPPI) SpeedupLinear SVM 43.12 % -IKSVM (binary search) 86.59 % 473×

IKSVM (piecewise linear) 86.59 % 2594×

IKSVM (piecewise constant) 86.59 % 3098×

Dalal & Triggs [10] 79.63 % -Dalal & Triggs [10]* 82.51 % -

TABLE 2Detection rate at 2 FPPI on INRIA person dataset. Thelast run of [10] is obtained by running the detector usinga finer “scaleratio” of 1.05 between successive layers of

the image pyramid, instead of the default 1.1.

(e.g. MIT Pedestrian Dataset) and is significantly harderbecause of wide variety of articulated poses, variableappearance/clothing, illumination changes and complexbackgrounds. Linear kernel SVMs with Histograms ofOriented Gradients (HOG) features achieve high accu-racy and speed on this dataset [10]. We use the multi-scale HOG features introduced in [35] and train a inter-section kernel SVM on these features. The single scaleHOG used in the original paper [10] when used withIKSVM provides small improvements over the linearkernel, similar to those observed by using the rbf-kernel.We also found that the HOG with l1-normalization of thegradient based features works better with the intersec-tion kernel. The multi-scale HOG however outperformsl1-normalized HOG. Results are shown in Table 2 using100 bin approximation. Figure 7 shows sample detec-tions on this dataset.

8.4 Daimler Chrysler Pedestrians

We use the Daimler Chrysler pedestrian benchmarkdataset, created by Munder and Gavrila [39]. The datasetis split into five disjoint sets, three for training and


9

Fig. 7. Sample pedestrian detections on the INRIA person dataset using spHOG + IKSVM classifier.

two for testing. Each training set has 5000 positive andnegative examples each, while each test set has 4900positive and negative examples each. We report resultsby training on two out of three training sets at a timeand testing on each of the test sets to obtain six train-testsplits. Due to small size of the images (18× 36), we onlycompute the multi-level features with only three levels(L = 3) of pyramid with cell sizes 18×18, 6×6 and 3×3 atlevels 1, 2 and 3 respectively. The block normalization isdone with a cell size of wn×hn = 18×18. The features atlevel l are weighted by a factor cl = 1/4(L−l) to obtain a656 dimensional vector, which is used to train an IKSVMclassifier.

The classification results using the exact methods andapproximations are shown in Table 3. Our results arecomparable to the best results for this task [39]. TheIKSVM classifier is comparable in accuracy to the rbf-kernel SVM, and significantly better than the linear SVM.The speedups obtained for this task are significant dueto large number of support vectors in each classifier. Thepiecewise linear with 30 bins is about 2000× faster andrequires 200× less memory, with no loss in classificationaccuracy. The piecewise constant approximation on theother hand requires about 100 bins for similar accuraciesand is even faster.

Our unoptimized MATLAB implementation for com-puting the features takes about about 17ms per imageand the time for classification (0.02ms) is negligiblecompared to this. Compared to the 250ms required bythe cascaded SVM based classifiers of [39], our pipelineis 15× faster. Figure 8 shows some of the errors madeby our classifier.

8.5 Caltech 101

Our next set of experiments are on Caltech-101 [15]. Theaim here is to show that existing methods can be made

Classification Method Accuracy(%) SpeedupLinear SVM 72.19± 4.40 -IKSVM (binary search) 89.06± 1.42 485×

IKSVM (piecewise linear) 89.03± 1.39 2253×

IKSVM (piecewise constant) 88.83± 1.39 3100×

RBF-SVM 88.85± 1.13 –

TABLE 3Accuracy on Daimler-Crysler pedestrian dataset.

significantly faster, even when the number of supportvectors in each classifier is small. We use the frameworkof [32] and use our own implementation of their ”weakfeatures” and achieve an accuracy of 56.49% (comparedto their 54%), with 30 training and test examples perclass and one-vs-all classifiers based on IKSVM. Theperformance of a linear SVM using the same featuresis about 44.33%, while that of a rbf kernel is 50.13%. TheIKSVM classifiers on average have 185 support vectorsand a piecewise linear approximation with 60 bins is 62×faster and the piecewise constant approximation is 76×faster than a standard implementation, with no loss inaccuracy (see Table 4).

It is interesting to note the performance of one-vs-oneclassifiers as they are faster to train. With 15 training and50 test examples per category, one-vs-one classifiers givean accuracy of 47.43± 0.37 for intersection, compared to39.58±0.78 for linear kernel, with 5-fold cross validation.Increasing with number of training examples to 30,improves the performance to 53.80±2.43 for intersectionkernel compared to 45.66± 2.63 for linear kernel.

8.6 UIUC Cars

This dataset was collected at UIUC [1] and containsimages of side views of cars. The training set consists


10

Fig. 8. (Top Row) False negatives and (Bottom Row) false positives on the Daimler-Chrysler pedestrian dataset.

15 examples, 115.2± 15 SVs 30 examples, 185.0± 26 SVsClassification Method Accuracy(%) Speedup Accuracy(%) SpeedupLinear SVM 38.79± 0.94 - 44.33± 1.33 -IKSVM (binary search) 50.15± 0.61 11× 56.49± 0.78 17×

IKSVM (piecewise linear) 50.10± 0.65 37× 56.59± 0.77 62×

IKSVM (piecewise constant) 49.83± 0.62 45× 56.11± 0.94 76×

TABLE 4Classification accuracy of various methods on Caltech-101 dataset using 15 and 30 training examples per category.

The piecewise linear classifiers are up to 60× faster without loss in accuracy over the exact method.

Classification Method Performance(%) SpeedupLinear SVM 89.8 -IKSVM (binary search) 98.5 23×

IKSVM (piecewise linear) 98.5 65×

IKSVM (piecewise constant) 98.5 83×

RBF-SVM 93.0 -Agarwal & Roth [1] 79.0 -Garg et al. [19] 88.0 -Fregus et al. [17] 88.5 -ISM [33] 97.5 -Mutch & Lowe [40] 99.6 -Lampert et al. [31] 98.5 -

TABLE 5Performance at Equal Error Rate on UIUC cars dataset.

of 550 car and 500 non-car images. We test our methodson the single scale image test set which contains 170images with 200 cars. The images are of different sizesthemselves but contain cars of approximately the samescale as in the training images. Results are shown inTable 5. Once again the IKSVM classifier outperformsboth the linear and the rbf kernel SVM and is comparableto the state of the art. Figure 9 shows some of thedetections and mis-detections on this dataset.

8.7 Comparison to the “square-root” kernel

As has been noted earlier [7] and recently by severalothers [45], [60], simply square-rooting the features,also known as the “Bhattacharyya” kernel, can improvethe performance of linear classifiers significantly. Thismethod has the advantage that it requires no specialmachinery on top of linear training and testing algo-rithms. For example, on Caltech-101 dataset, Vedaldi andZisserman [60] observe a 12% improvement over linearSVM by square-rooting the features and an additional

3% improvement using an additive kernel. On PASCALVOC 2007 dataset, [45] observe a 4.8% improvementusing ”square-root” kernel compared to linear, and anadditional 1.6% improvement using the intersection ker-nel and a similar trend on the ImageNet dataset.

9 EXTENSIONS AND APPLICATIONS

The techniques and analysis we propose can be appliedgenerally to settings where evaluation of weighted ad-ditive kernels is required. This includes kernelized ver-sions of PCA, LDA, regression and k-means. In additionone can use our method to speed up the inner loopof training SVM classifiers. We describe some of theseextensions next.

9.1 Efficient Training Algorithms

Though the focus of this paper is on efficient evaluationof the classifier, our method can be used for trainingthe classifier itself. Although for an application clas-sification speed is the most important factor, trainingthese classifiers can become a bottleneck. Several train-ing algorithms use classification as an inner loop toidentify misclassified examples to update the classifier.When the number of training examples is very large,for example while training object detectors, one oftenemploys bootstrapping to train successive classifiers andcollect ”hard” examples by running the classifiers onlarge amounts of data. In such scenarios the overallpipeline can be sped up using the techniques proposedin this paper.

Recent advancements in the learning community haslead to training algorithms for linear SVMs which scalelinearly with the training data (LIBLINEAR [13], PE-GASOS [51], SVMperf [29]). However for general kernelSVMs the training times can still be very high. Often


11

Fig. 9. Example detections (green) and mis-detections (red) of the detector on UIUC cars dataset.

the implicit Reproducing Kernel Hilbert Space (RKHS),denoted by Φ(x), such that, K(x,y) = Φ(x) ·Φ(y), of thekernel is very high dimensional (possibly infinite), whichmakes it impractical to use the linear SVM techniquesdirectly in the RKHS.

It is possible to construct low dimensional or sparseembeddings which preserve the dot products in kernelspace approximately (K(x,y) ∼ Φ(x) · Φ(y)) to leveragefast linear SVM training algorithms. For shift invariantkernels this has been addressed in [46]. These kernelshave a special form K(x,y) = f(|x−y|). The histogramintersection and χ2 kernels are not shift invariant, butthe approximate versions of the classifier namely thepiecewise constant/linear classifiers can be thought ofas embeddings that preserve the kernel dot productapproximately. In particular [34], describes a sparse em-bedding which has at most twice the number of nonzero entires as the original feature space which allowsefficient linear training techniques to be used. Morerecently [59] propose alternate embeddings which arelow dimensional for additive kernels.

Another approach is to use the efficient classifica-tion techniques internally to speed up the training pro-cess [63]. The performance of these approximate clas-sifiers are often identical to that of the trained kernelSVMs while taking a fraction of the training time asshown by the experiments in [34], [59]. On severalbenchmark datasets related to image classification andobject detection, these techniques provide a significantimprovement in accuracy over linear SVMs while payingonly a small cost in training and test time, which rendersthis paradigm of practical importance.

9.2 Classifier Cascades

One can order the class of kernels based on the rep-resentation power, Linear ⊂ Additive ⊂ General, tocreate a cascade of detectors where a linear classifier isused to first discard the easy examples and rest are thenpassed on to an additive kernel classifier and so on. Thisidea has been used for object detection by [58] to buildcascades of object detectors based on multiple kernelsderived from color, texture and shape features and wasthe top performing method on the PASCAL 2008 VOCobject detection challenge.

9.3 Additive Kernel Methods

Finally the additive kernel approximation ideas applygenerally to various other tasks like regression, cluster-ing, ranking, principal components, etc. Methods likekernel PCA, LDA, Gaussian processes, etc., can benefitfrom our analysis to vastly speed up the computationtimes as well as reduce the runtime memory and storagerequirements. We describe some of these in this section.

9.3.1 Kernel Regression/Gaussian Processes

This is analogous to the classification case, except theoutputs yi are real valued instead of {+1,−1} for theclassification case. The optimization problem often in-volves minimizing something like the squared-error be-tween the prediction and the true value [50]. Similar tothe classification case and the final regression functionis of the form :

f(x) =

n∑i=1

αik(xi,x) + b (23)

9.3.2 Kernel PCA/LDA

Dimensionality reduction techniques like Principal Com-ponent Analysis (PCA) have been extended to arbitraryHilbert spaces using the kernel trick [49]. These howevercome at a great increase in computational and memorycost. For an arbitrary kernel K , the projection of a newdata point x to the nth principal component takes theform [50] :

cn(x) =

m∑i=1

αni K(xi,x) (24)

where αn is the nth eigenvector of the kernel matrixKij := K(xi,xj). For additive kernels one can use ourproposed techniques to both represent these principalcomponents compactly as well as compute the projec-tions of a new data point onto the principal componentsefficiently. This idea has been used in [45] to computelow dimensional representations of features for imageclassification.

Fisher Discriminant Analysis or Linear DiscriminantAnalysis (LDA) in the kernel space has a similarform as one of the projection vectors in PCA, i.e.,∑m

i=1 αiK(xi,x). The α vector is the solution to thekernelized version of the optimization problem in LDA.We refer the readers to [38] for details.


12

9.3.3 Kernel Codebooks

Kernelized similarity can be used for unsupervised clus-tering to construct codebooks used bag-of-words modelsfor visual recognition. Methods like k-means or Gaussianmixture models represent cluster centers as a weightedcombination of the points within the cluster and evaluat-ing the similarity to a point involves the weighted com-bination of kernel similarity to points within the cluster,a step which can benefit from our analysis. In [65], theauthors demonstrate that histogram intersection kernelbased codebooks offer better accuracies on several imageclassification tasks over traditional clustering based onlinear kernel.

10 CONCLUSION

In this paper we showed that a class of non linearkernels called additive kernels lead to SVM classifierswhich can be approximately evaluated very efficiently.Additive kernels are strictly more general than linearkernels and often lead to significant improvements andour technique brings down the memory and time com-plexity of classification using additive kernels to only asmall constant multiple of that of a linear SVM. Additivekernels are widely used in computer vision and ourtechnique has found wide spread applications in manyclassification/detection tasks.

In addition our technique has lead to efficient trainingalgorithms for additive classifiers, and has sped upmany applications involving histogram based compar-ison, like near duplicate image detection, and kernel k-means/PCA/LDA/regression.

Acknowledgements: S.M. was supported by ARO MURIW911NF-06-1-0076, ONR MURI N00014-06-1-0734, and Google fellow-ship. A.C.B. acknowledges Stony Brook University start-up funding.

REFERENCES

[1] S. Agarwal and D. Roth. Learning a sparse representationfor object detection. In European Conference on Computer Vision(ECCV), 2002.

[2] S. Belongie, C. Fowlkes, F. Chung, and J. Malik. Spectral parti-tioning with indefinite kernels using the nystrom extension. InEuropean Conference on Computer Vision (ECCV), 2002.

[3] S. Boughorbel, J.-P. Tarel, and N. Boujemaa. Generalized his-togram intersection kernel for image recognition. In IEEE Con-ference on Image Processing (ICIP), 2005.

[4] L. Bourdev and J. Brandt. Robust object detection via soft cascade.In Computer Vision and Pattern Recognition (CVPR), 2005.

[5] C. Burges and B. Scholkopf. Improving the accuracy and speed ofsupport vector machines. In Neural Information Processing Systems(NIPS), 1997.

[6] C. J. C. Burges. Simplified support vector decision rules. InInternational Conference on Machine Learning (ICML), 1996.

[7] O. Chapelle, P. Haffner, and V. Vapnik. Support vector machinesfor histogram-based image classification. Neural Networks, IEEETransactions on, 10(5):1055–1064, 1999.

[8] O. Chum and A. Zisserman. Presented at pascal visual recognitionchallenge workshop. 2007.

[9] C. Cortes and V. Vapnik. Support-vector networks. MachineLearning, 20(3):273–297, 1995.

[10] N. Dalal and B. Triggs. Histograms of oriented gradients forhuman detection. In Computer Vision and Pattern Recognition(CVPR), 2005.

[11] D. Decoste and B. Scholkopf. Training invariant support vectormachines. Machine Learning, 46(1-3):161–190, 2002.

[12] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, andA. Zisserman. The pascal visual object classes (voc) challenge.International Journal of Computer Vision (IJCV), 88(2):303–338, jun.2010.

[13] R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin. Liblinear: Alibrary for large linear classification. Journal of Machine LearningResearch (JMLR), 9:1871–1874, 2008.

[14] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visualmodels from few training examples: an incremental bayesianapproach tested on 101 object categories. In Computer Vision andPattern Recognition (CVPR), 2004.

[15] L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of objectcategories. IEEE Transaction of Pattern Analysis and Machine Intel-ligence (PAMI), 28(4):594–611, 2006.

[16] P. Felzenszwalb, D. McAllester, and D. Ramanan. A discrimina-tively trained, multiscale, deformable part model. In ComputerVision and Pattern Recognition (CVPR), 2008.

[17] R. Fergus, P. Perona, and A. Zisserman. Object class recognitionby unsupervised scale-invariant learning. In Computer Vision andPattern Recognition (CVPR), 2003.

[18] J. Friedman and W. Stuetzle. Projection pursuit regression. Journalof the American statistical Association, pages 817–823, 1981.

[19] A. Garg, S. Agarwal, and T. S. Huang. Fusion of global andlocal information for object detection. In International Conferenceon Pattern Recogntion (ICPR), 2002.

[20] K. Grauman and T. Darrell. The pyramid match kernel: Discrimi-native classification with sets of image features. In Computer Visionand Pattern Recognition (CVPR), 2005.

[21] K. Grauman and T. Darrell. Unsupervised learning of categoriesfrom sets of partially matching image features. In Computer Visionand Pattern Recognition (CVPR), 2006.

[22] K. Grauman and T. Darrell. The pyramid match kernel: Efficientlearning with sets of features. Journal of Machine Learning Research(JMLR), 8:725–760, 2007.

[23] T. Hastie and R. Tibshirani. Generalized Additive Models. Chapman& Hall/CRC, 1990.

[24] B. Heisele, T. Serre, S. Prentice, and T. Poggio. Hierarchicalclassification and feature reduction for fast face detection withsupport vector machines. Pattern Recognition, 36:2007–2017(11),September 2003.

[25] M. Herbster. Learning additive models online with fast evaluatingkernels. In Computational Learning Theory, pages 444–460. Springer,2001.

[26] D. Hoiem, A. Efros, and M. Hebert. Putting Objects in Perspective.International Journal of Computer Vision, 80(1):3–15–15, Oct. 2008.

[27] P. Indyk and N. Thaper. Fast Image Retrieval via Embeddings. In3rd International Workshop on Statistical and Computational Theoriesof Vision, 2003.

[28] W. J. Deng, R. Dong, L.-J. Socher, K. L. Li, and L. Fei-Fei. Imagenet:A large-scale hierarchical image database. In Computer Vision andPattern Recognition (CVPR), 2009.

[29] T. Joachims. Training linear svms in linear time. In ACM SIGKDD,International Conference on Knowledge Discovery and Data Mining,2006.

[30] S. S. Keerthi, O. Chapelle, and D. DeCoste. Building supportvector machines with reduced classifier complexity. Journal ofMachine Learning Research (JMLR), 7:1493–1515, 2006.

[31] C. H. Lampert, M. B. Blaschko, and T. Hofmann. Beyond slidingwindows: Object localization by efficient subwindow search. InComputer Vision and Pattern Recognition (CVPR), 2008.

[32] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features:Spatial pyramid matching for recognizing natural scene cate-gories. In Computer Vision and Pattern Recognition (CVPR), 2006.

[33] B. Leibe, A. Leonardis, and B. Schiele. Combined object catego-rization and segmentation with an implicit shape model. In InECCV workshop on statistical learning in computer vision, pages 17–32, 2004.

[34] S. Maji and A. C. Berg. Max margin additive classifiers fordetection. In International Conference on Computer Vision (ICCV),2009.

[35] S. Maji, A. C. Berg, and J. Malik. Classification using intersectionkernel support vector machines is efficient. In Computer Visionand Pattern Recognition (CVPR), 2008.

[36] S. Maji and J. Malik. Fast and accurate digit classification. Tech-nical Report UCB/EECS-2009-159, EECS Department, Universityof California, Berkeley, Nov 2009.

[37] S. Maji and J. Malik. Object detection using a max-margin houghtransform. In Computer Vision and Pattern Recognition (CVPR),2009.


13

[38] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K. Mullers. Fisherdiscriminant analysis with kernels. In Neural Networks for SignalProcessing (NNSP), 1999.

[39] S. Munder and D. M. Gavrila. An experimental study onpedestrian classification. IEEE Transaction of Pattern Analysis andMachine Intelligence (PAMI), 28(11):1863–1868, 2006.

[40] J. Mutch and D. G. Lowe. Multiclass object recognition withsparse, localized features. In Computer Vision and Pattern Recogni-tion (CVPR), 2006.

[41] F. Odone, A. Barla, and A. Verri. Building kernels from binarystrings for image matching. IEEE Transactions on mage Processing,14(2):169–180, 2005.

[42] E. Osuna, R. Freund, and F. Girosi. Training support vectormachines: an application to face detection. 1997.

[43] E. E. Osuna and F. Girosi. Reducing the run-time complexity insupport vector machines. Advances in kernel methods: support vectorlearning, pages 271–283, 1999.

[44] C. Papageorgiou and T. Poggio. A trainable system for objectdetection. International Journal of Computer Vision (IJCV), 38(1):15–33, 2000.

[45] F. Perronnin, J. S andnchez, and Y. Liu. Large-scale imagecategorization with explicit data embedding. In Computer Visionand Pattern Recognition (CVPR), 2010.

[46] A. Rahimi and B. Recht. Random features for large-scale kernelmachines. In Neural Information Processing Systems (NIPS), 2007.

[47] S. Romdhani, P. Torr, B. Scholkopf, and A. Blake. Computationallyefficient face detection. In International Conference on ComputerVision (ICCV), 2001.

[48] R. Schapire. A brief introduction to boosting. In International JointConference on Artificial Intelligence, 1999.

[49] B. Scholkopf, A. Smola, and K. Muller. Nonlinear componentanalysis as a kernel eigenvalue problem. Neural computation,10(5):1299–1319, 1998.

[50] B. Scholkopf and A. J. Smola. Learning with Kernels: Support VectorMachines, Regularization, Optimization, and Beyond. MIT Press,Cambridge, MA, USA, 2001.

[51] S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primalestimated sub-gradient solver for svm. In International Conferenceon Machine Learning (ICML), 2007.

[52] L. Shang, L. Yang, F. Wang, K.-P. Chan, and X.-S. Hua. Real-time large scale near-duplicate web video retrieval. In ACMMultimedia, 2010.

[53] A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaignsand trecvid. In MIR ’06: Proceedings of the 8th ACM InternationalWorkshop on Multimedia Information Retrieval, pages 321–330, NewYork, NY, USA, 2006. ACM Press.

[54] M. J. Swain and D. H. Ballard. Color indexing. International Journalof Computer Vision (IJCV), 7(1):11–32, 1991.

[55] A. Torralba, K. Murphy, and W. Freeman. Sharing features:efficient boosting procedures for multiclass object detection. InComputer Vision and Pattern Recognition (CVPR), 2004.

[56] K. Van de Sande, J. Uijlings, T. Gevers, and A. Smeulders.Segmentation as selective search for object recognition. In In-ternational Conference on Computer Vision (ICCV), 2011.

[57] M. Varma and D. Ray. Learning the discriminative power-invariance trade-off. In International Conference on Computer Vision(ICCV), 2007.

[58] A. Vedaldi, V. Gulshan, M. Varma, and A. Zisserman. Multiplekernels for object detection. In International Conference on ComputerVision (ICCV), 2009.

[59] A. Vedaldi and A. Zisserman. Efficient additive kernels viaexplicit feature maps. In Computer Vision and Pattern Recognition(CVPR), 2010.

[60] A. Vedaldi and A. Zisserman. Efficient additive kernels viaexplicit feature maps. IEEE Transaction of Pattern Analysis andMachine Intelligence (PAMI), 2011.

[61] P. Viola and M. J. Jones. Robust real-time face detection. Interna-tional Journal of Computer Vision (IJCV), 57(2):137–154, 2004.

[62] S. Walk, N. Majer, K. Schindler, and B. Schiele. New features andinsights for pedestrian detection. In Computer Vision and PatternRecognition (CVPR), 2010.

[63] G. Wang, D. Hoiem, and D. Forsyth. Learning image similarityfrom flickr groups using stochastic intersection kernel machines.In International Conference on Computer Vision (ICCV), 2009.

[64] G. J. Z. Wei Zhang and D. Samaras. Real-time accurate objectdetection using multiple resolutions. In International Conferenceon Computer Vision (ICCV), 2007.

[65] J. Wu and J. M. Rehg. Beyond the euclidean distance: Creating ef-fective visual codebooks using the histogram intersection kernel.In International Conference on Computer Vision (ICCV), 2009.

[66] C. Yang, R. Duraiswami, N. A. Gumerov, and L. Davis. Improvedfast gauss transform and efficient kernel density estimation. InInternational Conference on Computer Vision (ICCV), 2003.

[67] J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid. Local fea-tures and kernels for classification of texture and object categories:A comprehensive study. International Journal of Computer Vision(IJCV), 73(2):213–238, 2007.

Subhransu Maji is currently a PhD studentin Computer Science at U.C. Berkeley advisedby Jitendra Malik. His primary interests are incomputer vision with focus on representationsand efficient algorithms for visual recognition. Heobtained his B.Tech in Computer Science andEngineering from Indian Institute of Technology,Kanpur in 2006. He was one of the recipientsof the Google Graduate Fellowship in 2008. Healso interned with the image search group atGoogle and was a visiting researcher at Mi-

crosoft research, Bangalore.

Alexander C. Berg completed the PhD degreein Computer Science from U.C. Berkeley in2005. He is currently an assistant professor atStony Brook University. Prior to that, Dr. Bergwas a research scientist at Yahoo! Researchand later Columbia University. His research ad-dresses challenges in visual recognition at alllevels, from features for alignment and objectrecognition, to action recognition in video, toobject detection, to high level semantics of hi-erarchical classification, attributes, and natural

language descriptions of images. Much of this work has a focus onefficient large scale solutions. He holds a BA and MA in Mathematicsfrom Johns Hopkins University.

Jitendra Malik was born in Mathura, India in1960. He received the B.Tech degree in Elec-trical Engineering from Indian Institute of Tech-nology, Kanpur in 1980 and the PhD degree inComputer Science from Stanford University in1985. In January 1986, he joined the universityof California at Berkeley, where he is currentlythe Arthur J. Chick Professor in the ComputerScience Division, Department of Electrical Engg.and Computer Sciences. He is also on the fac-ulty of the department of Bioengineering, and

the Cognitive Science and Vision Science groups. During 2002-2004 heserved as the Chair of the Computer Science Division and during 2004-2006 as the Department Chair of EECS. He serves on the advisoryboard of Microsoft Research India, and on the Governing Body of IIITBangalore.

His current research interests are in computer vision, computationalmodeling of human vision and analysis of biological images. His workhas spanned a range of topics in vision including image segmenta-tion, perceptual grouping, texture, stereopsis and object recognitionwith applications to image based modeling and rendering in computergraphics, intelligent vehicle highway systems, and biological imageanalysis. He has authored or co-authored more than a hundred and fiftyresearch papers on these topics, and graduated thirty PhD students whooccupy prominent places in academia and industry. According to GoogleScholar, five of his papers have received more than a thousand citationseach. He is one of ISI’s Highly Cited Researchers in Engineering.

He received the gold medal for the best graduating student in Elec-trical Engineering from IIT Kanpur in 1980 and a Presidential YoungInvestigator Award in 1989. At UC Berkeley, he was selected for theDiane S. McEntyre Award for Excellence in Teaching in 2000, a MillerResearch Professorship in 2001, and appointed to be the Arthur J. ChickProfessor in 2002. He received the Distinguished Alumnus Award fromIIT Kanpur in 2008. He was awarded the Longuet-Higgins Prize for acontribution that has stood the test of time twice, in 2007 and in 2008.He is a fellow of the IEEE and the ACM, and was elected to the NationalAcademy of Engineering in 2011 for ”contributions to computer visionand image analysis”.


Documents

Efficient Classification for Additive Kernel SVMsacberg.com/papers/mbm2012pami.pdf1 Efficient Classification for Additive Kernel SVMs Subhransu Maji 1, Alexander C. Berg2, and