Generalization performance of regularization networks …users.cecs.anu.edu.au/~williams/papers/P100published.pdf · Generalization Performance of Regularization Networks and Support

2516 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 47, NO. 6, SEPTEMBER 2001

Generalization Performance of RegularizationNetworks and Support Vector Machines Via

Entropy Numbers of Compact OperatorsRobert C. Williamson, Member, IEEE, Alex J. Smola, and Bernhard Schölkopf

Abstract—We derive new bounds for the generalization errorof kernel machines, such as support vector machines and relatedregularization networks by obtaining new bounds on their cov-ering numbers. The proofs make use of a viewpoint that is appar-ently novel in the field of statistical learning theory. The hypoth-esis class is described in terms of a linear operator mapping froma possibly infinite-dimensional unit ball in feature space into a fi-nite-dimensional space. The covering numbers of the class are thendetermined via the entropy numbers of the operator. These num-bers, which characterize the degree of compactness of the operator,can be bounded in terms of the eigenvalues of an integral operatorinduced by the kernel function used by the machine. As a conse-quence, we are able to theoretically explain the effect of the choiceof kernel function on the generalization performance of supportvector machines.

Index Terms—Covering numbers, -entropy, kernel methods,linear operators, metric entropy, statistical learning theory,support vector (SV) machines.

I. INTRODUCTION

I N this paper we give new bounds on the covering numbersfor kernel machines. This leads to improved bounds on

their generalization performance. Kernel machines perform amapping from input space into a feature space (see, e.g., [1],[34]), construct regression functions or decision boundariesbased on this mapping, and use constraints in feature space forcapacity control. Support vector (SV) machines, which haverecently been proposed as a new class of learning algorithmssolving problems of pattern recognition, regression estimation,and operator inversion [53] are a well-known example of thisclass. We will use SV machines as our model of choice toshow how bounds on the covering numbers can be obtained.We outline the relatively standard methods one can thenuse to hence bound their generalization performance. SV

Manuscript received April 2, 1999; revised August 1, 2000. This work wassupported in part by the Australian Research Council, the DFG (#Ja 379/71and Sm 62/1-1), and Neurocolt II. The material in this paper was presented inpart at the European Conference on Computational Learning Theory (EURO-COLT’99), Nordkirchen, Germany, March 29–31, 1999, and appeared (in sum-mary form) inAdvances in Kernel Methods, Cambridge, MA: MIT Press, 1999.

R. C. Williamson and A. J. Smola are with the Research School ofInformation Sciences and Engineering, Australian National University,Canberra, A.C.T. 0200, Australia (e-mail: Bob. [email protected];[email protected]).

B. Schölkopf was with Microsoft Research, Cambridge, CB2 3NH, U.K.He is now with Barnhill Technologies, New York, NY 10007 USA (e-mail:[email protected]).

Communicated by S. R. Kulkarni, Associate Editor for Nonparametric Esti-mation, Classification, and Neural Networks.

Publisher Item Identifier S 0018-9448(01)02835-8.

machines, like most kernel-based methods, possess the niceproperty of defining the feature map in a manner that allowsits computation implicitly at little additional computationalcost. Our reasoning also applies to similar algorithms such asregularization networks [16] or certain unsupervised learningalgorithms [41]. Let us now take a closer look at SV machines.Central to them are two ideas: capacity control by maximizingmargins, and the use of nonlinear kernel functions.

A. Capacity Control

In order to perform pattern recognition using linear hyper-planes, often a maximum margin of separation between theclasses is sought, as this leads to good generalization abilityindependent of the dimensionality [55], [53], [43]. It can beshown that for separable training data

(1)

this is achieved by minimizing subject to the constraintsfor , and some . The

decision function then takes the form

sgn (2)

Similarly, a linear regression

(3)

can be estimated from data

(4)

by finding the flattest function which approximates the datawithin some margin of error: in this case, one minimizessubject to , where the parameter playsthe role of the margin, albeit not in the space of the inputs, butin that of the outputs .

In both cases, generalizations for the nonseparable or nonre-alizable case exist, using various types of cost functions [14],[53], [46].

B. Nonlinear Kernels

In order to apply the above reasoning to a rather generalclass ofnonlinearfunctions, one can use kernels computing dotproducts in high-dimensional spaces nonlinearly related to inputspace [1], [10]. Under certain conditions on a kernel, to bestated below (Theorem 4), there exists a nonlinear mapinto a

0018–9448/01$10.00 © 2001 IEEE

WILLIAMSON et al.: GENERALIZATION PERFORMANCE OF REGULARIZATION NETWORKS AND SUPPORT VECTOR MACHINES 2517

reproducing kernel Hilbert space(see, e.g., [40]) such thatcomputes the dot product in, i.e.,

(5)

Given any algorithm which can be expressed in terms of dotproducts exclusively, one can thus construct a nonlinear versionof it by substituting a kernel for the dot product. Examples ofsuch machines include SV pattern recognition [10], SV regres-sion estimation [53], and kernel principal component analysis[41].

By using the kernel trick for SV machines, the maximummargin idea is thus extended to a large variety of nonlinear func-tion classes (e.g., radial basis function networks, polynomialnetworks, neural networks), which in the case of regression es-timation comprise functions written as kernel expansions

(6)

with , . It has been noticed that differentkernels can be characterized by their regularization properties[48]: SV machines are regularization networks minimizing theregularized risk

(with a regularization parameter , and a regularizationoperator ) over the set of functions of the form (6), providedthat and are interrelated by

To this end, is chosen as a Green’s function of whereis the adjoint of .

This provides insight into the regularization properties of SVkernels. However, it does not completely settle the issue of howto select a kernel for a given learning problem, and how using aspecific kernel might influence the performance of an SV ma-chine.

C. Outline of the Paper

In the present work, we show that properties of the spectrumof the kernel can be used to make statements about the general-ization error of the associated class of learning machines. Unlikein previous SV learning studies, the kernel is no longer merely ameans of broadening the class of functions used, e.g., by makinga nonseparable dataset separable in a feature space nonlinearlyrelated to input space. Rather, we now view it as a constructivehandle by which we can control the generalization error.

A key feature of the present paper is the manner in whichwe directly bound the covering numbers of interest ratherthan making use of a combinatorial dimension (such as theVapnik–Chervonenkis (VC) dimension or the fat-shatteringdimension) and subsequent application of a general resultrelating such dimensions to covering numbers. We boundcovering numbers directly by viewing the values induced bythe relevant class of functions as the image of a unit ball under aparticular compact operator. A general overview of the methodis given in Section III.

The remainder of the paper is organized as follows. We startby introducing notation and definitions (Section II). Section IVformulates generalization error bounds in terms of coveringnumbers. Section V contains the main result bounding entropynumbers in terms of the spectrum of a given kernel. The resultsin this paper rest on a connection between covering numbersof function classes and entropy numbers of suitably definedoperators. In particular, we derive an upper bound on theentropy numbers in terms of the size of the weight vector infeature space and the eigenvalues of the kernel used. Section VIshows how to make use of kernels such aswhich do not have a discrete spectrum. Section VII presentssome results on the entropy numbers obtained for given ratesof decay of eigenvalues and Section VIII shows how to extendthe results to several dimensions. The concluding section(Section IX) indicates how the various results in the papercan be glued together in order to obtain overall bounds onthe generalization error. All of the examples we provide forthe calculation of eigenvalues are for translation-invariantkernels (i.e., convolutional kernels); this is merely for conve-nience—the general theory is not restricted to such kernels.Key new results are labeled as propositions.

We do not present a single master generalization error the-orem for four key reasons: 1) the only novelty in the paper liesin the computation of covering numbers themselves; 2) the par-ticular statistical result one needs to use depends on the specificproblem situation; 3) many of the results obtained are in a formwhich, while quite amenable to ready computation on a com-puter, do not provide much direct insight by merely looking atthem, except perhaps in the asymptotic sense; and, finally, 4)some applications (such as classification) where further quanti-ties like margins are estimated in a data dependent fashion, needan additional luckiness argument [44] to apply the bounds.

Thus, although our goal has been theorems, we are ultimatelyforced to resort to a computer to make use of our results. Thisis not necessarily a disadvantage—it is both a strength and aweakness of structural risk minimization (SRM) [56] that a goodgeneralization error bound is both necessary and sufficient tomake the method work well. In [20], some more explicit for-mulas based on the present work and more suitable for SRMare developed.

II. DEFINITIONS AND NOTATION

For , denotes the -dimensional space of vectors. We define spaces as follows: as vector

spaces, they are identical to , in addition, they are endowedwith -norms: for

for

Note that a different normalization of the norm is used insome papers in learning theory (e.g., [51]). For ,


. We use the shorthand sequence notation.

Given points , we use the shorthand.

Suppose is a class of functions . The normwith respect to of is defined as

Likewise,

Given some set with a -algebra, a measureon , someand a function we define

if the integral exists and

For , we let

We let .If is a set and a metric on , then the -covering number

of with respect to the metric denoted isthe smallest number of elements of an-cover for using themetric . Given a metric space we will also write

. The th entropy number of a set , for, is

(7)

Let be the set of all bounded linear operatorsbetween the normed spaces and , i.e.,operators such that the image of the (closed) unit ball

(8)

is bounded. The smallest such bound is called theoperator norm

(9)

Theentropy numbers of an operator are definedas

(10)

Note that , and that certainly is well-definedfor all if is acompact operator, i.e., if for anythere exists a finite cover of with open balls on .

Thedyadic entropy numbers of an operatorare defined by

(11)

Similarly, the dyadic entropy numbers of a set are defined fromits entropy numbers. A very nice introduction to entropy num-bers of operators is [13].

In this paper, and will always beBanach spaces, i.e.,complete normed spaces (for instance,spaces with ).In some cases, they will beHilbert spaces , i.e., Banach spacesendowed with a dot product giving rise to its norm via

.By and , we denote the logarithms to baseand , re-

spectively. By , we denote the imaginary unit , willalways be a kernel, andand will be the input dimensionalityand the number of examples

(12)

respectively. We will map the input data into a feature space viaa mapping . We let .

III. OPERATORTHEORY METHODS FORENTROPYNUMBERS

In this section, we briefly explain the new viewpoint utilizedin the present paper. With reference to Fig. 1, consider the tra-ditional viewpoint in statistical learning theory. One is given aclass of functions , and the generalization performance attain-able using is determined via the covering numbers of. Moreprecisely, for some set , and for , definetheuniform covering numbersof the function class on by

(13)

where is the -covering number of with re-spect to . (Recall .) Many generaliza-tion error bounds can be expressed in terms of . Anexample is given in the following section.

The key novelty in the present work solely concerns themanner in which the covering numbers are computed. Tradi-tionally, appeal has been made to a result such as the so-calledSauer’s lemma (originally due to Vapnik and Chervonenkis).In the case of function learning, a generalization called theVC dimension of real-valued functions, or a variation due toPollard (called the pseudo-dimension), or a scale-sensitivegeneralization of that (called the fat-shattering dimension) isused to bound the covering numbers. These results reducethe computation of to the computation of a single“dimension-like” quantity (independent of ). An overviewof these various dimensions, some details of their history, andsome examples of their computation can be found in [5], [6].

In the present work, we view the classas being induced byan operator depending on some kernel function. Thus,is the image of a “base class”under . The analogy implicitin the picture is that the quantity that matters is the number of-distinguishable messages obtainable at the information sink.

(Recall the equivalence up to a constant factor ofin ofpacking and covering numbers [6].) In a typical communica-tions problem, one tries to maximize the number of distinguish-able messages (per unit time), in order to maximize the informa-tion transmission rate. But from the point of view of the receiver,the decoding job is made easier thesmallerthe number of dis-tinct messages that one needs to be concerned with decoding.(Of course, this lowers the information transmission rate.) Thesignificance of the picture is that the kernel in question is exactlythe kernel that is used, for example, in support vector machines.


Fig. 1. Schematic picture of the new viewpoint.

As a consequence, the determination of can be donein terms of properties of the operator. The latter thus plays aconstructive role in controlling the complexity of and hencethe difficulty of the learning task. We believe that the new view-point in itself is potentially very valuable, perhaps more so thanthe specific results in the paper. A further exploitation of the newviewpoint can be found in [62], [61], [49], [47].

We conclude this section with some historical remarks.The concept of the metric entropy of a set has been around

for some time. It seems to have been introduced by Pontriaginand Schnirelmann [37] and was studied in detail by Kolmogorovand others [27] (see also, e.g., [32, Ch. 15]). The use of metricentropy to say something about linear operators was developedindependently by several people. Prosser [38] appears to havebeen the first to make the idea explicit. He determined the effectof an operator’s spectrum on its entropy numbers. In particular,he proved a number of results concerning the asymptotic rateof decrease of the entropy numbers in terms of the asymptoticbehavior of the eigenvalues. A similar result is actually implicitin Shannon’s famous paper [42, Sec. 22], where he consideredthe effect of different convolution operators on the entropy of anensemble. Prosser’s paper [38] led to a handful of papers (see,e.g., [39], [22], [3], [29]) which studied various convolutionaloperators. A connection between Prosser’s-entropy of an op-erator and Kolmogorov’s-entropy of a stochastic process wasshown in [2]. Independently, another group of mathematiciansincluding Carl and Stephani [13] studied covering numbers [52]

and later entropy numbers [36] in the context of operator ideals.(They were unaware of Prosser’s work—see, e.g., [11, p. 136].)

Connections between the local theory of Banach spaces anduniform convergence of empirical means has been noted before(e.g., [35]). More recently, Gurvits [21] has obtained a resultrelating the Rademacher type of a Banach space to the fat-shat-tering dimension of linear functionals on that space and hencevia the key result in [4] to the covering numbers of the inducedclass. We will make further remarks concerning the relationshipbetween Gurvits’ approach and ours in [60]; for now, let us justnote that the equivalence of the type of an operator (or of thespace it maps to), and the rate of decay of its entropy numbershas been (independently) shown by Kolchinskiy[25], [26] andDefant and Junge [15], [23]. Note that the exact formulation oftheir results differs. Kolchinskiy was motivated by probabilisticproblems not unlike ours.

IV. GENERALIZATION BOUNDS VIA UNIFORM CONVERGENCE

The generalization performance of learning machines can bebounded via uniform convergence results as in [57], [56]. A re-cent review can be found in [5]; see also [30]. The key thingabout these results is the role of the covering numbers of the hy-pothesis class—the focus of the present paper. Results for bothclassification and regression are now known. For the sake ofconcreteness, we quote below a result suitable for regression


which was proved in [4]. For results on classifier performancein terms of covering numbers see [8]. Let

denote theempirical meanof on the sample .We make use of the following due to Alon, Ben-David, Cesa-

Bianchi, and Haussler [4].

Lemma 1: Let be a class of functions from intoand let be a distribution over . Then, for all and all

(14)

where denotes the probability with respect to the sampledrawn independent and identically distributred

(i.i.d.) from , and the expectation with respect to a secondsample also drawn i.i.d. from .

In order to use this lemma, one usually makes use of the factthat for any

(15)

The above result can be used to give a generalization error re-sult by applying it to the loss-function-induced class. The fol-lowing lemma, which is an improved version of [9, Lemma 17],is useful in this regard (a similar result appears in [6]).

Lemma 2: Let be a set of functions from to with, , and a loss function. Let

and

Then the following two statements hold.

1) Suppose satisfies the Lipschitz condition

for all (16)

Then for all

(17)

and

(18)

2) Suppose that for some , satisfies the “approx-imate Lipschitz condition”

for all (19)

then for all

(20)

Proof: We show that, for any sequence of pairsin and any functions and , if the restrictions ofand to are close, then the restrictions ofand toare close. Thus, given a cover of we can construct a coverof that is no bigger. For case 1) we get

In the second case we proceed similarly

for

The second case can be useful when the exact form of thecost function is not known, happens to be discontinuous, or isbadly behaved in some other way. Applying the result above topolynomial loss leads to the following corollary.

Corollary 3: Let the assumptions be as above in Lemma 2.Then for loss functions of type

with (21)

we have , in particular forand, therefore,

(22)

One can readily combine the uniform convergence results withthe above results to get overall bounds on generalization per-formance. We do not explicitly do this here since the particularuniform convergence result needed depends on the exact setup


of the learning problem. A typical uniform convergence resulttakes the form

(23)

where is the empirical risk and the expected riskof (see, e.g., [6], [54] Even the exponent in (23) de-pends on the setting:. In regression,can be set to, however,in agnostic learning [24] in general , except if the class isconvex in which case it can be set to[31]. Since our primary in-terest is in determining we will not try to summarizethe large body of results now available on uniform convergenceand generalization error.

These generalization bounds are typically used by setting theright-hand side equal toand solving for (whichis called the sample complexity). Another way to use these re-sults is as a learning curve bound where

We note here that the determination of is quite conve-nient in terms of , the dyadic entropy number associated withthe covering number in (23). Setting the right-handside of (23) equal to, we have

(24)

Thus, : (24) holds . Hence, the use of or(which will arise naturally from our techniques) is in fact a

convenient thing to do for finding learning curves.

V. ENTROPY NUMBERS FORKERNEL MACHINES

In the following, we will mainly consider machines where themapping into feature space is defined by Mercer kernelsas they are easier to deal with using functional analytic methods.(More general kernels are considered in [47].) Such machineshave become very popular due to the success of SV machines.

A. Mercer’s Theorem, Feature Spaces, and Scaling

Our goal is to make statements about the shape of the image ofthe input space under the feature map . We will make useof Mercer’s theorem. The version stated below is a special caseof the theorem proven in [28, p. 145]. In the following we willassume to be a finite measure space, i.e., .

Theorem 4 (Mercer):Suppose is a sym-metric kernel (that is, ) such that the integraloperator

(25)

is positive. Let be the eigenfunction of associ-ated with the eigenvalue and normalized by. Suppose is continuous for all . Then

1) for ;

2) and ;

3) holds for all ;

where the series converges absolutely and uniformly for all.

We will call a kernel satisfying the conditions of this theoremaMercer kernel.Note that if is compact and is continuous,then is continuous (cf., e.g., [7, p. 270]). Alternatively, ifis translation-invariant, then are scaled cosine functions andthus continuous. Thus, the assumption thatare continuous isnot very restrictive.

From statement 2) of Mercer’s theorem there exists someconstant depending on such that

for all and (26)

Moreover, from statement 3) it follows that correspondsto a dot product in , i.e., with

(27)

for all . In the following, we will (without loss of gener-ality) assume the sequence of is sorted in nonincreasingorder. From the argument above, one can see that livesnot only in but in an axis parallel parallelepiped with lengths

.We remark that the measureneed have nothing to do with

the distribution of examples. In particular, we may consider anyof the following kernels in our bounds.

Lemma 5 (Equivalent Kernels):Denote by a compact setand by a Mercer kernel. Then ,for any and surjec-tive map , the kernelalso satisfies Mercer’s condition and, moreover, the eigenvalues

and the coefficient of the integral operator

(28)

can be used equivalently in any application of.This means in particular that we could construct diffeomor-

phisms and look for the function such that theeigenvalues and are as small as possible.

Proof: The first part of the claim, namely, that also sat-isfies Mercer’s condition, follows immediately from the con-struction of . For the second claim, note that due to the factthat is surjective for any distribution on there mustexist an equivalent distribution on . Thus, we can al-ways consider the problem as being one onfrom the start.However, since and were chosen arbitrarily we can opti-mize over them.

Lemma 5 shows that the specific bounds we obtainwill de-pend on since that will affect the and the . Thequestion of the optimal to use and how it may be chosen if oneknows (the distribution from which the are drawn) is notconsidered here. In all cases considered in this paper, we will infact take to be the Lebesgue measure.

It will be useful to consider maps that map into balls ofsome radius centered at the origin. The following proposition


shows that the class of all these maps is determined by elementsof and the sequence of eigenvalues .

Proposition 6 (Mapping into ): Let be the diag-onal map

with (29)

Then maps into a ball of finite radius centered atthe origin if and only if .

Proof:( ) Suppose and let

For any

(30)

Hence .( ) Suppose is not in . Hence the sequencewith

is unbounded. Now define

(31)

Then due to the normalization conditionon . However, as there exists a set of nonzeromeasure such that

for all (32)

Combining the left-hand side of (30) with (31) we obtain

for all and all

Since is unbounded for a set with nonzero measure in, we can see that .

Once we know that is contained in the parallelepipeddescribed above we can use this result to construct a mapping

from the unit ball in to an ellipsoid such thatas in the following diagram (where we have slightly abused thetraditional notational convention).

(33)

The operator will be useful for computing the entropy num-bers of concatenations of operators. (Knowing the inverse willallow us to compute the forward operator, and that can be used tobound the covering numbers of the class of functions, as shownin the next subsection.) We thus seek an operatorsuch that

(34)

This means that will be such that . Thelatter can be ensured by constructingsuch that

with (35)

where and are chosen with respect to a specific kerneland where . From Proposition 6, itfollows that all those operators for which will sat-isfy (34). We call such scaling (inverse) operatorsadmissible.

B. Entropy Numbers

The next step is to compute the entropy numbers of the oper-ator and use this to obtain bounds on the entropy numbers forkernel machines like SV machines. We will make use of the fol-lowing theorem due to Gordon, König, and Schütt [17, p. 226](stated in the present form in [13, p. 17]).

Theorem 7: Let be a nonin-creasing sequence of nonnegative numbers and let

(36)

for be the diagonal operatorfrom into itself, generated by the sequence , where

. Then for all

(37)

We can exploit the freedom in choosingto minimize an en-tropy number as the following corollary shows. This will be akey ingredient of the calculation of the covering numbers for SVclasses, as shown below.

Proposition 8 (Scaling Operators):Let be aMercer kernel with eigenvalues . Choose forsuch that , and define

(38)

with . Then

(39)

This result follows immediately by identifying and . Wecan optimize (39) over all possible choices ofto obtain thefollowing proposition. (It turns out that the infimum is in factattainable [20] when is a Mercer kernel thus justifying writingthe inequality as we do. That is, we can minimize the right-handside of (39).)

Proposition 9: There exists an defined by (35) that satis-fies

(40)

As already described in Section I, the hypothesis that an SVmachine generates can be expressed as where both

and are defined in the feature space


and . The kernel trick, as introduced by [1], was thensuccessfully employed in [10] and [14] to extend the optimalmargin hyperplane classifier to what is now known as the SVmachine. We deal with the “ ” term in Section IX; for now weconsider the class

Note that depends implicitly on since does.We seek the covering numbers for the class induced

by the kernel in terms of the parameterwhich is the inverseof the size of the margin in feature space, or equivalently, thesize of the weight vector in feature space as defined by the dotproduct in (see [55], and [53] for details). In the following,we will call such hypothesis classes with length constraint on theweight vectors in feature spaceSV classes.Let be the operator

where and the operator is defined by

(41)

with for all . The following theorem is useful whencomputing entropy numbers in terms ofand . It is originallydue to Maurey, and was extended by Carl [12] and is given inalmost the form below by Carl and Stephani [13, p. 246].

Theorem 10 (Maurey):Let where is aHilbert space. Then, there exists a constant such that forall

(42)

(Carl and Stephani state an additional condition, namely, that. It turns out [62] that for , and even tighter bound

holds, and so it is not incorrect to state it as above. It shouldbe added that this tighter bound is of little value in learningtheory applications: it corresponds to determining the-cov-ering number for extremely smallfor which

.)An alternative proof of this result (given in [62]) provides a

small explicit value for the constant: . However, there isreason to believe thatshould be , the constant obtainablefor identity maps from into .

The restatement of Theorem 10 in terms of will beuseful in the following. Under the assumptions above we have

(43)

Now we can combine the bounds on entropy numbers ofand to obtain bounds for SV classes. First we need thefollowing lemma from [13, p. 11].

Lemma 11 (Carl and Stephani):Let be Banachspaces, , and . Then, for

(44)

(45)

(46)

Note that the latter two inequalities follow directly from (44)and the fact that for all .

Proposition 12 (Bounds for SV Classes):Let be a Mercerkernel, let be induced via (27), and let , where

is given by (41) and . Let be defined as in Propo-sition 9 and suppose for . Then, theentropy numbers of satisfy the following inequalities:

(47)

(48)

(49)

where is defined as in Theorem 10.This result gives several options for bounding . We shall

see in examples later that the best inequality to use dependson the rate of decay of the eigenvalues of. The result giveseffective bounds on since

Proof: We will use the following factorization of toupper-bound .

(50)

The top arrow in the diagram follows from the definition of.The fact that remainder commutes stems from the fact that since

is diagonal, it is self-adjoint and so for any

(51)

Instead of computing the entropy number of di-rectly, which is difficult or wasteful, as the bound on doesnot take into account that but just makes the assump-tion of for some , we will represent as

. This is more efficient as we constructedsuchthat filling a larger proportion of it than

does.By construction of and due to the Cauchy–Schwarz

inequality we have . Thus applying Lemma11 to the factorization of and using Theorem 10 proves thetheorem.

As we shall see in Section VII, one can give asymptotic ratesof decay for . (In fact, we give nonasymptotic results withexplicitly evaluable constants.) It is thus of some interest to giveoverall asymptotic rates of decay of in terms of the orderof . (By “asymptotic” here we mean asymptotic in; thiscorresponds to asking how scales as for fixed

.)

Lemma 13 (Rate Bounds on): Let be a Mercer kerneland suppose is the scaling operator associated with it as de-fined by (38).


1) If for some then for fixed

(52)

2) If for some then forfixed

(53)

This Lemma shows that in the first case, Maurey’s result (The-orem 10) allows an improvement in the exponent of the entropynumber of , whereas in the second, it affords none (since theentropy numbers decay so fast anyway). The Maurey result maystill help in that case though for nonasymptotic.

Proof: From theorem 10 we know that

Now use (49), ignoring constants and assumingis fixed, split-ting the index in the following way:

with (54)

For the first case this yields

In the second case we have

(55)

In a nutshell, we can always obtain rates of convergence betterthan those due to Maurey’s theorem because we are not dealingwith arbitrary mappings into infinite-dimensional spaces. Infact, for logarithmic dependency of on , the effect of thekernel is so strong that it completely dominates the be-havior for arbitrary Hilbert spaces. An example of such a kernelis ; see Proposition 17 and also Sec-tion VI for the discretization question.

VI. DISCRETESPECTRA OFCONVOLUTION OPERATORS

The results presented above show that if one knows the eigen-value sequence of a compact operator, one can bound itsentropy numbers. While it is always possible to assume that thedata fed into an SV machine have bounded support, the samecannot be said of the kernel ; a commonly used kernel is

which has noncompact support. Theinduced integral operator

(56)

then has a continuous spectrum (a nondenumerable infinity ofeigenvalues) and, thus, is not compact [7, p. 267]. The ques-tion arises: can we make use of such kernels in SV machinesand still obtain generalization error bounds of the form devel-oped above? Note that by a theorem of Widom [59], the eigen-value decay of any convolution operator defined on a compact

set via a kernel having compact support can decay no faster thanand thus if one seeks very rapid decay of eigen-

values (with concomitantly small entropy numbers), one mustuse convolution kernels with noncompact support.

We will resolve these issues in the present section. Beforedoing so, let us first consider the case thatfor some . Suppose further that the data pointssatisfy

for all . If is a convolution kernel (i.e.,which allows us to write with some abuse

of notation ), then the SV hypothesiscan be written

(57)

for where is the -periodic extension of(analogously, )

(58)

The -dimensional Fourier transform is defined by

(59)

Then, its inverse transform is given by

(60)

can be shown to be an isometry on .We now relate the eigenvalues of to the Fourier transform

of . We do so for the case of and then state the generalcase later.

Lemma 14: Let be a symmetric convolution kernel,let denote the Fourier transform of , and

denote the -periodical kernel derived from (also assumethat exists). Then has a representation as a Fourier serieswith and

(61)

Moreover, the eigenvalues of satisfy

for and .Proof: Clearly, the Fourier series coefficients of

exist (as exists) with


and therefore, by the definition of and the existence of ,we conclude

This and the fact that

forms an orthogonal basis in proves (61). (Notethat since we conclude .) Fur-thermore, we are interested in real-valued basis functions for

. The functions

and

(62)

for all satisfy , and form an eigensystemof the integral operator defined by with the corresponding

eigenvalues . Finally, one can see thatby computing the over and .

Thus, even though may not be compact, can be (if, for example). The above lemma can be

applied whenever we can form from . Clearly,for some suffices to ensure the sum in (58)

converges.Let us now consider how to choose. Note that the Rie-

mann–Lebesgue lemma tells us that for integrable ofbounded variation (surely any kernel one would use wouldsatisfy that assumption), one has . Thereis a tradeoff in choosing in that for large enough ,is a decreasing function of (at least as fast as ) and,thus, by Lemma 14, is an increasingfunction of . This suggests one should choose a small valueof . But a small will lead to high empirical error (as thekernel “wraps around” and its localization properties are lost)and large . There are several approaches to picking a valueof . One obvious one is toa priori pick some andchoose the smallest such that for all

. Thus, one would obtain a hypothesisuniformly within of where .

Remark 15: The above Lemma can be readily extended todimensions. Assume is -periodic in each direction (

), we get

(63)

for radially symmetric and finally for the eigenfunctions.

Finally, it is worth explicitly noting how the choice of a dif-ferent bandwidth of the kernel, i.e., letting ,affects the eigenspectrum of the corresponding operator. Wehave , hence scaling a kernel bymeansmore densely spaced eigenvalues in the spectrum of the integraloperator .

In conclusion, in order to obtain a discrete spectrum one needsto use a periodic kernel. For a given problem, one can alwaysperiodize a nonperiodic kernel in a way that changes the finalhypothesis in an arbitrarily small way. One can then make useof the results of the present paper.

VII. COVERING NUMBERS FORGIVEN DECAY RATES

In this section, we will show how the asymptotic behavior of, where is the scaling operator introduced

before, depends on the eigenvalues of.A similar analysis has been carried out by Prosser [38], in

order to compute the entropy numbers of integral operators.However, all of his operators mapped into . Further-more, while our propositions are stated as asymptotic results ashis were, the proofs actually give nonasymptotic informationwith explicit constants.

Note that we need to sort the eigenvalues in a nonincreasingmanner because of the requirements in Proposition 9. If theeigenvalues were unsorted one could obtain far too small num-bers in the geometrical mean of . Many one-dimen-sional kernels have nondegenerate systems of eigenvalues inwhich case it is straightforward to explicitly compute the geo-metrical means of the eigenvalues as will be shown below. Notethat while all of the examples below are for convolution kernels,i.e., , there is nothing in the formulations ofthe propositions themselves that requires this. When we con-sider the -dimensional case we shall see that with rotationallyinvariant kernels, degenerate systems of eigenvalues are generic.In Section VIII-B, we will show how to systematically deal withthat case.

Let us consider the special case where decays asymp-totically with some polynomial or exponential degree. In thiscase, we can choose a sequence for which we can evaluate(40) explicitly. In what follows, by the eigenvalues of a kernel

we mean the (sorted) eigenvalues of the induced integral op-erator .

Proposition 16 (Polynomial Decay):Let be a Mercerkernel with eigenvalues for some .Then for any we have

An example of such a kernel is . The proof can befound in Appendix I.

The next theorem covers a wide range of practically used ker-nels, namely, those with exponential polynomial decay in their


eigenvalues. For instance, the Gaussian kernel hasexponential quadratic decay in. The “damped harmonic os-cillator” kernel is another example, this time withjust exponential decay in its eigenvalues.

Proposition 17 (Exponential-Polynomial Decay):Supposeis a Mercer kernel with for some .Then

(64)

See Appendix I for a proof. (A more precise, but rather morecomplex, calculation is given in [20].) While this theorem givesthe guarantees on the learning rates of estimators using suchtypes of kernels (which is theoretically pleasing and leads to de-sirable sample complexity rates), it may not always be wise touse the theoretically obtained bounds. Instead, one should takeadvantage of the estimates based on an analysis of the distri-bution of the training data since the rates obtained by the lattermay turn out to be far superior with respect to the theoreticalpredictions (cf. Section VI and [61]).

VIII. H IGHER DIMENSIONS

Things get somewhat more complicated in higher dimen-sions. For simplicity, we will restrict ourselves to translation-in-variant kernels in what follows.

There are two simple ways to construct kernels inwith . First one could construct kernels by

(65)

This choice will usually lead to preferred directions in inputspace as the kernels are not rotationally invariant in general. Thesecond approach consists in setting

(65)

This approach also leads to translationally invariant kernelswhich are also rotationally invariant. In the following, wewill exploit this second approach to compute regularizationoperators and corresponding Green’s functions. It is quitestraightforward, however, to generalize our exposition to therotationally asymmetric case. Now let us introduce the basicingredients needed for the further calculations.

A. Basic Tools

Now introduce regularization operatorsdefined by

(67)

for some nonnegative function converging to for. It can be shown [48] that for a kernel to be a Green’s function

of , i.e.,

(68)

we need . For radially symmetric functions,i.e., , we can explicitly carry out the integra-

tion on the sphere to obtain the Fourier transform which is alsoradially symmetric (see, e.g., [50, p. 33]), namely,

(69)

where and is the Hankel transform over thepositive real line. The latter is defined by

(70)

Here is the Bessel function of the first kind defined by

(71)

Note that , i.e., (in ) due to theHankel inversion theorem [50].

B. Degenerate Systems

Computing the Fourier transform for a given kernelgivesus the continuous spectrum. As pointed out in Section VI, weare interested in the discrete spectrum of integral kernels de-fined on . This means that the eigenvalues are defined on thegrid with . Assuming is rotationallyinvariant, so is and, therefore, there are repeated eigen-values . Consequently, we have degenera-cies in the point spectrum of the integral operator given by(or , respectively) as all with equal length will have thesame eigenvalue. In order to deal with this case efficiently weslightly modify Theorem 7 for our purposes. The following the-orem allows proper account to be taken of the multiplicity ofeigenvalues, and thus allows a more refined calculation of thedesired entropy numbers.

Proposition 18: Let be an increasing sequencewith and be a nonincreasing sequence ofnonnegative numbers with

for and for

and let

(72)

for be the diagonal operatorfrom into itself, generated by the sequence , where

. Then for all

(73)

See Appendix II for a proof.

This proposition allows us to obtain a similar result to Propo-sition 9.

Proposition 19 (Degenerate Systems):Let bea Mercer kernel and let be defined by (38) with the additionalrestriction that the coefficients have to match the degeneracy


of , i.e., for and for .Then one can choose such that

(74)

This result by itself may not appear too useful. However, it is infact exactly what we need for the degenerate case (it is slightlytighter than the original statement, as the supremum effectivelyhas to be carried out only over a subset of). Finally, we have tocompute the degree of multiplicity that occurs for different in-dexes . For this purpose, consider shells of radiusin cen-tered at the origin, i.e., , which contain a nonzero numberof elements of . Denote the corresponding radii byand let

be the number of elements on these shells. Observe thatonly when . Thus

(75)

The determination of is a classical problem which iscompletely solved by the use of the-series (see, e.g., [19]).

Theorem 20 (Occupation Numbers of Shells):Let the formalpower series be defined by

(76)

Then

(77)

This theorem allows one to readily compute exactly; seeAppendix IV for some Maple code to do so. (Note that whilethere do exist closed-form asymptotic approximate formulas for

[19, p. 155], they are inordinately complicated and oflittle use for our purposes.)

We can now construct an index of the eigenvalues which sat-isfies the required ordering (at least for nonincreasing functions

) and we get the following result.

Proposition 21: Let be a Mercer kernel witheigenvalues given by a radially symmetric nonincreasing func-tion on a lattice, i.e., with and let bedefined by (38) with the additional restriction that the coeffi-cients have to match the degeneracy of, i.e., .Then

(78)

Note that this result, although it may seem straightforward,cannot be obtained from Proposition 9 directly as there thewould have to be carried out over instead of .

C. Bounds for Kernels in

Let us conclude this section with some examples of theeigenvalue sequences for kernels typically used in SV ma-chines. These can then be used to evaluate the right-hand sidein Corollary 21. Recall that . First we have tocompute the Fourier/Hankel transform for the kernels.

Example 22 (Gaussian RBFs):For Gaussian radial basis

functions (RBFs) in dimensions we haveand correspondingly

Example 23 (Exponential RBFs):In the case ofwe obtain

i.e., in the case of we recover the damped harmonic os-cillator (in the frequency domain). In general, we get a decay interms of the eigenvalues like . Moreover, we can con-clude from this that the Fourier transform of, viewed itself asa kernel, i.e., , yields the initial kernel asits corresponding power spectrum in Fourier domain.

Example 24 (Damped Harmonic Oscillator):Another wayto generalize the harmonic oscillator, this time in a way thatdoes not depend on the dimensionality, is to set .Following [58, Sec. 13.6] we get

where is the Bessel function of the second kind, defined by(see [50])

(79)

It is possible to upper-bound by utilizing the asymptoticrepresentation

(80)

(see, for example, [18, eq. (8.451.6)]) and we get exponentialdecay of the eigenvalues.

Using Theorem 20, Corollary 21, and Remark 15 one maycompute the entropy numbers numerically for a particular kerneland a particular set of parameters. This may seem unsatisfactory


from a theoretician’s point of view. However, as the ultimategoal is to use the obtained bounds for model selection, it is de-sirable to obtain as tight bounds (especially in the constants) aspossible. Hence, if much more precise bounds can be obtainedby some not too expensive numerical calculation it is definitelyworthwhile to use those instead of a theoretically nice but notsufficiently tight upper bound. The computational effort to cal-culate these quantities is typically negligible in comparison totraining the actual learning machine.

Notwithstanding the above, in order to give a feeling for theeffect of the decay of the Fourier transform of the kernel onthe entropy numbers of the operator, we conclude with thefollowing general result, the proof of which is in Appendix III.

Proposition 25 (Polynomial Exponential Decay in): Forkernels in with with

the entropy number of the corresponding scaling operator sat-isfies

IX. CONCLUSION

We have shown how to connect properties known about map-pings into feature spaces with bounds on the covering numbers.Exploiting the geometric structure of the feature-space map en-abled us to relate the properties of the kernel inducing the featurespace to the covering numbers of the class of functions imple-mented by SV machines based on such kernels.

The actual application of our results, perhaps for model selec-tion using structural risk minimization, is somewhat involved,but is certainly doable. Here, we outline one possible path. In[20] we present an application of the results to the performanceof SV machines for pattern classification.

A. One Way to Use the Results of this Paper

Choose and : The kernel may be chosen for a varietyof reasons, which we have nothing additional to say about here.The choice of should take account of the discussion in Sec-tion VI.

Choose the Period of the Kernel: One suggested procedureis outlined in Section VI.

Bound : This can be done using Proposition 9 (for thecase ) or Corollary 19 or 21 for the case . Someexamples of this sort of calculation are given in Section VII.

Bound : Using Theorem 12.Take Account of the “ ”: The key observation is that

given a class with known , one can boundas follows. (Here .)

Suppose is an -cover for and elements of areuniformly bounded by (this implies a limit on as well asa uniform bound on elements of). Then

is an -cover for and thus

Observe that this will only be “noticeable” for classeswithvery slowly growing covering numbers (polynomial in ).

Take Account of the Loss Function:Using Lemma 2 for ex-ample.

Plug into a Uniform Convergence Result:See the pointers tothe literature and the example in Section IV.

B. Further Work

The operator-theoretic viewpoint introduced in this paperseems fruitful. The overall bounds for SV classes can, via asomewhat involved argument, be considerably simplified [20].The general approach can be applied to various other learningmachines such as convex combinations of basis functions andmultilayer networks [47]. When combined with an appropriatestatistical argument [45], the approach yields bounds on thegeneralization that depend strongly on the particular sampleobserved [61]. The methods can also be applied to someproblems of unsupervised learning [49].

The results of the present paper hinge on the measurement ofthe size of the weight vector by an norm. In [62], we showthe effect of different norms for measuring the size of, as wellas presenting a number of related results.

We expect that further refinements and extensions to thesetechniques will continue to yield interesting results.

APPENDIX IPROOFS OFRESULTS IN SECTION VII

Proof (Proposition 16): The proof uses Proposition 9.Since there exists some with

. In this case, all sequenceswith lead to an admissible scaling property. One has

(81)

where is Riemann’s zeta function. Moreover, one can boundby

(82)

where is Euler’s constant. The next step is to evaluate theexpression

(83)

The Gamma function can be bounded as follows: for

(84)


Hence, one may bound

In order to avoid unneeded technicalities we will replaceby . This is no problem when computing

the upper bound, but it is an issue for the lower bound. How-ever, on is within a constant factor of of itscorresponding values on the integer domain, the biggestdiscrepancy being at .1 Thus, we may safely ignore theconcern. Next we compute

(85)

The maximum of the argument is obtained for , hence(85) holds for all , which is fine since we want tocompute bounds on as . For the lower boundson we obtain

(86)

This shows that is always bounded from below by. Computation of the upper bound is slightly more

effort, since one has to evaluate

(87)

Clearly, for any fixed we are able to obtain a rateof , thus, the theorem follows. For prac-tical purposes, a good approximation of thecan be found as

by computing the derivative of the argumentin (87) with respect to and dropping all terms independent of

and . However, numerical minimization of (87) is more ad-visable when small values of are crucial.

For the proof of Proposition 17 we need the following stan-dard Lemma.

1One may show [61] that

a � sup n (a ; . . . ; a ) � a

for that particularj wheresup is actually obtained. Hence, the maximumquotienta =a , which in the present case is2 , determines the value bywhich the bound has to be lowered in order to obtain a true lower bound.

Lemma (Summation and Integration in): Supposeis an integrable nonincreasing function. Then the following

inequality holds for any :

(88)

Proof: The proof relies on the fact that

due to the monotonicity of and a decomposition of the integral

The lemma is a direct consequence thereof.

Proof (Proposition 17): Since there existssome with . Similarly as before, we nowuse a series . Then by applying Lemma 26 wehave that for any

(89)

Next, we have to apply a similar bound to the product of the firstdiagonal entries of the scaling operator

(90)

The last inequality holds since . Next we compute

Differentiation of the exponent with respect toleads to

(91)

and, thus,

(92)

Replacing the domain from to is not aproblem when it comes to computing upper bounds on .As for the lower bounds, again, a similar reasoning to that in the


previous proof would have to be applied.2 (The proof is omittedhere.) Thus, can be bounded from below as follows:

(93)

Hence, a lower bound on the rate of is .Moreover, for the upper bound we obtain

(94)

One could evaluate (94) numerically. However, it can be seenthat for any fixed the rate of can bebounded by , which shows that the obtained ratesare tight.

APPENDIX IIPROOF OFTHEOREM 18

Proof: The first part of the inequality follows directlyfrom Theorem 7 as it is a weaker statement than the originalone. We prove the second part by closely mimicking the proofin [13, p. 17]. We define

(95)

and show that for all there is an index with .For this purpose, choose an indexsuch that and,thus, . Moreover, we have

(96)

because of the monotonicity of and, finally,

(97)

Using the definition of we thus conclude .If this happens to be the case for, we have whichproves the theorem.

2As in the previous theorem, the problem reduces to bounding the quotienta =a wherej is the variable for whichsup is obtained. However,

here the quotient can only be bounded bye . Fortunately, this is oflower order than the remaining terms, hence it will not change therate of thelower bounds.

If this is not the case, there exists an indexsuch that. Hence the corresponding sectional

operator

with

(98)

is of rank and the image of the closed unit ball

of is isometric to the subset of . In any case,is a precompact subset of. So let be

a maximal system of elements in with

for (99)

The maximality of this system guarantees that

(100)

and, thus, . In order to get an estimate for, we split the operator into two parts

which allows us to bound

(101)

Using and the bound onwe arrive at

(102)

The final step is to show that as then by substituting inthe definition of into (102) yields the result. This is againachieved by a comparison of volumes. Consider the sets

as subsets of the space which is possible sinceand . These sets are

obviously pairwise-disjoint. On the other hand, we have

(103)

as . Now a comparison of the-dimensional Eu-clidean volumes provides

vol

(104)and, therefore, . Using the defini-tion of this yields .

APPENDIX IIIPROOF OFPROPOSITION25

Proof: We will completely ignore the fact that we are ac-tually dealing with a countable set of eigenvalues on a latticeand replace all summations by integrals without further worry.Of course this is not accurate but still will give us the correctrates for the entropy numbers.

Denote the size of a unit cell, i.e.,the density of lattice points in frequency space as


given in Section VI. Then we get for infinitesimal volumesand numbers of points in frequency space

and, therefore,

(105)

(here denotes the volume of the -dimensional unitsphere) leading to

(106)

We introduce a scaling operator whose eigenvalues decay likefor . It is straightforward to check

that all these values lead to both useful and admissible scalingoperators. Now we will estimate the separate terms in (78).

(107)

Next, we have

(108)

and

(109)

(110)

This leads to

(111)

Computing the yields

(112)

and, therefore,

(113)

Already from this expression one can observe the rate boundson . What remains to be done is to compute the infimum over

. This can be done by differentiating (113) with respect to.Define

(114)

which leads to the optimality condition on

with (115)

which can be solved numerically.

APPENDIX IVMAPLE CODE TO COMPUTE

The following function can be used to compute :.

# This code defines a function t where

# t(m,d) is number of points on a sphere of

# radius 2̂=m from Z d̂

h:=n� >eval(‘if‘(isolve(m 2̂=n,m) =NULL,0,

‘if‘(n =0,1,2)),1):

powseries[powcreate](theta(n)=h(n)): t :=(m,d)-

coeff(convert(powseries[tpsform](

powseries[evalpow](theta d̂),

x,m+1),polynom),x,m):

Note Added in Proof

Steve Smale has pointed out to us that Claim 2 of Theorem 4due to König is false. This causes some of the intermediate re-sults in the paper to be false, but not the main theorems. One canget around the false result by redefining

(Note that most practically used kernels still have ).Only the “if” claim of Proposition 6 remains true (all we needfor or bounds on entropy numbers anyway). All of the upperbounds on entropy numbers still hold as long as(redefined asabove) is finite. A detailed correction can be found in Chapter 12of B. Schölkopf and A. Smola,Learning with Kernels, Cam-bridge, MA: MIT Press, 2001.

ACKNOWLEDGMENT

The authors wish to thank P. Bartlett, B. Carl, A. Ellisseff,Y. Guo, R. Herbrich, J. Shawe-Taylor, and A. Westerhoff forhelpful discussions and comments. This work would not havebeen possible had the European cup final not been held inLondon in 1996 immediately prior to COLT96.

REFERENCES

[1] M. A. Aizerman, E. M. Braverman, and L. I. Rozonoér, “Theoreticalfoundations of the potential function method in pattern recognitionlearning,”Autom. Remote Contr., vol. 25, pp. 821–837, 1964.

[2] S. Akashi, “Characterization of�-entropy in Gaussian processes,”KodaiMath. J., vol. 9, pp. 58–67, 1986.

[3] , “The asymptotic behavior of�-entropy of a compact positive op-erator,”J. Math. Anal. Applic., vol. 153, pp. 250–257, 1990.

[4] N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler, “Scale-sen-sitive dimensions, uniform convergence, and learnability,”J. Assoc.Comput. Mach., vol. 44, no. 4, pp. 615–631, 1997.


[5] M. Anthony, “Probabilistic analysis of learning in artificial neural net-works: The PAC model and its variants,”Neural Comput. Surv., vol.1, pp. 1–47, 1997. Also available [Online]: http://www.icsi.berkeley.edu/~jagota/NCS.

[6] M. Anthony and P. L. Bartlett,Artificial Neural Network Learning: The-oretical Foundations. Cambridge, U.K.: Cambridge Univ. Press, 1999.

[7] R. Ash, Information Theory. New York: Interscience , 1965.[8] P. Bartlett and J. Shawe-Taylor, “Generalization performance of support

vector machines and other pattern classifiers,” inAdvances in KernelMethods—Support Vector Learning, B. Schölkopf, C. J. C. Burges, andA. J. Smola, Eds. Cambridge, MA: MIT Press, 1999, pp. 43–54.

[9] P. L. Bartlett, P. Long, and R. C. Williamson, “Fat-shattering and thelearnability of real-valued functions,”J. Comput. Syst. Sci., vol. 52, no.3, pp. 434–452, 1996.

[10] B. E. Boser, I. M. Guyon, and V. N. Vapnik, “A training algorithm foroptimal margin classifiers,” in5th Annual ACM Workshop on COLT, D.Haussler, Ed. Pittsburgh, PA: ACM Press, 1992, pp. 144–152.

[11] B. Carl, “Entropy numbers of diagonal operators with an application toeigenvalue problems,”J. Approximation Theory, vol. 32, pp. 135–150,1981.

[12] , “Inequalities of Bernstein–Jackson-type and the degree of com-pactness of operators in Banach spaces,”Ann. Inst. Fourier, vol. 35, no.3, pp. 79–118, 1985.

[13] B. Carl and I. Stephani,Entropy, Compactness, and the Approximationof Operators. Cambridge, U.K.: Cambridge Univ. Press, 1990.

[14] C. Cortes and V. Vapnik, “Support vector networks,”Machine Learning,vol. 20, pp. 273–297, 1995.

[15] M. Defant and M. Junge, “Characterization of weak type by the entropydistribution of r-nuclear operators,”Stud. Math., vol. 107, no. 1, pp.1–14, 1993.

[16] F. Girosi, M. Jones, and T. Poggio, “Priors, stabilizers and basis func-tions: From regularization to radial, tensor and additive splines,” MIT,Cambridge, MA, A.I. Memo 1430, 1993.

[17] Y. Gordon, H. König, and C. Schütt, “Geometric and probabilistic esti-mates for entropy and approximation numbers of operators,”J. Approx-imation Theory, vol. 49, pp. 219–239, 1987.

[18] I. S. Gradshteyn and I. M. Ryzhik,Table of Integrals, Series, and Prod-ucts. New York: Academic, 1981.

[19] E. Grosswald,Representation of Integers as Sums of Squares. NewYork: Springer-Verlag, 1985.

[20] Y. Guo, P. L. Bartlett, J. Shawe-Taylor, and R. C. Williamson, “Coveringnumbers for support vector machines,” inProc. 12th Annu. Conf. Com-putational Learning Theory. New York: ACM, 1999, pp. 267–277.

[21] L. Gurvits, “A note on a scale-sensitive dimension of linear boundedfunctionals in Banach spaces,” inAlgorithmic Learning TheoryALT-97 (Lecture Notes in Artificial Intelligence), M. Li and A.Maruoka, Eds. Berlin, Germany: Springer-Verlag, 1997, vol. 1316,pp. 352–363.

[22] D. Jagerman, “�-entropy and approximation of bandlimited functions,”SIAM J. Appl. Math., vol. 17, no. 2, pp. 362–377, 1969.

[23] M. Junge and M. Defant, “Some estimates of entropy numbers,”IsraelJ. Math., vol. 84, pp. 417–433, 1993.

[24] M. J. Kearns, R. E. Schapire, and L. M. Sellie, “Toward efficient agnosticlearning,”Machine Learning, vol. 17, no. 2, pp. 115–141, 1994.

[25] V. I. Kolchinskiy, “Operators of typep and metric entropy” (in Rus-sian MR 89j:60007),Teoriya Veroyatnosteyi Matematicheskaya Statis-tika, vol. 38, pp. 69–76, 135, 1988.

[26] , “Entropic order of operators in Banach spaces and the central limittheorem,”Theory Probab. Its Applic., vol. 36, no. 2, pp. 303–315, 1991.

[27] A. N. Kolmogorov and V. M. Tihomirov, “�-entropy and�-capacity ofsets in functional spaces,”Amer. Math. Soc. Transl., ser. 2, vol. 17, pp.277–364, 1961.

[28] H. König, Eigenvalue Distribution of Compact Operators. Basel,Switzerland: Birkhäuser, 1986.

[29] T. Koski, L.-E. Persson, and J. Peetre, “�-entropy�-rate, and interpola-tion spaces revisited with an application to linear communication chan-nels,”J. Math. Anal. Applic., vol. 186, pp. 265–276, 1994.

[30] S. R. Kulkarni, G. Lugosi, and S. S. Venkatesh, “Learning patternclassification—A survey,”IEEE Trans. Inform. Theory, vol. 44, pp.2178–2206, Nov. 1998.

[31] W. S. Lee, P. L. Bartlett, and R. C. Williamson, “The importance ofconvexity in learning with squared loss,”IEEE Trans. Inform. Theory,vol. 44, pp. 1974–1980, Sept. 1998.

[32] G. G. Lorentz, M. v. Golitschek, and Y. Makovoz,Constructive Ap-proximation: Advanced Problems. Berlin, Germany: Springer-Verlag,1996.

[33] C. Müller, “Analysis of spherical symmetries in Euclidean spaces,” inApplied Mathematical Sciences. New York: Springer-Verlag, 1997,vol. 29.

[34] N. J. Nilsson,Learning machines: Foundations of Trainable PatternClassifying Systems. New York: McGraw-Hill, 1965.

[35] A. Pajor,Sous-Espaces̀ des Espaces de Banach. Paris, France: Her-mann, 1985.

[36] A. Pietsch.,Operator Ideals. Amsterdam, The Netherlands: North-Holland, 1980.

[37] L. S. Pontriagin and L. G. Schnirelmann, “Sur une propriété métriquede la dimension,”Ann. Math., vol. 33, pp. 156–162, 1932.

[38] R. T. Prosser, “The�-entropy and�-capacity of certain time-varyingchannels,”J. Math. Anal. Applic., vol. 16, pp. 553–573, 1966.

[39] R. T. Prosser and W. L. Root, “The�-entropy and�-capacity of certaintime-invariant channels,”J. Math. Anal. Applic., vol. 21, pp. 233–241,1968.

[40] S. Saitoh, Theory of Reproducing Kernels and its Applica-tions. Harlow, U.K.: Longman, 1988.

[41] B. Schölkopf, A. Smola, and K.-R. Müller, “Nonlinear componentanalysis as a kernel eigenvalue problem,”Neural Comput., vol. 10, pp.1299–1319, 1998.

[42] C. E. Shannon, “A mathematical theory of communication,”Bell Syst.Tech. J., vol. 27, pp. 379–423, 623–656, 1948.

[43] J. Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony, “Aframework for structural risk minimization,” inProc. 9th Annu. Conf.Computational Learning Theory. New York: ACM, 1996, pp. 68–76.

[44] , “Structural risk minimization over data-dependent hierarchies,”IEEE Trans. Inform. Theory, vol. 44, pp. 1926–1940, Sept. 1998.

[45] J. Shawe-Taylor and R. C. Williamson, “Generalization performance ofclassifiers in terms of observed covering numbers,” inProc. 4th Euro.Conf. Computational Learning Theory (EUROCOLT’99), 1999, pp.274–284.

[46] A. J. Smola and B. Schölkopf, “On a kernel-based method for patternrecognition, regression, approximation and operator inversion,”Algo-rithmica, vol. 22, pp. 211–231, 1998.

[47] A. J. Smola, A. Elisseff, B. Schölkopf, and R. C. Williamson, “En-tropy numbers for convex combinations and mlps,” inAdvances in LargeMargin Classifiers, A. Smola, P. Bartlett, B. Schölkopf, and D. Schuur-mans, Eds. Cambridge, MA: MIT Press, 2000.

[48] A. J. Smola, B. Schölkopf, and K.-R. Müller, “The connection betweenregularization operators and support vector kernels,”Neural Networks,vol. 11, pp. 637–649, 1998.

[49] A. J. Smola, R. C. Williamson, S. Mika, and B. Schölkopf, “Regular-ized principal manifolds,” inProc. 4th Euro. Workshop ComputationalLearning Theory (EUROCOLT’99), 1999, pp. 214–229.

[50] I. H. Sneddon,The Use of Integral Transforms. New York: McGraw-Hill, 1972.

[51] M. Talagrand, “The Glivenko–Cantelli problem, ten years later,”J.Theor. Probab., vol. 9, no. 2, pp. 371–384, 1996.

[52] H. Triebel, “Interpolationseigenschaften von Entropie- und Durch-messeridealen kompakter Operatoren,”Studia Math., vol. 34, pp.89–107, 1970.

[53] V. Vapnik, The Nature of Statistical Learning Theory. New York:Springer-Verlag, 1995.

[54] , Statistical Learning Theory. New York: Wiley, 1998.[55] V. Vapnik and A. Chervonenkis,Theory of Pattern Recognition(in

Russian). Moscow, U.S.S.R.: Nauka, 1974. German translation:W. Wapnik and A. Tscherwonenkis,Theorie der Zeichenerkennung,Berlin, Germany: Akademie-Verlag, 1979.

[56] V. N. Vapnik, Estimation of Dependences from Empirical Data. NewYork: Springer-Verlag, 1982.

[57] V. N. Vapnik and A. Ya. Chervonenkis, “Necessary and sufficient con-ditions for the uniform convergence of means to their expectations,”Theory Probab. Its Applic., vol. 26, no. 3, pp. 532–553, 1981.

[58] G. N. Watson,A Treatise on the Theory of Bessel Functions, 2ed. Cambridge, U.K.: Cambridge Univ. Press, 1958.

[59] H. Widom, “Asymptotic behavior of eigenvalues of certain integral op-erators,”Arch. Rational Mech. Anal., vol. 17, pp. 215–229, 1964.

[60] R. C. Williamson, B. Schölkopf, and A. J. Smola, “A maximum marginmiscellany,”Typescript, Mar. 1999.

[61] R. C. Williamson, J. Shawe-Taylor, B. Schölkopf, and A. J. Smola,“Sample based generalization bounds,”IEEE Trans. Inform. Theory,to be published.

[62] R. C. Williamson, A. J. Smola, and B. Schölkopf, “Entropy numbersof linear function classes,” inProc. 13th Annu. Conf. ComputationalLearning Theory. New York: ACM, to be published.

Documents

Generalization performance of regularization networks …users.cecs.anu.edu.au/~williams/papers/P100published.pdf · Generalization Performance of Regularization Networks and Support