Reproducing Kernel Hilbert spaces

Dr. M. Asaduzzaman Professor

Department of Mathematics University of Rajshahi

Rajshahi -6205, BangladeshE-mail: md_asaduzzaman@hotmail.com

DefinitionLet H be a Hilbert space comprising of complex valued functions on a set E. Then, H is called a reproducing kernel Hilbert space if there exists a function defined on EE such that

(a) for every y E, K(,y) H;(b) (reproducing property): for any f H,

.,),(,)( EyyKfyf H

In this case, K(x,y) is called a reproducing kernel of H.

2

Theorem A reproducing kernel Hilbert space admits a unique reproducing kernel.

Proof. Let H be a reproducing kernel Hilbert space comprising complex valued functions on a set E and let and be reproducing kernels of H. Then for any , we have

.0

),('),(),('),(),('),,('),(),(),,('),(

),('),(),,('),(),('),( 2

yyKyyKyyKyyKyKyKyKyKyKyK

yKyKyKyKyKyK

HH

HH

Therefore, for all .3

TheoremLet H be a Hilbert space comprising complex valued functions defined on a set E. Then, H is a reproducing kernel Hilbert space if and only if for every y E, the point evaluation at y is continuous on H; that is Ty:HC defined by Ty(f)=f(y) is a bounded linear functional on H, for every y E.Proof. Let H be a reproducing kernel Hilbert space and let K(x,y) be its reproducing kernel, then we have from the reproducing property and the Schwarz inequality,

.for,,(|),(,||)(||)(| 21

HfyyKfyKfyffTy

Hence, TyH*, for all y E.

Conversely, let TyH*, for all y E. Then, from the Riesz representation theorem there exists a unique Ky H such that f, Ky =Ty(f), for all f H. If we define K(x,y)= Ky, KxH for x,yE, then it is obvious that K(x,y) is a reproducing kernel of H.

4

ExampleLet be a finite dimensional complex vector space of functions. A finite dimensional space endowed with any inner product is always complete and therefore it is a Hilbert space.Let be an orthonormal basis in and define

.Then for any in , and for any function in , we have

.Any finite dimensional Hilbert space of functions has a reproducing kernel.

5

(2) Let and let be the set of complex sequences such that .

is a Hilbert space with the inner product if and . Consider.

and

6

(3) For any n-dimensional space Hn comprising complex valued functions on a set E, let {w1,w2,…,wn} be any basis on Hn. Then any member f of Hn is uniquely expressible in the form

n

jjjj cwcf

1

., C

Let A=(jk) be an n-square positive definite Hermitian matrix. Introduce normin Hn by the formula

)1(,α1 1

*2ji

n

i

n

jijH

ccACCfn

where C* is the conjugate transpose of the coordinate vector of f.

Let B=(ij) be the conjugate inverse of A and define

n

i

n

jjiij EEonqwpwqpK

1 1

.)()(),(

Then K(p,q) is the reproducing kernel for the space Hn equipped with the norm given by (1).

Then, we have ;,αnHjiij ww that is, A is the Gramm matrix the of the

system {w1,w2,…,wn}.

7

In particular, for any positive definite nn Hermitian matrix A=(a) ,

aK )μν,(

Is the reproducing kernel for the for the vector space Cn equipped withThe inner product

,~*, xAyyx

Where * denotes complex conjugate transpose and is the complex conjugate inverse of A.

A~

(4) For any abstract set E and for any complex valued function f(p) (|f(p)|<)on E, the functionis the reproducing kernel for the one dimensional space generated by f(p) and equipped with the inner product

)()(),( qfpfqpK

.βα)(β),(α qfpf

8

is a Hilbert space with inner product . Definition of reproducing kernel does not strictly apply in that case.(6) Let and

is defined almost everywhere as the derivative of . is a Hilbert space with the inner product .

belongs to the class of Sobolev spaces.

9

Consider . The weak derivative of is the function and

Then is a reproducing kernel. All infinite dimensional separable Hilbert spaces are isomorphic to which has a reproducing kernel. This property gives no guarantee that a given separable Hilbert space of functions has a reproducing kernel.

10

DefinitionLet E be a set and let be a complex valued function defined on EE. Then K(x,y) is called a positive matrix or a positive type function on E if, for any finite set of points in E and for any 1, 2,,n, in C

n

i

n

jjiji xxK

1 1

.0),(

niix 1

11

Theorem A positive matrix on a set E satisfies the following relations:

.,),(),(|,(|)(

.,for),(),()(

.for0),()(

2 EyxforyyKxxKyxKc

EyxxyKyxKb

ExxxKa

Kd )(

Proof. From the definition of a positive matrix (a) is clear. Again, from the definition of a positive matrix,

)1(0),(2

1

2

1

i j

jiji xxK

is a positive matrix.

12

for any x1, x2 E and 1, 2 C. If we substitute x1 and x2 by x and y respectively, and if we put 1=1 and 2=z , then from (1) we have is a nonnegative real number for arbitrary x,yE and z C. Hence,

,),(),(),(),(

),(),(),(),(

yxKxyKzyxKxyKz

zxyKzyxKzxyKzyxK

and so is real for an arbitrary z C. Thus, we have (b). ),(),( yxKxyKz

Next, let x, y E and let be a complex number such that | |=1 and K(y,x)=|K(x,y)|. Let r be any real number. Then by setting x1=x, x2=y, 1=- and 2= r in (1), we have

.0),(),(),(),( 2 yyKryxKrxyKrxxK That is )2(.0),(|),(|2),( 2 yyKrryxKxxK

If K(y,y)=0, we must have K(x,y)=0; otherwise (2) is false for a large positive number r. If K(y,y)>0, it is necessary that

,0),(),(,( 2 yyKxxKyxK and we have (c). 13

Since K(x,y) is a positive matrix on E, for any finite set of points in E and for any 1, 2,,n, in C

niix 1

n

i

n

j

n

i

n

jijjijiji xxKxxK

1 1 1 1

.0),(),(

Hence, we have (d).

14

Example(1) Any constant non negative function onis of positive type function.For every and for every

15

(2) Let be any non empty set. The delta function

is of positive type.

For every and

for every

16

Lemma : Let be some Hilbert space with inner product and let . Then , the function is of positive type.Proof: For every and for every

.Lemma tells us that writing in some is sufficient to prove definiteness of

17

Examples of positive matrices :

(1) Let then any complex valued function on E can be considered as an element of ℂm . Let e1, e2, e3, …,en be any vectors from ℂm Let A be the mn matrix whose i-th column vector is ei. Then the matrixK=AA* is a positive matrix on E, where A* denotes the conjugate transpose of A.

(2) For x,y and let Then K(x,y) is a positive type functionℝ on . For ℝ

.)1(2

121),( 2

)(

deeyxKyxi

yx

(3) For and let Then is a positive type function on . For ℝ

.21),( yxeyxK

which is the inner product of and in ixe iye .)1(2

, 22

dL R

.||141),( yxeyxyxK

.)1(2

1||141),( 22

)(

deeyxyxKyxi

yx

18

Reproducing Kernels and Positive type functionsLemma: Any reproducing Kernel is Positive type function.Proof: Suppose is the reproducing kernel of For every and for every

Hence is positive type function.

19

LemmaA real function defined on is a positive type function if and only if i.e. is symmetric and for every

20

LemmaIn a reproducing kernel Hilbert space a sequence converging in the norm sense converges point wise to the same limit.Proof: Let converges to in the norm sense . Then

Hence converges point wise to .

21

Theorem Let be any subspaces of ,the space of complex functions on , on which an inner product , with associate norm .In order that there exists a Hilbert space such that a) and the topology defined on by the inner product coincides with the topology induced on by b) has a reproducing kernel .It is necessary and sufficient that c) the evaluation functional are continuous on .d) any Cauchy sequence in converging pointwise to 0

converges also to 0 in norm sense.Assumption d) is equivalent tod’)for any function in and any Cauchy sequence in converging pointwise to , converges also to in the norm sense.

22

Proof Let is exist with conditions a) and b) are satisfied.Then the evaluation functionals are continuous on and therefore on .Let be a cauchy sequencein converging pointwise to 0.As is complete, converges in the norm sense to some .Thus we have

and Hence converges to 0 in norm sense.

23

Conversely , suppose c) and d) hold.Define as being the set of functions in for which there exists a Cauchy sequence in converging pointwise to .

Obviously . Lemma: Let and belong to . Let and be two Cauchy sequences in converging pointwise to and . Then the sequence is convergent and its limit only depends on and .ProofWe know that any Cauchy sequence is bounded.

, by Cauchy –Schwarz inequality. This shows that is a Cauchy sequence in and therefore convergent.

24

In the same way, if and are two other Cauchy sequences in converging pointwise respectively to and , we have ,

and are Cauchy sequences in converging pointwise to

From assumption d) they also converges to 0 in the norm sense . It follows that and have the same limit.

25

•

26

LemmaLet and be a Cauchy sequence in converging pointwise to . Then converges to in the norm sense.

ProofLet and let be such that and . Fix . Then sequence is a Cauchy sequence in converging pointwise to .Therefore .Thus converges to in the norm sense.

Corollary : is dense in .ProofBy definition, for any there exists a Cauchy sequence in converging pointwise to .Again converges to in the norm sense. Hence is dense in . 27

Lemma: The evaluation functionals are continuous on .Proof: As the evaluation functionals are linear it suffices to show that they are continuous at 0.Let . The evaluation functionals are continuous on .Fix and let such that and .

For any function in with there exists a function in such that and .

This entails Hence and . Thus is continuous on .

28

Lemma: is a reproducing kernel Hilbert space.ProofIt remains to prove that is complete.Let be a Cauchy sequence in and let . As evaluation functional is linear and continuous , is a Cauchy sequence in and thus converges to some . One has to prove that such defined f belongs to . Let be any sequence of positive numbers tending to zero as n tends to . As is dense in such that .From the inequalities

and from the properties of it follows that tends to as tends to .

29

We have

.Thus is a Cauchy sequence in converging pointwise to and so. Again converges to in the norm sense.Now .Therefore converges to in the norm sense and is complete.

30

Remark: As is dense in is isomorphic to the completion of.It is the smallest Hilbert space of functions on satisfying (a) .is called the functional completion of .

Theorem: (Moore- Aronszajn) Let be a positive type function on .There exists only one Hilbert space of functions on with as a reproducing kernel. The subspace of spanned by the functions is dense in and is the set of functions on which are pointwise limits of Cauchy sequences in with the inner product

where and

31

Proof The complex number defined does not depend on the representations not necessarily unique of and :

,this shows that depends onand only through their values. Then, takingand we get

Thus the inner product with ‘ reproduces ’ the values of functions in . In particular .

32

As is a positive type function, is a semi-positive Hermitian from on .Now, suppose that . From the Cauchy-Schwarz inequality we have

and . Let us consider endowed with the topology associated with the inner product and check conditions : The evaluation functional are continuous on . Any Cauchy sequence in converging pointwise to 0 converges also to 0 in norm sense.

33

Let and in

.Therefore the evaluation functionals are continuous on .Let be a Cauchy sequencein converging pointwise to 0 and let be an upper bound for . Let and such that .Fix , and such that .

34

As

we have, for , hence . As is arbitrary This implies that converges to 0 in norm sense.Then there exist a Hilbert space of functions on satisfying and the topology defined on by the inner product coincides with the topology induced on by and

has a reproducing kernel .

35

is the set of functions in for which there exists a Cauchy sequence in converging pointwise to . Then such a sequence converges to in norm sense: is dense in . Therefore is unique and

Thus is the reproducing kernel of .

Theorem claims that a RKHS of functions on a set is characterized by its kernel on and that the property for of being a reproducing kernel is equivalent to the property of being a positive type function.

36

: The space is the set of complex sequencessatisfying

endowed with the inner product

Theorem: A complex function defined on is a reproducing kernel or a positive type function if and only if there exists a mapping from to some space such that

.

37

Proof Let be a RKHS of functions on a set with kernel . Consider the mapping

Since any Hilbert space, is isometric to some space .If denotes any isometry from to , the mappingmeets the requirements.

38

Conversely, Suppose that there is a mapping such that

Since is a Hilbert space with inner product

.This implies that is a positive type function and hence is Reproducing Kernel of a Hilbert Space.This theorem provides a characterization of ALL reproducing kernels on an abstract set . It turns out that the definition of a positive type function or of a reproducing kernel on or of a RKHS of functions on is equivalent to the definition of a mapping on with values in some .

39

Particularizing to a set , a pre-Hilbert space which can be considered through a suitable isomorphism as a part of a space and mapping from to , one can construct as many reproducing kernels as desired.

Example: Let and By previous Theorem we get that defined on by

is a reproducing kernel. where if if 0 and

40

Integral transforms and isometrical identities:

Let dm be a positive measure on a set D. For a set E, let h(t,x) be a complex valued function defined on DE such that, for every x E, h(t,x) belongs to as a function of t. Then it is obvious that the function),(2 dmDL

D

tdmxthythyxK )(),(),(),(

is a positive matrix on E, and so it is interesting to determine the reproducing kernel Hilbert space admitting K(x,y) as a reproducing kernel.We consider the problem under more general situation.Let H be a Hilbert space and let h: E H be a function from an abstract set E. Let F(E) be the complex vector space comprising all complex valued functions on E. Then the null space N(L) of the linear map L: H F(E) defined by HfphfpLf H ,)(,))((

is a closed subset of H. For, the point evaluation map defined by Lp(f)=(Lf)(p) is bounded. Hence, N(Lp) is a closed subset of H. Since N(L) is a closed subset of H. Thus the quotient space is a Hilbert space.

),()( pEpLNLN

)(LNH

41

Therefore, by the fundamental theorem of linear maps, there is a linear bijection map defined by If we define a norm in R(L) byThen becomes an isometry and hence,becomes a Hilbert space. In fact we have

)()(

: LRLN

H .),())(( HffLLNf

)(LR ).(,)(

)()(

fLgLNfgLN

HLR

)(

),(LR

LR

Theorem The space is a reproducing kernel Hilbert space with reproducing kernelFurthermore, the mapping L is an isometry from H into R(L) if and only if is complete in H.

)(

),(LR

LR

.)(),(),( HphqhqpK

}:)({ Epph

42