A NEURAL NETWORK APPROACH TO OFF-LINE SIGNATURE VERIFICATION USING DIRECTIONAL PDF

8/20/2019 A NEURAL NETWORK APPROACH TO OFF-LINE SIGNATURE VERIFICATION USING DIRECTIONAL PDF

http://slidepdf.com/reader/full/a-neural-network-approach-to-off-line-signature-verification-using-directional 1/10

Pergamon

Pattern Recognition Vol. 29, No. 3, pp. 415 424, 1996

Elsevier Science Ltd

Copyright © 1996 Pattern Recognition Society

Printed in Great Britain. All rights reserved

0031 3203/96 15.00+.00

0031-3203 95)00092-5

NEURAL NETWORK APPROACH TO OFF LINE

SIGNATURE VERIFICATION USING DIRECTIONAL PD F

J .- P. D R O U H A R D , R . S A B O U R I N a n d M . G O D B O U T

Labo ratoire d 'Im agerie , de Vision et d 'Intelligence Artific ielle (LIVIA), Drpartem ent de g~nie de la

produc tion au tomatisre , l~cole de technologie sup+rieure, 4750 Henri-Julien, M ontrr al (Quebec), Canada

H2T 2C8

Received 24 February 1994; revised 11 April 1995; received or publication 3 July 1995)

Ab str aet --A neural network a pproac h is propo sed to build the firs t s tage of an A utom atic Handwritten

Signature Verification System. The directional Pro bability Density Functio n was used as a g lobal shape

factor and its discriminating power w as enhanced b y reducing its cardin ality via filtering. Various

experimental proto cols were used to implement the ba ckpro pagation network (BPN ) classifier. A com pari-

son, on the same da tabase and with the same decis ion rule, shows that the B PN classifier s c learly better than

the thresh old classifier and co mp ares favourably with the k-Nearest-Neighbour classifier.

Pattern recognition Classifiers Neural networks Backp ropagation

Autom atic s ignature verification Directional proba bility density function

1. INTRODUCTION

T h e d e s i gn o f a c o m p l e t e A u t o m a t i c H a n d w r i t t e n

S i g n a t u r e V e r i f i c a t i o n S y s t e m ( A H S V S ) t h a t w i l l b e

a b le to t a k e in to a c c o u n t a l l c l a s s e s o f fo rg e r i e s i s a v e ry

d i f f i c u lt t a s k . ~1) In d e e d , a c o m p le te A H S V S s h o u ld b e

a b l e t o d i s c r i m i n a t e b e t w e e n g e n u i n e s i g n a t u r e s a n d

th e fo l lo w in g fo rg e r ie s :

r a n d o m f o r g e r i e s ,

c h a r a c t e r i z e d

b y a d i f f er e n t se m a n t i c m e a n i n g a n d c o n s e q u e n t l y b y

a d i f fe r e n t o v e r a l l s h a p e w h e n c o m p a r e d t o g e n u i n e

s ig n a tu re s ; s i mp l e f o r g e r i e s , w i t h t h e s a m e s e m a n t i c

m e a n i n g a s g e n u i n e s i g n a t u r e s b u t a n o v e r a l l s h a p e

th a t d i f f e r s g re a t ly ;

f r e e h a n d a n d s i mu l a t e d f o r g e r i e s ,

p r o d u c e d w i t h t h e a pr ior i k n o w l e d g e o f b o t h t h e

s e m a n t i c m e a n i n g a n d t h e g r a p h i c a l m o d e l o f a t a r g e t

s i g n a t u r e b y a s k i l le d o r a n o c c a s i o n a l f o r g e r r e s p e ct -

ive ly ; f ina l ly , t r a c in g f o r g e r i e s a n d p h o t o c o p i e s , w i t h

a l m o s t t h e s a m e g r a p h i c a l a s p e c t a s g e n u i n e s i g n a -

t u r e s , b u t w i t h d i f f e r e n t p s e u d o - d y n a m i c p r o p e r t i e s

s u c h a s d i s s im i l a r i t i e s n g re y - l e v e l - r e l a t e d e a tu re s l i k e

t e x tu re , c o n t ra s t .

I n s u c h a s y s te m , in o r d e r t o t a k e i n t o a c c o u n t a l l

c l a s s e s o f fo rg e r ie s , t h e d e c i s io n i s m a d e o n ly a t t h e e n d

o f th e v e r i f i c a t io n p ro c e s s . C o n s e q u e n t ly , t h i s a p -

p r o a c h i s a v e r y c o s t l y s o l u t i o n i n t e r m s o f c o m p u t a -

t i o n a l r e s o u r c e s a n d i n t e r m s o f r e l a t e d a l g o r i t h m i c

c o m p l e x i t y .12) S in c e r a n d o m a n d s im p le fo rg e r i e s r e p -

r e s e n t a l m o s t 9 5 o f t h e c a s es g e n e r a l l y e n c o u n t e r e d

in p ra c t i c e , 13'41 a b e t t e r s o lu t io n m ig h t b e to s u b d iv id e

th e v e r i f i c a t io n p ro c e s s in s u c h a w a y to r a p id ly e l im i -

n a t e g r o s s f o r g e ri e s . T h u s , a t w o - s t a g e A H S V S s e e m s

to b e a m o re p ra c t i c a l s o lu t io n , w h e re th e f i r s t s t a g e

w o u l d b e r e s p o n s i b l e f o r t h is r a p i d e l i m i n a t i o n a n d t h e

s e c o n d s t a g e u s e d o n l y i n c o m p l i c a t e d c a s e s . T h e

d e s i g n o f t h i s f i r s t s t a g e w a s m a d e w i t h r a n d o m

fo rg e r i e s b a s e d o n th e f a c t t h a t a v e r i f i c a t io n s y s t e m

a b l e t o c o p e w i t h r a n d o m f o r g e r i e s w i l l b e a b l e t o

s u c c e s s fu l ly d i s c r im in a te s im p le fo rg e r i e s .

T h e f i r s t s t a g e o f t h i s c o m p l e t e A H S V S t h u s h a s t w o

m a i n o b j e ct i v e s: f i rs t ly , t o c o n s i d e r o n l y r a n d o m a n d

s im p le fo rg e r i e s a n d , s e c o n d ly , t o m a k e a r a p id d e c i -

s io n . T o m e e t t h e f i r s t o b je c t iv e , a c h a ra c te r i s t i c d e a l -

i n g w i t h t h e o v e r a l l sh a p e o f h a n d w r i t t e n s i g n a t u r e s

s e e m s a p p r o p r i a t e . A c c o r d i n g l y , w e h a v e c h o s e n t o u s e

t h e d i r e c ti o n a l P r o b a b i l i t y D e n si t y F u n c t i o n ( P D F ) a s

a g lo b a l s h a p e f a c to r . ~51 I t s d i s c r im in a t in g p o w e r i s n o t

o p t i m u m b e c a u se , e ve n t h o u g h i t is i n v a r i a n t i n t r a n s -

l a t io n a n d in s c a l e , i t i s n o t i n v a r i a n t i n ro t a t io n . O n

t h e o t h e r h a n d , i t d o e s n o t r e q u i r e t o o m u c h c o m p u t e r

t im e , t h e re b y s a t i s fy in g th e s e c o n d o b je c t iv e .

T o m e e t t h e s e c o n d o b je c t iv e , w e h a v e c h o s e n to u s e

a B a c k P r o p a g a t i o n N e t w o r k ( B P N ) a s a si g n a tu r e

c l a s si f i e r . In d e e d , o n c e t r a in e d , u n l ik e c o n v e n t io n a l

c l a ss i fi e rs s u c h a s t h e k N e a r e s t N e i g h b o u r ( k N N )

c la s s if i e r , i t h a s a v e ry f a s t r e s p o n s e t im e s in c e i t d o e s

n o t h a v e to m e m o r iz e in fu l l a l l s ig n a tu re s p e c im e n s .

H o w e v e r , t h e l e a rn in g p h a s e o f t h e s e c la s s i f i e rs i s

a r e l a t iv e ly d if f i c u l t t a s k in th i s a p p l i c a t io n . A s a m a t -

t e r o f f a c t , w i th th i s t y p e o f c l a s s if i e r , w e m u s t k n o w

a pr ior i a l l t h e f a l s e s ig n a tu re s a n d h a v e m a n y

e x a m p l e s p r i o r t o t r a i n i n g . I n a d d i t i o n , d u e t o t h e v e r y

h i g h v a r i a b i l i t y o f h a n d w r i t t e n s i g n a t ur e s , t h e s e p a r -

a t i o n b e t w e e n t r u e a n d f a ls e si g n a t u r e s is n o t a s h a r p

o n e . T h i s r e s u l t s i n v e ry lo n g c o n v e rg e n c e t im e s a n d

th e o v e ra l l p e r fo rm a n c e w i l l n e v e r b e p e r fe c t . N e v e r -

th e l e s s, a s s h o w n in a f e a s ib i l i t y s tu d y , ~5) the se diffic ul-

t i es c a n b e d i m i n i s h e d i f s o m e p r e c a u t i o n s a r e t a k e n

a n d t h e r e s u l t s o b t a i n e d c o r r e s p o n d t o t h e m a i n

415



416 J .-P . DR OU HA RD et al

o b je c t iv e o f t h e f i r s t s t a g e o f t h e d e c i s io n , w h ic h i s t h e

ra p id e l im i n a t io n o f g ro s s fo rg e r ie s .

T h i s p a p e r a d d r e s s e s t h e p r o b l e m s r e l a t e d t o t h e

d e s i g n o f th e f i r st s t a g e o f a c o m p l e t e A H S V S . I n

S e c t io n 2 , w e g iv e a l l t h e d e f in i t i o n s th a t a re u s e d

t h r o u g h o u t t h i s p a p e r . T h e c h o i c e o f t h e p r e - t r e a t m e n t

p e r f o r m e d o n t h e fu l l d i r e c t i o n a l P D F i n o r d e r t o

e n h a n c e i t s p e r fo rm a n c e s i s b r i e fly p re s e n te d in S e c -

t io n 3 . T h e d e s ig n o f t h e B P N u s e d a s a s ig n a tu re

c l a s s i f ie r i s s u m m a r iz e d in S e c t io n 4 . F in a l ly , in S e c t io n

5 th e B P N c l a ss i f ie r p e r f o r m a n c e s a r e c o m p a r e d , o n

t h e s a m e d a t a b a s e , t o t h o s e o b t a i n e d w i t h t h e c o n v e n -

t io n a l k N N a n d T c l a s s i f i e r s .

2. DEFINITIONS

2 1 D a t a s e ts

A s t a n d a r d s i g n a t u r e d a t a b a s e o f 4 0 s i g n a t u r e s w r i t -

t e n b y 2 0 in d iv id u a l s 8 0 0 im a g e s ) i s u s e d in th i s s tu d y .

F o r th e c h o ic e o f p re - t r e a t m e n t S e c t io n 3 ), t h e f i r st 2 0

a n d th e l a s t 2 0 s ig n a tu re s o f e a c h w r i t e r w e re u s e d to

b u i ld th e r e fe re n c e s e t a n d th e t e s t s e t , r e s p e c t iv e ly . F o r

th e d e s ig n o f t h e B P N S e c t io n 4) , t h e f i r st 20 s ig n a tu re s

w e re u s e d to b u i ld th e t r a in in g s e t , b u t t h e l a s t 2 0

s i g n a t u r e s w e r e d i v i d e d i n t o t w o g r o u p s t o b u i l d t h e

tes t se t f i rs t 10) an d the va l i da t ion se t la s t 10) .

2.2. R e f e r e n c e a n d t r a i n i n o s e ts

T h e re fe re n c e s e t fo r t h e k N N c la s s i f ie r a n d th e

t r a i n i n g s e t fo r t h e B P N c l a s si f i er w e r e c o m p o s e d o f

2 8 0 e x a m p le s : 1 4 0 r e l a t e d to th e g e n u in e s ig n a tu re s o f

a n in d i v id u a l c l a s s 0 )1 ) a n d a n o th e r 1 4 0 r e l a t e d to

r a n d o m f o r g e ri e s d ef i n e d a s a s u b s e t o f s i g n a t u r e s f r o m

a l l t h e o th e r i n d iv id u a l s c l a s s 0 )2 ). F o r e a c h w r i t e r , t h e

1 4 0 e x a m p le s in c l a s s 0 )1 w e re o b ta i n e d b y ro t a t in g a l l

t h e f u l l d i r e c t i o n a l P D F i s s u e d f r o m t h e 2 0 r e f e r e nc e

s ig n a tu re s in a c i r c u la r f a s h io n f ro m - 6 to + 6 ° i n 2 °

in c re m e n t s . In th e c a s e o f c l a s s 0 )2 , t h e 1 4 0 e x a m p le s

w e r e o b t a i n e d b y c h o o s i n g s e v e n o r e i g h t r e f e r e n c e

s i g n a t u r e s at r a n d o m f r o m 1 9 o t h e r i n d i v i d u a l s . T h e

c a rd i n a l i ty o f c l a s se s 0)1 a h d 0 )2 is e q u a l i n o rd e r n o t t o

fa v o u r o n e c l a s s o v e r th e o th e r .

2.3.

T e s t s e t f o r t h e k N N c l as s if ie r

T h e t e s t s e t fo r t h e k N N c la s s i f ie r u s e d in S e c t io n

3 c o n ta in s 1 6 0 e x a m p le s : 2 0 g e n u in e t e s t s ig n a tu re s

c l a s s 0 )1 ) a n d 1 4 0 s ig n a tu re s t a k e n a t r a n d o m f ro m a l l

t h e o th e r w r i t e r s c l a s s 0 )2 ), i n th e s a m e w a y a s fo r t h e

re fe re n c e a n d t r a in in g s e t s . In th i s c a s e , w e d o n o t

in t ro d u c e a n y ro t a t io n in c l a s s 0)1, s in c e i t i s n e c e s s a ry

t o c h e c k t h e c l a s si f i er p e r f o r m a n c e w i t h s t a t i s ti c a l l y

i n d e p e n d e n t s i g n a tu r e s .

2.4. T e s t s e t f o r t h e B P N c l a s s if i e r

T h e t e s t s e t fo r t h e B P N c la s s i f i e r u s e d in S e c t io n


c l a s s 0 )1 ) a n d 9 5 s ig n a tu re s t a k e n a t r a n d o m f ro m a l l

t h e o th e r w r i t e r s c l a s s

0 2 .

T h e 1 0 e x a m p le s o f c l a s s 0) 1

a re th e f i r st 1 0 s ig n a tu re s o f t h e t e s t s e t o f o n e in d iv id -

u a l . T h e 9 5 e x a m p le s o f c l a s s 0 )2 a re f iv e r a n d o m

e x a m p le s o f t h e l a s t 1 0 s ig n a tu re s o f th e t e s t s e t o f t h e

o th e r 1 9 in d iv id u a l s . T h e c a rd in a l i ty o f c l a ss 0)2 w a s

r e d u c e d t o o f fs e t t h e d i m i n u t i o n i n t h e c a r d i n a l i t y o f

c l a s s 0 )1 , b u t n o t t o o m u c h in o rd e r n o t t o a f f e ct t h e

a c c u r a c y o f t h e p e r f o r m a n c e m e a s u r e .

2.5.

Val ida t ion se t fo r a l l c lass i fi e rs

T h e v a l id a t io n s e t fo r a l l c l a s si f i e r s u s e d in S e c t io n


c l a s s 0 )1 ) a n d 9 5 s ig n a tu re s t a k e n a t r a n d o m f ro m a l l

t h e o th e r w r i t e r s c l a s s 0 )2 ). T h e 1 0 e x a m p le s o f c l a s s 0) 1

a re th e l a s t 1 0 s ig n a tu re s o f t h e t e s t s e t o f o n e in d iv id -

u a l . T h e 9 5 e x a m p le s o f c l a s s 0 )2 a re f iv e r a n d o m

e x a m p le s o f t h e l a s t 1 0 s ig n a tu r e s o f t h e t e s t s e t o f t h e

o th e r 1 9 in d iv id u a l s .

2.6. P e r f o r m a n c e m e a s u r e s

T h e p e r f o r m a n c e o f e a c h c l a s si f i er is e v a l u a t e d g l o -

b a l ly fo r t h e 2 0 w r i t e r s a n d fo r 2 5 e x p e r im e n t s fo r

w h ic h th e r e fe re n c e , t r a in in g a n d t e s t s e ts a re c h a n g e d

e a c h t im e . T h u s , i n e a c h e x p e r im e n t c l a s s 0 ) t w i l l

a lw a y s b e th e s a m e , b u t c l a s s 0 )2 w i l l b e a lw a y s d i f f e r-

e n t . In th i s w a y , i t i s p o s s ib l e to r e d u c e th e b i a s th a t

c o u l d h a v e b e e n i n t r o d u c e d b y p a r t i c u l a r r a n d o m

f o r g er i e s. C l a s si f i e r p e r f o r m a n c e i s m e a s u r e d b y m e a n s

o f th e to t a l e r r o r r a t e e e x p re s s e d in t e rm s o f e 1 ty p e

I e r r o r r a t e , t h e f a l s e c l a s s i f i c a t io n o f g e n u in e s ig n h -

tu re s) , e : t y p e I I e r ro r r a t e , t h e f a l s e c l a s s i f i c a t io n o f

r a n d o m f o r g er i es ) a n d P [ 0 ) i ] , th e a pr ior i p r o b a b i l i t y

for c lasses 0 ) i , wh ich is se t a t 0 .5 in o ur case :

e t = E 1 X P [0 )1 ] ) + e 2 x P [0 )2 ] ) ). W h e n a p p ro p r i a t e ,

w e u s e th e to t a l r e j e c t io n r a t e R t , w h i c h i s o b t a i n e d

w i t h a n e q u a t i o n s i m i l a r t o t h e l a t t e r o n e w h e r e e r r o r

ra t e s e i a r e s u b s t i t u t e d w i th r e j e c t io n r a t e s R i R 1

r e j e c ti o n o f g e n u i n e s i g n a t u r e s a n d R 2 r e j e c ti o n o f

r a n d o m f o rg e r ie s ). F i n a l l y , i n o r d e r t o t a k e i n t o a c -

c o u n t i n a s in g l e p a r a m e t e r t h e t o t a l e r r o r a n d r e je c -

t ion ra tes , a re l ia b i l i t y fac t o r 161 de f ine d as fo l lows :

R F = 100 - e t - R t ) / 1 0 0 - R t ) i s u s e d to f in d th e b e s t

c o n f i g u r a t i o n o f t h e B P N c l as s if i er .

3 C H O I C E O F PRE-TREATMENTON THE FULL

D I R E C T I O N A L P D F

T h e g o a l o f p r e - t r e a t m e n t o n t h e f u l l d i r e c t i o n a l

P D F i s t o e n h a n c e i t s d i s c r i m i n a t i n g p o w e r , i n o t h e r

w o r d s , t o i m p r o v e t h e p e r f o r m a n c e s o f a c l a s si f i er th a t

u s e s th e P D F a s a n in p u t v e c to r . A c l a s s i f i e r w i l l b e

m o r e e f f ic i e nt if t h e d i m e n s i o n o f t h e i n p u t v e c t o r i s n o t

t o o b i g u n n e c e s s a ry d a t a ) a n d i f t h e v a r i a t i o n s i n t h e

i n p u t v e c t o r a r e n o t t o o a b r u p t n o i s y d a t a ) . U n f o r t u -

n a t e l y , t h e f u l l d i r e c t i o n a l P D F d o e s n o t h a v e t h e s e

c h a ra c te r i s t i c s F ig . 1). T h u s , p re - t r e a tm e n t , i n c lu d in g

f i l te r i n g a n d c o m p r e s s i o n , m u s t b e c a r r i e d o u t . T h e

p u r p o s e o f t h i s s e c t i o n is t o d e t e r m i n e w h i c h i s t h e b e s t



Off-line signature verification 417

0 0 3 0

0 0 2 5

0 0 2 0

0 0 1 5

0 0 1 0

0 0 0 6

0 0 0 0

i

I F u l l P D F

P r e t r e a t e d P D F

i L L

0 2 0 4 0 6 0 8 0 1 0 0 1 2 0 1 4 0 1 6 0 1 8 0

d e g r e e s

Fig. 1. Full and pretreated (with a 6B7 filter) directional PDF of the handwritten signature shown in the

upper part of this figure.

comb ina tion based on the following heuristic: the

best performance having the higher compression rate

and the shorter computer time .

3.1.

Description of the pre-treatments

The treatments that we have chosen to evaluate

can be classified in three categories. The first, called

the integration filter (I) , consists of making a summa-

tion of the full directional PDF samples inside a win-

dow, whose width is equal to the compression step

which can take the values 1, 2, 3, 4, 5, 6, 9, 10 (A) or

12 (B). The second, call ed the undersamplin f i l ter

(U), consists of making a simple undersampling of

the full directional PD F with the value of the compres-

sion step. The third, called weighted filter, consists

of first smoothing the full directional PDF with

three types of filters: rectangular (R), triangular (T)

and binom ial (B), whose window width can take vari-

ous values: 3, 5, 7, 9 or I I(A), and then perfo rming

a simple undersa mplin g of this smoothed curve. In

each case the process is the same and only the coeffi-

cient number varies with the window width and its

value will depend on the filter used. In addition, to

facilitate the comparisons, the s um of the coefficients is

always equal to one to maintain the window area

constant.

3.2. Experimental protocol

To find the pre-treatment that improves the dis-

criminatingpower of the directional PD F the most, we

used the kN N classifier because it permits the evalu-

ation of a lower limit of the total error rate when the

maxim um available information is kept in memory. 7t

The reference set and the test set needed by a kNN

classifier have been described in Secton 2.2 and 2.3,

respectively. The total error rate e,, as defined in Sec-

tion 2.6, is evaluated for each writer in the signature

database. The mean of these e, is then determined to

obtain the globa l total error (eg,) of the verificat ion

system. This procedure is repeated 25 times with 25

different reference and test sets. Finally , it is the mean

of these 25 egt that is used to choose the best pre-

treatment. In this part of the study, no rejection is

permitted and it is the first mi nimu m distance met that

is taken in to consideration.

3.3. Results analysis

We note a two-fold reduction in the error as soon as

a smoothing is performed. In addition, up to a com-

pression step of 6, the width of the win dow does not

affect the result, while above this the performance is

slightly lower when the width diminishes. Except for

the integration filter, the performance seems to reach



4 18 J .- P. D R O U H A R D et al

a m a x i m u m a t t h i s c o m p r e s s i o n v a l u e . T h e s i m p l e

u n d e r s a m p l i n g , a t t h e o t h e r h a n d , r e d u c e s t h e e r r o r

l e ss a n d e v e n a f t e r a c o m p r e s s i o n s t e p o f 6 t h e e r r o r

i n c r ea s e s. C o n s e q u e n t l y , w e a l r e a d y k n o w t h a t s i m p l e

u n d e r s a m p l i n g is n o t a g o o d p r e - t r e a t m e n t t o a p p l y t o

t h e f u l l d i r e c t i o n a l P D F . F i n a l l y , i n a l l c a se s t h e s t a n -

d a rd d e v ia t io n is l o w e n o u g h (b e tw e e n 0 . 2 a n d 0 .3 5) s o

t h a t w e m a y c o n c l u d e t h a t t h e c l a s s if i c a t io n r e s u lt s a r e

s t a b l e f o r e ac h p r e - t r e a t m e n t s t u d i e d .

F r o m o u r s t a t i s ti c a l a n a l y s i s w e r e t a i n e d t h e i n t e -

g ra t io n f i l t e r w i th a c o m p re s s io n s t e p o f 1 0 , w h ic h

c o r r e s p o n d s v e r y w e l l t o o u r h e u r i st i c . C o n s e q u e n t l y ,

b y a p p l y i n g th i s p r e - t r e a t m e n t o n t h e f u ll d i r e c t io n a l

P D F , w e i n c re a s e i t s d i s c r i m i n a t i n g p o w e r r o u g h l y

tw o fo ld (e t d ro p s f ro m 5 .9 4 to 2 . 6 9 ) a n d , o v e ra l l , w e

re d u c e th e in p u t v e c to r d im e n s io n f ro m 1 8 0 to 1 8. T h i s

i s v e ry im p o r t a n t s in c e i t w i l l g re a t ly f a c i l i t a t e t h e

d e s i g n a n d t r a i n i n g o f t h e B P N c l as s if i e r.

4 . B AC K PR OPA GAT ION NE T WOR K C L ASSIFIE R

4.1.

Backpropagation network BP N )

N e u r a l n e t w o r k s p r e s en t a c o m p u t a t i o n a l p a r a d i g m

f o r c o n s t r u c t i n g c l a s s if i e rs t h a t c a n p e r f o r m a s a c c u -

ra t e ly a s c o n v e n t io n a l t e c h n iq u e s , ca) F o r th e f i r s t st a g e

o f th e A H S V S w e u s e d a c o m p l e t e l y c o n n e c t e d f e e d-

f o r w a r d n e u r a l n e t w o r k w i t h t h e c l a s s ic a l b a c k p r o p a -

g a t i o n l e a r n i n g a l g o r i t h m , 9) m o r e s i m p l y k n o w n a s

t h e B a c k p r o p a g a t i o n N e t w o r k ( B P N ) w h i c h i s d e -

s c r i b e d i n d e t a i l i n m a n y t e x t b o o k s , l° - t 3 ) T o b u i l d

a B P N , t h e r e a r e m a n y p a r a m e t e r s t o c h o o s e f r o m

d e a l i n g w i t h t h e n e t w o r k s i ze o r t h e l e a r n i n g l a w .

U n f o r t u n a t e l y , t h e r e i s n o w a y t o d e t e r m i n e t h e m

r i g o r o u s l y s i n ce t h e y a r e s t r o n g l y d e p e n d e n t o n t h e

a p p l i c a t i o n . T h e f i r s t is t h e n u m b e r o f h i d d e n l a y e r s ,

w h ic h h a s b e e n s e t t l e d to o n e (1 ) s in c e m a n y

a u th o r s l a° -1 3 ) c o n s id e r t h a t a s in g le h id d e n l a y e r i s

s u f fi c ie n t f o r m o s t a p p l i c a t i o n s . T h e n u m b e r o f n e u -

r o n s o n t h e i n p u t l a y e r (N i ) i s 1 8, w h i c h c o r r e s p o n d s t o

t h e d i m e n s i o n o f t h e v e c t o r

F Oi)

f t e r a p p l y i n g t h e b e s t

p r e - t r e a t m e n t f o u n d i n S e c t i o n 3 . 3 . T h e n u m b e r o f

n e u r o n s o f t h e o u t p u t l a y e r ( N o) is t w o , s i n ce w e h a v e

tw o c l a s s e s (to 1 a n d ~ o2 ) a n d w e w a n t to u s e v a r io u s

r e j e c t io n m e t h o d s . I t i s n o t s o e a s y t o f i n d t h e n u m b e r

o f n e u r o n s o n t h e h i d d e n l a y e r ( Nh ) w h o s e u p p e r l i m i t

i s t h e o re t i c a l ly 2 N i + 1 . 12) U p t o n o w , o n ly ru l e s o f

t h u m b h a v e b e e n p r o p o s e d t o d e t e r m i n e N h, a n d w e

h a v e a r b i t r a r i l y s e t t l e d N h = 1 2, a n u m b e r i n c l u d e d

b e t w e en t h e m a x i m u m a n d t h e m i n i m u m p r o p o s e d b y

t h e v a r i o u s m e t h o d s . L a t e r , w e h a v e a d j u s t e d t h i s

n u m b e r b y e x a m i n i n g i t s i nf l u e n ce o n t h e g l o b a l p e r -

f o r m a n c e o f t h e A H S V S ( s e e S e c t i o n 4.4 ). C o n c e r n i n g

t h e l e a r n i n g la w , t h e r e a r e t w o p a r a m e t e r s t o c h o o s e :

t h e l e ar n i n g ra t e r / a n d t h e s m o o t h i n g r a t e o r m o m e n -

tu m 0t. A g a in , t h e re i s n o w a y to f in d a r ig o ro u s v a lu e

f o r t h e s e p a r a m e t e r s . M o r e o v e r , t h e e m p i r i c a l r u l es

p r o p o s e d a r e o f t e n c o n t r a d i c t o r y . C o n s e q u e n t l y , a f t e r

a f e w p r e l i m i n a r y t r i al s , w e a r b i t r a r i l y d e c i d e d t o s e t tl e

= 0 . 6 a n d ct = 0 .0 . In o rd e r to f a c i l i t a t e t h e s t a r t o f t h e

t r a i n i n g p h a s e , t h e w e i g h t s s h o u l d b e i n i t i a l i z e d t o

s m a l l r a n d o m v a lu e s , . 3' 14 ) a n d th e b i a s t e rm s h o u ld

b e u s e d t o a v o i d a s a t u r a t i o n o f th e o u t p u t o f t h e

n e u ro n s 3 s '1 3 ) W e th e re fo re u s e d a b i a s t e rm fo r t h e

h i d d e n a n d o u t p u t l a y e r s a n d a l l w e i g h t s w e re i n -

i t i a l iz e d r a n d o m l y i n t h e - 0 . 1 t o 0 .1 r a n g e . O n e l a s t

w a y t o i m p r o v e t h e c o n v e r g e n c e t i m e o f t h e B P N

d u r i n g t h e t r a i n i n g p h a s e i s t o n o r m a l i z e t h e i n p u t

v e c to r s b e tw e e n 0 a n d 1 . ~ 5) T h i s i s p a r t i c u la r ly t ru e

w h e n th e d a ta v a lu e s a re v e ry s im i l a r . S in c e th i s i s s o in

o u r c a s e , w e h a v e n o r m a l i z e d a l l t r a i n i n g , t e s t a n d

p e r f o r m a n c e s e ts u s e d b y t h e B P N .

W i t h t h e B P N , t h e t r a i n i n g p h a s e i s c r i t ic a l , e s -

p e c ia l ly w h e n th e d a t a to b e c l a s s if i e d a re n o t c l e a r ly

d i s t i n g u i s h a b l e a n d w h e n t h e r e a r e n o t e n o u g h

e x a m p l e s t o c o n d u c t t r a i n i n g . I n t h i s c a se , t h e t r a i n i n g

p h a s e c a n b e v e r y lo n g a n d i t m a y e v e n b e i m p o s s i b l e

t o o b t a i n a n a c c e p t a b l e p e r f o r m a n c e . S i n c e t h i s is t h e

c a s e fo r o u r a p p l i c a t io n (f ew s ig n a tu re s w i th h ig h

v a r i a b i l i t y ) : f i r s t, w e h a v e d e f in e d a c r i t e r io n fo r s to p -

p in g th e t r a in in g p h a s e (S e c t io n 4 .2 ); s e c o n d , w e h a v e

e v a l u a t e d s e v e r a l r e j e c t i o n m e t h o d s t o i m p r o v e t h e

d e c i s io n t a k e n b y th i s t y p e o f c l a s s i fi e r (S e c t io n 4 .3 );

f i n al l y, w e h a v e a d j u s t e d t h e n u m b e r o f n e u r o n s i n

t h e h i d d e n l a y e r o f t h e B P N i n o r d e r t o i n c r e a s e t h e

g l o b a l p e r f o r m a n c e o f t h e f i r s t s t a g e o f t h e A H S V S

(Sec t io n 4 .4) .

4.2. Stopping criterion

I t i s w e ll k n o w n t h a t t h e t r a i n i n g p h a s e i s c r u c i al i n

t h e d e s i g n o f a B P N c l as s if i er . T h e m a j o r d i f f i c u lt y s t o

d e c i d e o n w h a t b a s i s t o s t o p t r a i n i n g . F o r t h e B P N w e

c a n u s e th e e r ro r v a lu e e ( th e d i f f e re n c e b e tw e e n th e

d e s i r e d o u t p u t a n d t h e a c t u a l o u t p u t. ) t o s t o p t r a i n i n g

w h e n t h i s i s l o w e r t h a n a p r e - e s t a b l i s h e d l i m i t ( id e a l l y

0 ) f o r a l l e x a m p l e s i n c l u d e d i n t h e t r a i n i n g s e t. H o w -

e v e r , i t is n o t a lw a y s p o s s ib l e to r e a c h th i s s to p p in g

c r i t e r io n , e sp e c i a l ly w h e n t h e i n p u t d a t a a r e n o t c l e a r l y

s e p a r a t e d . A v a r i a n t o f t h i s m e t h o d , w h i c h t a k e s t h i s

p r o b l e m i n t o a c c o u n t , i s t o s t o p t r a i n i n g w h e n t h e

R o o t M e a n S q u a r e ( R M S ) e r r o r o n t h e t r a i n i n g s e t i s

lo w e r th a n a f ix e d th re s h o ld ( id e a l ly 0 ). T h i s v e ry

p o p u l a r m e t h o d w a s n o t s e l e ct e d b e c a u s e i t i s n o t w e l l

s u i t e d t o o u r a p p l i c a t i o n . In d e e d , i t is n o t a b s o l u t e l y

n e c e s s a r y f o r u s t o h a v e a s m a l l R M S e r r o r t o o b t a i n

a n a c c e p t a b l e p e r f o r m a n c e o f t h e c l a ss i fi e r. I n o t h e r

w o r d s , w e d o n o t w a n t t h e B P N t o l e a r n t o g i ve t h e

e x a c t o u t p u t l e ve l b u t t o m a k e a g o o d d e c i s i o n . T h i s

m a y b e d o n e w e l l b e f o r e t h e R M S e r r o r b e c o m e s w e a k .

C o n s e q u e n t l y , w e h a v e b a s e d o u r s t o p p i n g c r i t e r i o n

o n th e p e r f o rm a n c e m e a s u re e d e f in e d in S e c t io n 2 .6 .

T h u s , t h e n e t w o r k w e i g h t s a r e a d j u s t e d f o r e a c h

e x a m p l e o f t h e t r a i n i n g s e t a n d o n c e a l l e x a m p l e s h a v e

b e e n p r e s e n t e d t o t h e n e t w o r k ( l a t e r r e f e r r e d t o a s

a p re s e n ta t io n ) , w e f r e e z e th e w e ig h t s a n d w e e v a lu a te

e o n th i s t r a in in g s e t ( l a t e r r e fe r r e d to a s etm . In th i s

w a y , w e m e a s u r e t h e m e m o r i z a t i o n p e r f o r m a n c e o f th e

B P N c l as s if i er . H o w e v e r , t h e f i r s t o b j e c t iv e o f th e B P N

c l a ss i f ie r i s t o h a v e a g o o d g e n e r a l i z a t o n p e r f o r m a n c e ,




v

a

i

. Q

O

r -

ig

I M e a n

11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M e a n + 1- S D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

/

8 . . . . . . . . . t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

n

7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . l . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . n e r a z a ii o n . . . . . . . . . .. . . . .

2

0 . . . .. . . .. . . . .. . . . .. . . .. . . . .. . . . .. . . .. . . . .. . . . .. . . . .. . . .. . . . .. . . . .. . . .. . . . .. . . . Y . . . .

I I I I t I I I I I I I I I I I i I I I i I ]

0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0 9 0 0 1 0 0 0

N u m b e r o f p r e s e n t a t i o n s ( N B P )

Fig . 2 . M ean g loba l er ror i n m emo r i za t i on e tm) and g enera l i za t i on e ta) as a func t i on o f the numb er o f

presenta t i ons NBP) . T he bo ld l i ne shows the mean va lue and the fi ne li nes on both s ides show the s tanda rd

dev ia t i on SD) .

that is, on examples not seen duri ng the train ing phase.

It is thus logical to stop traini ng on the ge neralization

perfo rmance measure of the BPN classifier. To do this,

we evaluate e on the test set (late r referred to as etg).

The drawback of the BPN, which is known as the

overtraining phenomenon ,~12) is that we must stop

training when

8 t g

goes through a global min imum. We

have not retained this criterion since preliminary re-

suits showed that for our data et, was quite noisy and

the curve was relatively fiat after a few hundre d presen-

tations, and that the overtraining phenom enon was

often imperceptible even after man y tho usand s of pres-

entations. Und er these conditions , it would be best to

stop training as soon as 8tm and etg are both almost

stable (i.e. when the slope of the curve is very weak).

However, this method needs a great deal of compu ting

time and is very unstabl e when the data are noisy. For

these reasons, we decided to stop training when ~,g is

stable for all 20 writers. This consists of finding the

num ber of pres entations (NBP = Tp) after which the

generalizatio n performance for each writer is not sig-

nificantly mproved with l onger training. Nevertheless,

in some cases the training phase could be greatly

reduced if we stopped it when the memorization and

generalization performances are acceptable, that is,

when etm < T m and etg < Tg. The thresholds T m and Tg

must be settled to 0, since we do no t wan t to impose

a limit on the BPN classifier's performance in either

memori zatio n or in generaliza tion. Tp, the maxim um

numb er of presentations used in the train ing phase, is

determined by means of an experimental protocol.

Figure 2 illustrates the results found with this experi-

ment al protocol. Since the ex pone ntia l etg curve is very

noisy, we smooth ed it before calculating its slope by

curve-fitting . We have tried the two following slope

values 10- a and 10 _4 and the resulting Tp values were

250 and 650, respectively. As we can see in Fig. 2, the

first value is quite low and the second is a little too

high. Thus, we have set Tp at 500 to o bta in a sufficient-

ly stable curve without excessive co mput atio n time. In

conclusion, the st opping criterion used in this study is

stated as follows: the tra ining phase will be stopped

when

gtm=Tm=0) a n d

/ ; t g : T g = 0 ) ) o r when

(NBP > Tp = 500) .

4.3.

ejection criterion

For systems not requiring an immediate decision,

the addit ion of a rejection criterion to the decision rule

allows significant improvement in classifier perform-

ance by re fusing to classify doubtful cases, t16'17) How-

ever, even if the cost of a rejection is lower tha n tha t of

an error, 16~ the rejection rate mu st be as weak as

possible for two reasons. The first is that we can also

reject good decisions and if the rejection rate of good

decisions becomes higher than that of bad decisions,

the classifier's performance will be decreased ins tead of

increased. The second con cerns classifier utility. In-

deed, in an extreme case the classifier can have an error

rate of 0% but a re jection rate of 100%. In this case, the






p e r f o r m a n c e o f th e B P N c l a s si f i er w h e n t h e t h r e s h o l d

v a l u e s i n cr e a se . T h i s b e t t e r p e r f o r m a n c e , c o r r e s p o n d -

i n g t o a h i g h e r r e l i a b i l i t y ( R F ) a n d a l o w e r t o t a l e r r o r

ra t e ( e, ), i s o b ta in e d a t t h e e x p e n s e o f a t o t a l r e j e c t io n

ra t e (R 0 th a t i s c l e a r ly h ig h e r . A s s e e n in F ig . 3 , u n l ik e

R C 2 a n d R C 3 , R C 1 b e c o m e s e f f e ct iv e o n l y w h e n t h e

th re s h o ld T M i s h ig h e r th a n 0 . 5 . F ro m F ig . 3 w e c a n

a l s o c o n c l u d e t h a t R C 1 , R C 2 a n d R C 3 a r e e q u i v a le n t

w h e n a p p l i e d t o o u r d a t a . T h e r e s u l ts d e a l i n g w i t h

R C 5 a n d R C 6 s i m p l y s h o w t h a t t h e a d d i t i o n o f R C 2 o r

R C 3 to R C 1 i s o n ly e ff e c tiv e w h e n th e th re s h o ld T A o r

T R~ i s h ig h e r t h a n a v a lu e fo r w h ic h i t a l lo w s a b e t t e r

p e r f o r m a n c e t h a n t h a t a l r e a d y o b t a i n e d w i t h t h e

th re s h o ld TM ( se e, fo r e x a m p le , R C 5 in F ig . 3 ). T h i s

m e a n s t h a t b e l o w t h i s p a r t i c u l a r t h r e s h o l d T A o r T R y,

t h e r e j e c ti o n c r i t e r i a R C 1 a n d R C 2 o r R C 3 r e j ec t

e x a c t ly th e s a m e e x a m p le s . A l l t h e s e r e s u l t s c a n e a s i ly

b e e x p l a i n e d b y t h e f a c t t h a t o u r o u t p u t l e v el s a r e

c o m p l e m e n t a r y .

I n c o n c l u s i o n , w e c a n s a y t h a t , d u e t o t h e o u t p u t

c o m p l e m e n t a r i t y , t h e r e j e c ti o n c ri t e r i a R C 1 , R C 2 a n d

R C 3 a r e s t r i c t l y e q u i v a l e n t a n d t h e r e j e c t i o n c r i te r i a

R C 5 a n d R t ~ 6 a r e o n l y u s e fu l w h e n w e w a n t t o i m p o s e

a l o w e r l i m i t o n t h e B P N c l a ss i f ie r p e r f o r m a n c e v i a

R C 1 . C o n s e q u e n t l y , t h e r e j e c t io n c r i t e r io n R C 1 h a s

b e e n c h o s e n o n ly b e c a u s e o f it s p ra c t i c a l a s p e c t ( l e ss

c o m p u t a t i o n t i m e ) . W e n o w h a v e t o d e t e r m i n e t h e

t h r e s h o l d T M , b u t t h i s d e t e r m i n a t i o n m u s t b e m a d e

w i t h c a u t i o n . A c a r e f u l e x a m i n a t i o n o f o u r r e s u lt s

p r o v e s t h a t t h e a p p l i c a t i o n o f a r e j e c t i o n cr i t e r i o n m a y

d e g r a d e B P N c l a ss i fi e r p e r f o r m a n c e a n d a l s o i n d i c at e s

t h a t t h e t h r e s h o l d v a l u e m u s t b e h i g h en o u g h , b u t n o t

t o o h i g h . A s a l r e a d y m e n t i o n e d , B P N c l a ss i f ie r p e r -

f o r m a n c e i n c r e a se s a t t h e e x p e n s e o f a n i n c r e a s e i n t h e

r e j e c ti o n r a t e . C o n s e q u e n t l y , w e c h o s e t h e t h r e s h o l d

T M a c c o r d i n g t o t h e f o l l o w i n g h e u ri s t ic : o b t a i n t h e

b e s t p e r f o r m a n c e w i t h a n a c c e p t a b l e r e j e c t i o n r a t e .

F o r o u r A H S V S a p p l i c a t i o n a r e j e ct i o n ra t e o f 5 %

c o u l d b e c o n s i d e r e d a s a c c e p t a b l e . I n t h i s c a s e f r o m

F ig . 3 w e f in d a v a lu e o f 0 . 9 4 fo r t h e th re s h o ld T u . In

a d d i t i o n , o u r r e s u l t s s h o w t h a t t h e i n t r o d u c t i o n o f

a r e j e c ti o n c r i t e r i o n i m p r o v e s t h e r e l i a b i l i ty ( R F ) o f t h e

A H S V S s l i g h t ly , b u t s h a r p l y d e c r e a s e s it s t o t a l e r r o r

ra t e ( e , ) v i a th e ty p e I e r ro r r a t e ( e l ) .

4.4. Backpropagation network BPN) optimization

E s s e n t i al l y , t h e r e a r e t w o w a y s t o o p t i m i z e a B P N :

(1) w e c a n a d ju s t t h e l e a rn i n g a lg o r i th m l s -2 ~ ) a n d (2)

w e c a n m o d i fy i t s s t ru c tu r e b e fo re z 2-25 ~ o r d u r -

i n g 26 -2 8) th e t r a i n i n g p h a s e . W e o n l y p e r f o r m e d t h e

B P N o p t i m i z a t i o n o n t h e s t r u c t u r a l a s p e c t o f t h e

n e t w o r k . A t t h e t i m e t h e B P N w a s d e f in e d , w e m e n -

t i o n e d t he i m p o r t a n c e o n B P N p e r f o r m a n c e o f t h e

h id d e n n e u ro n s ( s e e S e c t io n 4 .2 ). U n fo r t u n a te ly , t h e re

a r e n o t h e o r e t i c a l m e a n s f o r f in d i n g th e o p t i m u m

n u m b e r o f h id d e n n e u r o n s ( N h o) . H o w e v e r , w e k n o w

i t s u p p e r l im i t , w h ic h i s

N h m =

2 N i + 1~2~ = 37, in our

c a s e . C o n s e q u e n t ly ,

Nho

m u s t b e f o u n d b y e x p e r i m e n -

t a t i o n i n w h i c h N hv a r i e s g r a d u a l l y b e t w e e n 0 a n d Nhm

T h e r e s u l ts o f th i s e x p e r i m e n t a t i o n a r e s h o w n

g ra p h ic a l ly in F ig . 4. F ro m th e s e r e s u l t s , w e c a n s e e

t h a t t h e B P N p e r f o r m a n c e s c l e a r l y i m p r o v e a s s o o n a s

w e h a v e a h i d d e n l a y e r a n d t h a t , f o r o u r d a t a b a s e , t h e

e ff ec t o f t he n u m b e r o f h i d d e n n e u r o n s i s n o t v e r y

p r o n o u n c e d . A s t a t i s t i c a l a n a l y s i s s h o w s t h a t t h e c o n -

f i g u r a t i o n w i t h e i g h t h i d d e n n e u r o n s i s t h e b e s t b e -

c a u s e i t h a s a l o w e r e r ro r r a t e (~ t= 1 -2 4%) a n d

re j e c t io n r a t e (R = 4 . 5 0 6 %) a n d a h ig h e r r e l i a b i l i t y

fa c to r (R F = 0 .9 8 7) . T h u s , t h e e ig h t -h id d e n -n e u ro n

c o n f i g u r a t i o n w a s u s e d f o r t h e c o m p a r a t i v e s t u d y

d e s c r ib e d in th e d i s c u s s io n .

5 D I S C U S S I O N

I n o r d e r t o e v a l u a t e t h e b e h a v i o u r o f t h e B P N

c l a ss i f ie r m o r e a c c u r a t e l y , w e c a r r i e d o u t a c o m p a r a -

t iv e s tu d y o f t h e k N N c la s s if i e r , t h e T c l a s s i f i e r a n d th e

B P N c l a ss i f ie r o n t h e s a m e d a t a s e t s. T h e k N N a n d

B P N c la s s if i e r s u s e a n e w d e f in i t i o n o f t h e d a ta s e t s , a s

def ined in Sec t ions 2 .2 , 2 .4 and 2 .5 . There fore , the re is

n o c o m p a r i s o n b e t w e e n p e r f o r m a n c e s o b t a i n e d n o w

i n t h i s c o m p a r a t i v e s t u d y a n d t h o s e o b t a i n e d p r e v i -

o u s ly in S e c t io n 3 fo r t h e k N N c la s s i f ie r a n d in S e c t io n

4 fo r t h e B P N c la s si f i e r. T h e T c l a s s i f i e r o n th e o th e r

h a n d , e x c e p t f o r t h e v a l i d a t i o n s e t w h i c h i s r i g o r o u s l y

th e s a m e fo r a l l c l a s s if i e r s , u s e s th e fo l lo w in g r e fe re n c e

a n d t e s t s e ts . T o b u i ld th e T r e fe re n c e se t , w e s e l e c t a t

r a n d o m a f ew e x a m p l e s f r o m t h e f i rs t 20 g e n u i n e

s i g n a t u r e s a n d a p p l y a r o t a t i o n a s i n S e c t io n 2 .2 . F r r

t h i s st u d y , t h e n u m b e r o f e x a m p l e s b e f o r e r o t a t i o n f o r

th e T r e fe re n c e se t w a s v a r i e d b e tw e e n 1 a n d 1 0 w i th a n

in c re m e n t o f 1 , t h u s fo rm in g 1 0 T c l a s s i f i e rs . A s fo r t h e

T t e s t s e t , w h ic h i s u s e d o n ly to f in d th e th re s h o ld

va lue , c las s ~o1 c o m p r i s e s t h e r e m a i n i n g e x a m p l e s o f

th e f i r st 2 0 g e n u in e s ig n a tu re s a n d c l a s s ~ 2 c o m p r i s e s

f iv e r a n d o m e x a m p le s o f t h e f i r s t 2 0 s ig n a tu re s o f t h e

1 9 o th e r i n d iv id u a l s . C o n s e q u e n t ly , c l a s s ~ o2 a l w a y s

h a s th e s a m e d im e n s io n (9 5 ), b u t c l a s s c o1 h a s a d i m e n -

s i o n t h a t v a r i e s a c c o r d i n g t o t h e n u m b e r o f e x a m p l e s

a l r e a d y p ic k e d u p fo r t h e T r e fe re n c e s e t.

B e f o re d o i n g t h e c o m p a r a t i v e s t u d y , w e h a v e t o f i n d

th e b e s t T c l a s s i f ie r a m o n g th e 1 0 w e h a v e s im u la t e d .

B e tw e e n th re e a n d s e v e n e x a m p le s , t h e d i f f e ren c e i s n o t

v e r y p r o n o u n c e d a n d w e h a v e c h o se n t h a t w h i c h g i v e s

t h e b e s t p e r f o r m a n c e , t h a t i s, fi v e e x a m p l e s t o b u i l d t h e

T re fe re n c e se t . N o w , i t i s t h i s b e s t T c l a s s i f i e r t h a t w i l l

b e u s e d f o r t h e c o m p a r i s o n . A s e x p e c t e d , t h e B P N

c la s s i f i e r g iv e s a r e s u l t ( et = 3 . 2 2 %) th a t l i e s b e tw e e n

th e b e s t g iv e n b y th e k N N c la s s i f ie r ( et = 1 . 6 8 %) a n d

th e w o rs t g iv e n b y th e T c l a s s i f ie r ( et = 5 . 6 1 %) . T h i s i s

a l s o t ru e fo r t h e s t a b i l i t y o f t h e d e c i s io n w h ic h i s

r e f l ec t e d i n t h e s t a n d a r d d e v i a t i o n . O u r r e s u l ts c l e a r l y

s h o w t h a t a l l cl a ss i f ie r s a r e m o r e u n s t a b l e b e t w e e n

w r i t e r s t h a n b e t w e e n e x p e r i m e n t s . T h is h i g h s t a n d a r d

d e v i a t io n v a lu e in d ic a t e s th e d i f f i c u l ty th a t t h e c l a s s i -

f i e r s h a v e in id e n t i fy in g s o m e w r i t e r s . In th e tw o c a s e s ,

t h e B P N c l a ss i fi e r i s c l o s e r t o t h e k N N c l a s si f i er t h a n

to the T c lass i f ie r .



4 J .-P . D R O U H A R D e t a l

10

i th reject ion

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . 0 0

. . . . . . . . . . . . . . . . . . . .

• RF 0 96

. . . . . . . . . . . . . . . . . . ' . . . . . . . . . . . . . . . . . . . . . . .

0 94

I i i i I I i i I I i i 0 . 9 0

0 4 8 12 16 20 24 28 32 36

Nh

u l

10

i thout reject ion

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . 1 . 0 0

~ I I L T I I I

l : / r i 2 1 1

± z

1 9 2

i , I , I , . . . . . • • 1 . 9 0

0 4 8 12 16 20 24 28 32 36

N h

Fig. 4. Effect of the numb er of hidden n u r o n s Nh) on the performance E, R, and R F) of the BP N evaluated

without a rejection criterion or with the rejection criterion RC 1 with a threshold level T , of 0.94.

6. CONCLUSIONS

T h e m a i n o b j e c t i v e o f t h i s w o r k w a s t o d e t e r m i n e

w h e t h e r o r n o t a B P N c l a ss i f ie r c o u l d b e u s e d i n t h e

d e s i g n o f t h e f i r s t s t a g e o f a c o m p l e t e A u t o m a t i c

H a n d w r i t t e n S i g n a t u r e V e r i f i c a t i o n S y s t e m A H S V S ) .

T o d o t h i s, w e h a v e c h o s e n t h e d i r e c t i o n a l P r o b a b i l i t y

D e n s i t y F u n c t i o n P D F ) a s a g l o b a l s h a p e f a ct o r a n d

t h e c o m p l e t e l y c o n n e c t e d f e e d - f o r w a r d n e u r a l n e t -

w o r k w i t h t h e c l a s si c a l b a c k p r o p a g a t i o n l e a r n i n g a l -

g o r i t h m r e f e r re d t o a s t h e B a c k p r o p a g a t i o n N e t w o r k

B P N ) .

H o w e v e r , th e d i m e n s i o n 1 80 ) o f t h e P D F i s t o o

l a r g e t o b e p r o p e r l y m a n i p u l a t e d b y a B P N c l as s if i er .

C o n s e q u e n t l y , i n a f ir s t a t t e m p t a n d b y m e a n s o f th e

k N e a r e s t N e i g h b o u r k N N ) c l as s if i e r, w e h a v e d e te r -

m i n e d t h e p r e - t r e a t m e n t t h a t i m p r o v e s t h e c l a s si f ic a -

t i o n w h i l e d e c r e a s i n g t h e d i m e n s i o n o f t h e i n p u t

v e c to r . T h e r e s u l t s i n S e c t io n 3 s h o w th a t t h e p re -

t r e a t m e n t w i t h a n i n t e g r a t i o n f i l te r w i th a s t e p o f t en

g a v e th e b e s t r e s u l t .

I n t h e c a s e o f t h e B P N c l a ss i fi e r, t h e t r a i n i n g p h a s e i s

c ru c ia l a n d d i f f i c u l t t o c o n t ro l . T h e m a jo r d i f f i c u l ty i s

to d e c id e w h e n to s to p t r a in in g . In S e c t io n 4 . 2 , w e

d e f i n e d a s t o p p i n g c r i t e r i o n b a s e d o n a m e a s u r e o f t h e

p e r f o r m a n c e o f t h e B P N c l a ss i f ie r b o t h i n m e m o r -

i z a t io n e m ) a n d in g e n e ra l i z a t io n e t, ), a s w e l l a s o n

a m a x i m u m n u m b e r o f p re s e n t a ti o n s N B P ) o f t h e

t r a i n i n g s e t. U s i n g a n e x p e r i m e n t a l p r o t o c o l , w e d e -

c i d e d t h a t t h e t r a i n i n g p h a s e w o u l d b e s t o p p e d w h e n



Off-line signature verif icat ion 423

(e tm = 0 a n d e ts = 0 ) o r w h e n ( N B P > 5 0 0) . S t a t i s t i c a l

a n a l y s i s c a r r i e d o u t l a t e r , d u r i n g t h e e v a l u a t i o n o f t h e

p e r f o r m a n c e o f t h e B P N , s h o w e d t h a t t h e m a j o r i t y

( 8 1 ) o f t h e n e t w o r k s s t o p p e d t h e i r tr a i n i n g o n t h e

s e c o n d c o n d i t i o n a n d 1 4 . 2 o f t h e m s t o p p e d th e i r

t r a i n i n g o n t h e f ir s t c o n d i t i o n w i t h f ew e r t h a n 1 0 0

p r e s e n t a t i o n s .

I n o r d e r t o i m p r o v e t h e r e l ia b i l it y o f th e B P N

c l a ss i fi e r, w e h a v e i n t r o d u c e d a r e j e c t i o n c r i t e r i o n i n

t h e d e c i s i o n r u l e . I n S e c t i o n 4 .3 , w e s h o w e d t h a t i n a l l

c a s e s t h e a d d i t i o n o f a r e j e c t io n c r i t e r i o n s h a r p l y

d e c r e a s e d t h e t o t a l e r r o r r a t e a n d s l i g h t l y i n c r e a s e d t h e

r e l i a b i l i t y a t t h e e x p e n s e o f a r e j e c t i o n r a t e t h a t i s

s o m e t i m e s h i g h . D u e t o t h e f a c t t h a t o u r d a t a a r e

c o m p l e m e n t a r y , al l t h e r e j e c t i o n c r i t e r ia p r o p o s e d a r e

s i m i l a r . T h u s , w e h a v e c h o s e n t h e e a s i e s t o n e ( R C 1 )

w i t h a t h r e s h o l d ( T u --- 0 . 9 4 ) g i v i n g a n a c c e p t a b l e r e j e c -

t i o n ra t e ( 5 ) f o r o u r a p p l i c a t i o n .

I1: s w e ll k n o w n t h a t t h e n u m b e r o f n e u r o n s o n t h e

h i d d e n l a y e r p l a y s a n i m p o r t a n t r o l e o n B P N p e r f o r m -

a n c e . I n S e c t i o n 4 .4 , w e e v a l u a t e d t h i s r o l e b y m e a s u r -

i n g B P N p e r f o r m a n c e w i t h a n d w i t h o u t t h e r e j e c ti o n

c r i t e r i o n p r e v i o u s l y d e f i n e d w h e n t h e n u m b e r o f

h i d d e n n e u r o n s i s v a r i e d . I n o u r c a s e , t h e a d d i t i o n o r

n o t o f th e r e j e c t i o n c r i t e r i o n d o e s n o t s i g n i f i c a n t l y

a f fe c t t h e n u m b e r o f h i d d e n n e u r o n s t h a t g i v e s t h e b e s t

p e r f o r m a n c e . A s t a t i s t i c a l a n a l y s i s o n o u r d a t a h a s

s h o w n t h a t t h e b e s t c o n f i g u r a t i o n w a s w i t h e i g h t

h i d d e n n e u r o n s .

F i n a l l y , i n o r d e r t o b e t t e r a s s es s t h e p o t e n t i a l o f t h e

B P N c l a ss i fi e r, w e m a d e a c o m p a r a t i v e s t u d y o f t h is

o n e ', t h e k N N c l as s if i er , w h i c h s h o u l d g i v e a n u p p e r

l i m i t f or t h e p e r f o r m a n c e , a n d t h e T c l a ss i fi e r, w h i c h i s

v e r y p o p u l a r i n s p i t e o f i t s f ai l in g s . S i n ce t h e s e l a t t e r d o

n o t u s e a r e j e c t i o n c r i t e r i o n i n m a k i n g t h e i r d e c i s i o n ,

w e h a v e u s e d , in S e c t i o n 5 , a B P N c l a ss i fi e r w i t h o u t t h e

r e j e c t i o n c r i t e r i o n p r e v i o u s l y d e f i n e d . I n e f f e c t , t h e

B P N c l a ss i fi e r b e h a v e s m u c h b e t t e r t h a n t h e T c l a s si -

f ie r a n d a l i t t le le s s w e ll t h a n t h e k N N c l a s s i f i e r .

I n c o n c l u s i o n , w e c a n s a y t h a t , o n c e t r a i n e d , t h e

B P N c la s si fi e r c o m p a r e s f a v o u r a b l y w i t h t h e k N N

c l a ss i fi e r s i n c e i t h a s a l m o s t t h e s a m e p e r f o r m a n c e b u t

w i t h a s h o r t e r r e s p o n s e t i m e a t g e n e r a l i z a t i o n , e s -

p e c ia l ly i f w e u s e a h a r d w a r e i m p l e m e n t a t i o n o f t h e

B P N c l as s if ie r . H o w e v e r , f o r o u r a p p l i c a t i o n t h e d r a w -

b a c k o f t h e B P N c l a ss i fi e r i s t h e w a y i n w h i c h i t d e a l s

w i t h t h e t r a i n i n g p h a s e a n d t h is c o u l d u n d e r m i n e t hi s

c o n c l u s i o n s o m e w h a t .

A c k n o w l e d g e m e n t s - - T h i s work was suppor ted in pa r t by

a PSIR gran t f rom the Eco le de t echno log ie supr r ieu re to

Jean-P ie r re Drouhard and Rober t Sabour in , and by g ran t

O G P 0 1 0 6 4 5 6 t o R o b e r t S a b o u r i n f ro m t h e N S E R C o f C a n a -

da . Mas te r ' s s tuden t M ar io Go dbou t , who took pa r t in th i s

research project , a lso received a scholarsh ip from the E cole de

technologic sup~rieure.

R E F E R E N E S

1 . R . Sabour in , Une Approache de Type Comp rrhens ion de

Scrne Appl iqure au Prob l rm e de l a Vr r i f i ca tion Auto-

mat ique de l ' Iden t i t6 pa r l ' Imag e de l a S igna tu re Ma nus-

crite . Ph.D. Thesis , l~cole Polytec hniqu e de M ontr ral

(1990).

2. R. Sabou rin, R. Pla m on do n an d G. Loret te , Off- line

identif icat ion with ha ndw rit ten signa ture images: survey

and perspectives,

Structured Document Image Analysis.

H. S. Baird, H. B unke an d K. Y ama mo to, eds., pp. 219

234, Springer-Verlag, Ge rm any (1992).

3. W. R. Harrison,

Suspect Docum ents Their Scientific

Examination.

Nelso n-Ha ll Publishers , Ch icago ( 1981).

4 . G . Pou l in , Pe rsona l Com mun ica ton , Labora to i re Jud i -

caire de la GRC, 4229 boul . Dorchester guest , Wes-

tmount , QC, H3 Z 2T4 .

5 . R . Sabour in , J. -P . Drou hard , L . Gagn6 and N. Paque t ,

Autom at ic handw r i t t en s igna tu re ve r i f ica t ion us ing d i -

rect ional PDF and neural networks: a feasibi l i ty s tudy,

Proc. Conf. Exhibit. lndust. Automat.

Montr ra l , 20 . 17-

20. 20 (1992).

6 . L . Xu , A . Krzyzak and C . Y . Suen , Methods o f combin ing

mu lt iple classifiers and their applicat io ns to hand writ in g

recognit ion,

IEEE Trans. Syst . Man Cybernet.

22, 418

435 (1992).

7 . K . Fuku naga ,

Introduction to Statistical Pattern Recogni-

tion 2nd ed n. Academic Press Inc., New Y ork (1990).

8. D. J. Burr , Experim ents on neu ral net recognit ion of

spoke n and w ri t ten text ,

I EE E Trans. Acoust. Speech Sig.

Process.

36, 1162-1168 (1988).

9. D. E. Rumelhart , G. E. Hinton and R. J . Will iams,

Learn ing in te rna l r ep resen ta tion by e r ro r p ropaga t ion , in

Parallel Distributed Processing

D. E . Rum elhar t and J . L .

McC lelland, eds, Vol. 1 , pp. 318 362. M IT Press, M ass-

achussetts (1986).

10. R . P . L ippm ann , An in t roduc t ion to com put ing wi th

neura l ne t s ,

I E E E A S S P M a g .

4-22 (April 1987).

11. P. D. Wasserman,

Neural Computing: Theory and Prac-

tice. Van N os t rand Re inho ld , New York (1989).

12. R. Hecht-Nielsen,

Neurocomputing.

Addison-Wesley,

New York (1990).

13. J . A. Freeman and D. M. Skapura,

Neural Networks:

Algorithms Applications and Program ming Techniques.

Addison-Wesley, New York (1991).

14. J . Hertz , A Krogh and R. G. Palmer,

Introduction to the

Theory o f Neural Computation. Addison-Wesley, New

York (1991).

15. A . J . M aren , D . Jones and F . F ran k l in , Conf igur ing and

op t imiz ing the ba ck-propaga t ion ne twork ,

Handbook of

Neural Computing Applications.

Academic Press, San-

tiago (1990).

16. M. Nadler and E. P. Smith,

Pattern Recognition Enoi-

neering Joh n W iley & Son, New York (1993).

17. B. Dubuisson,

Diagnostic et reconnaissance des formes.

Hermes, Paris (1990).

18. A .V. Ooyen and B . Ne inhu is , Improv in g he convergence

of the backpropag a t ion a lgor i thm,

Neura l Ne tworks

5,

465-471 (1992).

19. T. P. Vogl, J . K. Mangis, A. K. Rigler , W. T. Zink and

D. L. Alkon, Accelerat ing the convergence of the

back p ropaga t ion method ,

Biol. Cybernet.

59, 257 263

(1988).

20. R. A. Jacobs, Increased rates of convergence throug h

lea rn ing ra te adap ta t ion ,

Neura l Ne tworks

l , 295-307

(1988).

21 . N . Baba , A new approach fo r f ind ing he g loba l min imu m

of error fu nction of neur al networks,

Neural Networks 2

367 373 (1989).

22 . S . J . Marsha l l and R . F . Har r i son , Opt im iza t ion and

t ra in ing o f the feedforward neura l ne tworks by gene t ic

algori thms, 2nd Int. Conf. Artific. Neural Networks 349

39 43 (1991).

23. S. Akaho and S. Amari , On the capacity of three-layer

ne twork ,

Proc. Int . Joint. Conf. Neural Networks

3, 1-6

(1990).

24. E. B. Baum and D. Haussier , What s ize net gives val id

general izat ion?

Neural Comput.

l, 151 160 (1989).



4 24 J .- P. D R O U H A R D

e t al

25. H. Akaike , A new look at th e statist ical mod el identif ica-

tion,

I E E E T r an s A u t o m a t C o n t r o l

19, 716-7 23 1974).

26 . T . Ash , Dynam ic node c rea t ion in backprop aga t ion ne t -

works ,

P r o c I n t J o i n t C o n f N e u r a l N e t w o r k s

2, 623 628

1989).

27 . M . Hag iwara , Nove l backpropag a t ion a lgor i thm fo r re -

duc t ion o f h idden un i t s and acce le ra t ion o f convergence

using artif icial selection,

P r o c I n t J o i n t C o n f N e u r a l

N e t w o r k s

2, 625 630 1990).

28 . O. Fu j i ta , Op t im isa t ion o f the h idden un i t func t ion in

feedfo rward neura l ne tworks ,

N e u r a l N e t w o r k s

5, 755

764 1992).

bout the

A u t h o r - - J E A N - P I E R R E D R O U H A R D r e c e iv e d t h e M . S c. d eg r e e in e le c t ro n i c , e l e ct r ic a l a n d

con t ro l eng ineer ing f rom the U nivers i ty o f Caen , F rance , in 1972 and the M.Sc .A. and Ph .D. degrees in

b iomed ica l eng ineer ing f rom the Eco le Po ly techn iqu e de Mo ntr6a l in 1975 and 1979, re spect ive ly . F rom 1980

to 1989 he was a resea rch assoc ia te fo r the B iomedica l Eng ineer ing Ins t i tu te o f the E co le Po ly techn ique de

Mo ntrea l . In 1989 , he jo ine d the s ta f f o f the l~co le de Tec hno log ie Sup6rieure , Un ivers i t6 du Q uebec ,

Montr6a l , P .Q. , Canada , where he i s cu r ren t ly P ro fesso r in the D6par temen t de G6n ie de la P roduc t ion

Autom at ism. His cu r ren t re sea rch in te res ts a re in the app l ica t ion o f the a r t i f ic ia l in te l l igence techn ics such as

exper t systems , a r t i f ic ia l neura l ne tworks and fuzzy systems in the f ields o f pa t te rn recogn i t ion and com pute r

vision.

bout the

A u t h o r - - R O B E R T S A B O U R I N r e c e i v e d B .i ng ., M . S c. A . an d P h . D . d e g r e es i n e le c t ri c a l

eng ineer ing f rom the l~co le Po ly techn ique de Montr6a l in 1977 , 1980 and 1991 respec t ive ly . In 1977 he jo ined

the physics depar tm en t o f the Univers i t6 de M ontr6a l where he was respons ib le fo r the des ign and

dev e lo pm ent o f sc ien ti fic in s t rumenta t ion fo r the Observ a to i re du M ont M6gan tic . In 1983 , he jo ined the

s ta f f o f the Eco le de Techno log ie Sup~rieure , Un ivers i t~ Qu6bec , M ontr6a l , P .Q , Cana da , w here he i s

cu r ren t ly P ro fesso r in the D6par tem en t de G~ nie de la P rodu c t ion Automatis6e . His resea rch in te res ts a re in

the a reas o f compu te r v is ion , scene unders tand ing , segmenta t ion , s t ruc tu ra l pa t te rn recogn i t ion , neura l

ne two rks and fuzzy systems , charac te r recogn i t ion and s ign a tu re ve r i f ica tion .

bout the

A u t h o r - - M A R I O G O D B O U T r e ce i ve d t h e B A ng . d e g re e in P r o d u c t io n A u t o m a t i s ~ f r o m t h e

l~cole de Techno log ie Sup6r ieu re de Montr6a l in 1992 . He i s cu r ren t ly mak ing a m as te r degree a t tha t same

school. H is researc h intere sts are in the f ield o f artif icial intell igence.

Documents

A NEURAL NETWORK APPROACH TO OFF-LINE SIGNATURE VERIFICATION USING DIRECTIONAL PDF