Novel Approaches Discrimination

8/12/2019 Novel Approaches Discrimination

http://slidepdf.com/reader/full/novel-approaches-discrimination 1/29

Z O R - M e t h o d s a n d M o d e l s o f O p e r a t io n s R e s e a rc h ( 19 9 2) 3 6 :5 1 7 - 5 4 5

N o v e l p p ro a ch es t o t h e D i s cr im i n a t io n P ro b l em

P . M a r c o t t e a n d G . S a v a r d 1

Abstract W e con sider the pro blem of determ ining a hype rplane that separates , as we l l as possible,

two f in i t e se t s o f po in t s i n R . We ana lyze two c r i t e r i a fo r j udg ing the qua l i t y o f a cand ida t e

hype rplane ( i ) the m axim al d is tance of a misclassit ied po int to th e hyperp lane ( i i ) the nu mb er o f

misc lassi fi ed po in t s . I n each case, we inves t iga t e t he com puta t iona l com plex i ty o f t he co r r espond ing

ma them at i ca l p rograms, g ive equ iva l en t f o rmula t ions , suggest so lu t ion a lgor i thms and p resen t

prel im inary nu me r ical results.

Key words Discr iminan t ana lys i s . Quadra t i c p rogramming . Complex i ty . I n t eger p rogramming .

Bi level programming.

Introduct ion

Co n s i d e r r a n d o m s am p l e s X {xi} i~1 an d Y --- J{Y } j ~s f ro m t w o d i s t i n c t p o p u -

l a t i o n s 3 f an d qr i n R (x i e 5 f, y J ~ q/ ) w i t h r e s p ec t iv e d i s t r i b u t i o n fu n c t i o n s F x

a n d F t . G i v e n a r a n d o m l y g e n e ra t e d v e c t o r z o r ig i n a t in g fr o m ei ther p o p u l a t i o n

5 ( o r qr s t a t i s ti c a l l i n ea r d i s c r i m i n an t an a l y s i s u s e s a h y p e r p l an e a s a t o o l t o

d ec i d e w h e t h e r z e &r o r z e qr n am e l y z e X i f z l ie s ab o v e t h e h y p e rp l an e an dz e qr i f z l ie s b e l o w t h e h y p e rp l an e . S t a t is t i c a l d i s c r i m i n an t an a l y s i s i s an

es t ab l i s h ed f i e ld o f s ta t i s t ic a l t h eo ry . I n i ts s i mp l e s t , l in ea r fo rm , i t co n s i s t s i n

d e t e r m i n i n g t h e b e s t h y p e r p l a n e , i . e . t h e h y p e r p l a n e t h a t m a x i m i z e s t h e p r o b -

a b i l i ty o f c o r r e c t l y a s si g n i n g z t o p o p u l a t i o n 5 f o r qr W h e n {x i} i~ i a n d J

a r e r a n d o m s a m p l e s d r a w n f r o m m u l t iv a r i a te n o r m a l p o p u l a t i o n s w i t h re s pe c ti ve

m e a n s / ~ x a n d # r a n d common v a r i a n c e - c o v a r i a n c e m a t r i x 2~, i t c a n b e s h o w n

t h a t t h e o p t i m a l h y p e rp l an e i s g i v en b y t h e fo rm u l a (s ee F i s h e r 1 4 ]):

p t z a = O

R e s e a rc h s u p p o r t e d b y N S E R C g r a n ts 5 7 8 9 a n d 4 64 05 , th e A c a d e m i c R e s e a rc h P r o g r a m o f t h e

De par tm en t o f Na t iona l D efense (Canada) and FC A R gran t 91NC0510 . (Quebec) .

1 P ro fesso r s Dr . Pa t r i ce M arco t t e and Gi l l e s Savard , Drp ar t em en t de M ath+mat iques , Co ll~ge

mi l i t a i re roya l de Sa in t -Jean , Riche la in Qurb ec J0J 1R0 , Cana da .

03 40 - 9422 /92 /6 /51 7- 545 $2.50 9 1992 Phys i ca -Ver lag , H e ide lberg



518

w h e r e

P. M arcotte and G . Savard

P = L ' - l ( # x - # r )

a = - 5 ( x + r ) r - l ( ~ x - ~ r ) - I n

a n d p a n d 1 - p a r e p r i o r p ro b a b i l it i e s a s s ig n e d t o p o p u l a t i o n s Y a n d ~1

respec t ive ly .

I n t h i s p a p e r w e f o ll o w a n a l t o g e t h e r d i f f e re n t a p p r o a c h (se e a ls o C a v a l i e r e t

a l. [3 ] ). W e d o n o t a s s u m e a n y f u n c t i o n a l f o r m f o r t h e d i s t r ib u t i o n s F x a n d F t .A c t u a l l y w e d o n o t e v e n a s s u m e t h e p o i n t s x i a n d y J t o b e r a n d o m . R a t h e r w e

l o o k f o r a d i s c r i m i n a t i n g h y p e r p l a n e t h a t m a x i m i z e s a f u n c t i o n r e p r e s e n t i n g ,

i n a s en s e to b e d e t e rm i n ed , t h e l ev e l o f d i s c r i m i n a t i o n b e t w een t h e t w o s e t s X

and Y.

T h e p a p e r i s d i v i d e d i n t o t w o s y m m e t r i c p a r t s, e a c h d e v o t e d t o a m a t h e m a t i c a l

p r o g r a m m i n g m o d e l o f t h e d i s c r im i n a t i o n p r o bl e m . T h e p a p e r is en t ir e ly d e v o t e d

t o a l g o r i t h m s a n d d o e s n o t a d d r e s s t h e i m p o r t a n t , b u t o f a v e r y d if f e re n t n a t u r e ,

q u e s t io n o f t h e r e le v a n ce o f th e m a t h e m a t i c a l p r o g r a m m i n g a p p r o a c h o v er , sa y ,

t h e s t a ti s ti c a l a p p r o a c h t o t h e d i s c r i m i n a t i o n p r o b l e m .F o r e a c h m o d e l , w e a n a l y z e w o r s t- c a s e c o m p l e x i ty , gi ve r e la t e d f o r m u l a t i o n s ,

p r o p o s e s o l u t i o n a l g o r i t h m s a n d p r e s e n t p r e l i m i n a r y c o m p u t a t i o n a l re s ul ts .

Fir s t Mode l

2 . 1 F o r m u l a t i o n

I n t h i s m o d e l , a s s u m i n g t h a t a s e p a r a t i n g h y p e r p l a n e H e x i s t s , w e s t r i v e t o

m a x i m i z e t h e m i n i m u m d i s t an ce o f an y g i v en p o i n t t o H = {z lp tz a = O, z ~ R }.

I f n o s u c h h y p e r p l a n e e x i st s, w e l o o k f o r a h y p e r p l a n e H s u c h t h a t t h e m a x i m u m

d i s t a n c e o f a n y m i s c la s si fi e d p o i n t t o t h e h y p e r p l a n e H b e m i n i m i z e d . I n b o t h

c a se s, th e h y p e r p l a n e H is d e t e r m i n e d b y s o lv i n g t h e m a t h e m a t i c a l p r o g r a m m i n gpro b le m (see Ca val i e r e t a l . I-3] ).



Novel Approaches to the Discr iminat ion Problem 519

m a x w

s. t . p x i + a > w i e I = {1 . . . , r} (1 )

p y J + a < - w j e J = {1 . . . . , s }

Ilpl122 = 1

w h e r e t h e n o n l i n e a r c o n s t r a i n t IlP ll2 = 1 e n s u r e s t h a t p t x ~ + a a n d p ty j a

r e p r e s e n t t h e e u c l id i a n d i s t an c e f r o m p o i n t s x ~ a n d y J t o t h e h y p e r p l a n e H =

{ p z + a = 0 } . I g n o r i n g t h is c o n s t r a i n t i s t e m p t i n g s i n ce th e r e s u l t i n g r e l a x e dp r o g r a m i s l in e a r ( s ee F r e e d a n d G l o v e r 1 -5 ] 1 6 ], G l o r f e l d a n d G a i t h e r [ 9 ]) .

H o w e v e r it l e a d s t o in c o n s i s t e n c i e s t h a t c a n n o t b e s a t i s fa c t o r i ly r e s o l v e d , i n s p i t e

o f n u m e r o u s a t t e m p t s ( s e e K o e h l e r [ 1 6 ] , R u b i n 1 -2 1] a n d M a r k o w s k i a n d

M a r k o w s k i 1 -20]). A r e c e n t p a p e r a l o n g t h i s li n e, w h i c h r e m o v e s s o m e o f t h e

d i ff ic u lt ie s a s s o c i a t e d w i t h t h e l i n e a r p r o g r a m m i n g f o r m u l a t i o n , i s t h a t o f G l o v e r

[ 1 0 ] w h e r e t h e n o r m a l i z i n g c o n s t r a i n t

- s ~ , p x i + r ~ p y J = 1i ~ l j ~

is su b s t i t u t e d t o t h e r e v e r s e c o n v e x c o n s t r a i n t

Iip21122 ~ 1 .

I f a s e p a r a t i n g h y p e r p l a n e e x i s ts , w i s n o n n e g a t i v e a t t h e o p t i m u m , a n d ( 1) is

e q u i v a le n t t o th e re l a x e d c o n v e x p r o g r a m m i n g p r o b l e m

m a x wp a w

s . t . p~x ~ + a > w

p t y j + a < - - w

Ilp[122 < 1

i e l

j e J

(2 )

w h e r e t h e n o n l i n e a r c o n s t r a i n t i s o b v i o u s l y t ig h t a t th e o p t i m u m . W e r e fe r t o

t h i s c a s e a s t h e c o n v e x c a se . I f n o s e p a r a t i n g h y p e r p l a n e e x is ts , i.e . t h a t t h e

i n t e r s e c t i o n o f t h e c o n v e x h u l l s o f X a n d Y h a s a n o n e m p t y i n t e r i o r , t h e n ( 1 ) is

e q u i v a l e n t t o t h e r e v e rs e c o n v e x p r o g r a m :



520

m a x wp a~w

s. t . p t x i + a > w

p t y j + a < - - w

IlPll 2 _ 1

i e I

j ~ J

P. Marco t te and G . Savard

(3)

o r , a f t e r h a v i n g p e r f o r m e d a n o b v i o u s c h a n g e o f v a r ia b l e :

m i n wp,a,w

s. t . p x i + a > - w i ~ I (4 )

p t y J + a < w j ~ J

I l p l l ~ > 1

w h o s e o p t i m u m w i s s t r i c t l y p o s i t i v e . T h i s l a t te r p r o g r a m b e i n g r e v e r s e c o n v e x

w e w i l l r e fe r t o t h e n o n s e p a r a b l e s i t u a t i o n a s th e r e v e r se c o n v e x c a s e .

W e w i l l a l s o r e f e r t o t h e s t r i c t l y c o n v e x c a s e i n t h e c a s e w h e r e t h e r e e x i s t s a

s t r ic t s e p a r a t i o n h y p e r p l a n e p t z + a - - 0 , i .e . t h a t w i s s t r i c t l y p o s i t i v e i n (2 ). T h e

l i n e a r c a s e w i ll c o r r e s p o n d t o t h a t s i t u a t i o n w h e r e t h e s e t s X a n d Y c a n b e

s e p a r a t e d , b u t n o t s t r ic t l y , i.e . t h a t t h e o p t i m a l v a l u e o f w i n ( 2) i s z e r o . U n d e r

o u r t e r m i n o l o g y , t h e r e i s n o s t r ic t l y r e v e r s e c o n v e x c a se .

2 .2 E q u i v a l e n t F o r m u l a t i o n

W e f ir st s h o w t h a t t h e f ir s t v e r s i o n o f t h e d i s c r i m i n a t i o n p r o b l e m D P 1 i s

e q u i v a l e n t t o s o m e d i s t a n c e p r o b l e m .

T h e o r e m 1 : T h e d i s c r i m i n a ti o n p r o b l e m D P 1 is e q u i v a l e n t t o th e m i n i m u m n o r m

p r o b l e m

m in I ]q l]2q b

s t q t x i b > 1 i ~ I

5 )

qt y J + b <_ - I j ~ J



No vel Approaches to the Discrimination Problem

i n t h e s t r i c t l y c o n v e x c a se , t o t he m a x i m u m n o r m p r o b l e m

m a x Ilqll22q b

s . t . q t x i + b > - 1 i ~ l

q~yJ + b < l j e J

i n t h e r e v er s e c o n v e x c a se , a n d t o th e m a x i m u m n o r m p r o b l e m

m ax I lq ll2q b

s t qtxi 4- b >_ i ~ I

q t yJ + b _ < O j e J

- l _ < q k _ < l k = l . . . . . n

i n t h e l i n e a r c a s e .

521

(6)

( 7 )

Proof.

(a) S t r i c t l y c o n v e x c a s e . F i r s t n o t i c e t h a t a n y o p t i m a l s o l u t i o n t o (2 ) m u s t s a t is f y

p 0 . O t h e r w i s e w e w o u l d h a v e a > w a n d a _< - w , i.e . w = 0 , a n i m p o s s i b i l i t y

s i n c e w > 0 a t t h e o p t i m u m i n t h e s t r i c t ly c o n v e x c a s e . S i m i l a r l y , q 0 a t th e

o p t i m u m o f (5 ).

L e t ( p *, a * , w * ) b e a n o p t i m a l v e c t o r f o r (2 ). T h e n ( q* , b * ) = ( p * / w * , a * / w * ) is

w e l l d e f i n e d f o r ( 5 ) . A s s u m e i t i s n o t o p t i m a l f o r ( 5 ) , a n d l e t ( ~ , / ~ ) b e o p t i m a l

ins tea d . T h e n I1 11 < I Iq*l l a n d ( if , a , i f ) = ( / 1 1 ~ 1 1 , b / l l l l , 1 / 1 1 1 1 ) i s f e a s i b l e f o r ( 2 )

w i t h f f = 1 / 1 1 ~ t l > 1/llq*ll = w / l l p l l = w * ( t h e c o n s t r a i n t I I p l l < 1 i s t i g h t a t

t h e o p t i m u m ) , i n c o n t r a d i c t i o n w i t h t h e o p t i m a l i t y o f w * . H e n c e ( q* , b * ) i s

o p t i m a l f o r (5 ).

N o w l e t ( q *, b * ) b e o p t i m a l f o r (5 ) a n d d e f i n e (p * , a * , w * ) = ( q * / l l q * 1 1 , a / l l q I I ,1 /llq *ll). A g a i n , a s s u m e t h a t ( p * , a * , w * ) i s s u b o p t i m a l , l e t (/~ , a , f ) b e o p t i m a l

w i t h f > w * a n d d e f i n e ( iT , b ) = ( / ~ /f , a / f ) . W e h a v e : 1 1 4 1 1= I l P l l / f = l i f t <

I / w * = IIq * II, i n c o n t r a d i c t i o n w i t h t h e o p t i m a l i t y o f q * .

(b) R e v e r s e c o n v e x c a s e . F i r s t, n o t e t h a t , i n th e r e v e r s e c o n v e x c a s e , p r o b l e m (6 )

is b o u n d e d . O t h e r w i s e l e t (q(k), b(k)) b e a fe a si bl e, u n b o u n d e d s e q u e n ce . W e h a v e :

q k ) t b k ) - - 1

x i + >tl(q k), btkJ)l II(q k), btk))lr - - iI(qtk) , btk~)l

L e t (q *, b * ) b e a n y c o n v e r g e n t s u b s e q u e n c e o f t h e b o u n d e d s e q u e n c e



522

(q(k)/ll(q (k), b(k))ll, b(k)/ll(qtk) b(k))ll) .

W e t h e n h a v e , b y c o n t in u i t y ,

q t x i + b * > 0

P. M arcotte and G. S avard

an d , s i mi l a rl y :

q , t y j + b * <_ 0

an d (q * , b * ) d e t e rmi n es a (n o t n eces s a r i l y s t r i c t ) s ep a ra t i o n h y p e rp l an e , an i m-

p o s s i b i l i t y s i n ce w e a r e i n t h e r ev e r s e co n v ex ca s e . N o w , u s i n g t h e o n e - t o -o n e

re l a t i o n s h i p b e t w een f ea s ib l e s o l u t i o n s t o (3) (w i t h w > 0 ) an d f ea s i b le s o l u t i o n s

t o 6 ) :

(q, b) = ( p / w , a / w ) ,

t h e p r o o f i s s i m i la r t o t h e p r e v i o u s o n e , m o d u l o s i gn r e ve r sa l.(c) L i n e a r c a s e . A ny feas ib le so lu t ion to (7) wi l l p ro v ide a feas ib le so lu t io n , i .e . a

s o l u t i o n w i t h a n o n z e r o v e c t o r p . [ ]

R e m a r k : In t h e s t r i c t l y co n v ex ca s e , t h e d i s c r i m i n a t i o n p ro b l em i s eq u i v a l en t t o

p r o j e c ti n g t h e o r i g i n o n t h e c lo s e d , c o n v e x p o l y h e d r o n

{ q e R n l q t x + b > l V i e I

q t y J + b < - i V j ~ J

fo r s o m e b ~ R} .

T h i s p o l y h e d ro n d o es n o t co n t a i n t h e o r i g i n (s ee f ig u re 1 ).

I n t h e r ev e r s e co n v ex ca se , i t r ed u ces t o f i n d i n g t h e ( ex tr eme) p o i n t o f m ax i m u m

e u c l id i a n n o r m o f th e c lo s ed c o n v e x p o l y h e d r o n

{ q E R n [ q t x i + b > - 1 V i ~

q ty J + b < l V j ~ J

fo r s o m e b ~ R}



P 2

N o v e l A p p r o a c h e s t o t h e D i s c r i m i n a t i o n P r o b l e m 5 23

O

F i g 1 : T h e c o n v e x c a s e

I J

P l

h a v i n g t h e o r i g i n a s a n i n t e r i o r p o i n t ( se e f i g u r e 2 ). I n t h e l i n e a r c a s e , p r o b l e m

(5 ) is in f e a si b le , w h i le p r o b l e m (6 ) is u n b o u n d e d .I t is a l s o p o s s i b l e to r e f o r m u l a t e t h e d i s c r i m i n a t i o n p r o b l e m , i n t h e r e v e r s e

c o n v e x c a s e , a s a b i l ev e l p r o g r a m m i n g p r o b l e m .

Theorem 2: I n t h e r e v e r s e c o n v e x c a s e , (3 ) a n d (6 ) a r e e q u i v a l e n t t o t h e l i n e a r

b i l e v e l p r o g r a m :

~ 1 t . e 2 t u 2m a x c u +

p a

s.t . A p + ae I >_ - e 1

Bp + ae 2 <_ e 2

( u l , u z ) 6 a r g m a x -e l~ u 1 e2 u 2 (8)

s . t . - A t u 1 + B~u 2 = p

e l t u I J e 2 t u 2 = 0

u 1, u 2 >_ 0

w h e r e A = ( x 1 . . . , x ) t, B = ( y l . . . . . y~)r, e 1 = ( 1 . . . . . 1 )r e R r a n d e 2 =

1 . . . . . 1 ) t ~ R s .



P2

F i g 2 : T h e r e v e r se c o n v e x c a s e

5 2 4 P M a r c o t t e a n d G S a v a r d

Pl

Proof The transformation of a concave quadrat ic program into a linear bilevel

program has been shown to be valid in Konno [181 using linear dualityarguments. []

2.3 Com plexity Ana lysis

Theorem 3: The discrimination problem DP1 is in P in the convex (strictly convex

9 r linear) case and strongly NP-complete in the reverse convex case.

Proof (a) In the strictly convex case, the result follows from the existence of

polynomial algorithms for the monotone linear complementari ty problem which

constitutes an extension of the convex quadratic programming problem (see

Kojima et al. [14]).

(b) In the linear case, there must exist a nonzero vector p and a scalar a such

that the set of linear inequalities

p t x i + a > _ O i ~ I

p t y ~ + a < 0 j ~ J

is satisfied. One way to enforce the condition p ~ 0 is simply to impose the



Novel Approaches to the DiscriminationProblem 525

a d d i t i o n a l c o n d i t i o n Pk > 1 or Pk < -- 1) f o r s o m e i n d e x k . S in c e it is u n k n o w n

a priori w h i c h i n d e x k w i ll c o r r e s p o n d t o a n o n z e r o c o e t ti e ie n t P k, o n e h a s t o

s o lv e , in t h e w o r s t c a s e , t h e 2 n l i n ea r f ea s i b i l it y p ro b l em s :

p t x i + a > _ O i ~ I

pty~ + a < 0 j ~ J 9a)

pk > 1

a n d

p t x i + a > O i e I

pty~ + a < 0 j e J 9b)

Pk <-~ - -1

f o r k = 1 . . . , n , a n d p o l y n o m i a l i t y i s a c o n s e q u e n c e o f t h e p o l y n o m i a l i t y o f l i n e a rp r o g r a m m i n g see K h a c h y a n [1 5 ]) .

c ) I n t h e r e v e rs e c o n v e x ca s e, th e s t r o n g N P - c o m p l e t e n e s s o f D P 1 f o l lo w s

f r om t h e s tr o n g N P - c o m p l e t e n e s s o f t h e m a x i m u m n o r m p r o b le m se e F r e u n d

a n d O d i n [ 7 ] o r B o d l a e n d e r e t a l. [ 2 ]) . [ ]

Coro l lary: T h e e x i s te n c e o r n o n - e x i s t e n c e o f a d i s c r i m i n a t i n g h y p e r p l a n e c a n b e

d e t e r m i n e d i n p o l y n o m i a l ti m e .

Proof. F ro m t h eo r em 3 , p a r t b ), t h e r e fo l lo w s t h a t t h e exi s ten ce o f a d i s c r i mi n a t i n g

h y p e r p l a n e c a n b e d e t e r m i n e d b y s o l v in g fo r t h e e x i s te n c e o f a f e as ib l e s o l u t i o n

t o o n e o f 2 n l i n ea r s y s t ems . H en c e t h e r e s u lt . [ ]

2 .4 S o lu t i o n A lg o r i t h m s a n d N u m er i ca l R es u l t s

So l u t i o n a l g o r i t h ms a r e b a s ed o n t h eo rem 1, w h i ch s t a t e s t h e eq u i v a l en ce b e t w een

D P 1 a n d q u a d r a t i c p r o g r a m m i n g . I n t h e c o n v e x c a se i t i s e a s il y s o l v e d a s a

p r o j e c t i o n p r o b l e m f o r w h i c h e ff ic ie n t a l g o r i t h m s e x is t se e W o l f e [ 23 ] o r t h e

s u r v e y b y L i n a n d P a n g [ 1 9 ]) . I n t h e r e v e r se c o n v e x ca s e, w e s o l v e d D P 1 u s i n g

a c o d e i n i ti a ll y d e s i g n e d f o r so l v i n g l i n e a r bi le v e l p r o g r a m m i n g p r o b l e m s se e



526

Table 1: Possible param eters combinations


#x 0.0 0.0 0.0 0.0 0.0 0 .0

Dispersion # r 0.5 1.0 2.0 4.0 8.0 i6.0parameters Ex I I I I I IZr I I 41 41 161 161

Sam plesize (r,s) (20,10) (20,15) (20,20) (30,15) (30,22) (30,30) (40,20) (40,30) (40,40)

Dimension n 2 3 4

H a n s e n e t a l [ 11 ]) . T h i s a l g o r i t h m i m p l e m e n t s i d e as o f l o gi c p r o g r a m m i n g w i t h i n

a B r a n c h - a n d - B o u n d s c h em e a n d c o n v e rg e s t o a g l o b l o p t i m u m i n f in i te l y m a n y

s te p s. N e c e s s a r y o p t i m a l i t y c o n d i t i o n s e x p r e s s e d i n t e r m s o f t i g h t n e s s o f t h e

s e c o n d l e v e l ' s c o n s t r a i n t s a r e u s e d f o r f a t h o m i n g o r s i m p l i f y i n g s u b p r o b l e m s

t h r o u g h e l i m i n a t i o n o f v a r i a b l e s, b r a n c h i n g , a n d o b t a i n i n g p e n a l t i e s si m i l a r to

t h o s e u s e d i n m i x e d i n t e g e r p r o g r a m m i n g . F o r t h o s e p r e l i m i n a r y r e s u l t s , n o

a t t e m p t s h a v e b e e n m a d e a t s p e c ia l iz i ng th i s a l g o r i t h m t o t h e p a r t i c u l a r s t r u c t u r e

o f 8 ) .O u r e x p e r i m e n t a l d e s i gn f o l lo w s c lo s e ly t h a t o f B a jg i er a n d H i l l [ 1 ]. T h e

d i s c r i m i n a t i n g p o i n t s a r e d r a w n f r o m m u l t i v a r i a t e n o r m a l p o p u l a t i o n s w i t h

v a r i a b l e d i s p e r s i o n p a ram e t e r s (# x , / ~ r , 2 7x , 2 7 r ), s am p l e s i ze s r , s an d s p ace

d i m e n s i o n n . T a b l e 1 s h o w s t h e c o m b i n a t i o n s u s e d f o r t h e a c t u a l e x p e r i m e n t s .

F o r e a c h o n e o f t h e 6 x 9 x 3 = 1 62 c o m b i n a t i o n s t h a t h a v e b e e n t e s t e d , a s e t

o f 1 0 r a n d o m p r o b l e m s h a s b e e n g e n e r a t e d . N u m e r i c a l r e s u lt s a r e p r e s e n t e d i n

t a bl e s 2, 3 a n d 4 , w h e r e m r e f er s t o t h e a v e r a g e ( C P U - t i m e a n d n u m b e r o f n o d e s

e x p l o r e d i n th e b r a n c h - a n d - b o u n d s c h em e ) o f t h e c o r r e s p o n d i n g s e t o f p r o b l e m s ,

s , , t o t h e s t a n d a r d d e v i a t i o n a n d m e t o t h e m e d i a n o f t h e o b s e r v a t io n s , T h e

b il ev e l p r o g r a m m i n g c o d e, w r i tt e n in F o r t r a n , h a s b e e n r u n o n a S U N S P A R K

w o r k s t a t i o n .

A s e x p e c t e d , th e p r o b l e m ' s d i f f ic u l ty i s c l o s e ly r e l a t e d t o t h e n u m b e r o f n o d e s

e x p l o r e d in t h e b r a n c h - a n d - b o u n d s c h em e u n d e r l y i n g t he b i le v el p r o g r a m m i n g

a l g o r i th m , a n d i n c re a s e s w i t h t h e d e g r e e o f o v e r l a p b e t w e e n t h e t w o s et s o f p o i n t s

X a n d Y. W h e n t h e o v e r l a p i s w e a k , t h e n t h e n u m b e r o f n o d e s t o b e e x p l o r e d in

o r d e r t o o b t a i n a n o p t i m a l s o l u t i o n is a l m o s t i n s e ns it iv e t o t h e s p a c e d i m e n s i o n

n. T h i s h o w e v e r is n o t t h e c a s e w h e n t h e o v e r l a p i s s t r o n g , i.e . w h e n t h e m e a n s

# x an d # r a r e c l o s e . F o r i n s t an ce , w i t h t h e d a t a s e ts (# x , # r , 2 7x , Z ' r ) = (0, .5 .; I , I )

a n d ( # x , # r , S x , 2 ; r ) = (0 , 1, I , I ) t h e n u m b e r o f n o d e s e x p l o r e d i s a p p r o x i m a t e l y

t h e s a m e i n d i m e n s i o n 2 ( n = 2 ) w h i l e i t i s i n th e r a t i o 2 to 1 w h e n t h e d i m e n s i o n

i s i n c r e a s e d f r o m n = 2 t o n = 4 . W e a l s o n o t e d t h a t t h e m o s t d i ff ic u l t p r o b l e m s

u s u a l ly c o r r e s p o n d e d t o t h e s i t u a t io n w h e r e t h e n u m b e r o f p o i n t s i n e a c h s a m p l e

we re c l o s e t o eac h o t h e r ( r ~ s ). Fo r t h e ea s i e r p ro b l em s ( r << s o r s << r ) t h e

m e d i a n w a s c l o s e t o t h e a v e r a g e a n d t h e s t a n d a r d d e v i a t i o n w a s s m a ll , a n

u n u s u a l f e a t u r e i n t h e s o lu t i o n o f p r o g r a m s v i a b r a n c h - a n d - b o u n d m e t h o d s .



T

e2

T

b

e

a

o

oD

P

d

as

# /Z E En

m

r=

2

B

sm

s=

1

me

r=

2

B

s=

m

Sm

me m

r=

2

B

sm

s=

2

me

r=

3

B

s=

i m

me m

r=

3

B

Sm

s=

2

m

e

r=

3

B

s=

3

r=

4

B

s=

2

r=

4

B

s=

3

r=

4

B

s=4

m m me m Sm

me m Sm

me m m me

00

05 I I 2

00

80 I

12

00

1

0I

12

n

c

n

c

n

n

c

4

8

3

5

1

6

31

3

3

3

5

6

6

1

1

8

1

6

5

6

8

5

6

8

0

1

8

2

4

5

8

0

7

4

1

3

2

9

3

4

7

1

1

1

4

2

0

2

1

4

8

1

2

5

1

8

4

3

6

1

1

3

1

3

8

1

6

3

2

3

0

9

3

1

3

0

4

2

1

0

3 5

8

1

5

5 6

4

1

3

6 7

8

2

3

71

4

3

7

1 1

2

4

0

1 1

6

4

0

1 2

4

9

3

1 2

0

1

1

2

00

00

10

20

I

4

2

2

c

n

3

3

2

2

1

7

6

3

3

2

5

6

3

6

1

0

8

7

6

0

3

9

0

5

8

1

7

1

1

8

0

4

1

9

5

8

3

2

1

2

1

0

4

2

3

8

6

7

4

2

8

2

7

7

4

7

8

0

1

4

1

8

3

2

8

3

9

9

4

1

1

2

7

3

4

9

7

1

1

2

3

1

3

8

6

0

1

1

3

1

2

8

3

3

3

9

2

1

2

367

2

3

4

4

1

6

4

8

7

2

2

0

7

0

8

0

2

9

8

5

1

6

5

1

1

5

2

1

6

8

2

1

2

6

8

3

3

4

5

6

1

4

4

9

8

6

2

5

7

4

2

8

7

5

2 3

8

9

0

3 3

2

1

8

3 3

8

t1

3

3 7

0

1

4

6 7

0

3

5

8 5

6

2

5

5 113o 8

2

5

1 1

8

6

7

1

00

40 I 4 2

c

n

2

3

2

2

79

4

1

2

8

2

4

0

2

6

1

4

7

0

3

5

2

5

1

3

0

1

5

1

8

5

5

3

7

6

4

8

2

8

1

6

7

2

3

1

9

4

6

4

3

2

0

1

8

3

2

5

4

4

1

5

2

3

2

0

5

1

5

6

2

7

8

2

9

1

4

6

4

2

7

8

1

2

3

1

4

9

7

8

6

6

8

4

1

2

7

6

4

7

1

0

6

9

6

6

2

0

1

5

0

1

1

4

6

9

8

1

6

9

5

2

2

2

1

6

2

1

0

3

3

1

8

3

4

5

2

2

3

5

5

c 2

4

1

6

39

6

0

2

2

1

3

1

2

4

89

7

9

2

4

2

4

7

2

0

2

6

7

5

5

3

2

7

6

2

8

2

0

1

3

7

0

2 4

0

1

5

4 5

2

1

4

5 4

4

2

2

5 6

6

1

3

6 7

0

2

6

7

2

4

1

1

6

3

3

8

2

1

7

1

3

56

48

2

888

2

5

4

7

1

5

4

8

5

7

2

6

4

9

1

1

2

8

1

6

1

4

4

5

1

3

2

0

8

4

1

8

2

9

7

3

3

2

4

7

1

3

4

4

ZO ;> , Oc 4



T

e3

T

b

e

a

o

oD

P

s

d

as

r=

2

s=

1

r=

2

s=

r=

2

s=

2

r=

3

s=1

r=

3

s=

2

r=

3

s=

3

r=

s=

2

r

=

4

s=

3

r=

4

s=

4

00

10

I I 3

00

20

I 4 3

00

40 1 4 3

00

80 I

13

00

1

0I

13

c

n

c

n

c

n

c

n

c

n

c

#

00

#

05

Ex

1

E

I

n

3

n

m

6

2

sm

1

4

me

6

m

9

4

sm

2

1

me

1

m

1

8

sm

2

7

me

1

m

1

0

s~

2

9

me

1

m

2

4

s

6

8

me

2

m

2

4

s

7

6

me

2

m

2

8

sm

8

0

me

2

m

3

4

sm

8

6

me

3

m

3

8

s=

6

0

me

3

6

7

1

6

6

4

5

8

4

3

1

9

1

5

5

4

5

1

2

4

8

1

9

4

2

4

4

1

2

1

0

4

4

1

B

1

9

7

2

9

9

6

2

8

2

B

3

0

2

6

3

4

2

6

2

6

1

9

7

9

8

5

8

0

B

2

5

5

9

2

8

4

6

1

9

4

6

8

4

2

9

7 8

3

2

0

6

3

B B

1

2

1

3

2

2

3

5

9

1

5

1

6

1

6

4

9

7

6

8

1

4

1

2

4

4

7

0

1

5

1

3

6

2

6

7

3

6

8

2

5

2

6

B

2

6

6

9

7

4

2

1

2

7

3

3

9

1

5

1

3

5

5

3

1

1

2

9

1

7

1

7

8

7

2

1

6

B B B

8

0

1

5

2

7

4

8

7

1

3

9

8

1

0

3

4

5

8

8

1

1

1

6

3

3

5

6

1

8

1

2

4

1

8

5

3

3

0

1

5

1

5

5

1

9

3

5

4

1

1

5

1

4

8

2

6

8

8

9

7

3

7

2

8

3

2

4

9

7

7

2

1

9

4

7

1

5

B

1

4

1

5

1

1

1

8

8

79

2

3

6

1

8

2

3

2

3

7

2

6

2

7

1

3

1

1

2

2

3

3

4

4

8

1

3

1

5

3

4

4

9

2

96

1

4

1

1

3

45

1

2

3

3

1

3

1

6

2

6

1

2

1

4

9

1

9

4

4

7

8

7

4

1

3

2

2

4

6

2

1

4

4

1

4

3

6

1

5

2

9

4

7

1

"6

1

9

1

3

5

4

4

8

4

5

0

1

7

1

8

3

6

2

2

4

9

2

2

4

7

1

8

2

6

3

6

8

5

9

6

1

3

1

7

6

1

0

9

4

2

1

3

0

1

2

3

4

9

7

3

8

6

7

2

6

7

0

7

1

7

8

2

1

2

6

7

9

6

3

5

1

8

3

1

7

2

2

9

4

2

1

5

2

5

7

5

3

7

1

4

8

2

A

1

3

3

5

2

6

7

6

2

3

8

6

2

1

2

6

9

8

4

6

1

7

4

0

1

2

2

7

7

4

4

1

1

8

2

6

1

6

2

1

0

6

1

3

2

1

1

5

9

1

6

1

6

8

3

5

A

1

1

9

1

1

2

9

1

6

1

0

5

8

5

1

1

4

1

4

3

1

8

8

3

6

9

3

9

1

8

8

9

4

4

5

2

0

1

6

9

4

7

8

8

5

9

5

6

3

8

8

4

4

o~

c o~



Ta

e4

Th

b

e

a

o

oD

PI

h

d

d

a

s

fx

la n m

r=

2

BLP

s

s

1

me

r=

2

BLP

s=1

r= 2

BLP

s=

2

r= 3

BLP

s=1

In Sm

me m

Sm

me

Sm

me m

r=

3

BLP

sm

s=

2

me m

r=

3

BLP

sm

s=

3

me

r= 4

BLP

s=

2

r= 4

BLP

r=

3

r=

4

BLP

s=

4

m

Sm

me m

m

me m

Sm

me

0

0

0

5I I 4

0

010

1 I 4

0

0

2

0

I 4 4

0

0

4

0I 4

0

08

0

14

n

c

n

c

n

c

n

e

n

c

9

6

1

5

3

8

3

2

9

9

5

1

0

2

7

6

2

8

3

1

2

4

2

2

3

3

6

5

1

3

1

2

3

2

8

5

1

7

7

1

9

2

4

9

4

4

1

0

4

1

0

3

2

1

4

2

3

5

8

2

2

1

9

4

9

1

7

3

2

5

1

2

0

4

2

1

3

4

7

6

0

1

3

6

4

2

5

6

3

7

4

3

2

3

5

9

6

1

9

4

3

5

0

8

1

9

1

2

1

8

3

0

6

0

1

1

7

1

2

2

3

7

6

1

2

1

2

6

2

4

9

5

1 3

2

1

3

3 3

2

8

1

2 6

8

2

4

5 6

4

2

3

4

5

2

2

6

5

2

1

3

3

1

1

7

9

2

2

1

8

7

2

5

8

3

2

6

3

8

1

7

2

2

4

8

5

0

2

1

2

8

3

4

9

9

2

1

6

2

7

3

3

8

1

3

1

0

2

7

4

2

7

0

1

2

0

1

6

2

9

3

9

6

3

1

1

0

828

9

6

9

0

1

0

3

51

1

0

2

5

2

7

3

4

4

75

2

6

4

2

2

0

3

6

1

3

8

5

8

1

1

5 7

7

1

7

7

1

6

3

8

1

1

3

4

1

2

1

2

8

1

3

2 5

2

3

3

3 9

6

3

7

9 4

0

3

4

3

7

9

1

319

1

9

2

38

7

2

6

2

4

1

7

2

4

2

8

1

2

2

0

5

3

5

6

1

7

1

6

6

5

8

0

2

3

5

4

1

6

7

9

6

5

1

4

0

2

6

3

9

2

2

2

8

9

0

9

0

3

4

1

6

1

7

1

8

3

7

7

5

2

7

7

5

1

2

2

9

2

7

2

2

1

1

5

2

2

8

5

6

2

2

5

1

4

8

8

9

6

3

0

7

3

2

0

5

2

1

0

2

8

4

4

1

0

1

3

9

4

2

5

I 838

0

0

1

01

14

c

n

2

6

6

4

1

9

1

9

2

1

1

7

5

1

0

4

0

1

4

4

0

6

1

0

1

0

5

2

1

2

1

5

5

1

4

1

4

7

6

1

3

8

4

4

2

7

2

0

1

3

2

7

2

8

6

6

3

4

8

2

4

5

3

6

5

2

4

7

2

2

1

7

3

2

4

5

6

1

9

5

0

4

0

5

4

1

6

4

1

4

4

6

9

6

6

4

1

3

1

4

0

1

9

8

7

3

5

4

1

8

5

6

3

2

1

1

8

1

5

3

4

1

4

5

4

1

3

4

1

6

2

6

2

4

1

3

2

1

2

1

2

[

2

2

1

3

7

9

I

1

5

1

605 6

I 1

2

4

2

2

2

7

2

4

3

0

7

2

Z0 1 0 l



53

3 S e c o n d M o d e l

P. M arcotte and G. S avard

3 . 1 F o r m u l a t i o n s

M i n i m i z i n g t h e n u m b e r o f m i c la s si fi e d p o i n t s r e s ul ts i n t h e m i x e d i n t e g e r p r o -

g r a m m i n g p ro b l e m D P 2 :

m i n n x + n rp a

w h e r e n x = c a r d { x ~ l p t x ~ + a < 0 } an d n r = c a r d { y J [ p t y j + a > 0 } . I n t r o d u c i n g

b i n a ry v a r iab l e s ~ (~ = 0 i f x i i s no t m isc lassi f i ed , 1 o therw ise) an d t/ j (qj = 0 i f

y J is n o t mi s c l a s si f ied , 1 o t h e rw i se ) , an d o b s e rv i n g t h a t w e m u s t i m p o s e in s o m e

w a y th e c o n s t r a i n t p 0 , w e o b t a i n t h a t D P 2 c a n be s o lv e d b y s o l v in g t h e t w o

m i x e d i n t eg e r p r o g r a m s

m i n ~ ~ + ~ t / jp a ~ ~l i~ l jE J

s.t. M ~ i >_ - p t x i + a ) i ~ I

M t l j >_ p t y j + a J ~ J ( l l a )

~ j e { o 1}

- - l < p k < _ l k = l . . . . , n

P l = - 3

a n d

m i n 2 ~ , + 2 r tJp a ~ ~l i~ l j~ J

s.t. M~i >_ - - p tx i + a) i e I

M t l j > p t y ~ + a J ~ J (1 lb)

~i, ~/j ~ {0, 1}

- - l < _ p k < _ l k - - 1 . . . . . n

P i = 6



N o v e l A p p r o a c h e s t o t h e D i s c r i m i n a t i o n P r o b l e m 5 3

w h e r e M is a p o s i t iv e c o n s t a n t s u f f ic i e nt ly l a r g e t o e n s u r e t h a t p r o b l e m s l l a )

a n d 1 l b ) a r e f e a s ib l e f o r a ll a d m i s s i b le v a l u e s o f p a n d a a n d 6 i s a s m a l l p o s it i v e

c o n s t a n t . W e h a v e t h e f o ll o w i n g t h e o r e t i c a l r e s u lt :

T h e o r e m 4 : F o r D P 2 t o b e e q u i v a l e n t t o l l a ) a n d l l b ) i t is s u ff ic i en t t h a t th e

c o n d i t i o n

M > 2 t m a x ~ xgl m a x ~ tY ~ t}i~l k = l j ~ J k = l

b e s a t i s f ie d .

Proof . I f a < - m a x / ~ k I x~ l < ~ k p k X ~ f o r a l l p o i n t s x ' i n x , r e c al l t h a t [PRI < 1)

t h e n a l l p o i n t s i n X a r e m i s c la s s i fi e d . O n t h e o t h e r h a n d , i f a > m a x ~ ~ k l x ~ l , t h e n

a l l p o i n t s i n X a r e w e l l c l a s si f ie d . A s i m i l a r r e s u l t h o l d s f o r p o i n t s i n Y , r e p l a c i n g

x i b y y J i n th e a b o v e i n e q u a li ti e s . W e c a n t h e r e f o r e c o n c l u d e t h a t , w i t h o u t l o s s

o f g e n er al it y , la l c a n b e b o u n d e d b y m a x , ~ , ~ = l l x ~ l + m a x i ~ s ~ = x l Y ~ l :

C o n s e q u e n t l y , I p t x i + a l a n d I p ' y j + a l w i l l a l w a y s a s s u m e v a l u e s l e s s t h a n2 { m a x , ~ , ~ = ~ Ix~ l + m a x j ~ s ~ = ~ lY/~I}, a s c l ai m e d . [ ]

O n e c a n s u b s t i t u te t o th e m i x e d i n te g e r p r o g r a m s 1 l a ) a n d l l b ) t h e s in g le

b i le v e l p r o g r a m :

m i n ~ , + ~ q j + L [ ~ i + X [ 3 i + 7 1p,a,~,~,lt i~l j~ J L eI je J

s.t. M ~ i > - - ( p t x i + a )

M r l j > p t y j + a

2/~ + p l = 3

- - l < p k < l

i ~ I

j ~ J

k = 1 , . . . , n

~ , f l , ? ) ~ a r g m a x ~ o q + ~ f l j +

i ~ l j ~ J

s.t. ~i -< r ~i < 1 - ~i

r > 0 i ~ I ,

,8j <_ rlj ,Sj <_ l - rli

1 2 )



532 P. M arcotte and G. Savard

t l j > O j e J

? _ < p ? < 1 - / ~

? _ > 0

w h e re L i s a s u i t ab l y l a rg e co n s t an t . T h e c o n s t r a i n t s e t o f th e l o w e r lev e l f o r ce s

v a r ia b l e s ~ , f lj a n d ? t o t a k e n o n z e r o v a l u e s w h e n e v e r a b i n a r y v a r i a b l e ~ , r/i

o r is n o t e q u a l to z e r o o r o n e . I n o r d e r t o m i n i m i z e t h e s u m o f t h e ~ ' s , fi rs

an d ? , t h e u p p e r l ev el w i ll s et t h e co n t i n u o u s v a r i ab l e s (~ i's , ~ / fs an d / ~ ) , t o e i t h e r

0 o r 1. I n d e e d , t h e l o w e r l e v el s i m p l y t r a n s la t e s i n t o a n o p t i m i z a t i o n f o r m t h e

re l a t i o n s h i p s

~i = m in{ ~i, 1 - ~i}

fli = min{ ~/j, 1 - ~/j}

? = m i n { , 1 - / ~ } .

T h e l o w e r le v el p r o g r a m , t o g e t h e r w i t h t h e u p p e r l ev e l c o n s t r a i n t

2/~ + p l = 1

e n s u r e s t h a t P l w i ll t a k e e it h e r v a l u e - 1 o r 1, t h u s a v o i d i n g t h e d e g e n e r a t e

s o l u ti o n . T h e s o l u t i o n t o D P 2 w i ll n o t i n g e n e r a l be u n i q u e , a n d s o l u t i o n m e t h o d s

b a s e d o n l i n e a r p r o g r a m m i n g r e l a x a t i o n ( b r a n c h - a n d - b o u n d , c u t t i n g p l a n e s ,

L ag ran g ean r e l ax a t i o n ) w i l l f av o r ex t r ema l s o l u t i o n s i . e . , i n t h i s co n t ex t ,

h y p e r p l a n es g o i n g t h r o u g h a t l e a s t n p o i n t s o f e i t h e r X o r Y (s ee f i g u re 3 fo r a

t w o - d i m e n s i o n a l e xa m p l e ). I n t h i s c a s e it is n a t u r a l t o s e e k, a m o n g t h e s e t o f

h y p e r p l a n e s m i n i m i z i n g t h e n u m b e r o f m i s c la s si fi e d p o i n ts , a n h y p e r p l a n e t h a t

mi n i m i zes a s w e l l t h e s e t o f w e ll -c l a ss if i ed p o i n t s . T h u s w e o b t a i n t h e m e t a -b i l ev e l

f o r m u l a t i o n :

m a x m i n ( m in f x i a,p a Ix i r welliX)

s .t . (p , a) ~ ar g m in n x + nr

m i n _ p t y J _ a }yJ r well Y)

1 3 )

IlPll2 ~ 1

w h e re w e l l (X ) an d w e l l(Y ) d e n o t e t h e s e ts o f w e l l -c l a ss i fi ed p o i n t s i n X an d Y

respec t ive ly , whi l e n x ( r e s p ect i v e ly n y ) d en o t e s t h e n u m b e r o f mi s c l a s s if i ed p o i n t s

in X (resp ect ively in Y).



Novel Approaches to the DiscriminationProblem 533

P~ 0

..o%o0 0 O

9 ~ a = O

m

P l

Fig. 3: An exterm al solution

In v i ew o f t h e d i f f i cu lt y o f s o l v i n g (1 3 ) , w e w i ll in s t ead co n s i d e r a g o a l p ro -

g r a m m i n g f o r m u l a t i o n w i t h p r i m a r y o b je c ti ve t h e n u m b e r o f m i sc l as s if ic a ti o nsan d s eco n d a ry o b j ec t iv e t h e s ep a ra t i o n o f t h e se t s o f w e l l -c l a ss if i ed p o i n t s . T h i s

r e s u l t s in t h e fo l l o w i n g p ro ced u re :

G o a l P r o g r a m m i n g A l g o ri th m

1. L et (p*, a* ) e ar g m inp, a n x + n r a n d I x ( r e s p ec ti v e l y Iv ) d en o t e t h e i n d ex s e t

o f wel l -c lass i f ied po in t in X ( respec t ive ly Y).

2 . So l v e t h e p ro j ec t i o n p ro b l em :

min t lql l2q b

s.t. q tx l + b >_ 1 i E I x

q t y J + b < - I j e l r 9

(14)

I f (14) i s feas ib le ( s t r i c tly con vex case) l e t (q* , b*) be i t s so lu t io n an d se t: p* =

q* /tlq * II 2, a * = a*/llq*}l 2.3 . O u tp u t (p* , a* ) . [ ]



534

3 .2 Com plex i ty Ana l ys i s


I n t h e c o n v e x c a se , t h e o p t i m u m o f (14) is z e r o , a n d o u r g o a l p r o g r a m m i n g

f o r m u l a t i o n r e d u c e s D P 2 t o D P 1 . I n t h e r e v e r s e c o n v e x ( o r i n t h e g e ne r a l) ca s e,

w e o b t a i n t h e f o l l o w i n g w o r s t - c a s e r e s u lt :

T heorem 4 : T h e d e c i s i o n p r o b l e m c o r r e s p o n d i n g t o D P 2 i s s t r o n g l y N P -

c o m p l e t e .

Proof T h e d e c i s io n p r o b l e m c o r r e s p o n d i n g t o D P 2 c a n b e s t a te d a s:

D o e s t h e r e e x is t a h y p e r p l a n e H s u c h t h a t t h e n u m b e r o f m i s c l a s si f ic a t io n s

i s l es s o r e q u a l t o K ?

(a ) D P 2 i s i n N P .

L e t I d e n o t e t h e c a r d i n a l i t y o f a m a x i m a l s e t S o f a f f in e l y i n d e p e n d e n t p o i n t s i n

X w Y an d l e t n ' = ca rd (S ) = r a in { l , n }. In p rac t i ce we wi l l u s u a l l y h av e t h a t r + s

i s m u c h l a r g e r t h a n n a n d w e e x p e c t n ' = n . T h e r e a l w a y s e x i s ts a s o l u t i o n ( p, a )t o D P 2 s u c h t h a t t h e n ' p o in t s i n S li e o n t h e h y p e r p l a n e ptz + a = 0 . T h e re fo r e

a s o l u t i o n t o D P 2 c a n b e c h a r a c t e r i z e d b y s p e c i f y in g t h e f i n it e s e t S . F o r a g i v e n

S , t h e v a r i a b l e s p a n d a a r e d e t e r m i n e d a s s o l u t i o n s o f t h e ( p o s s ib l y u n d e r -

d e t e r m i n e d i f n ' < n ) l i n e a r s y s t e m

tz + a = 0 fo r al l z e S ,

a n d , p r o v i d e d t h a t t h e i n p u t p o i n t s x ~ n d y J a r e r a t i o n a l , p a n d a a r e a l so r a t i o n a lw i t h l e n g t h s p o l y n o m i a l l y b o u n d e d i n t h e l e n g t h o f t h e i n p u t . F o r g i v e n ( p, a ),

c h e c k i n g t h a t t h e n u m b e r o f m i s c l a s s if i c a ti o n s d o e s n o t e x c e e d K c a n b e p e r -

f o r m e d i n t i m e p r o p o r t i o n a l t o t h e l e n g t h o f (p , a ), th e l e n g t h o f t h e i n p u t a n d

t h e n u m b e r r + s o f p o i n t s i n X u Y, i.e . i n p o l y n o m i a l ti m e w i t h r e s p e c t t o th e

i n p u t s iz e, t h e n u m b e r o f b i ts r e q u i r e d t o e n c o d e t h e d a t a x i, i ~ I a n d yJ, j ~ J .

( b) D P 2 i s N P - c o m p l e t e .

T h i s r e s u l t w il l b e o b t a i n e d b y p o l y n o m i a l ( a c t u a l ly li n e a r ) r e d u c t i o n o f t h e c l o s e d

d e n s es t h em i s p h e r e p r o b l e m ( C D H ) , w h o s e N P - c o m p l e t e n e s s h a s b e e n p r o v e d

b y J o h n s o n a n d P r e p a r a t a [ 1 3 ] , t o D P 2 . T h e c l o s ed d e n s e s t h e m i s p h e r e p r o b l e m ,i n it s sa t is f ia b i li ty f o r m , c a n b e s t a t e d a s ( se e G a r e y a n d J o h n s o n [ 8 ] ) :

G i v e n a s e t W o f m p o i n t s i n R , d o e s t h e r e e x i st a v e c t o r q i n R a n d a

s c a l a r b s u c h t h a t a t l e a s t L p o i n t s i n W l ie o n t h e s a m e s id e o f ( o r o n )

t h e h y p e r p l a n e H = {q ' z = 0}, i .e .: ca rd {w ~ W l q 'w > 0} > L?



No vel Approaches to the Discrimination Problem 535

O

~ 0

Fig 4: Solving CD H

C o r r e s p o n d i n g t o C D H w e f o r m u l a t e a n i ns ta n c e o f D P 2 w i th X = W w T a n d

Y = T w h e re T co n t a i n s m - L + 1 co p i e s o f t h e o r i g in , an d we s e t K = m - L .

T h e n a n y s o l u t i o n t o t h i s D P 2 m u s t b e a h y p e r p l a n e g o i n g t h r o u g h t h e o r i g i n

a = 0 ) an d l eav e a t m o s t m - L p o in t s o f X m i s c l a s si f ied , i.e . t h a t a t l e a s t L p o i n t s

i n X e q u i v a l e n t l y L p o i n t s in W ) li e o n t h e s a m e s i d e o f H , t h u s s o l v i n g C D H

see f igure 4 ). [ ]

3 .3 S o l u t i o n A l g o r i t h m s

M i n i m i z i n g t h e n u m b e r o f m i s c la s s if i ed p o i n t s c a n b e a c h i e v e d b y s o l v in g t h e

b i le v e l p r o g r a m s 1 2). H o w e v e r , f o r n = 2 , a n O r + s ) 2 l g r + s ) ) a l g o r i t h m i s

a v a il a b le , b a s e d o n t h e f a ct t h a t t h e d i s c r im i n a t i n g h y p e r p l a n e h a s t o g o t h r o u g h

a t l e a s t t w o p o i n t s o f X w Y. I n i t s b as i c f o r m t h i s a l g o r i t h m c a n b e e x p r e s s e d a s

A l g o r i th m 2 - P o l

fo r a ll p o in t s z i n X w Y o



536 P. M arcotte and G. Savard

0. S e t S = X u Y - { z } .

1. S e l e c t a n a r b i t r a r y a x i s g o i n g t h r o u g h t h e p o l e z.

2 . S o r t a l l p o i n t s i n S a c c o r d i n g t o t h e i r p o l a r a n g l e w i t h r e s p e c t t o t h e a x i s .

3 . S e l e c t a n a r b i t r a r y p o i n t z I i n S a n l e t H b e t h e h y p e r p l a n e s t r a i g h t l in e )

g o i n g t h r o u g h z a n d z 1.

4 . L e t N d e n o t e t h e n u m b e r o f m i s c l a s s i fi c a ti o n s w i t h r e s p e c t t o H .

5. le t S = S - {z 1 }.

i f S = ~ t h e n s t o p a n d o u t p u t N

ls s e t z I t o t h e p o i n t i n S c l o s e s t t o H a n d r e t u r n t o 4 .

I n a lg o r i th m 2 - P O L , s t e p 2 h a s c o m p l e x i t y O r + s) l g r + s )) . S ince , a f t e r s t e p

5 h a s b e e n p e r f o r m e d , t h e n u m b e r o f m i s c l a s s if i c a ti o n s c a n o n l y i n c r e a s e o r

d e c r e a s e b y 1 o r 2 , s t e p 4 c a n b e i m p l e m e n t e d i n c o n s t a n t t im e , a n d t h e o v e r a l l

w o r s t -c a s e c o m p l e x it y o f 2 - P O L is O r + s ) 2 l g r + s )) , a s p r e v i o u s l y c l a i m e d .

R e m a r k s : 1. I n t h e r e v e r se c o n v e x c a s e , 2 - P O L is a c o m p l e t e e n u m e r a t i o n s c h e m e .

I t is p o s s i b l e to i n c r e a s e i ts e ff ic i e nc y b y u s i n g s y m m e t r y r e l a t io n s h i p s a n d t h e

f a c t t h a t w e c a n r e st r ic t o u r s e a r c h t o h y p e r p l a n e s g o i n g t h r o u g h c o u p l e s x , y )

w i t h x ~ X a n d y ~ Y . A l s o , if p o i n t s u s e d a s p o l e s a r e s e l e c t e d in a s u i t a b l e o r d e r

c l o se n e i g h b o r s ) t h e n t h e s o r t in g s t e p 2 c a n b e i m p l e m e n t e d u s i n g r e o p t i m i z a t i o n

t e c h n i q u e s .

2 . A l g o r i t h m 2 - P O L c a n b e g e n e r al iz e d t o p r o b l e m s i n R n t o y ie ld a n

O r + s ) ~ l g r + s )) a l g o r i t h m , f o l l o w i n g t h e li n es d e v e l o p e d p r e v i o u s l y a n d t h e

r e c u rs iv e p r o c e d u r e m e n t i o n e d i n t h e p a p e r b y J o h n s o n a n d P r e p a r a t a 1 1 3] f o r

t h e d e n s e s t h e m i s p h e r e p r o b l e m . H o w e v e r s u c h a lg o r i th m s a r e n o t c o m p e t i ti v e

w i t h i m p l i c it e n u m e r a t i o n s c h e m e s i n h i g h d i m e n s i o n s , a s w e w i ll s ee .

3 .4 N u m e r i c a l R e s u l t s

W e c o m p a r e d t he ef fic ie nc y o f t h e b ile v el a p p r o a c h a n d t he 2 - P O L a n d N - P O L

e n u m e r a t i o n s c h e m e s fo r s o lv i n g D P 2 o n t h e se ts o f r a n d o m l y g e n e r a t e d p r o b -

l e m s p r e v i o u s l y d e s c r i b e d i n s e c t i o n 2 . 4 . A g a i n , n o a t t e m p t s h a v e b e e n m a d e t o

s p e c ia l iz e t h e B L P c o d e t o e x p l o it t h e s t r u c t u r e o f t h e b il e v e l p r o b l e m 1 2).

I n t a b l e s 5 , 6 a n d 7 w e p r e s e n t t h e n u m e r i c a l r e su l ts . N o t e t h a t , s in c e a l g o r i th m s

2 - P O L a n d N - P O L a re b a s e d o n e x p l i c i t i n c o n t r a s t w i t h implic i t ) e n u m e r a t i o n ,

s o l u t i o n t i m e s f o r t h o s e a l g o r i t h m s s h o u l d t h e o r e t i c a l l y b e e q u i v a l e n t f o r p r o b -

l e m s w i t h i d e n t i c a l r , s a n d n v a l u e s , a l b e i t d i ff e r e n t X a n d Y d a t a . T h e r e f o r e , f o r

g i v e n s a m p l e s iz e s, t h o s e 2 a l g o r i th m s w e r e n o t t e s t e d o n a l l t e n r a n d o m l y g e n e r -

a t e d s u b p r o b l e m s , a s i n g le o n e b e i n g s u ff ic ie n t t o s h o w t h e b e h a v i o r o f t h e m e t h o d s .



N o v e l A p p r o a c h e s t o t h e D i s c r i m i n a ti o n P r o b l e m 5 7

Co n seq u en t ly , m ed ian an d s t an d a r d d ev ia t io n f ig u r es a r e n o t g iv en , b e in g eq u a l

to th e av e r ag e an d th e n u m b er ze r o r e sp ec tiv e ly .

I n v i ew o f t ab l e 5 , i t i s c l ea r t h a t 2 - P O L o u tp e r f o r m s th e b i l eve l ap p r o ac h f o r

so lv in g d i sc r im in a t io n p r o b le m s o n th e p l an e (n = 2). I n d eed , w h a tev e r t h e

d eg r ee o f o v e r l ap b e tw een X an d Y, p r o b lem s in v o lv in g h u n d r ed s o f p o in t s i n

each g roup cou ld be so lved wi thou t d i f f icu l ty . Never the less th is approach fa i l s

in d im en s io n s h ig h e r t h an 3 . I n d im en s io n 4 i t i s o u tp e r f o r m ed b y th e b i l ev e l

a lg o r i th m o n a ll t e st p r o b lem s , w i th th e ex cep t io n o f t h o se p r o b lem s co r r e -

spo ndin g to the f i r s t se t o f d ispers ion coef ficien ts . In d im ensions h igher th an 4 ,

a l g o ri th m N - P O L b e c o m e s i n tr a ct a b le . A r o u g h c a l c u l a ti o n s h o w s t h a t s o m e

2 x l0 s seco n d s w o u ld b e r eq u i r ed f o r so lv in g a D P 2 o f size 4 0 x 4 0 an d

d im en s io n 5 . Fo r so lv in g a D P 2 in v o lv in g th e sam e n u m b er o f p o in t s , 2 x 1 0 x3

s e c o n d s o f C P U t im e w o u l d b e r e q u ir e d i n d i m e n s i o n 1 0.

I n th e b i lev e l ap p r o ach , t h e p r o b lem s d i ff icu l ty i s d i r ec t ly p r o p o r t io n a l t o t h e

n u m b er o f n o d es ex p lo r ed in t h e b r an ch - an d - b o u n d t ree an d , a s i n d ica t ed in

sec t io n 2 .4 , i t i s r e l a t ed to t h e d eg r ee o f o v e r l ap b e twe en th e two g r o u p s o f p o in ts .

F o r p r o b lem s in v o lv in g a l a rg e n u m b er o f m i sc la s s if ied p o in t s , t h e av e r ag e

n u m b e r o f b r a n c h - a n d - b o u n d n o d e s c a n v a r y b y a f a c to r la r ge r t h a n 1 00 0 f o r

p r o b lem s o f s im i la r d im en s io n s g en e r a t ed th r o u g h d i ff e ren t m ea n s an d v a r i an ce-

cov ar ianc e m at r ices . F or the se t o f p rob lem s hav ing d ime nsions (40 , 20 , 4 ) ( see

tab le 7 ) th is fac to r go t as la rge as 1157 , whi le i t was a lways lower than 23 fo r

D P1 . Th i s can b e ex p la in ed b y th e f ac t th a t t h e n u m b er o f m i sc la s s if ica t io n s is

a f fec ted , t o a g r ea t e r ex t en t t h a t t h e o p t im a l v a lu e o f DP1 , b y th e d eg r ee o f

o v e r l ap b e tw een X an d Y. I n co n t r a s t t h e o b jec t iv e D P 2 is v e r y m u ch d ep en d en t

o n th e d eg r ee o f d i sp e r s io n o f t h e sam p le p o in t s.

These p re l imina ry resu l t s a re encourag ing , espec ia l ly in v iew of those resu l t s

o n DP 2 o b ta in ed o n m a in f r am e co m p u te r s f o r p r o b lem s o f s ize r = s = 5 0, n = 3

a n d r e p o r t e d i n t h e p a p e r s b y K o e h l e r a n d E r e n g u c [ 1 7 ] a n d S t a r e a n d

Jo ach im s th a le r [ 2 2 ] .

Conc lus ion

I n th i s p a p e r we in v es tig a ted , t h eo r e t i ca lly an d n u m er i ca lly , tw o f o r m s o f a

n o n p a r am e t r i c d i sc r im in a t io n p r o b lem , f o r wh ich n ew, a l t e r n a tiv e f o r m u la t io n s

wer e g iv en , an d sev e r a l a lg o r i th m s p r o p o sed . P r e l im in a r y r e su l t s o n sm a l l t o

m ed iu m - s i ze p r o b lem s a r e en co u r ag in g an d we in ten d , a s a seq u e l t o t h is w o r k ,

to t a i lo r so m e o f t h e m o r e p r o m is in g a lg o r i th m s to th e p a r t i cu la r s t r u c tu r e o f

th e d i sc r im in a t io n p r o b lem . I n p a r t i cu l a r we b e l i eve th a t t h e in c lu s io n o f n ew

o p t im a l i ty te s t s an d b o u n d s in t h e b il ev el p r o g r am m in g p r o ce d u r e th a t ex p lo i t

t h e s t r u c tu r e o f D P 2 co u ld s ign i fi can t ly d ec r ease co m p u t in g t im es .



Table D

P2

d

a

s

oo

r

=

2

s=

1

r

=

2

s=

1

r

=

2

s

=

2

r

=

3

s=1

~x

:Cx

~r n

0

0

0

5I I 2

0

01

0 I I 2

0

0

2

0I 4 2

0

0

4

0I 4 2

0

0

8

0I

12

0

0

1

0I

12

n

c

n

c

n

c

n

c

n

c

n

c

m

1

0

4

7

4

6

2

0

3

4

1

4

1

2

6

6

1

8

9

6

8

8

4

9

BLP

sm

4

3

2

2

2

8

1

6

2

0

9

1

1

2

6

7

8

7

4

6

1

4

6

9

m

1

4

9

4

2

7

2

1

7

9

4

0

1

9

0

3

1

7

2

PO

L

m

-

0

0

0

0

-

0

0

-

0

0

-

0

0

-

0

0

m

1

8

8

1

6

2

3

4

5

0

3

1

9

4

6

1

2

0

1

5

6

0

4

2

BLP

Sm

1

3

5

0

4

6

2

4

3

4

2

7

5

6

3

8

1

7

1

8

4

8

3

5

m

1

6

9

2

1

5

4

2

7

9

5

4

2

1

5

3

1

7

2

PO

L

rn

-

0

0

0

0

-

0

0

-

0

0

-

0

0

-

0

0

m

2

2

1

3

9

8

6

2

8

8

6

3

1

4

9

8

4

4

3

2

9

6

7

8

BLP

sm

1

5

9

3

6

9

4

8

4

0

2

2

6

8

51

2

1

1

7

4

1

3

4

m

1

1

8

7

4

4

8

6

3

9

7

5

3

3

6

7

6

2

2

PO

L

m

-

0

0

0

0

-

0

0

-

0

0

-

0

0

-

0

0

m

2

6

1

7

7

8

7

1

5

8

5

0

1

6

1

8

2

2

3

8

7

8

7

6

BLP

s

9

8

8

5

6

7

5

5

5

4

5

7

7

0

6

7

2

0

3

8

6

2

5

5

m

1

1

0

4

3

0

3

3

2

9

9

7

2

1

3

5

4

3

2

PO

L

m

-

0

0

0

0

-

0

0

-

0

0

-

0

0

-

0

0

<



m

r=3

BLP

s~

s=2

me

4

9

53

15

14

4

43

12

11

80

82

1

13

18

14

71

75

1

12

32

33

27

38

2

29

66

84

46

52

5

79

16

18

16

14

1

18

2POL

m

-

00

00

-

00

-

00

-

00

-

00

18

29

91

12

1

16

m

r=3

BLP

s~

s=3

m

46

62

31

50

2

33

28

2

0

10

14

1

25

10

17

61

15

8

18

6

0

8

5

14

2

2

5

78

24

43

31

47

1

26

2POL

m

-

00

00

-

00

-

00

-

00

-

00

26

33

28

31

9

19

4

6

61

15

2

6

4

67

16

12

76

13

1

13

42

60

30

59

2

40

12

2

3

92

19

1

1

m

r=4

BLP

s

s=2

me

12

23

27

47

5

87

2POL

m

-

00

00

-

00

-

00

-

00

-

00

2

2

47

14

29

2

41

58

17

35

66

4

90

10

27

57

12

9

2

6

m

r=4

BLP

s~

s=3

m

9

6

1

0

25

4

3

8

1

9

38

6

4

14

11

2

54

16

22

72

14

9

16

2POL

m

m

00

00

-

00

-

00

-

00

-

00

4

2

1

8

21

61

3

8

401

1

1

2

32

73

1

2

1

56

17

42

12

4

12

01

0.14

16

4

7

10

29

1

38

01

m

r=4

BLP

sm

s=4

m

4

8

1

1

11

41

4

9

901

2POL

m

12

48

10

37

1

33

01

Z O Q O ~



Ta

e

6

D

P2

s

d

a

s

r 2

s=

1

r

=

2

s=

1

r

=

2

s

=

2

r

=

3

s=1

g gn

0

0

0

5I I 3

0

01

0I I 3

0

0

2

0

I 4 3

0

0

4

0I 4 3

0

0

8

0I

13

0

0

1

0I

13

n

c

n

c

n

c

n

c

n

c

n

c

m

1

0

6

9

3

8

1

4

2

4

1

9

8

4

4

9

1

0

6

7

8

2

4

8

BLP

sm

1

8

4

7

6

1

2

7

4

2

4

1

8

5

4

1

7

7

0

1

2

5

3

m

e

8

4

6

1

6

7

7

3

8

3

2

4

3

2

1

3

2

1

N

-PO

L

m

-

5

2

5

2

-

5

2

-

5

2

5

2

-

5

2

m

4

2

2

5

1

8

7

5

1

8

6

6

2

2

1

3

5

6

3

4

1

6

7

8

BLP

sm

2

3

1

5

1

2

1

3

1

8

6

5

5

4

3

9

5

7

3

5

1

6

1

3

m

2

1

6

5

3

5

4

2

6

7

6

6

2

2

9

3

2

6

N

-PO

L

m

-

8

8

8

8

m

8

8

-

8

8

8

8

-

8

8

m

6

4

4

8

1

8

8

5

1

4

1

5

2

6

1

5

7

8

6

7

1

0

1

4

BLP

sm

4

5

3

3

8

8

6

1

1

4

9

5

3

7

2

6

6

4

4

1

1

6

1

4

m

3

2

0

8

6

8

1

1

6

1

1

6

6

5

9

3

4

1

N

-PO

L

m

-

1

9

1

9

-

1

9

-

1

9

1

9

-

1

9

m

5

4

5

8

1

4

1

6

8

6

8

4

5

4

7

1

2

4

2

4

3

2

4

2

BLP

sm

4

1

4

7

9

0

9

0

8

4

8

7

4

1

4

7

2

1

2

9

1

8

2

0

m

e

4

3

8

6

6

4

4

4

4

3

4

7

1

1

5

3

3

8

N

-PO

L

m

-

2

2

-

2

2

-

2

2

-

2

2

-

2

2

-

2

2

O g



m

r

=

3

BLP

Sm

S

=

2

me

N

-PO

L

rn

m

r

=

3

BLP

sm

s

=

3

me

N

-PO

L

m

m

r

=

4

BLP

sm

s

=

2

me

N

-PO

L

m

1

9

1

7

1

1

1

2

1

1

1

2

4

3

5

2

6

3

2

1

1

0

-

3

9

2

2

3

4

1

1

2

5

1

2

4

-

3

9

6

2

1

6

2

7

4

2

7

1

1

2

6

2

6

1

5

2

8

1

2

8

3

9

9

8

1

4

6

2

9

2

6

1

2

1

2

1

2

3

m

r

=

4

BLP

sm

s

=

3

me

9

2

9

3

5

1

3

1

0

7

7

3

9

1

2

1

1

9

0

7

2

1

8

7

3

9

3

3

5

0

-

3

9

4

4

7

3

2

7

4

4

3

5

9

6 0

5 8

3

4

8

5

2

1

9

9

8

6

4

7

3

9

7

3

9

0

3

4

-

6

7

-

6

7

6

7

6

7

-

6

7

-

6

7

1

0

1

4

3

0

5

1

3

2

5

4

4

8

8

9

1

2

1

1

2

0

3

9

6

5

9

7

2

8

3

1

3

8

5

4

6

5

9

9

9

6

1

1

3

6

6

0

8

1

3

2

4

8

2

3

3

2

3

0

7

1

2

9

1

3

7

9

-

7

9

-

7

9

-

7

9

-

7

9

7

9

7

2

1

2

1

4

2

2

2

5

1

7

2

1

1

4

0

8

7

5

1

0

4

4

8

2

1

0

2

3

4

8

4

8

1

1

0

4

3

9

1

6

1

9

N

-PO

L

m

9

8

-

9

8

-

9

8

9

8

1

4

2

3

1

7

2

6

5

1

9

2

0

6

9

1

3

4

0

1

4

5

1

2

3

4

1

4

3

0

5

1

9

m

r

=

4

BLP

Sm

S

=

4

me

9

0

2

8

5

0

1

2

7

1

1

9

8

9

8

4

8

1

3

1

8

5

6

3

4

9

2

2

0

7

4

3

9

4

9

2

8

-

1

6

-

1

6

m

_

m

9

8

2

4

6

4

1

9

6

1

5

-

1

6

1

6

1

6

N

-PO

L

m

-

1

6

ZQ, O

O :1

O c



5 4 2 P M a r c o t t c a n d G S a v a rd

t ~

0

0

0

0

- t

0

~ i c 5 e , i

C ~ C b ~ ,

~ ~

9

~ . o . ~

~,.,i t.~l -xl

, 1 , .. .1 , . 1 , - 1

C b

I I It...

r 0

I I I II I I I I I I



m

r=

3

BL

s

s=

2

m

2

2

3

5

1

3

1

6

3

4

0

1

8

2

7

2

0

3

6

5

7

3

2

2

3

5

3

8

4

4

9

1

5

62

1

5

75

1

3

3

52

7

2

1

8

1

1

2

9

1

2

4

30

56

23

36

30

46

N-POL

m

-

6

9

-

6

9

-

6

9

-

6

9

'

6

9

-

6

9

8

9

1

3

5

7

9

4

6

1

8

m

r=

3

BL

s~

s=

3

m

1

0

3

3

2

3

4

8

9

2

6

6

2

1

8

4

8

7

1

4

8

3

2

6

4

4

2

1

4

1

2

4

4

6

6

1

9

2

4

4

1

6

9

4

72

1

8

1

5

2

4

3

78

N-POL

m

-

1

4

-

1

4

1

4

-

1

4

1

4

-

1

4

40

1

7

39

67

3

8

6

9

1

4

2

0

3

8

5

4

2

5

4

4

6

0

1

7

4

7

8

7

3

4

6

1

2

2

4

5

m

r=

4

BL

sm

32

81

29

58

s=

2

m

N-POL

m

m

r=

4

BL

sm

s=

3

m

N-POL

m

m

r=

4

BL

sm

s=

4

m

N-POL

m

3

5

2

3

6

1

1

3

1

3

66

6

1

8

3

59

-

1

7

-

1

7

1

7

-

1

7

-

1

7

-

1

7

-

4

4

1

2

7

0

1

0

58

1

7

1

6

4

1

38

1

0

-

-

5

0

1

1

6

3

1

6

~

1

0

2

2

4

3

25

74

-

2

5

1

4

1

0

3

1

9

1

3

1

3

73

2

4

-

2

4

2

4

-

2

4

-

2

4

-

2

4

m m

1

8

,3

9

1

7

~3

4

8

2

2

I

8

8

5

8

9

4

2

2

1

5

6

4

0

1

7

8

3

2

1

9

3

6

5

2

1

4

4

4

1

8

2

8

8

1

0

3

7

2

5

~

6

4

3

~

92

-

5

2

-

5

2

5

2

-

5

2

-

5

2

-

5

2

ZO > O g O to



544 P. Marcotte and G. Savard

Fina lly i t would be interest ing to co mpar e the practical performance of the

proposed bi level prog ramm ing approac h to that of more es tabl ished methods of

global optimizat ion, such as the cutt in g-plan e meth ods of Horst an d T uy [12].

Acknowledgments: We would like to thank Grrald Marquis and Xavier Haurie for their help in

conducting the numerical experiments.

eferences

[1] Bajgier SM and Hill AV 1982) An experimental comparison of statistical and linear program-

ming approaches to the discriminant problem. Decision Sciences 13:604-618

[2] Bodlaender HL, Gritzmann P, Klee V and Van Leeuwen J 1990) Computational complexity

of norm-maximization.Combinatorica 10:203-225

[3] Cavalier TM, Ignizio JP and Soyster AL 1989) Discriminant analysis via mathematical pro-

gramming: certain problems and their causes.Com puters and Operations Research 16:353-362

[4] Fisher R A 19 36) Th e use of mu l t ip le measure men ts in taxono mic problems. Annals o f Eugenics

7:179-188

[5] Freed N and Glover F 1981) A linear programming approach to the discriminant problem.Decision Sciences 12:68-74

[6] Freed N and Glover F 1981) Simple but powerful goal programming models for discriminant

problems. European Journal of Operational Research 7:44-60

[7] Freund RM and Orlin JB 1985) On the complexity of four polyhedral set containment

problems. Mathematical Programming 33:139-145

I-8] Garey MR and Johnson DS 1979)Computers and intactahility, Freeman WH, San Francisco

[9] Gaither N and Glorfeld W 1982) On using linear programming in discriminant problems.

Decision Sciences 13:167-171

[10] Glover F 1990) Improved linear programming models for discriminant analysis. Decision

Sciences 21 : 771-785

[11] Hansen P, Jaumard B and Savard G 1990) New branch-and-bound rules for linear bilevel

programming, Cahier du GERAD G-89-09. Ecole Polytechnique, Montrral, submitted forpublication

[12] Horst R and Tuy H 1990) Global optimization. Deterministic approaches. Springer-Verlag,

New-York

[13] Johnson DS and Preparata FP 1978) The densest hemisphere problem. Theoretical Computer

Science 6: 93 - 107

[14] Kojima M, Mizuno S and Noma T. A polynomial-time algorithm for a class of linear comple-mentarity problems, to appear in Mathematical Programming

[15] Khachyan LG 1979) A polynomial algorithm in linear programming in Russian),Doklady

Akademi ia Nauk SSSR 244:1093-1096; English translation: Soviet Mathematics Doklady 20:191-194

[16] Koehler GJ 1989) Characterization of unacceptable solutions in LP discriminant analysis.Decision Sciences 20: 239-257

[17] Koehler GJ and Erenguc SS 1990) Minimizing misclassifications in linear discriminantanalysis.Decision Sciences 21:63-85

[18] Konno H 1976) Maximization of a convex quadratic function subject to linear constraints.Mathematical Programming 11 : 117-127



No v e l Ap p r o a c h es t o t h e D i s c r im in a t i o n P r o b l e m 5 45

[19] L in YY a nd Pan g JS (1987) I te ra t ive me thods fo r large convex quadra t ic p rograms: a su rvey .

S I M Journal on Control and Optimization 2 5 :3 8 3 - 4 1 1

[20] M arkow sk i CA an d M arkow sk i EP (1987) An exper imen ta l com par ison o f severa l approaches

to t h e d i s c r im in a n tp rob le m w i th bo th qua l i ta t ive and quan t i ta t ive var iab les . European Journalo f O perational Research 2 8 : 7 4 - 7 8

[ 2 1 ] Ru b in P A ( 19 8 9) E v a lu a t in g t he m a x im iz e m in im u m d i s ta n c e f o r m u la t i o n o f t h e l i n e a r

d i s c r im in a n t p r o b le m . European Journal o f Operational Research 4 1 :2 4 0 - 2 4 8

[22] S tam A a nd Joach im stha le r EA (1990) A com par ison o f a rob us t mixed- in teger approach to

exist ing m etho ds for e stablishing classif ication rules for the disc r im inan t problem . European

Journal o f O perational Research 4 6 : 1 1 3 - 2 2

[23] Wolfe P (1976) Fin din g he nearest point in a polytope. Mathem atical Programming 11 : 128-14 9

Received: Fe bru ary 1991

Revised ve rsion received: Octo ber 1991

Documents

Novel Approaches Discrimination