A mixture of experts network structure for modelling Doppler ultrasound blood flow signals

Computers in Biology and Medicine 35 (2005) 565–582http://www.intl.elsevierhealth.com/journals/cobm

A mixture of experts network structure for modelling Dopplerultrasound blood *ow signals

Inan G-uler∗, Elif Derya -Ubeyl2Department of Electronics and Computer Education, Faculty of Technical Education, Gazi University,

06500 Teknikokullar, Ankara, Turkey

Received 12 January 2004; accepted 13 April 2004

Abstract

Mixture of experts (ME) is a modular neural network architecture for supervised learning. This paper illus-trates the use of ME network structure to guide modelling Doppler ultrasound blood *ow signals. Expectation–Maximization (EM) algorithm was used for training the ME so that the learning process is decoupled in amanner that ;ts well with the modular structure. The ophthalmic and internal carotid arterial Doppler signalswere decomposed into time–frequency representations using discrete wavelet transform and statistical featureswere calculated to depict their distribution. The ME network structures were implemented for diagnosis ofophthalmic and internal carotid arterial disorders using the statistical features as inputs. To improve diagnosticaccuracy, the outputs of expert networks were combined by a gating network simultaneously trained in orderto stochastically select the expert that is performing the best at solving the problem. The ME network structureachieved accuracy rates which were higher than that of the stand-alone neural network models.? 2004 Elsevier Ltd. All rights reserved.

Keywords: Mixture of experts; Expectation–Maximization algorithm; Diagnostic accuracy; Discrete wavelet transform;Doppler signal; Ophthalmic artery; Internal carotid artery

1. Introduction

There have recently been widespread interests in the use of multiple models for pattern classi;-cation and regression in statistics and neural network communities. The basic idea underlying thesemethods is the application of a so-called divide-and-conquer principle that is often used to tacklea complex problem by dividing it into simpler problems whose solutions can be combined to yield

∗ Corresponding author. Tel.: +90-312-212-3976; fax: +90-312-212-0059.E-mail address: [email protected] (I. G-uler).

0010-4825/$ - see front matter ? 2004 Elsevier Ltd. All rights reserved.doi:10.1016/j.compbiomed.2004.04.001

mailto:[email protected]

566 I. G'uler, E.D. 'Ubeyl* / Computers in Biology and Medicine 35 (2005) 565–582

a ;nal solution. Utilizing this principle, Jacobs et al. [1] proposed a modular neural network archi-tecture called mixture of experts (ME). The ME models the conditional probability density of thetarget output by mixing the outputs from a set of local experts, each of which separately derivesa conditional probability density of the target output. The ME weights the input space by usingthe posterior probabilities that expert networks generated for getting the output from the input. Theoutputs of expert networks are combined by a gating network simultaneously trained in order tostochastically select the expert that is performing the best at solving the problem [2,3]. As pointedout by Jordan and Jacobs [4], the gating network performs a typical multiclass classi;cation task[5–7].Expectation–Maximization (EM) algorithm have been introduced to the ME architecture so that

the learning process is decoupled in a manner that ;ts well with the modular structure [2–4]. TheEM algorithm can be extended to provide an eFective training mechanism for the MEs based ona Gaussian probability assumption. Though originally the model structure is predetermined and thetraining algorithm is based on the Gaussian probability assumption for each expert model output, theME framework is a powerful concept that can be extended to a wide variety of applications includingmedical diagnostic decision support system applications due to numerous inherent advantages suchas (i) a global model can be decomposed into a set of simple local models, from which controllerdesign is straightforward. Each model can represent a diFerent data source with an associated stateestimator/predictor. In this case, the ME system can be viewed as a data fusion algorithm. (ii) Thelocal models operate independently but provide output correlated information that can be stronglycorrelated with each other, so that the overall system performance can be enhanced in terms ofreliability or fault tolerance. (iii) The global output of the ME system is derived as a convex com-bination of the outputs from a set of N experts, in which the overall system predictive performanceis generally superior to any of the individual experts [2–5].Neural networks have been successfully used in a variety of medical applications [8,9]. Recent

advances in the ;eld of neural networks have made them attractive for analyzing signals. Theapplication of neural networks has opened a new area for solving problems not resolvable by othersignal processing techniques [10,11]. However, neural network analysis of Doppler shift signals isa relatively new approach [12–14]. Doppler ultrasound is widely used as a noninvasive method forthe assessment of blood *ow both in the central and peripheral circulation [15,16]. It may be usedto estimate blood *ow, to image regions of blood *ow and to locate sites of arterial disease as wellas *ow characteristics and resistance of ophthalmic and internal carotid arteries [17–21].Up to now, there is no study in the literature relating to the assessment of ME accuracy in analysis

of Doppler shift signals. In this study, experimental results on ME predictions for diagnosis ofophthalmic arterial diseases and internal carotid arterial diseases were presented. In the con;gurationof ME for the diagnosis of ophthalmic arterial disorders, we used four local experts and a gatingnetwork, which were in the form of multilayer perceptron neural networks (MLPNNs), since therewere four possible outcomes of the diagnosis of ophthalmic arterial conditions (healthy, ophthalmicartery stenosis, ocular Behcet disease, uveitis disease). In the development of ME for the diagnosisof internal carotid arterial disorders, we used three local experts and a gating network, which werein the form of MLPNNs, because there were three possible outcomes of the diagnosis of internalcarotid arterial conditions (healthy, internal carotid artery stenosis, internal carotid artery occlusion).We were able to achieve signi;cant improvement in accuracy by using the ME network structurescompared to the stand-alone neural networks used in our previous studies [13,14].

I. G'uler, E.D. 'Ubeyl* / Computers in Biology and Medicine 35 (2005) 565–582 567

The outline of this study is as follows. In Section 2, we explain spectral analysis of signalsusing discrete wavelet transform (DWT) in order to extract features characterizing the behavior ofthe signal under study. In Section 3, we present description of neural network models includingMLPNN and ME architecture used in this study. We also explain EM algorithm used for trainingthe ME networks. In Section 4, we present the application results of ME networks to ophthalmicand internal carotid arterial Doppler signals. Finally, in Section 5 we conclude the study.

2. Spectral analysis using discrete wavelet transform

The Doppler shift signal contains a wealth of information about blood *ow occurring within thesample volume of the Doppler ultrasonography. The most complete way to display this informationis to perform spectral analysis. The wavelet transform (WT) provides very general techniques whichcan be applied to many tasks in signal processing. One very important application is the abilityto compute and manipulate data in compressed parameters which are often called features [22–24].Thus, the Doppler signal, consisting of many data points, can be compressed into a few parameters.These parameters characterize the behavior of the Doppler signal. This feature of using a smallernumber of parameters to represent the Doppler signal is particularly important for recognition anddiagnostic purposes. The WT can be thought of as an extension of the classic Fourier transform,except that, instead of working on a single scale (time or frequency), it works on a multi-scalebasis. This multi-scale feature of the WT allows the decomposition of a signal into a number ofscales, each scale representing a particular coarseness of the signal under study. The procedureof multiresolution decomposition of a signal x[n] is schematically shown in Fig. 1. Each stage ofthis scheme consists of two digital ;lters and two downsamplers by 2. The ;rst ;lter, g[ · ] is thediscrete mother wavelet, high-pass in nature, and the second, h[ · ] is its mirror version, low-passin nature. The downsampled outputs of ;rst high-pass and low-pass ;lters provide the detail, D1and the approximation, A1, respectively. The ;rst approximation, A1 is further decomposed and thisprocess is continued as shown in Fig. 1.All wavelet transforms can be speci;ed in terms of a low-pass ;lter h, which satis;es the standard

quadrature mirror ;lter condition:

H (z)H (z−1) + H (−z)H (−z−1) = 1; (1)

where H (z) denotes the z-transform of the ;lter h. Its complementary high-pass ;lter can be de;nedas

G(z) = zH (−z−1): (2)

A sequence of ;lters with increasing length (indexed by i) can be obtained:

Hi+1(z) = H (z2i)Hi(z);

Gi+1(z) = G(z2i)Hi(z); i = 0; : : : ; I − 1 (3)

with the initial condition H0(z) = 1. It is expressed as a two-scale relation in time domain

hi+1(k) = [h]↑2i ∗ hi(k);

gi+1(k) = [g]↑2i ∗ hi(k); (4)


x[n]

g[n]

h[n] A1

D1

g[n]

h[n]A2

D2

g[n]

h[n]A3

D32

2

2

2

2

2

Fig. 1. Subband decomposition of discrete wavelet transform implementation; g[n] is the high-pass ;lter, h[n] is thelow-pass ;lter.

where the subscript [ · ]↑m indicates the up-sampling by a factor of m and k is the equally sampleddiscrete time.The normalized wavelet and scale basis functions ’i;l(k), i; l(k) can be de;ned as

’i;l(k) = 2i=2hi(k − 2il);

i; l(k) = 2i=2gi(k − 2il); (5)

where the factor 2i=2 is an inner product normalization, i and l are the scale parameter and thetranslation parameter, respectively. The DWT decomposition can be described as

a(i)(l) = x(k) ∗ ’i;l(k);

d(i)(l) = x(k) ∗ i; l(k); (6)

where a(i)(l) and di(l) are the approximation coeNcients and the detail coeNcients at resolution i,respectively [22–24].

3. Description of neural network models

3.1. Multilayer perceptron neural network

Arti;cial neural networks (ANNs) may be de;ned as structures comprised of densely intercon-nected adaptive simple processing elements (neurons) that are capable of performing massivelyparallel computations for data processing and knowledge representation. ANNs can be trained torecognize patterns and the nonlinear models developed during training allow neural networks to gen-eralize their conclusions and to make application to patterns not previously encountered [10,11,25].The MLPNNs are the most commonly used neural network arhitectures since they have featuressuch as the ability to learn and generalize, smaller training set requirements, fast operation, ease ofimplementation. A MLPNN consists of (i) an input layer with neurons representing input variablesto the problem, (ii) an output layer with neurons representing the dependent variables (what is beingmodeled), and (iii) one or more hidden layers containing neurons to help capture the nonlinearity


Expert Network

1

Expert Network

N

O(x)

X X

Gating Network

X

Fig. 2. The architecture of mixture of experts.

in the data. The MLPNN is a nonparametric technique for performing a wide variety of detectionand estimation tasks [10,11,25].

3.2. Mixture of experts and expectation–maximization algorithm

As illustrated in Fig. 2, the ME architecture is composed of a gating network and several expertnetworks. The gating network receives the vector x as input and produces scalar outputs that arepartition of unity at each point in the input space. Each expert network produces an output vector foran input vector. The gating network provides linear combination coeNcients as veridical probabilitiesfor expert networks and, therefore, the ;nal output of the ME architecture is a convex weighted sumof all the output vectors produced by expert networks. Suppose that there are N expert networks inthe ME architecture. All the expert networks are linear with a single output nonlinearity that is alsoreferred to as “generalized linear”. The ith expert network produces its output oi(x) as a generalizedlinear function of the input x:

oi(x) = f(Wix); (7)

where Wi is a weight matrix and f(·) is a ;xed continuous nonlinearity. The gating network is alsogeneralized linear function, and its ith output, g(x; vi), is the multinomial logit or softmax functionof intermediate variables �i:

g(x; vi) =e�i

∑Nk=1 e

�k; (8)

where �i = vTi x and vi is a weight vector. The overall output o(x) of the ME architecture is

o(x) =N∑

k=1

g(x; vk)ok(x): (9)

The ME architecture can be given a probabilistic interpretation. For an input–output pair (x; y), thevalues of g(vi; x) are interpreted as the multinomial probabilities associated with the decision that


terminates in a regressive process that maps x to y. Once the decision has been made, resulting ina choice of regressive process i, the output y is then chosen from a probability density P(y|x;Wi),where Wi denotes the set of parameters or weight matrix of the i th expert network in the model.Therefore, the total probability of generating y from x is the mixture of the probabilities of generatingy from each component densities, where the mixing proportions are multinomial probabilities:

P(y|x; �) =N∑

k=1

g(x; vk)P(y|x;Wk); (10)

where � is the set of all the parameters including both expert and gating network parameters. More-over, the probabilistic component of the model is generally assumed to be a Gaussian distribution inthe case of regression, a Bernoulli distribution in the case of binary classi;cation, and a multinomialdistribution in the case of multiclass classi;cation [1–3].Based on the probabilistic model in Eq. (10), learning in the ME architecture is treated as a

maximum likelihood problem. Jordan and Jacobs [4] have proposed an EM algorithm for adjustingthe parameters of the architecture. Suppose that the training set is given as � = {(xt ; yt)}T

t=1. TheEM algorithm consists of two steps. For the sth epoch, the posterior probabilities h(t)i (i = 1; : : : ; N ),which can be interpreted as the probabilities P(i|xt ; yt), are computed in the E-step as

h(t)i =g(xt ; v

(s)i )P(yt|xt ;W

(s)i )∑N

k=1 g(xt ; v(s)k )P(yt|xt ;W

(s)k )

: (11)

The M-step solves the following maximization problems:

W(s+1)i = argmax

Wi

T∑

t=1

h(t)i logP(yt|xt ;Wi) (12)

and

V (s+1) = argmaxV

T∑

t=1

N∑

k=1

h(t)k log g(xt ; vk); (13)

where V is the set of all the parameters in the gating network. Therefore, the EM algorithm issummarized as

1. For each data pair (xt ; yt), compute the posterior probabilities h(t)i using the current values of theparameters.

2. For each expert network i, solve a maximization problem in Eq. (12) with observations{(xt ; yt)}T

t=1 and observation weights {h(t)i }Tt=1.

3. For the gating network, solve the maximization problem in Eq. (13) with observations{(xt; h(t)k )}T

t=1.4. Iterate by using the updated parameter values.

In this framework, a number of relatively small expert networks can be used together with a gatingnetwork designed to divide the global classi;cation task into simpler subtasks (Fig. 2) [1–4]. In thepresent study, both the gating and expert networks were MLPNNs consisting of neurons arranged incontiguous layers. This con;guration occured on the theory that MLPNN has features such as the


ability to learn and generalize, smaller training set requirements, fast operation, ease of implemen-tation. The ME network structure proposed for the diagnosis of ophthalmic arterial disorders andinternal carotid arterial disorders were implemented by using MATLAB software package (MATLABversion 6.5 with neural networks toolbox).

4. Experimental results

4.1. Feature extraction using discrete wavelet transform

Diagnosis of arterial diseases is feasible by analysis of spectral shape and parameters. Since *owin arteries is pulsatile and the moving targets have a random spatial distribution, the Doppler signal istime-varying and random. It is known that the WT is better suited to analyzing nonstationary signals,since it is well localized in time and frequency. The property of time and frequency localizationis known as compact support and is one of the most attractive features of the WT. The mainadvantage of the WT is that it has a varying window size, being broad at low frequencies andnarrow at high frequencies, thus leading to an optimal time–frequency resolution in all frequencyranges [22–24]. Therefore, spectral analysis of the ophthalmic and internal carotid arterial Dopplersignals was performed using the DWT as described in Section 2.Selection of appropriate wavelet and the number of decomposition levels is very important in

analysis of signals using the WT. The number of decomposition levels is chosen based on thedominant frequency components of the signal. The levels are chosen such that those parts of thesignal that correlate well with the frequencies required for classi;cation of the signal are retainedin the wavelet coeNcients. In the present study, since the Doppler signals do not have any usefulfrequency components below 40 Hz, the number of decomposition levels was chosen to be 7. Thus,the ophthalmic and internal carotid arterial Doppler signals were decomposed into details D1–D7 andone ;nal approximation, A7. The ranges of various frequency bands are given in Table 1. Usually,tests are performed with diFerent types of wavelets and the one which gives maximum eNciencyis selected for the particular application. The smoothing feature of the Daubechies wavelet of order1 (db1) made it more suitable to detect changes of arterial Doppler signals. Therefore, the waveletcoeNcients were computed using the db1 in the present study. In order to investigate the eFect ofother wavelets on classi;cations accuracy, tests were carried out using other wavelets also. Apartfrom db1, Symmlet of order 10 (sym10), Coi*et of order 4 (coif4), and Daubechies of order 8 (db8)were also tried. It was seen that the Daubechies wavelet oFers better accuracy than the others, anddb1 is marginally better than db8. The discrete wavelet coeNcients were computed using MATLABsoftware package.Feature selection is an important component of designing the neural network based on pattern

classi;cation since even the best classi;er will perform poorly if the features used as inputs arenot selected well. The computed discrete wavelet coeNcients provide a compact representation thatshows the energy distribution of the signal in time and frequency. Therefore, the computed detailwavelet coeNcients of the ophthalmic and internal carotid arterial Doppler signals of each subjectwere used as the feature vectors representing the signals. It was observed that the values of thecoeNcients are very close to zero in A7. So the coeNcients corresponding to the frequency band,A7 were discarded, thus reducing the number of feature vectors representing the signal. In order


Table 1Ranges of frequency bands in wavelet decomposition

Decomposed signal Frequency range (Hz)

D1 2500–5000D2 1250–2500D3 625–1250D4 312.5–625D5 156.25–312.5D6 78.13–156.25D7 39.07–78.13A7 0–39.07

to further reduce the dimensionality of the extracted feature vectors, statistics over the set of thewavelet coeNcients was used. The following statistical features were used to represent the time–frequency distribution of the Doppler signals:

1. mean of the absolute values of the coeNcients in each subband;2. maximum of the absolute values of the coeNcients in each subband;3. average power of the wavelet coeNcients in each subband;4. standard deviation of the coeNcients in each subband;5. ratio of the absolute mean values of adjacent subbands;6. distribution distortion of the coeNcients in each subband.

Features 1–3 represent the frequency distribution of the signal and the features 4–6 the amount ofchanges in frequency distribution. These feature vectors, calculated for the frequency bands D1–D7, were used for classi;cation of the ophthalmic and internal carotid arterial Doppler signals. Insome applications, in order to further reduce the dimensionality of the extracted feature vectors, onlysome of the statistical features given in this section can be used to represent the time–frequencydistribution of the signal under study. However, in our applications all of the statistical features(6 statistical features) were used to represent the ophthalmic and internal carotid arterial Dopplersignals.

4.2. Application of mixture of experts to ophthalmic arterial Doppler signals

The ophthalmic arterial Doppler signals were obtained from 214 subjects. The group consisted of103 females and 111 males with ages ranging from 19 to 65 years and a mean age of 33.5 years(standard deviation-SD 9.8). Diasonics Synergy color Doppler ultrasonography was used duringexaminations and sonograms were taken into consideration. According to the examination results,52 of 214 subjects suFered from ophthalmic artery stenosis, 54 of them suFered from ocular Behcetdisease, 45 of them suFered from uveitis disease, and the rest were healthy subjects (control group)who had no ocular or systemic disease. The group suFering from ophthalmic artery stenosis consistedof 25 females and 27 males with a mean age 35.5 years (SD 8.3, range 23–65), the group suFeringfrom ocular Behcet disease consisted of 25 females and 29 males with a mean age 35.5 years (SD


8.5, range 21–63), the group suFering from uveitis disease consisted of 22 females and 23 maleswith a mean age 34.5 years (SD 8.1, range 22–62), and the healthy subjects were 31 females and32 males with a mean age 30.0 years (SD 9.3, range 19–64).Ophthalmic artery examinations were performed with a Doppler unit using a 10 MHz ultrasonic

transducer. The measurement system consisted of ;ve units. These were 10 MHz ultrasonic trans-ducer, analog Doppler unit (Diasonics Synergy), recorder (Sony), analog/digital interface board(Sound Blaster Pro-16 bit), and a personal computer with a printer. The ultrasonic transducer wasapplied on a horizontal plane to the closed eyelids using sterile methylcellulose as a coupling gel.Care was taken not to apply pressure to the eye in order to avoid artifacts. The probe was most oftenplaced at an angle of 60◦ from the midline pointing towards the orbital apex. Good and consistentsignals were obtained at 37–42 mm depth. The ophthalmic arterial Doppler signals were sampled in5 kHz and framed by equal time intervals. The frame length was chosen as 256.The ME architecture used for the diagnosis of ophthalmic arterial disorders is shown in Fig. 2.

Since we investigated four-group classi;cation exclusively, the ME was con;gured with four localexperts and a gating network which were in the form of MLPNNs. ANN architectures are derived bytrial and error and the complexity of the neural network is characterized by the number of hiddenlayers. There is no general rule for selection of appropriate number of hidden layers. A neuralnetwork with a small number of neurons may not be suNciently powerful to model a complexfunction. On the other hand, a neural network with too many neurons may lead to over;tting thetraining sets and lose its ability to generalize which is the main desired characteristic of a neuralnetwork. The most popular approach to ;nding the optimal number of hidden layers is by trial anderror. Our architecture studies con;rmed that for the ophthalmic arterial Doppler signals, a minimalnetwork has better generalization properties and results in higher classi;cation accuracy. For thisdata, MLPNNs with one hidden layer were superior to models with two hidden layers. The mostsuitable network con;guration found was 10 neurons for the hidden layer and the number of outputwas 4. Samples with target outputs healthy, ophthalmic artery stenosis, ocular Behcet disease, anduveitis disease were given the binary target values of (0; 0; 0; 1), (0; 0; 1; 0), (0; 1; 0; 0), and (1; 0; 0; 0),respectively.The adequate functioning of neural networks depends on the sizes of the training set and test set.

In the ME, 80 of 214 subjects were used for training and the rest for testing. A practical way to;nd a point of better generalization is to use a small percentage (around 20%) of the training setfor cross validation. For obtaining a better network generalization 16 training subjects were selectedrandomly to be used as a cross validation set. The training set consisted of 20 subjects suFering fromophthalmic artery stenosis, 20 subjects suFering from ocular Behcet disease, 20 subjects suFeringfrom uveitis disease, and 20 healthy subjects. The testing set consisted of 32 subjects suFering fromophthalmic artery stenosis, 34 subjects suFering from ocular Behcet disease, 25 subjects suFeringfrom uveitis disease, and 43 healthy subjects. The cross validation set consisted of 4 subjects suFeringfrom ophthalmic artery stenosis, 4 subjects suFering from ocular Behcet disease, 4 subjects suFeringfrom uveitis disease, and 4 healthy subjects.The computed discrete wavelet coeNcients were used as the inputs of the MLPNNs employed in

the architecture of ME. In order to extract features, the wavelet coeNcients corresponding to theD1–D7 frequency bands of the ophthalmic arterial Doppler signals were computed. For each oph-thalmic arterial Doppler signal frame (256 samples), the detail wavelet coeNcients (dk; k =1; 2; 3; 4;5; 6; 7) at the ;rst, second, third, fourth, ;fth, sixth and seventh levels (128 + 64 + 32 + 16 +


0 20 40 60 80 100 120 140-40

-30

-20

-10

0

10

20

30

40

50

Number of detail wavelet coefficients

Det

ail w

avel

et c

oeffi

cien

ts

0 20 40 60 80 100 120 140-20

-15

-10

-5

0

5

10

15


Det

ail w

avel

et c

oeffi

cien

ts

0 20 40 60 80 100 120 140-10

-8

-6

-4

-2

0

2

4

6

8

10


Det

ail w

avel

et c

oeffi

cien

ts

0 20 40 60 80 100 120 140-6

-5

-4

-3

-2

-1

0

1

2

3


Det

ail w

avel

et c

oeffi

cien

ts

(a) (b)

(c) (d)

Fig. 3. The detail wavelet coeNcients corresponding to the D1 frequency band of the ophthalmic arterial Doppler signalsrecorded from: (a) 35-year-old healthy subject (subject no: 12), (b) 29-year-old subject suFering from ophthalmic arterystenosis (subject no: 17), (c) 37-year-old subject suFering from ocular Behcet disease (subject no: 35), (d) 34-year-oldsubject suFering from uveitis disease (subject no: 41).

8 + 4 + 2 coeNcients) were computed. Then 254 detail wavelet coeNcients were obtained for eachophthalmic arterial Doppler signal frame. In order to reduce the dimensionality of the extracted fea-ture vectors, statistics explained in Section 4.1 over the set of the wavelet coeNcients was used. Thenthe MLPNNs had 41 inputs, equal to the number of input feature vectors. The detail wavelet coef-;cients corresponding to the D1 frequency band of the ophthalmic arterial Doppler signals obtainedfrom 35-year-old healthy subject (subject no: 12), 29-year-old subject suFering from ophthalmicartery stenosis (subject no: 17), 37-year-old subject suFering from ocular Behcet disease (subjectno: 35), and 34-year-old subject suFering from uveitis disease (subject no: 41) are given in Figs.3(a)–(d), respectively. It can be noted that the detail wavelet coeNcients of the ophthalmic arterialDoppler signals obtained from a healthy subject (Fig. 3(a)) and subjects suFering from ophthalmicarterial diseases (Figs. 3(b)–(d)) are diFerent from each other.The training holds the key to an accurate solution, so the criterion to stop training must be very

well described. When the network is trained too much, the network memorizes the training patternsand does not generalize well. Cross validation is a highly recommended criterion for stopping thetraining of a network. When the error in the cross validation increases, the training should bestopped because the point of best generalization has been reached. Training of the ME was donein 400 epochs since the cross validation errors began to rise at 400 epochs. Owing to the valuesof MSE converged to small constants approximately zero in 400 epochs, training of the ME wasdetermined to be successful. However, in our previous study [13] the stand-alone MLPNN trained


with the least-mean squares backpropagation algorithm had a slow convergence and MSE convergedto a small constant of approximately zero in 3000 epochs. The backpropagation algorithm searchedglobal optimal solution for the classi;cation problem so that the number of epochs required forconvergence increased. However, in the ME the classi;cation problem was divided into simplerproblems and then each solution was combined. In addition to this, the training algorithm of the MEis a general technique for maximum likelihood estimation that ;ts well with the modular structureand enables a signi;cant speed up over the backpropagation algorithm. Thus, the convergence rateof ME presented in this study was found to be higher than that of the stand-alone MLPNN used inthe previous study [13].In classi;cation, the aim is to assign the input patterns to one of several classes, usually represented

by outputs restricted to lie in the range from 0 to 1, so that they represent the probability of classmembership. While the classi;cation is carried out, a speci;c pattern is assigned to a speci;c classaccording to the characteristic features selected for it. In this application, there were four classes:healthy, ophthalmic artery stenosis, ocular Behcet disease, uveitis disease. Classi;cation results of theME were displayed by a confusion matrix. The confusion matrix showing the classi;cation resultsof the ME is given below.

Confusion matrix

Output/desired Result Result Result Result (uveitis disease)(healthy) (ophthalmic (ocular Behcet

artery stenosis) disease)

Result 42 0 0 0(healthy)Result 1 31 1 0(ophthalmic artery stenosis)Result 0 1 32 1(ocular Behcet disease)Result 0 0 1 24(uveitis disease)

According to the confusion matrix, one healthy subject was classi;ed incorrectly by the ME asa subject suFering from ophthalmic artery stenosis, one subject suFering from ophthalmic arterystenosis was classi;ed as a subject suFering from ocular Behcet disease, one subject suFering fromocular Behcet disease was classi;ed as a subject suFering from ophthalmic artery stenosis, one subjectsuFering from ocular Behcet disease was classi;ed as a subject suFering from uveitis disease, andone subject suFering from uveitis disease was classi;ed as a subject suFering from ocular Behcetdisease.The test performance of the ME was determined by the computation of the following statistical

parameters:Speci9city: number of correct classi;ed healthy subjects/number of total healthy subjects.Sensitivity (ophthalmic artery stenosis): number of correct classi;ed subjects suFering from

ophthalmic artery stenosis/number of total subjects suFering from ophthalmic artery stenosis.


Table 2The values of statistical parameters of the ME used for the diagnosis of ophthalmic arterial disorders

Statistical parameters Values

Speci;city 97.67%Sensitivity (ophthalmic artery stenosis) 96.88%Sensitivity (ocular Behcet disease) 94.12%Sensitivity (uveitis disease) 96.00%Total classi;cation accuracy 96.27%

Sensitivity (ocular Behcet disease): number of correct classi;ed subjects suFering from ocularBehcet disease/number of total subjects suFering from ocular Behcet disease.Sensitivity (uveitis disease): number of correct classi;ed subjects suFering from uveitis

disease/number of total subjects suFering from uveitis disease.Total classi9cation accuracy: number of correct classi;ed subjects/number of total subjects.The values of these statistical parameters are given in Table 2. As it is seen from Table 2, the

ME classi;ed healthy subjects, subjects suFering from ophthalmic artery stenosis, subjects suFeringfrom ocular Behcet disease and subjects suFering from uveitis disease with the accuracy of 97.67%,96.88%, 94.12% and 96.00%, respectively. The healthy subjects, subjects suFering from ophthalmicartery stenosis, subjects suFering from ocular Behcet disease and subjects suFering from uveitisdisease were classi;ed with the accuracy of 96.27%. The correct classi;cation rates of the stand-aloneMLPNN presented in our previous study [13] were 90.63% for healthy subjects and 88.89% forsubjects having ophthalmic artery stenosis. Thus, the accuracy rates of the ME network structurepresented for this application were found to be higher than that of the stand-alone MLPNN used inthe previous study [13].The performance of a test can be evaluated by plotting a ROC curve for the test. For a given

result obtained by a classi;er system, four possible alternatives exist that describe the nature of theresult: (i) true positive (TP), (ii) false positive (FP), (iii) true negative (TN), and (iv) false negative(FN) [26]. In this study, a TP decision occured when the positive detection of the ME coincidedwith a positive detection of the physician. A FP decision occured when the ME made a positivedetection that did not agree with the physician. A TN decision occured when both the ME and thephysician suggested the absence of a positive detection. A FN decision occured when the ME madea negative detection that did not agree with the physician. A good test is one for which sensitivityrises rapidly and 1-speci;city hardly increases at all until sensitivity becomes high. ROC curveswhich are shown in Fig. 4 represent performances of the stand-alone MLPNN and ME networkstructure on the ophthalmic arterial Doppler signals test ;le. Fig. 4 shows that the performance ofthe ME is higher than that of the stand-alone MLPNN.

4.3. Application of mixture of experts to internal carotid arterial Doppler signals

The internal carotid arterial Doppler signals were obtained from 160 subjects. The group consistedof 78 females and 82 males with ages ranging from 18 to 67 years and a mean age of 32.0 years(SD 9.6). Toshiba 140A color Doppler ultrasonography was used during examinations and sonograms


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1-Specificity

Sen

sitiv

ityMEMLPNN

Fig. 4. ROC curves of the stand-alone MLPNN and ME network structure used for the diagnosis of ophthalmic arterialdisorders.

were taken into consideration. According to the examination results, 59 of 160 subjects suFered frominternal carotid artery stenosis, 53 of them suFered from internal carotid artery occlusion, and therest were healthy subjects (control group) who had no arterial disease. The group suFering frominternal carotid artery stenosis consisted of 26 females and 33 males with a mean age 33.0 years(SD 8.6, range 21–67), the group suFering from internal carotid artery occlusion consisted of 29females and 24 males with a mean age 32.5 years (SD 8.5, range 20–65), and the healthy subjectswere 23 females and 25 males with a mean age 31.5 years (SD 9.4, range 18–65).Internal carotid artery examinations were performed with a Doppler unit using a 5 MHz ultrasonic

transducer. The measurement system consisted of ;ve units. These were 5 MHz ultrasonic transducer,analog Doppler unit (Toshiba 140A color Doppler ultrasonography), recorder (Sony), analog/digitalinterface board (Sound Blaster Pro-16 bit), a personal computer with a printer. The ultrasonic trans-ducer was applied on a horizontal plane to the neck using water-soluble gel as a coupling gel. Carewas taken not to apply pressure to the neck in order to avoid artifacts. The probe was most oftenplaced at an angle of 60◦ towards the internal carotid artery. The internal carotid arterial Dopplersignals were sampled in 5 kHz and framed by equal time intervals. The frame length was chosen as256.The ME architecture used for the diagnosis of internal carotid arterial disorders is shown in Fig.

2. Since we investigated three-group classi;cation exclusively, the ME was con;gured with threelocal experts and a gating network which were in the form of MLPNNs. For this data, MLPNNswith one hidden layer were superior to models with two hidden layers. The most suitable networkcon;guration found was 10 neurons for the hidden layer and the number of output was 3. Sampleswith target outputs healthy, internal carotid artery stenosis, and internal carotid artery occlusion weregiven the binary target values of (0; 0; 1), (0; 1; 0), and (1; 0; 0; ), respectively.In the ME, 60 of 160 subjects were used for training and the rest for testing. For obtaining

a better network generalization 12 training subjects were selected randomly to be used as a crossvalidation set. The training set consisted of 20 subjects suFering from internal carotid artery stenosis,


0 20 40 60 80 100 120 140-60

-40

-20

0

20

40

60

80


Det

ail w

avel

et c

oeffi

cien

ts

0 20 40 60 80 100 120

-100

-150

-50

0

50

100

150


Det

ail w

avel

et c

oeffi

cien

ts

0 20 40 60 80 100 120 140-60

-50

-40

-30

-20

-10

0

10

20

30

40


Det

ail w

avel

et c

oeffi

cien

ts

(a) (b)

(c)

Fig. 5. The detail wavelet coeNcients corresponding to the D1 frequency band of the internal carotid arterial Dopplersignals recorded from: (a) 33-year-old healthy subject (subject no: 10), (b) 35-year-old subject suFering from internalcarotid artery stenosis (subject no: 23), (c) 36-year-old subject suFering from internal carotid artery occlusion (subjectno: 28).

20 subjects suFering from internal carotid artery occlusion, and 20 healthy subjects. The testing setconsisted of 39 subjects suFering from internal carotid artery stenosis, 33 subjects suFering frominternal carotid artery occlusion, and 28 healthy subjects. The cross validation set consisted of foursubjects suFering from internal carotid artery stenosis, four subjects suFering from internal carotidartery occlusion, and four healthy subjects.The computed discrete wavelet coeNcients were used as the inputs of the MLPNNs employed

in the architecture of ME. In order to extract features, the wavelet coeNcients corresponding tothe D1–D7 frequency bands of the internal carotid arterial Doppler signals were computed. Foreach internal carotid arterial Doppler signal frame (256 samples), the detail wavelet coeNcients(dk; k = 1; 2; 3; 4; 5; 6; 7) at the ;rst, second, third, fourth, ;fth, sixth and seventh levels (128 + 64 +32+16+8+4+2 coeNcients) were computed. Then 254 detail wavelet coeNcients were obtainedfor each internal carotid arterial Doppler signal frame. In order to reduce the dimensionality of theextracted feature vectors, statistics explained in Section 4.1 over the set of the wavelet coeNcientswas used. Then the MLPNNs had 41 inputs, equal to the number of input feature vectors. The detailwavelet coeNcients corresponding to the D1 frequency band of the internal carotid arterial Dopplersignals obtained from 33-year-old healthy subject (subject no: 10), 35-year-old subject suFeringfrom internal carotid artery stenosis (subject no: 23), and 36-year-old subject suFering from internalcarotid artery occlusion (subject no: 28) are given in Figs. 5(a)–(c), respectively. It can be noted


that the detail wavelet coeNcients of the internal carotid arterial Doppler signals obtained from ahealthy subject (Fig. 5(a)) and subjects suFering from internal carotid arterial diseases (Figs. 5(b)and (c)) are diFerent from each other.Training of the ME was done in 300 epochs since the cross validation errors began to rise at

300 epochs. Owing to the values of MSE converged to small constants approximately zero in 300epochs, training of the ME was determined to be successful. However, in our previous study [14]the stand-alone MLPNN trained with the backpropagation algorithm had a slow convergence andMSE converged to a small constant approximately zero in 5000 epochs. Since the EM algorithmused in training the ME enabled a signi;cant speed up, the convergence rate of ME presented inthis study was found to be higher than that of the stand-alone MLPNN used in the previous study[14].In this application, there were three classes: healthy, internal carotid artery stenosis, internal carotid

artery occlusion. Classi;cation results of the ME were displayed by a confusion matrix. The confusionmatrix showing the classi;cation results of the ME is given below.

Confusion matrix

Output/desired Result Result Result(healthy) (internal carotid (internal carotid

artery stenosis) artery occlusion)

Result (healthy) 27 0 0Result 1 38 1(internal carotidartery stenosis)Result 0 1 32(internal carotidartery occlusion)

According to the confusion matrix, one healthy subject was classi;ed incorrectly by the ME asa subject suFering from internal carotid artery stenosis, one subject suFering from internal carotidartery stenosis was classi;ed as a subject suFering from internal carotid artery occlusion, and onesubject suFering from internal carotid artery occlusion was classi;ed as a subject suFering frominternal carotid artery stenosis.The test performance of the ME was determined by the computation of the following statistical

parameters:Speci9city: number of correct classi;ed healthy subjects/number of total healthy subjects.Sensitivity (internal carotid artery stenosis): number of correct classi;ed subjects suFering from

stenosis/number of total subjects suFering from stenosis.Sensitivity (internal carotid artery occlusion): number of correct classi;ed subjects suFering from

occlusion/number of total subjects suFering from occlusion.Total classi9cation accuracy: number of correct classi;ed subjects/number of total subjects.The values of these statistical parameters are given in Table 3. As it is seen from Table 3, the

ME classi;ed healthy subjects, subjects suFering from internal carotid artery stenosis, and subjects


Table 3The values of statistical parameters of the ME used for the diagnosis of internal carotid arterial disorders

Statistical parameters Values

Speci;city 96.43%Sensitivity (internal carotid artery stenosis) 97.44%Sensitivity (internal carotid artery occlusion) 96.97%Total classi;cation accuracy 97.00%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1-Specificity

Sen

sitiv

ity

MEMLPNN

Fig. 6. ROC curves of the stand-alone MLPNN and ME network structure used for the diagnosis of internal carotid arterialdisorders.

suFering from internal carotid artery occlusion with the accuracy of 96.43%, 97.44%, 96.97%,respectively. The healthy subjects, subjects suFering from internal carotid artery stenosis, and sub-jects suFering from internal carotid artery occlusion were classi;ed with the accuracy of 97.00%.The correct classi;cation rates of the stand-alone MLPNN presented in our previous study [14] were95.24% for healthy subjects, 91.30% for subjects having internal carotid artery stenosis, and 91.67%for subjects having internal carotid artery occlusion. Thus, the accuracy rates of the ME presentedfor this application were found to be higher than that of the stand-alone MLPNN in the previousstudy [14].ROC curves which are shown in Fig. 6 represent performances of the stand-alone MLPNN and

ME network structure on the internal carotid arterial Doppler signals test ;le. Fig. 6 shows that theperformance of the ME is higher than that of the stand-alone MLPNN.

5. Conclusion

This paper presented the use of ME network structures to improve diagnostic accuracy ofophthalmic and internal carotid arterial disorders since the overall structure predictive performance is


generally superior to any of the individual experts. Towards achieving the diagnosis of ophthalmicarterial conditions four local experts and a gating network, which were in the form of MLPNNs,were used in the con;guration of ME architecture. In order to diagnose internal carotid arterial con-ditions, three local experts and a gating network, which were in the form of MLPNNs, were usedin the con;guration of ME architecture. EM algorithm was used for training the ME networks sothat the learning process is decoupled in a manner that ;ts well with the modular structure. The MEused for the diagnosis of ophthalmic arterial disorders was trained, cross validated and tested withthe extracted features using DWT of the ophthalmic arterial Doppler signals obtained from healthysubjects, subjects suFering from ophthalmic artery stenosis, subjects suFering from ocular Behcet dis-ease, and subjects suFering from uveitis disease. The ME used for the diagnosis of internal carotidarterial disorders was trained, cross validated and tested with the extracted features using DWT ofthe internal carotid arterial Doppler signals obtained from healthy subjects, subjects suFering frominternal carotid artery stenosis, and subjects suFering from internal carotid artery occlusion. Theclassi;cation results, the values of statistical parameters, and ROC curves were used for evaluatingperformances of the classi;ers. The accuracy rates achieved by the ME network structures presentedfor the diagnosis of ophthalmic and internal carotid arterial disorders were found to be higher thanthat of the stand-alone neural network models used in the previous studies.

Acknowledgements

This study has been supported by the Scienti;c Research Project of Gazi University (Project no:07/2003-03).

References

[1] R.A. Jacobs, M.I. Jordan, S.J. Nowlan, G.E. Hinton, Adaptive mixtures of local experts, Neural Comput. 3 (1)(1991) 79–87.

[2] K. Chen, L. Xu, H. Chi, Improved learning algorithms for mixture of experts in multiclass classi;cation, NeuralNetworks 12 (9) (1999) 1229–1252.

[3] X. Hong, C.J. Harris, A mixture of experts network structure construction algorithm for modelling and control, Appl.Intell. 16 (1) (2002) 59–69.

[4] M.I. Jordan, R.A. Jacobs, Hierarchical mixture of experts and the EM algorithm, Neural Comput. 6 (2) (1994)181–214.

[5] P. Mangiameli, D. West, An improved neural classi;cation network for the two-group problem, Comput. Oper. Res.26 (5) (1999) 443–460.

[6] Y.H. Hu, S. Palreddy, W.J. Tompkins, A patient-adaptable ECG beat classi;er using a mixture of experts approach,IEEE Trans. Biomed. Eng. 44 (9) (1997) 891–900.

[7] G. Viardot, R. Lengelle, C. Richard, Mixture of experts for automated detection of phasic arousals in sleep signals,IEEE International Conference on Systems, Man and Cybernetics, Vol. 1, Hammamet, Tunisia, 2002, pp. 551–555.

[8] W.G. Baxt, Use of an arti;cial neural network for data analysis in clinical decision making: the diagnosis of acutecoronary occlusion, Neural Comput. 2 (1990) 480–489.

[9] A.S. Miller, B.H. Blott, T.K. Hames, Review of neural network applications in medical imaging and signal processing,Med. Biol. Eng. Comput. 30 (1992) 449–464.

[10] I.A. Basheer, M. Hajmeer, Arti;cial neural networks: fundamentals, computing, design, and application, J. Microbiol.Methods 43 (1) (2000) 3–31.


[11] B.B. Chaudhuri, U. Bhattacharya, ENcient training and improved performance of multilayer perceptron in patternclassi;cation, Neurocomputing 34 (2000) 11–27.

[12] I.A. Wright, N.A.J. Gough, F. Rakebrandt, M. Wahab, J.P. Woodcock, Neural network analysis of Doppler ultrasoundblood *ow signals: a pilot study, Ultrasound Med. Biol. 23 (5) (1997) 683–690.

[13] I. G-uler, E.D. -Ubeyli, Detection of ophthalmic artery stenosis by least-mean squares backpropagation neural network,Comput. Biol. Med. 33 (4) (2003) 333–343.

[14] E.D. -Ubeyli, I. G-uler, Neural network analysis of internal carotid arterial Doppler signals: predictions of stenosisand occlusion, Expert Systems Appl. 25 (1) (2003) 1–13.

[15] I. G-uler, N.F. G-uler, The electronic detail of a pulsed Doppler blood *ow measurement system, Meas. Sci. Technol.1 (10) (1990) 1087–1092.

[16] B. Sigel, A brief history of Doppler ultrasound in the diagnosis of peripheral vascular disease, Ultrasound Med.Biol. 24 (2) (1998) 169–176.

[17] I. G-uler, F. HardalaVc, E.D. -Ubeyli, Determination of Behcet disease with the application of FFT and AR methods,Comput. Biol. Med. 32 (6) (2002) 419–434.

[18] I. G-uler, E.D. -Ubeyli, Application of classical and model-based spectral methods to ophthalmic arterial Dopplersignals with uveitis disease, Comput. Biol. Med. 33 (6) (2003) 455–471.

[19] E.D. -Ubeyli, I. G-uler, Spectral broadening of ophthalmic arterial Doppler signals using STFT and wavelet transform,Comput. Biol. Med. 34 (4) (2004) 345–354.

[20] E.D. -Ubeyli, I. G-uler, Comparison of eigenvector methods with classical and model-based methods in analysis ofinternal carotid arterial Doppler signals, Comput. Biol. Med. 33 (6) (2003) 473–493.

[21] E.D. -Ubeyli, I. G-uler, Spectral analysis of internal carotid arterial Doppler signals using FFT, AR, MA, and ARMAmethods, Comput. Biol. Med. 34 (4) (2004) 293–306.

[22] I. Daubechies, The wavelet transform, time-frequency localization and signal analysis, IEEE Trans. Inform. Theory36 (5) (1990) 961–1005.

[23] M. Akay, Wavelet applications in medicine, IEEE Spectrum 34 (5) (1997) 50–56.[24] Y. Zhang, Y. Wang, W. Wang, B. Liu, Doppler ultrasound signal denoising based on wavelet frames, IEEE Trans.

Ultrason. Ferroelectrics, Frequency Control 48 (3) (2001) 709–716.[25] S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan, New York, 1994.[26] M.H. Zweig, G. Campbell, Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical

medicine, Clin. Chem. 39 (4) (1993) 561–577.

Inan G&uler graduated from Erciyes University in 1981. He took his M.S. degree from Middle East Technical Universityin 1985, and his Ph.D. degree from Istanbul Technical University in 1990, all in Electronic Engineering. He is a professorat Gazi University where he is Head of Department. His interest areas include biomedical systems, biomedical signalprocessing, biomedical instrumentation, electronic circuit design, neural networks, and arti;cial intelligence. He has writtenmore than 100 articles on biomedical engineering.

Elif Derya &Ubeyli graduated from CV ukurova University in 1996. She took her M.S. degree in 1998, all in electronicengineering. She took her Ph.D. degree from Gazi University, electronics and computer technology. She is a researchassistant at the Department of Electronics and Computer Education at Gazi University. Her interest areas are biomedicalsignal processing, neural networks, and arti;cial intelligence.

Documents

A mixture of experts network structure for modelling Doppler ultrasound blood flow signals