IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

A Knowledge Based System for Analysis of GatedBlood Pool Studies

HEINRICH NIEMANN, MEMBER, IEEE, HORST BUNKE, INGRID HOFMANN, GERHARD SAGERER,FRIEDRICH WOLF, AND HERBERT FEISTEL

Abstract-A system for obtaining a complete diagnostic descriptionof an image sequence taken in nuclear medicine from the human hearthas been developed, implemented, and tested. The knowledge aboutthese images is represented in a semantic net, conclusions are drawn bya production rule approach, and scoring of alternative diagnoses is basedon fuzzy membership functions. On the low level, image pixels aresmoothed and organ contours are extracted; these are the input for thehigh level processing. Tests with several image sequences gave correctdescriptions as compared to the diagnosis of a physician.

Index Terms -Expert system, fuzzy membership function, gated bloodpool studies, graph search, knowledge based system, medical imageanalysis, semantic net.

I. INTROI)UCTION

THE problem of image analysis, that is, the problem of ob-taining a symbolic description of an image or a sequence

of images. has been treated extensively in past years. It is wellknown that this requires a combination of picture processingmethods starting at the pixel level with techniques of knowl-edge representation and utilization ranging to the symboliclevel, and there are a number of articles and books related tothis field, e.g., [1] -[5] . The general goal is to obtain power-ful and flexible vision systems [6] -[8] which are suited forvarious applications. One important field of applications ismedical image processing and analysis. Medical images areroutinely examined, for example, in counting blood cells, caryo-typing chromosomes, prescreening cervical smears, evaluatingradiographic and sciintigraphic images, and inspecting micro-scopic slides of tissue. It has been reported that diagnosticaccuracy for radiographic images is about 70 percent, that is, aradiologist tends to disagree with his own diagnosis when shownthe same image again at about 15 percent [9], and that thenumber ot medical images is extremely large and steadily in-creasing. Because of these facts, computer assisted or com-pletely automated systems for description of medical imageswould be extremely useful; it will assist the physician in mak-ing a diagnosis, free him from evident routine cases, allow himto spend more time on really ambiguous cases, and it willprovide reproducible results and measurements. Besides, anautomatic system may also be useful in training medicalstudents because it has a well defined and explicit processingand decisioni strategy.

Manuscript received April 30, 1984; revised January 21, 1985. Rec-ommended for acceptance by J. Mylopoulos.H. Niemann, H. Bunke, I. Hofmann, and G. Sagerer are with Lehrstuhl

fur Informatik 5 (Mustererkennung), University Erlangen-Nurnberg,West Germany.

F.- Wolf and H. F eistel are with Institut and Poliklinik fur Nuklear-medizin, University Erlangen-Nuirnberg, West Germany.

This paper describes a system for the automatic analysis ofscintigraphic image sequences of the human heart. Scinti-graphic methods play an important role because they are non-invasive. A radionuclide is injected into the patient's blood, itdistributes through the body and thereby allows one to studythe behavior of various organs. In our case, the radionuclideTc-99m is used. It is suited for studying the beating heart, inparticular the interior of the ventricles. Images are taken by aspecial scintillation camera. The time interval between two R-waves of a ECG determines one cycle of the heart, and thiscycle is divided into n segments, 12 < n < 32, of equal length.The R-wave triggers the scintillation camera to record oneimage per segment. In order to improve the signal-to-noiseratio, images in about 500 cycles are recorded and added insynchronous segments. An image sequence represents an aver-age heart cycle, starting in the first image with maximal expan-sion (diastole) of the heart, going to maximal contraction(systole) and back to expansion. Concentration of radioactivematerial in the patient's body should be at the lowest possiblelevel, and, therefore, image quality is fairly bad (as comparedto radiographic images) and image resolution is low, typically64 X 64 pixels. The goal is to evaluate the motility of the leftventricle, to describe it using common medical terminology,and to suggest possible diagnoses.General medical knowledge about scintigraphic images is

described in [10] , [ 1] , and the rules used here for inferringdiagnostic descriptions are mainly based on this work. Generalimage processing methods are treated, e.g., in [2] , special ap-proaches and results for scintigraphic images are presented in[12], [13]. The artificial intelligence (Al) aspects of motionunderstanding in radiographic images are discussed in [14],and our system also uses the semantic net approach to knowl-edge representation. The contribution of our paper is a com-plete operational knowledge based system which starts withthe pixels of an image sequence and gives, without any humaninteraction, a symbolic description of medical evidence. Thenext section gives a general overview of the system, Section IIIdescribes the methods for low level processing, that is, smooth-ing, contour extraction, and segmentation of the left ventricle.The representation of problem specific knowledge is treated inSection IV, and its utilization for knowledge guided analysis inSection V. Results of processing several image sequences arepresented in Section VI.

II. SYSTEM OVERVIEWIn analogy to the general system structure introduced in [5],

the system consists of four main modules as indicated in Fig. 1.

0162-8828/85/0500-0246$01.00 © 1985 IEEE

246

247NIEMANN et al.: KNOWLEI)GE BASED SYSTEM FOR GATEI) BLOOD POOL STUDIES

INSTANCES(RESULTSOFPROCESSING)

LOW LEVEL METHODS KNOWLEDGEKNOWLEOOE

BASIC MODEL COMPONEN TS

|PREPROCESSINGI

NUT IMAGE SEOUENCE Il

Fig. 1. System structure for knowledge based analysis of image se-

quences in gated blood pool studies.

Problem specific knowledge about the above mentioned type

of image sequences and their possible interpretations is con-

tained in a mode' which consists of declarative and proceduralknowledge. The declarative part contains a description ofstructural properties of the heart and the left ventricle, of a

heart cycle and its motional phases, and of medical evidencewhich can be inferred from them. A semantic net structurewas chosen to represent this knowledge [3], [14], [15]. Ithas about 170 concepts arranged in a "necessary-part-of"hierarchy (the part-of structure) which is eight levels deep, andon each level a "specialization" hierarchy (the is-a structure)which is between one and three levels deep. The lowest levelin the necessary-part-of hierarchy is the interface to the resultsof low level processing methods; the highest level is the descrip-tion of nmedical evidence. The procedural part is linked to thedeclarative part by means of proceduiral attachment and con-

tains algorithms for computing values of attributes, structures,

and certainty factors. Inferences about medical evidence are

made by a set of rules which have the general "IF (conditions,certainty factors), THEN (conclusion, certainty factor)" format[5], [16].

It was menitioned already that the lowest level in the modelprovides the interface to the results of low level processingmethods. This module incorporates standard algorithms forimage smoothing and contour detection [2] as well as speciallytailored routines for detection of the left ventricular contourand for obtaining segmeints of this contour [12], [17]The third module in Fig. 1 is called instances. Whereas the

declarative model contains generic descriptions of parts andevents, the instances contain descriptions and parameter valuescomputed from one particular image sequence. An instanceis mainly a copy of the corresponding declarative model but,for example, default values of parameters are replaced byactually computed parameter values, and also procedures forcomputing certainty factors are replaced by their actual values.The whole analysis is guided by the control module in Fig.

which incorporates several main functions. It provides an in-terface to the user who may initialize system activity by re-

questing an instance of one out of several possible diagnosticfindings or who may also request a complete description ofmedical evidence found in an image sequence. In any case, the

user request causes the system to try to compute one or moreinstances of a particular concept in the semantic net. If therequested concept has specializations, then also instances of allits more specialized concepts are computed. Instantiation ofconcepts is done with the help of a so called "expanded model"which contains that part of declarative knowledge which wasalready evaluated. Furthermore, a "search tree" represents thecomplete history of analysis; a node in this tree is a certainstate of the expanded model as well as the instances generatedso far. Several competing instances give rise to several nodesin the search tree. In principle, competing instances are evalu-ated by a variant of the well known A*-algorithm [41, [5].The control module also creates instances and initializes theappropriate procedural knowledge in the model.More details about the above mentioned modules of the sys-

tem are given in the following sections.

III. Low LEVEL PROCESSING METHODS

Low level processing methods are applied in order to detectthe contour of the left ventricle and to segment the left ven-tricular area into particular regions. While the contour of theleft ventricle as a whole is used for a global motility assessment,the regions resulting from segmentation are the basis on whicha local evaluation of the motional behavior is performed.Input to our system is a sequence consisting of n images,

12 < n < 32, with a spatial resolution of 64 X 64 pixel and anintensity resolution of 8 bits. Since image quality is poor, asmoothing operation is performed first. It improves the sub-jective quality of the input data and facilitates further process-ing steps. Smoothing is done by a median filter of size 7 X 7.The window size has been heuristically determined based onsubjective inspection. An example of an input sequence con-sisting of 12 images is shown in Fig. 2. The first image of thissequence after median filtering is given in Fig. 3. Followingmedian filtering, methods for left ventricular contour detec-tion and segmentation are applied. These methods are de-scribed in detail in Section Ill-A and III-B, respectively.

A. Detection of the Left Ventricular Contour

There are several approaches to the detection of the leftventricular contour in scintigraphic images known from theliterature. They are based on first and second derivatives ofthe image intensity function [12] , [20], [21 ], or on threshold-ing [17] . In our system we are using a modification of [12] .This method has proven better results than the thresholdingprocedure described in [ 17] The detection of the left ventri-cular contour consists of the following six major steps.

a) A poirnt (xo, yo) belonging to the left ventricle is detected.b) A polar coordinate transform of the original image is per-

formed using the point (xo,yo) as origin.c) A one-dimensional local edge detector based on the first

derivative of the intensity function is applied to the image re-sulting from step b).d) Zero crossings are located and their intensity is changed

from zero to a value Imm in the image resulting from step c).e) An optimal path from the bottom to the top line in the

image resulting from step d) is detected.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

Fig. 2. An input image sequence.

Fig. 4. The contour of the left ventricle in Fig. 3 in polar coordinatespace.

Fig. 3. The first image of the sequence shown in Fig. 2 after 7 X 7median filtering.

f) A backtransform into the original x, y-coordinate systemis applied with the path detected in step e) corresponding tothe left ventricular contour.While the methods described in [201, [211 utilize rather

general edge detection techniques, the above procedure is a

"special tailored" approach to the problem in question. Thekey in the above edge detection mefhod is the polar coordinatetransform performed in step b), which is motivated by the factthat the left ventricle has a circular or an elliptical outline. Asa consequence, a one-dimensional local edge detector is suf-ficient in the transformed image. This saves considerable com-

puting time as compared to the application of several direc-tional sensitive local edge detectors in the original x, y-domain,or the application of one directional insensitive local edgeoperator based on second derivatives which requires, however,a larger window for computation than the one-dimensionaloperator used in our method. The second benefit resultingfrom the polar coordinate transform is a great simplification ofthe detection of an optimal path along those points where thelocal edge operator yields maximum response. Our procedureis closely related to [12] with steps b) and e) being identical to[12] and the other stepsbeingextensionsand/or modifications.Following is a more detailed description of steps a)-f).Step a): The procedure for finding a point (xo, yo) within

the left ventricle as origin for the polar coordinate transform isbased on the heuristics that the left ventricle is always located

in the right-hand part of the image and shows higher intensityvalues than its surrounding region.Step b): The polar coordinate transform is done in discrete

steps of 0 = 60 and r = 2 pixel yielding an image of dimension32 X 60 in the r, 0-domain.Step c): The one-dimensional local edge operator consists of

three identical lines with weights -2, -1, 0, 1, 2 in consecutivepixels of each line. It is applied along horizontal lines in the r,0-domain, i.e., along lines with constant 0.Step d): At its right-hand side and its top and bottom, the

left ventricle is bordered by a ramp-like edge, while at its left-hand side, it is neighboring the septum which appears as a valleyin the intensity function. A point belonging to a ramp-likeedge corresponds to a local minimum under the edge operatorin step c). By contrast, a point in the septum corresponds to azero crossing. In order to keep track of local minima (corre-sponding to top, bottom, and right-hand side border) as well asof zero crossings (corresponding to the septum) in the contourfollowing the procedure in step e), a new intensity value Imniis assigned to zero crossings with Imin being equal to the overallminimum of the r, 0-image after application of the edge oper-ator in step c).Step e): This is a dynamic programming procedure identical

to that described in [12], yielding an optimnal path in the r, 0-image obtained after step d). Fig. 4 shows the optimal pathafter applying steps a)-e) to Fig. 3.Step f): The backtransform is based on the same principles

as the transform in step b). The result of the backtransformapplied to every image in the sequence in Fig. 2 is shown inFig. 5.Due to interpolation in steps b) and f), local irregularities in

the lett ventricular contour sometimes occur. They are cor-rected by means of contour smoothing using the well knownoperations EXPAND and SHRINK [5].

B. Segmentation ofthe Left VentricleThe segmentation described in the following is performed

for the purpose of facilitating local motility assessment of par-ticular parts of the left ventricle. There are several approachesto left ventricular segmentation known from literature, e.g.,

248

NIEMANN etal.: KNOWLEDGE BASED SYSTEM FOR GATED BLOOD POOL STUI)IES

Fig. 5. Left ventricular contours superimposed on the images in Fig. 2. Fig. 7. Anatomically meaningful segmentation of the left ventricle.

23

4

5

Fig. 6. Left ventricular segmentation using fixed sectors.

[221, [231. In our system we are using two different ways ofsupplementing each other. The first way is similar to [22].The center of gravity of the left ventricular area is determinedand a segmentation into n sectors, each corresponding to an

angle of 2iT/n, is done. As an example, the segmentation re-

sulting from the contour obtained from Fig. 4 after smoothingis shown in Fig. 6 for n = 12.Resulting trom the second type of segmentation used are sec-

tors which each have a particular anatomical meaning, corre-

sponding to the septal (SE), inferioapical (IA), posterolateral(PL), and basal (BA) region of the left ventricle, respectively.The result of segmenting Fig. 4 according to the second tech-nique is shown in Fig. 7. The sectors in question are arrangedin the following way:

SE: left of the center,IA: below the center,PL: right of the center,BA: above the center.

While we have fixed angles in Fig. 6, the angles defining thesectors in Fig. 7 vary from patient to patient, in general. Asegmentation as shown in Fig. 7 is needed since the anatomicalsectors SE, IA, PL, and BA are relevant with respect to a di-agnosis. In the following, the segmentation method yieldingthe result in Fig. 7 will be explained.

6Fig. 8. Chain code.

TABLE IEXPECTED CONTOUR DIRECTION FOR THE ANATOMICAL SECTOR S, IA, PL,

B

expected contour directionanatomical sector

chain code angle

IA 0 00

PL 2 90°B 5.1 2300S 6.2 2800

The borders of the sectors SE, IA, PL, and BA depend on

anatomical criteria which are not directly visible in an image.However, there is an expected contour direction and an ex-

pected range of occurrence for each of the anatomical sectorsin an image. Those are utilized for segmentation. Assumingcontour tracing in an anticlockwise manner and assuming a

chain code representation of the left ventricular contour ac-

cording to Fig. 8, the expected contour directions are shownin Table 1. (These expected directions have been determinedfrom a training set of images; see Fig. 7 for an example.) FromTable I, one can compute for each contour point the degree ofcoincidence between the actual and the expected contour di-rection for each of the anatomical sectors SE, IA, PL, and BA.An example of the anatomical section PL is shown in Fig. 9.

The measure of coincidence has been transformed into a range

between 4 and 8, with 8 corresponding to identity betweenexpected and actual contour direction, and 4 corresponding to

1

-/ 9 0

\\47

249

IEEE TRANSACTIONS ON PATTERN ANALYSIS ANI) MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

Fig. 9. Coincidence between expected and actual contour directions torthe anatomical sector PL.

Fig. 10. Certainty measure for each contour point for belonging to theanatomical sector PL.

the maximum possible difference of ±ir. The leftmost contourpoint on the horizontal axis in Fig. 9 corresponds to the pointdirectly below the center of gravity in Fig. 7.

In the next step, the function shown in Fig. 9 is multipliedwith another function which represents the expected range ofoccurrence for the sector PL. This function is of the same typeas the function shown in Fig. 16. The result of the multipli-cation is given in Fig. 10, which can be interpreted as fuzzymembership function in the sense of [261, giving for each con-

tour point a measure of certainty for belonging to the anatom-ical section PL. (A more detailed discussion of fuzzy member-shiip functions of the type shown in Fig. 16 will be given inSection IV-B.) Using the expected contour direction and range

of occurrence for the other anatomical sectors SE, IA, and BA,functions similar to that in Fig. 10 are obtained. The super-

position of three of these functions is shown in Fig. 11. Letdc denote a function according to Figs. 10 and 11, with XE{SE, IA, PL, BA}. A point p on the contour is assigned to thatanatomical sector for which d_(p)= max {dSE(p), dIA(P),dPL(p), dBA(P)}. Making this decision for each contour pointp, the segmentation shown in Fig. 7 results. A detailed descrip-tion of the segmentation algorithm can be found in [24] .

Fig. 11. Certainty measures for the three anatomical sectors super-imposed.

In summary, for an image sequence the contour of the leftventricle as a whole and the contours of the sectors accordingto Figs. 6 and 7 are the final result obtained by means of thelow level processing methods of our system. These contoursare input data to further knowledge based analysis. The verifi-cation of the left ventricular contour detection and segmenta-tion methods will be discussed in Section VI.

IV. KNOWLEDGE REPRESENTATIONThe first question arising for knowledge representation is

"What needs to be represented?", i.e., what is the problemspecific knowledge needed for analyzing an image sequence ofthe type described above? There are different objects occurringin one image of the human heart. Looking at a sequence, addi-tionally, the motility of the heart or some parts of it is char-acterized by a number of events. Last but not least, in themedical world there are diagnoses associated with the motionalbehavior of the objects and their form. Therefore, the questionabove leads to the subquestion "What objects, events, and di-agnoses occur in the world of scintigraphic images and whatare their interrelations?" The following statements give anidea of the answer:

a) The left ventricle consists of four segments namedposterolateral, inferioapical, basal, and septal. During a healtcycle, contractions, expansions, and stagnations may occur foran object.b) The left ventricle is an object. Hypokinesis is a diagnosis.c) To talk about an object requires the contours. To talk

about motility requires objects detected in different images.If the objects, events, and diagnoses summarized as concepts

are selected, then the next question is "What are the quantita-tive properties of a concept?"d) An object occupies an area at each image of the sequence.e) A motility phase, e.g., a contraction, is mainly character-

ized by a time interval.f) A diagnosis has some certainty and a textual description.To the aim of an image analysis system generating a descrip-

tion of the input image sequence automatically, procedural at-tachment to a descriptive model of such quantitative propertiesof concepts is necessary. Beside these two classes of informa-

250

251NIEMANN etal.: KNOWLEDGE BASED SYSTEM FOR GATED BLOOD POOL STUDIES

name of concept

( number of parts - integernome text

descriptive Inform, text

Instance - o. instances

generalization o. more general concepts

specialization o. more special concepts

similarity - Il. oa similar concepts

semantic part - part concept

sem. attri'bute -

sem. structure - pairt Lsem. pSrt of r namenecessary part _attri'b. I -L descri'pt. inform.nec. attributek ( T range of valuesnec. structure attrib. M Inters r .

abcbaon 10me it.of. (Fo dIls mensecioncomput.~ ~of.C. defoult vcalue

re.N-X comput. of value -l~ ~~~~~~~~~res tric t ionsa) t ~~b)

Fig. 12. (a) Syntax of a concept used for semantic net representationof declarative knowledge. (b) The entities referenced by the slots of aconcept. Arrows marked with a * are pointers to procedures. Theabbreviation 1.0 means list of. (For details see Section IV-A.)

tion, concepts and their quantitative properties, a third kind ofknowledge is available. For example, the following restrictionsand relations, respectively, hold for the problem looked at.

g) Two objects are only valid to be "left ventricle" and"right ventricle," if left ventricle "is left of" right ventricle.h) The systole of the left ventricle starts at the beginning of

a heart cycle and occupies about half of a cycle.Similar to quantitative properties, it is not sufficient in an

image analysis system to say "'the left ventricle is left of theright ventricle." There is also a need for a procedure to testwhether this relation holds for the two objects. Due to thefacts described, the module model of the system is divided inthe two submodules "declarative knowledge" and "proceduralknowledge." Procedures are referred to by declarative struc-tures, i.e., by means of procedural attachment. In this paper a

particular example of an image analysis system as well as theparticular knowledge required by it is considered. However,the general idea of representing objects, events, and interpreta-tions (in our case: diagnoses) and their attributes and structuralproperties is applicable and useful also in other applications.In this sense the approach to knowledge representation is ofgeneral applicability, but, of course, the actual knowledge repre-

sented in the system is problem specific.

A. Declarative Knowledge

The declarative part of problem specific knowledge is repre-sented in a semantic net [1 8], [19]. Semantic networks are

labeled directed graphs, where the nodes represent descriptionsof objects or events and edges or links represent relationshipsbetween them. A node describing an object or event is termeda concept; it is the basic unit in the semantic net and has thestructure shown in Fig. 12(a). A concept may be referencedby its name and have a number of slots arranged into severalgroups. Each slot has an associated type that refers to anotherconcept, to a relation, or to a simple type like an array of in-

CYCLE_OF. LV

part part

S YSTOLE port DIA STOLE

part port

L VFig. 13. A relational "part" between four concepts describing the cycleof the left ventricle, the systole and the diastole phases, and the leftventricle as object.

tegers. The head of a concept contains the number of its parts,the name-e.g., "NORMLV"- and descriptive information-e.g., "normal left ventricle."The next slot is a pointer to a list of instances of this con-

cept. The following three slots refer to lists of more generalconcepts, more special concepts, and similar concepts, respec-tively. For example, the concept NORMLV has the more gen-eral concept RMILV (regional motility interpretation of theleft ventricle), and in this case there are no specialized and nosimilar concepts, (see statement b) above). The specializationslot is the inverse of geineralization, and establishes a doublelinking of concepts in the net.Although it may be sufficient to use "part-of" only, it was

considered to distinguish two different kinds of parts of aconcept. In image (sequence) analysis, usually there are dif-ferent levels of concepts to be represented. Besides objectsand their decomposition in parts in a conceptual or physicalsense, also concepts of motion exist, which may be dividedinto a number of refined motions. Additionally, in the systemlooked at there are concepts representing diagnosis. If only"part-of" relation is used, networks as shown in Fig. 13 mayoccur. The cycle of the left ventricle is divided into its naturalparts, systole and diastole. But all these three concepts havethe concept LV as part, additionally. These "part-of" relationsbetween LV and the three motion concepts are necessary dueto the fact that it is impossible to detect motion phases of anobject not known by the different motion phases themselves.This leads to a relation "part-of" with two semantics. By thefirst one, concepts are said to be in the relation if the partsbuild up a natural decomposition (see also statement a) above).In the second case, different leveis of abstraction or complex-ity of concepts are connected (in Fig. 13, object and motions).In order to distinguish these two meanings of the relation"part-of," it is split up into so called semantic parts and neces-sary parts. Semantic parts of a concept are parts in a naturalor physical sense. On the contrary, necessary parts of a con-cept reference concepts on another level of abstraction. If aconcept A has the necessary parts B and C, the relation can berewritten into the following statement. "To talk about A re-quires B and C " (see also statement C) above). In the tollowingtwo sectioi-is a few examples iased on the network realized asmodels in the system are given. Physicians subdivide the con-tour of the left ventricle into four anatomical segments whichare called the septal, basal, posterolateral, and inferioapical

IEEE TRANSACTIONS ON P'ATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

segments as mentioned in Section III. Therefore, the conceptLV, that is the left ventricle, has the tour semantic parts SE,BA, PL, and IA. Of course, the number of semantic parts maybe different for different concepts. To give another example,the time interval of one cycle of the beating heart is subdividedinto the motional phases of contraction, stagnation, and ex-pansion of the heart. Therefore, the concept CYCL, that isone cycle of the beating heart, has the semantic parts CONT,STAG, and EXPA. Again, the slot semantic part-of is the in-verse of the semantic part slot and established a double linkingof semantic parts in the net.The concept BASMO, that is basic motion, has an object

which is moving as a necessary part. The concept RMIB, thatis, regional motility interpretation of the basal segment ofCYCLBA, is a cycle of the basal segment. The slot necessarypart of gives the inverse linking to the slot necessary part.Whereas a semantic part in our realization of the net always isanother concept, there is one level in the net where the slotnecessary part does not reference a concept but a primitiveentity. This level provides the interface to the results of lowlevel processing, and in this application, primitive entities arecontour segments of the left ventricle in the n images coveringone cycle of the heart.

In addition to the more or less qualitative description of aconcept provided by the slots' semantic parts and niecessaryparts, every concept also contains some quantitative descrip-tions concerning attributes, statements d), e), f), and structures,statements g), h), both split into the semantic ones and neces-sary ones. As indicated in Fig. 12(b), an attribute has besidessome other information a default value and a procedure forcomputing the actual value of an attribute when instantiatingthe concept. For example, the above mentioned concept CONThas the semantic attributes "start time" and "end time" andthe necessary attribute "number of images in an image se-quence." The semantic and necessary structures are relationswhich exist between parts or attributes. For example, the abovementioned segments of the left ventricle must be adjacent.

Finally, every concept references a procedure for computa-tion of its certainty factor CF, and every instance of a concepthas an associated value of the CF.Therefore, concepts are linked to each other and have quanti-

tative attributes and relations. By means of procedural attach-ment each concept refers procedures for calculating attributes,the CF, and for testing relations. Fig. 12 shows the result-ing structure of a concept in the network. Because all conceptssatisfy this structure, it is sufficient for the control module ofthe system to know how a concept is represented but not whatit represents. Procedures referred to in a concept are to beactivated if an instance of the concept has to be created. Dis-tinguishing declarative knowledge (the network) and pro-cedural knowledge allows a flexible use of different knowledgerepresentation techniques. For example, anatomically possiblecycle types of parts of the heart are easy to describe by aregular grammar. Therefore, to select cycle types for a givenobject can be done directly by a kind of parsing due to theregular grammar. The parsing procedure and the grammar buildup the algorithm for calculation of an attribute in the network.To give another example, production rules based on fuzzy set

speci alizati on

necessarypart COMPLETE sPec pecial

DESCRIPTION descriptjrn5np \ np

s pecialGENERAL LOCAL local motility\ 'Iotit ityMOTILITY DESCRIPTION description descriptio

np rp anatomi ca npinterpretationof leftventr. motion

On p np

ooec special __c special motionMOTION motion of anat.objects

inp s|np Qnp /spec.basic

BASIC pec speci a] SPc motion of1lOTION bas.mot. anat.objects

np np np DESCRIPTION OFspec medicalFORM AND eevidence of

3 PROPORTION form and prop.

nrp

TIME COMPLETE spec special time2 OBJECT compl. objects

np \q?n p Wnp OP

TIME GENERAL Opec interface SPACE1 REFERENCE INITERFACE of objects REFERENCE

Fig. 14. The two hierarchies of specialization (spec) and necessary parts(np) in the semantic network. Capital letters denote single generalconcepts, lower case letters denote collections of specialized concepts.The semantic part links are omitted.

theory are an easy way to calculate the certainty of a diagnosis(see Section IV-B). These rules are referred to in terms of pro-cedural knowledge bounded by attributes. Because of this fact,the problem of which rule is to be activated does not exist inthe system explicitly. A rule is activated by creating an in-stance of the concept in which the attribute bounding the ruleis defined.The necessary-part-of and the specialization hierarchy of the

whole net is shown in Fig. 14. It integrates knowledge aboutobjects, motion, and medical evidence. Since there are about170 concepts and 450 links for specializations, necessary parts,and semantic parts, Fig. 14 can only give a condensed view ofthe general structure of the network. In order to give a moredetailed idea, Fig. 15 shows a section of different levels of thenetwork. This figure is explained at the end of part A of thissection. As is common practice for semantic nets, we considerall parts, attributes, and relations of a concept also to belongto all specializations of this concept, unless a different defini-tion is stated explicitly for a particular concept.We now turn to a short description of the eight levels of the

net shown in Fig. 14. The concepts on level 1 provide the inter-face to the low level processing methods. TIME REFERENCEdescribes general time parameters of the image sequence; forexample, the number of images. SPACE REFERENCE definesthe angles of anatomical segments and "interface of objects"gives the contours of the left ventricle, its four anatomicalsegments, and 16 subsegments spaced at equal angles. Theclass TIME COMPLETE OBJECT on level 2 defines the objectsheart, left ventricle, heart without left ventricle, and the fouranatomical segments. Among the main attributes is the areaof the left ventricle and its segments.The third level only describes form and size of the left ven-

252

NIEMANN etal.: KNOWLEDGE BASEI) SYSTEM FOR GATED BLOOD POOL STUDIES

LV >TIME COMPLETE

OBJECT

IA sp sp

TIME

REFERENCE -L

SE

Fig. 15. A detailed view on the semantic network shown in Fig. 14.The arrows denoted spec, np, and sp represent specializations (spec),necessary parts (np), and semantic parts (sp).

contraction

-0.01

CF

1 expansion

s tagnation

XlX,0.01 area

change

Fig. 16. Fuzzy membership function to assign one more of the motionphases contraction, stagnation, and expansion to an observed area

change. The value of the function is the certainty factor CF.

tricle and its segments. This may be used on level 8 to inferstatements like "left ventricle is enlarged."Levels 4 and 5 define BASIC MOTION and MOTION, special-

ized for a cycle in general and its phases of contraction, stagna-tion, and expansion, and further specialized for a cycle of theleft ventricle and its segments. The main attributes of BASICMOTION are the change of area between consecutive imagesand direction and strength of motion obtained from the centerof gravity of objects in consecutive images. Based on area

changes, the three phases of motion are attributes by a fuzzymembership function according to Fig. 16. Using a set of ana-

tomically possible motion cycles, the best fitting sequence ofmotion phases is selected. The concept MOTION is definedsimilarly. The main difference is that BASIC MOTION always

relates two consecutive images, i and i + 1, whereas MOTIONdescribes larger entities, for example, a contraction phase forimages 1 to 7. Several alternative descriptions of the motionof anatomical objects may arise during this step.Level 6 gives a further refinement of motion description and

a representation in medical terms. The general statement isthat an anatomical interpretation of a motion cycle of the leftventricle consists of systolic and diastolic phase, and thesephases are further subdivided into pre-ejection period, ejectionperiod, endsystolic stagnation, fast filling period, iso volumeexpansion, and slow filling period. Attributes are start andend times and ejection fractions. No ambiguities in motionphases are allowed on this level.The last two levels relate motion phases, form, and propor-

tion to statements about medical evidence. On level 7, localmotility descriptions are derived, for example, "hypokineticposterolateral segment" or "acinetic basal segment." On level8 the local description and descriptions of form and propor-tion are combined to a complete description. This allows oneto infer new diagnoses like aneurysma and to modify individualfindings. For example, if a posterolateral hypokinesis and aninferioapical akinesis are confirmed with high certainty, itshould be checked whether the first one is only a side effect ofthe second one (because both segments are mechanically cou-pled). If motion deficiency of the posterolateral segment isstrongest in the vicinity of the inferioapical segment, only theinferioapical akinesis will be maintained.Whereas Fig. 14 gives a condensed description of the total

network, Fig. 15 shows a few concepts out of the network andtheir interrelations in terms of the relations "necessary part"(np), "semantic part" (sp), and "specialization" (spec). Theconcept of highest level in this figure is the interpretation ofthe cycle of the left ventricle in anatomical terms, ANCYCLLV.This concept has the semantic parts SYSTOLE and DIASTOLEwhich are themselves divided into different phases (see alsoSection IV-B). The necessary parts of these concepts are de-fined to be the contraction-, stagnation-, and expansion-phasesof the left ventricle: the concepts CONTLV, STAGLV, andEXPALV. As described in previous sections, these conceptsare specializations of the more general motion concepts CON-Traction, STAGnation, and EXPAnsions, respectively. Thesetogether build up the general concept describing a CYCLE.Therefore, they are semantic parts of this concept. CYCLE,STAG, EXPA, and CONT are specializations of the most gen-eral concept MOTION of this level. The next level below, de-scribing frame-to-frame motions, has the same internal struc-ture: one most general concept, called BASic MOtion withfour specializations-BASCONT, BASSTAG, BASEXPA, andBASCYCL. Their interrelations are quite similar to those ofthe motion concepts. In the same way they have specializa-tions for the objects LV, shown in Fig. 15, and IA, PL, BA, andSE (not shown in the figure). Each concept at the "motionlevel" has this concept of the "basic motion level" as neces-sary part, which corresponds in a direct manner. That is,CYCLE has the necessary part BASCYL, STAGLV has thenecessary part BSTAGLV, and so on. Necessary parts of thebasic motion concepts themselves are time complete objects.For the general concepts not requiring a special object, the

I r- l

253

24IEIE TRANSACTIONS ON PATTI RN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

11. 36

.14 C.39 1. D.99

.9-6.9 -7.9

I ig. 17. A set of possible cycle types and their representation by regularexpressions, C: contraction, E: expansion, S: stagnation.

concept TIME COMPLETE-OBJECT is referred to as a neces-sary part. For the concepts requiring the left ventricle, namelyBCYCLLV, BCONTLV, BSTAGLV, and BEXPALV, the con-cept IV is a necessary part. At the "object level," the conceptLV consists of the four anatomical segments, described by theconcepts IA, PL, BA, and SE. All these five concepts arespecializations of TIME_COMPLETE_OBJECT.

B. Procedural KnowledgeA large amount of procedural knowledge is attached to the

semantic net-in fact, the ratio of procedural to declarativeknowledge is about 10 to 1 measured by required storage spaceand excluding low level methods. Therefore, a full descriptionis impossible here and probably not necessary because a gooddeal is fairly straightforward: for example, computation ofareas, centers of gravity, length of contours, axes of an ellipseapproximating the left ventricle, and so on. Two less straight-forward problems are the interpretation of a heart cycle (onthe levels 4 and 5 of Fig. 14) and inferences about medical evi-dence (on the levels 7 and 8). These two points will be treatedin the following.Interpretation of a heart cycle is a two phase process executed

on levels 4 and 5 of the net. The basic idea is to use optimalsearch techniques [4], [5] . Using general medical knowledge,the set of anatomically possible cycle types shown in Fig. 17is specified and represented by regular expressions. After com-puting the area of the left ventricle in every image sequence,the image to image area change is obtained, and from Fig. 16a certainty factor for each of the phases of contraction, expan-sion, and stagnation is derived. This gives an initial labeling asshown in Fig. 18(c). Due to the overlap of the fuzzy member-ship functions in Fig. 16, this initial labeling is highly ambig-uous. For every prototype cycle shown in Fig. 17, the bestscoring set of labels covering a cycle is computed by dynamicprogramming on level 4. The score is simply the sum of thecertainty factors along a path through the initial labelings ofFig. 18(c). This path is constrained to select C, S, E labels inthe order required by the chosen prototype, for example, asshown in Fig. 18(d). Such a path is computed for every proto-type, and all paths whose score is above a threshold are re-tained. More details are given in [20] .

On level 5 the labels attributed to each pair of consecutiveimages are summarized to cover larger intervals of time. Labelswith CF = 1 are retained, but labels with CF < 1 are split intoa set of alternatives. If the label L, L C { C, S, E} has CF < 1and is followed by the samne label, then the set is { L, [ L ] }, whichmeans that the motion phase could be L, or undefined. If thelabel L, L E { C, S, E} has CF < 1 and is followed by a label

3.9 - 9. 9

1JO5-

Fig. 18. The basic steps for finding motion phases in an image sequence.Details are given in the text. (a) Number of images in the sequence.(b) Area of the left ventricle. (c) Initial labeling obtained from image-to-image area change and fuzzy functions in Fig. 16. (d) Best scoringpath for one of the prototypes shown in Fig. 17. (e) Insertion of al-ternatives if certainty is below one. (f) New optimal path. (g) Finalmotion phases.

L! #L, then the set is {L, [L],[L ] }, which means that themotion phase could be L, undefined, or an overlap of L andL'. The introduction of L' may cause a proliferation of thisprocess, since the succession L, L, L' now is modified to L, L,L or L, [L], L', or L, [LI], L'. Certainty factors are attri-buted similarly to Fig. 16, and this way the net shown in Fig.18(e) is derived from 18(d). Again, the best path is computedby dynamic programming giving the path shown in Fig. 1 8(f).This path corresponds to the motion phases shown in Fig.18(g). The ambiguities are retained and may give rise to com-peting diagnoses on higher levels. The main attributes of themotion phases are their begin and end times.Currently the system knows about 45 individual descriptions

on levels 3, 7, and 8. These are of the type "the left venticleis normally beating," "the left ventricle is deformed," or "thebasal segment of the left ventricle is nearly motionless." Thecertainty of each statement is measured by a certainty factorCF which is computed by fuzzy algebra [26]. For example,given a rule of inference of the form

IF (A AB) V (7 C) THEN D

the certainty factor of D is computed from those of A, B, C by

CF(D) = max {min { CF(A), CF(B)}, 1 - CF(C)}

or. in short-hand notation,

CF(D) = (CF(A) A CF(B)) V (1 - CF(C)).

254

r+r+

I \\

c+s+E+s+E+s+c+s+E+s- C+E+S+

- E

. .\-. . - . .

'\\. E

.F

6 0 --)-- 7 0 --),, ',

NII1MANN etal.: KNOWLEDCGE BASIED SYSTEM FOR GATED BLOOD POOL STUDIES

In the above equations, A, B, C, and D stand for names of con-cepts in the net. Since there is an obvious one-to-one corre-spondence between a rule for deriving a conclusion and the CFof this conclusion, it suffices to state only the equation forthe CF.The rules on level 8 "COMPLETE DESCRIPTIONS" use

concepts on level 7 "GENERAL LOCAL MOTILITY DE-SCRIPTIONS" as arguments. For example, an "aneurysma" ischaracterized by an enlarged and nearly akinetic (motionless)inferoapical or posterolateral segment combined with weakmotion of the left ventricle as well as an enlarged left ventricle.Therefore, the CF of aneurysma is

CF (ANEU) = CF (LVWWEAK) A CF (LV_ENLA) A

[(CF (IA_AKIN A CF (IA_ENLA)) V (CF (PL_AKIN) /\

CF (PL_ENLA))].

Similar rules are available for 7 more diagnoses on level 8.Things are somewhat more complicated on level 7 because

on this level descriptions have to be derived from motionphases. On this level the motion of individual segments of theleft ventricle is described. This is done with two functions-tand u defined below. The cycle of the left ventricle is repre-sented by systolic and diastolic phases on level 6, and these arefurther subdivided into pre-ejection period, ejection period,fast filling period, iso volume expansion, and slow filling pe-riod. Let P be an element of the set of anatomical motionphases, that is,

P E { CYCL, SYST, DIAS, PEP, EP, FFP, IVE, SFP}.

Each phase has an unambiguous start and end time determinedfrom general medical knowledge and the ambiguous interpreta-tions on level 5. If the motion phases on level 5 are highly ir-regular, default values are used on level 6. On level 5 the cycleof an observed image sequence is represented by { C, E, SI asillustrated in Fig. 18. This is available for the set of segments.Let s be an element of this set, that is

s E { septal, basal, inferoapical, posterolateral}= {SE, BA, IA, PL}.

The function t(B, s, P), with B an element of the power set ofrC. E. S}, gives the nercentage of' motional phases of B in seg-ment s during anatomical motion phase P. For example, t({ C,S}, IA, PEP)= 0.9 means that the inferioapical segment con-tracts or stagnates during 90 percent of the pre-ejection period.Let u(d, s, k) (x) be a fuzzy membership function of similarshape as shown in Fig. 16, where

d E {normal, hypokinetic, akinetic, dyskinetic, phaseshiftedlis the set of diagnostic terms, s is a segment as defined above,k is the criterion used (for example, ejection fraction or amountof stagnation), and x is the independent variable for which u isto be evaluated. For example, u(AKIN, IA, SA) (t({S}, IA,CYCL)) is the certainty factor for akinesis (nearly no motion)of the inferioapical segment when the amount of stagnation(SA) is considered and the independent variable has the valuet({S}, IA, CYCL), where t is as defined above. The functionsu are determined from general medical knowledge [10], [1] .

The functions u and t are the basis for determining the CF ofindividual diagnostic terms. As an example, the CF of akinesisin segment s is considered. A segment is termed akinetic if itis nearly motionless, that is, if contraction and expansion arevery weak. Since undefined and overlapping motion phasesmay result on level 5 (see above), it is not sufficient to measureonly length of motion phases or consider only the complementof such phases, but a combination must be used. This resultsin the rule

CF(s_AKIN) = u(AKIN, s, EF) (ef) A

u(AKIN, s, SA) t({S}, s, CYCL) A

7t({ C}, s, PEP) A t({ E}, s, IVE) A

[t({ S}, s, EP) V t({ C, E}, s, EP)] /\

[t({S}, s, FFP) V t({ C, E}, s, FFP)] /\

[t({S}, s, SFP) V X t({ C, E}, s, SFP)] /\

7t({E}, s, EP) /\n t({C}, s, FFP)

A7 t({C}, s, SFP).

It should be noted that this rule and also the other rules in thesystem were derived in close cooperation between physiciansand computer scientists using medical knowledge available inthe literature. Slightly modified versions are conceivable and,in fact, experiments with modifications are planned in the nearfuture. However, as discussed in more detail in Section VI, thepresent rules gave correct descriptions of the data tested so far.

V. MODEL GUIDED ANALYSISThe declarative knowledge described above is used to guide

analysis of an image sequence. Initially the user states a con-cept contained in the network. The system accepts this as arequest to compute an instance of the stated concept. For ex-ample, if the concept is HYPOIA (hypokinesis of the inferio-apical segment) and an instance of HYPOIA is found and givenCF = 0.8, then there is evidence of 0.8 that the patient has ahypokinetic inferioapical segment of his left ventricle. If thestated concept is RMIIA (regional motility interpretation ofthe inferioapical segment), the system is given a concept whichhas 6 specialized concepts, among the 6 is HYPOIA andUNDFIA (undefined inferioapical segment; that is, the systemdid not make a decision). Now the system tries to instantiateRMIIA and all the 6 specializations of it. However, computa-tion of an instance of a concept is only possible if instances ofall parts are already available. Since initially this is not thecase, the system tries to instantiate the parts of the stated con-cept. This process is repeated until a primitive concept is found.A concept is primitive if it has no necessary parts, no semanticparts, and no more general concepts. The requirement that aprimitive concept should have no parts is obvious because apart is itself a concept as shown in Fig. 12. There should beno more general concept because this might have parts whichare transferred to the more special concept. The attributes ofprimitive concepts are results of low level processing as men-tioned above.The basic control strategy is a top-down search through the

semantic net. If asked for an instance of a certain concept, the

255

I2t111 TRANSACFIONS ON I'ATTLRN ANALYSIS AND MACHINE INTELLIGENCE, VOL. IPAMI-7, NO. 3, MAY 1985

begininitialize SEARCH-SPACE; initialize EXPANDED

MODEL;while (not al'1 concepts in EXPANDED-MODEL have

been focused or )do cloose ACTUAL-CONCEPT from EXPANDED-MODEL;

if instancesOr ACTUAL-CONCEPT don't exist

then if ACTUAL-CONCEPT iS primitivethen cal1 methods for creating instances

of primitive concepts; link instan-ces to model; expand SEARCH-SPACE:pick up best state in SEARCH-SPACE;

if ACTUAL-CONCEPT isn't primitive and all

parts of ACTUAL -CON CEPT are in stanti-

ated

then create instances of ACTUAL-CONCEPT;link instances to model; expanSEARCH-SPACE; pick up best state in

SEARCH-SPACE;if ACTUAL-CONCEPT isn't primitive and not

al 1 parts of ACTUAL-CONCEPT are instan-tiated

ther exmpand model;

EXPANDED MODEL

YCLLV

TAGL V

HYCL V

(STAGLE);O.,,8I

\;c., I:, T(sAGL V) ;

'3STAGLV ),

STAGLV EXPALV STAGLV.;0.33 1 (ST L;

,TAGLV BEXPALV ;O .34

HY LLV

Fig. 19. An overview of the control algorithm.

system goes through the semantic parts and necessary partshierarchy, thereby steps down to level 1 in Fig. 14 where primi-tive concepts are found, and instantiates the primitive con-cepts. Now the system goes up again, successively instantiatingconcepts and their specializations on the next higher level.This is done until the stated concept is instantiated. If it hasno specializations, analysis is finished. Otherwise, all specializa-tions of the stated concept are also instantiated. It was men-tioned already that alternative results may be obtained on var-ious levels of the net by the procedural knowledge. This givesrise to competing instances of concepts. For combinatorialreasons it is impossible to follow every alternative through allstages of the net. Therefore, the control algorithm selects at eachstep only the best scoring instance for further computations.This strategy is adopted from the well known A *-algorithm.The control algorithm uses two special data structures, the

search tree and the expanded model, in addition to the semanticnet and the instances. The expanded model is a kind of note-book showing the amount of declarative knowledge used al-ready and showing where additional knowledge is required. Itis initialized with the concept stated by the user, and new con-cepts are added as the system steps down to the primitive con-cepts. The search tree contains the complete history of analysis.Every state of the expanded model together with the generatedinstances gives a node in this tree. Competing instances arekept in different nodes. Analysis proceeds at the leaf of thesearch tree containing the best scoring instance.

Fig. 19 shows the control algorithm [27] and Fig. 20 gives asimple snapshot on the expanded model and the search treewhich is a section from an actual run of the system. In thisfigure it is assumed that an instance of the concept LV alreadyexists. The goal concept is chosen to the CYCLLV. The evalu-ation of the expanded model is based on the network shown inFig. 15. Since the generalizations of a concept are instantiatedimplicitly if the concept itself is instantiated, just the semanticand necessary parts must be looked at. Beginning with the goalconcept CYCLLV, the two concepts STAGLV and BSTAGLV

(STAGLLV);O.

STAGLV EXPALV

BSTAGLV BEXPALV XPALV,

IL,(STAGLV) 0 .34

I(BEXALV ;0.34

XPALV)

I BSTAGLV)

GLV);0 .13: 1. ( STFAGLV) ;0 .34

I (BEXPALV);0. LXPA LV .34

LV) (EXPALV)O .27stdrt node

A actual concept

assumed status of instance datawhen starting the process

Fig. 20. An example of the expanded model (left) and the search tree(right). Only the instances of concepts are stated explicitly in thesearch tree. The relation to instantiated concepts is made bh pointersnot shown in the figure.

are put into the expanded model. Since BSTAGLV only re-quires an instance of LV and this instance exists, BSTAGLV isinstantiated. The CF of the corresponding search tree node,represented by I(BSTAGLV), is calculated and its value is 0.88.No competing instances are created in this step. After instanti-ating BSTAGLV it is possible to instantiate STAGLV and newACTUAL_CONCEPT. This process leads to two new nodes inthe search tree with competing instances I, (STAGLV) and12(STAGLV) with different CF's 0.33 and 0.34, respectively.The next actual concept in the expanded model is CYCLLV,not shown in Fig. 20. This actual concept cannot be instanti-ated yet, because there are parts not instantiated. Furtherevaluation shows the concept EXPALV and the actual conceptBEXPALV. This is able to be instantiated and results in thenew node I(BEXPALV) in the search space, whiclh becomesthe best so far existing node, i.e., the node with highest CF.Instantiating EXPALV in the next step results once more intwo competing instances with CF 0.22 and 0.27, respectively.Since I, (STAGLV) is now the best node in the search tree,this path has to be evaluated. In a similar way this processcontinues until the best instance of CYCLLV is found. Usuallythe search tree has about 250 nodes, if a complete diagnosis is

256

SEARC

NIEMANN etal.: KNOWLEI)GE BASIED SYSTEM IFOR GATED BLOOD POOL STUDIES

chosen to be the goal concept. But there are also extreme caseswhere up to 700 nodes are generated in an actual run.

VI. RESULTSThe system described in the previous sections has been com-

pletely implemented in Ratfor on a PDP 11/34 following thearchitecture sketched in Fig. 1. Due to storage limitations,procedural knowledge as well as the control module has beenspilled into a number of individual processes communicatingamong each other under the multitasking environment of theRSX-1 1/M operating system. The complete image understand-ing system according to Fig. 1 resides on about 4 Mbyte storageincluding a comfortable interface to the user for displaying thefinal and intermnediate results obtained in the analysis process.During system development, a database containing about 40

image sequences exhibiting different diseases has been collected.This database serves as training and test sample set. Two dif-ferent series of tests have been performed so far. The firstseries of tests has been concerned with the verification of thecontours of the left ventricle automatically derived by methodsas described in Section III. A second series of tests has beenperformed for the purpose of assessing the correctness of thediagnostic interpretations inferred by the system.

A. Verification ofLeft Ventricular ContoursIn contrast with industrial or natural scenes where everybody

is an expert in assessing the correctness of contours which areautomatically detected by means of some algorithm, things aremore complicated in the domain considered in this paper. Here,intensive training is required for a human operator in order tobe able to correctly locate the left ventricular contour. Sinceall model guided processing in our system is based on left ven-tricular contours, the validation of the methods for automaticcontour detection is of crucial importance.There are several methods for contour validation in medical

image analysis known from literature. One of the standardtechniques for image sequences showing the human heart,which we have adopted, is based on the parameter "ejectionfraction" (EF) [13] . This parameter directly depends on theleft ventricular contour. It is defined in the following way.

EDV- ESVEF=

EDV

with EDV and ESV being the left ventricular volume in theend-diastolic and end-systolic image in a sequence showing theleft ventricle at maximum expansion and contraction, respec-tively. In all image sequences in our database, this parameterhas been determined, based on contours which were interac-tively drawn by a physician.The method for automatic left ventricular contour detection

described in Section III has been applied to a total of 420images. Based on subjective visual inspection, 93 percent ofthe images have correctly been processed. These images havebeen used for the calculation of EF, based on the contoursautomatically detected. There has been a correlation of r=0.93 between the EF based on automatic and manual contours,respectively. A graphical illustration is given in Fig. 21. Othercomparable results in the literature are ranging between r= 0.8

EF1

80

70-

60

50

40

30

20

10-

x x/ x

x / xy=ax+ba =0,83b=6,0r=0,93

EF210 20 30 40 50 60 70 80 90

Fig. 21. Correlation between EF based on nianually outlined contours(EFI) and EF based on automatically detected contours (El 2).

TABLE IIDIAGNOSTIC INTERPRETATIONS GIVEN BY A PHYSICIAN FOR NINE IMAGE

SEQUENCI'S

sequence regional motility interpretation

1 ia-akinesis, s-akinesi s, pl-hypokinesis2 ia-akinesis, s-akinesis, pl-hypokinesis3 ia-(akinesis-hypokinesis)4 ia-(akinesis-hypokinesis)5 ia- (akinesi s-di skinesi s), s-akinesi s

6 ia-(akines is-diskinesis), s- akinesis7 normal

8 normal9 ia-hypokinesis, s -hypokinesis

and r= 0.98 [13] In conclusion, our method of automaticcontour detection can be considered satisfactory.

B. Verification ofRegional Motility Interpretatioais

Further tests have been performed for assessing the correut-ness of diagnostic descriptions which are automatically inferredby the system. For this reason, nine image sequences havebeen randomly chosen from the database. For each image se-quence, the system has been requested to give a regional mo-tility interpretation.A regional motility interpretation largely depends on fuzzy

membership functions as discussed in Section IV-B and graph-ically displayed in Fig. 16. After a proper adjustment of theparameters which define those functions, a good agreement be-tween the diagnostic interpretation given by the system andthat given by a physician has been achieved. In Table II, re-gional motility interpretations given by a physician for each ofthe test sequences are shown. Each of the special regionalmotility interpretations "ia-akinesis," "se-akinesis," etc., has acorresponding concept in the semantic net. For all sequencesunder study, exactly those concepts which have been found bythe physician have been instantiated by the system with a cer-tainty significantly above the certainty which as been inferredfor other concepts not found by the physician. For sequence1, e.g., the concepts "ia-akinesis,". "se-akinesis," and "pl-hypo-

w

257

x

x

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 3, MAY 1985

kinesis" have been instantiated by the system with a certaintyequal to one while the certainties for other concepts like "ia-hypokinesis" or "ia-dyskinesis," which are competing with"ia-akinesis," "se-akinesis," and "pl-hypokinesis," are equal tozero.Except for sequence 7 and 8, more than one disease has been

reported for one sequence by the physician, and by the systemas well. There are four sequences, namely 3, 4, 5, and 6,whereno "hard" decision was given by the physician. Instead, twoconcepts have been found suitable for describing the motionalbehavior. The regional motion interpretation "ia-(akinesis-hypokinesis)" for sequence 3 means, for example, the inferio-apical part of the left ventricle being hypokinetic (restricted inmotility) with a tendency to being akinetic (without motion)but not being completely akinetic. In sequence 3, 4, 5, and 6,only one of the two interpretations given by the physician hasbeen inferred by the system, respectively. These are the onlydifferences between man and machine observed so far. Theyare, however, not severe from a medical point of view.

In summary, the experiments performed until now havebeen very encouraging. A high agreement has been observedbetween a physician's results and those of the system, with re-spect to the location of contours and diagnostic interpretationsas well.

REFERENCES[1] A. V. Oppenheim and R. W. Schafer, Digital Signal Processing.

Englewood Cliffs. NJ- Pr-ntice-Hall. 1975.[2] W. K. Pratt, Digital Image Processing. New York: Wiley-Inte.

science, 1978.[3] N.V.Findler,Ed.,AssociativeNetworks. NewYork:Academic,

1979.[4] N. J. Nilsson, Principles of Artificial Intelligence. New York:

Springer, 1983.[5] H. Niemann, Pattern Analysis. Springer Series in Information

Sciences, vol. 4. New York: Springer, 1981.16] A. R. Hanson and E. M. Riseman, Eds., Computer Vision Sys-

tems. New York: Academic, 1978.[7] D. H. Ballard and C. M. Brown, Computer Vision. Englewood

Cliffs, NJ: Prentice Hall, 1982.[81 H. Niemann and Y. T. Chien, Guest Eds., Special Issue on Knowl-

edge Based Image Analysis, Pattern Recognition, vol. 17, no. 1,1984.

[9] R. W. Conners, C. A. Hailow, and S. J. Dwyer, "Radiographicimage analysis: Past and present," in Proc. 6th Int. Conf. PatternRecognition, Munich, West Germany, 1982, pp. 1152-1169.

[101 H. Feistel, "Nuklearmedizinische Herzfunktionsdiagnostik-Unter-suchungen wahrend der ersten Tracer Passage und im Verteilung-sgleichgewicht," Ph.D. dissertation, Med. Fakultat der UniversitatErlangen-Nurnberg, Niirnberg. West Germany, 1 982.

[11] E. Sauer and H. Sebening, 'Myokard- und Ventrikelszintigraphie,"Boehringer GmbH., Mannheim, West Germay, 1980.

[12] J. J. Gerbrands and C. Hoek, "Automated left ventricular bound-ary extraction from technetium-99m gated blood pool scintigramswith fixed or moving region of interest," in Proc. 2nd Int. Conf.Visual Psychophys. Med. Imaging, Brussels, Belgium, 1981. pp.155-159.

[13] H. Geffers, W. E. Adam, F. Bitter, H. Sigel, an H. Kampmann,"Data processing and functional imaging in radionuclide ventri-culographie," in 5th Int. Conf. Inform. ProcessingMed. Imaging,Nashville, TN, 1978. pp. 322-331.

[14] J. K. Tsotsos, H. D. Myleroulos, H. D. Cowey, and S. W. Zucker,'A framework for visual motion understanding," IEEE Itans.Pattern Anal. Mach. Intell., vol. PAMI-2, pp. 563-573,1980.

[15] Special Issue on Knowledge Representation, Computer, vol. lb,no. 1i,1l988.

[16] D. A. Waterman and F. Hayes-Roth, Eds., Pattern Directed In-ference Systems. New York: Academic, 1978.

[17] H. Bunke, H. Feistel, H. Niemnann, G. Sagerer, F. Wolf, and G. X.Zhou, "Smoothing, thresholding and contour extraction in imagesfrom gated blood pool studies," in Proc. Ist IEEE Comp. Soc.Int. Symp. Med. Imaging Image Interpretation, Berlin, West Ger-many, 1982, pp. 146-151.

[18] R. Brachman "On the epistemological status of semantic net-works." in Associative Networks, N. V. Findler, Ed. New York:Academic, 1979.

[19] H. Niemann and G. Sagerer, "A model as part of a system for in-terpretation of gated blood pool studies of the human heart." inProc. 6th int. r7onf. Pattern Recognition, Mtunich. West Geuis,any1982, pp. 16-i8.

[20] E. G. Hawman, "Digital boundary detection techniques for theanalysis of gated cardiac scintigrams," Opt. Eng., vol. 20, no. 5,pp. 719-725, 1981.

[21] M. L. Goris, J. H. McKillop, and P. A. Briandet, "A fully auto-mated detennination of the left ventricular region of interest innuclear angiocardiography," Cardiovascular Int. Radiol., vol. 4,pp. 117-123, 1981.

[22] 1. Azancot, D. Vannier, A. W. Kedra, M. Rosengarten, R. Slama,and Y. Bouvrain, "Angiographic frame-by-frame computerizedtvaluation of overall and regional left ventricular wall motion,"Comput. Cardiol., 1978, pp. 281-284.

[23] R. W. Brower and G. T. Meester, "Spatial resolution and correla-tion between segments in regional wall motion studies of the leftventricle," Comput. Cardiol., pp. 69-75, 1978.

[24] H. Bunke. "Segmentation of the left ventricle in ;cintigraphicimage sequunces," presented at Digital Signal Processing, 1984,Florence, Italy.

[25] H. Bunke, K. Grebner, and G. Sagerer, "Syntactic analysis ofnoisy input strings with an application to the analysis of heart-volume cuives," presented at 7th mnt. Conf. Pattern Recognition,Montreal, P.Q., Canada, 1984.

[26] L. A. Zadeh, "Fuzzy sets," Inform. Contr., vol. 8, pp. 338-353,1965.

[27] H. Bunke, G. Sagerer, and H. Niemann, "Model based analysis ofscintigraphic image sequences of the human heart," in Image Se-quence Processing and Dynamic Scene Analysis, T. Huang, Ed.New York: Springer, 1983, pp. 724-740.

Heinrich Niemann received the Dipl.-lng. de-gree in electrical engineering in 1966 and theDr.-lng. degree in 1969 from Technische Uni-versitat Hannover, llannover, West Germany.From 1966-1967 he was at the University of

Illinois, Urbana, and from 1967-1972 at Fraun-hofer Institut fur Informationsverarbeitung inTechnik und Biologic in Tiibinven and Karlsruhe,West Germany, and trom 1973-197T at I-ach-hochschule Giessen. Since 1975 he has beenProfessor in Informatik (computer science) at

Universitat Erlangen-Niirnberg, Erlangen, West Germany. His field ofresearch is speech understanding and knowledge-based image analysis.

Horst Bunke was born in Langenzenn, WestGermany, on July 30, 1949. He received thediploma and the Ph.D. in computer sciencefrom the University of Erlangen, Erlangen,West Germany in 1974 and 1979, respectively.From 1976 to 1984 he was with the Univer-

sity of Erlangen. In 1980-1981 he was onleave visiting Purdue University, West Lafayete,IN. In 1983 he temporarily was with the Uni-versity of Hamburg, West Germany. Since 1984,he has been Professor of Computer Science at

the University of Berne, Switzerland. His research interests includesyntactic and structural pattern recognition and artificial intelligence.

258

NIEMANN etaaL: KNOWLEDGE BASED SYSTEM FOR GATED BLOOD POOL STUDIES

Ingrid Hofmann received the Dipl.-lnf. degreein informatics (computer science) in 1983 fromthe University of Erlangen-NUrnberg, Erlangen,West Germany.Since 1983 she has been with the Institut fUr

Informatik, Mustererkennung (pattern recogni-tion), University of Erlangen-NUrnberg. Herfield of research is knowledge-based image anal-ysis and artificial intelligence.

Gerhard Sagerer received the Dipl.-Inf. degreein informatics (computer science) in 1980 fromthe University of Erlangen-NUrnberg, Erlangen,West Germany. Since 1980 he has been withthe Institut fUr Informatik, Mustererkennung(pattern recognition) at the University ofErlangen-Nffrnberg. His field of research isknowledge-based image analysis, speech analy-sis, and artificial intelligence.

a . Friedrich Wolf was born on August 5, 1930.i-l He received the M.D. degree in 1954.

He was a Lecturer on Internal Medicine and_ Nuclear Medicine in 1965 and served as chair-

professor and director of the Institut andPoliclinic of Nuclear Medicine at the Universityof Erlangen-Nirnberg, Erlangen, West Ger-many, since 1973. In 1983 he became Dean ofthe Faculty of Medicine at the same institute, aposition he currently holds. His main interestsare radionuclides in pathophysiology and diag-

nostics, emission tomography.In 1980 he was elected President of the European Society of Nuclear

Medicine.

Herbert Feistel received the informatics di-- _ ploma (computer science) and the M.D. degree

from the University of Erlangen-Nurnberg,Erlangen, West Germany, in 1975 and 1982,

- respectively.Since 1980 he has been an Assistant Physician

at the Institut fur Nuklearmedizin of the Uni-versity of Erlangen-Niirnberg. He has been aSpecialist in Nuclear Medicine since 1984.

259