Diagnostic Systems (I):Rule-Based Expert Systems

and the MYCIN Project

Rule-Based Expert Systems:Suitable Domains

• Many Rules• No Unifying Theorem• Knowledge can be easily separated from the way

it is used• Updating the knowledge base has to be easy• The knowledge base can be the only [indirect]

communication channel among rules• Clinical/psychological and other domains, rather

than mathematical/physical domains

MYCIN: The Problem

• Roberts & Visconti [1972]:– Only 13% of patients are treated rationally– 66% are being given irrational treatment– 21% are being given questionable treatment

• Irrationality means, for example:– Using a contra-indicated combination– Using the wrong agent for a specific organism– Not taking the required cultures

Stages in Diagnosis and Treatment

• Decide if there is a significant infection

• Identify the causing organism(s) by clinical and laboratory evidence

• Decide what antibiotic agent the organisms are sensitive to

• Prescribe the optimal drug combination for the particular case

A MYCIN Runtime Example

The MYCIN Architecture

Consultation program

Explanation program

Knowledge-acquisition program

Dynamic patient data

Static factual & judgmental knowledge

Physician user

Infectious diseases expert

The Knowledge Base• Inferential knowledge stored in decision rules

– If Premise then Action (Certainty Factor [CF])– If A&B then C (0.6)– The CF represents the inferential certainty

• Static knowledge:– Natural language dictionary– Lists (e.g., Sterile Sites)– Tables (e.g., gram stain, morphology, aerobicity)

• Dynamic knowledge stored in the context tree– Patient specific– Hierarchical structures: Patient, cultures, organisms– <Object, Attribute, Value> triples: <Org1, Identity, Strep>– A CF used for factual certainty <Org1, Identity, Staph, 0.6>

Example of a Decision RuleRULE-507

IF:1. The infection which requires therapy is

meningitis2. Organisms were not seen on the stain of the

culture3. The type of the infection is bacterial4. The patient does not have a head injury

defect5. The age of the patient is between 15 and 55

yearsThen:

The organisms that might be causing the infection are diplococcus-pneumoniae and neisseria-meningitidis

A Sample Context Tree

The Rule Interpreter

• Control structure: goal driven, backward chaining• Attempt to establish values of clinical parameters at

the leaf nodes• The interpreter retrieves a list of rules whose

conclusions bear on current goals, and tries to evaluate these rules

• Questions are asked only when the rules fail to deduce the necessary information

• If the user cannot supply the information, the rule is ignored

The Goal Rule

RULE-092

IF:1. There is an organism which requires therapy2. Consideration has been given to the possible existence

of additional organisms requiring therapy, even though they have not actually been recovered from any current cultures

Then: Do the following:1. Compile the list of possible therapies which, based

upon sensitivity data, may be effective against the organisms requiring treatment

2. Determine the best therapy recommendations from the compiled list

Else: Indicate that the patient does not require therapy

A MYCIN Reasoning Tree

The Main MYCIN Algorithm

-Uses Monitor and FindOut to recursively invoke each rule when relevant

The Monitor Mechanism

The FindOut Mechanism

Certainty Factors• Not a Bayesian probability measure, but rather a Certainty Factor

(CF) with its update functions• A Conclude function uses

– The CF of the rule used for making the inference– The minimal CF of the premises (using the Tally function)– The context node about which the conclusion is made– The clinical parameter whose value is added to the dynamic DB– The value of the clinical parameter

• Conclude derives a conclusion including the CF of the result– E.g., “There is suggestive evidence (0.7) that the identity of the organism is

streptococcus”– The CF is mapped into English

• The CF of a context is updated by other evidence (relevant rules)• It is always true that -1 ≤ CF ≤ +1• If CF = +1 then all other hypotheses are rejected

Question Answering

• General question answering module– Used after a consultation regarding any aspect of it– General questions about the knowledge base (e.g., “is

blood a sterile site?”)– Uses a simple keyword pattern-matching approach

• Online reasoning status checker provided during consultation– A limited set of explanations for the reasoning line– HOW: recapitulates all relevant successful rules used on

the path to the conclusion asked about– WHY: backtracks to the reason that a question (a

particular bit of information) was asked

In this series, how many of the cultures were positive for the e.coli {typical system prompt}(ORGANISM-1)?

—WHY {request from user to determine motivation for question}[i.e. WHY is it important to determine the number of cultures which {system’s rephrasing of the WHY question, were positive for the organism?] and answer}

[1.0] This will help to determine whether there is significant diseaseassociated with this occurrence of the organism.

It has already been established that[1.1] the culture was taken from a sterile source[1.2] the number of cultures in the series is greater than 2Therefore, if[1.3] the number of cultures which were positive for ORGANISM-1 is greater than 2Then there is suggestive evidence (.5) that there is significant diseaseassociated with this occurrence of the e.coli (ORGANISM-1).

[RULE049]— WHY {repeated WHY to determine relevance of previous explanation}

[i.e. WHY is it important to determine if there is significant disease associated with this occurrence of the organism?][2.0] This will help to determine a diagnosis for which the infection should be treated.it has already been established that[2.1] the infection is not one of: primary-bacteremia meningitis[2.21 for some current culture of the patient it is true that the site of the culture is bloodTherefore, if[2.3] there is significant disease associated with this occurrence of the organismThenit is definite (1.0) that the diagnosis for which the infection should be treated is secondary-bacteremia[RULE103]

Example: The WHY Explanation Capability

Knowledge Acquisition

• The knowledge base (KB) has to be modified and expanded continuously– The knowledge-acquisition (KA) bottleneck

• The TEIRESIAS module enabled interactive KA from medical experts– Included a rule model for different types of rules,

which creates expectations about the structure of the acquired rule

– A new or old rule can be created or modified interactively using meta-knowledge about rule models

– KA important also in the context of a reasoning error

Therapy Selection

• Originally a combination of Therapy Rules and a LISP procedure– A list of potential therapies is created– The best combination of drugs is selected

• Resulted in a context tree of possible therapies under the “current organism” node

• Replaced by a clearer version enabling explicit explanations, as well as optimal therapy, using a Plan, Generate, and Test strategy– more appropriate, since the therapy-planning task is really

a configuration task

Improved Therapy Selection

• Plan by re-ranking the potential drugs, using local factors (e.g., organism sensitivity, drug toxicity, current therapy continuity)

• Propose a recommendation and Test it, using global factors (e.g., minimize total number of drugs)– Proposals are managed using a canonical instruction set– Testing uses rules to check for proper coverage, unique drug classes, and

patient-specific recommendations– The first satisfactory proposal is chosen

• Prescribe final recommendation– Uses algorithmic dosage calculation and patient-specific adjustment

Clinical Evaluation of MYCIN[Yu et al., Comp. Prog. Biomed. 9, 1979]

• Objective assessment of the basic (bacteremia) system

• Examined the main three decision-making steps:1. Decide if there is a significant organism

2. Determine organism identity

3. Recommend therapy, including alternatives

The Evaluation Method

• 15 patients with positive blood cultures (at least one organism)

• 5 Stanford infectious disease experts• 5 experts from other hospitals• All data recorded and given, if asked for, by

the computer or a human expert• All decisions by the computer or the experts

recorded, including the majority opinion

Results of the Evaluation Study• Significant-organism decision:

– MYCIN decided identically in 97% of the 150 (15x10) decisions– MYCIN decided identically in 1000% of the 15 majority decisions

• Organism-identification decision:– MYCIN identified 45 organisms in the 11 cases requiring therapy, thus 450 (45x10) organism

judgements– Agreement in 80% of Stanford experts’ judgements, 72% of others– Agreement by the majority of experts: 90%

• Treatment decision: treatment was suggested to all 11 patients requiring it, thus 110 (11x10) decision instances to compare

– Agreement with MYCIN’s recommendation was 76% for Stanford experts, 69% for the others– Agreement by a majority: 90% for Stanford experts, 73% for the others– In one case, agreement of 4/5 Stanford experts, disagreement of 4/5 others! the KB represented well the Stanford prescription habits

• Overall “acceptable” performance was rated in 93% (14/15) of cases• Several design problems, such as unblinded evaluation of the program’s and experts’

performance, were corrected in a later study• MYCIN was actually ranked best, relative to all other experts, in a blinded evaluation

Summary: Rule-Based Expert Systems and the MYCIN Project

• Task: Diagnosis and treatment of infectious diseases• Problem solving method: Heuristic Classification [Clancey, 1985]

– Data->Abstracted data=>Abstracted solutions->Solutions• Implementation: Backward-chaining production rules• Evaluation results: Surprisingly good for a research tool

– Different evaluation by Stanford and Other experts stems probably from different local practices. This might be actually considered as a representational success. (Rules and tables can be modified easily).

• Many technical and conceptual problems prevented clinical use (small memory, slow CPU, medical DB communication problems, stand-alone system, etc.), several of which are now solvable

• At the time of the first study, MYCIN rules included only bacteremia (meningitis and endocarditis were added later), thus never tested in a real clinical environment with general infections

• Practically no temporal reasoning• Implicit control — hard to modify• Probabilistic model was not Bayesian and not intuitive• The knowledge-acquisition bottleneck remained significant