Knowledge Engineering and Data MiningJiajuan LinJohn LarimerLopa NathNick RemishSean RuckTyler Purdom
What Is Knowledge Engineering?John Larimer
Knowledge Engineering
It is the process of building intelligent systems. 1. Problem Assessment2. Data & knowledge acquisition3. Prototype4. Complete System5. Evaluation & Revise6. Integration & Maintain System
Knowledge Engineering
1. Problem Assessment● Determine the problem's characteristics● Identify the main participants in the project● Specify the project's objectives● Determine the resources need for building the
system Types of problems: Diagnosis, selection, prediction, classification, clustering, optimization, control E.g. Diagnosis -> domain knowledge, explanation facilities
Knowledge Engineering
2. Data and Knowledge Acquisition● Collect and analyse data and knowledge● Make key concepts of the system design more
explicit
Intelligent System
Knowledge Engineering
3. Development of a Prototype System● Choose a tool for building an intelligent system● Transform data and represent knowledge● Design and implement a prototype● Test the prototype
Knowledge Engineering
4. Development of a complete system● Prepare a detailed design for a full-scale system● Collect additional data and knowledge● Develop the user interface● Implement the complete System
Knowledge Engineering
5. Evaluation and revision of the system● Evaluate the system against the performance
criteria● Revise the system as necessary
Knowledge Engineering
6. Integration and Maintenance of the System● Make arrangements for technology transfer● Establish an effective maintenance program
Case Study 1: Diagnostic Expert System
I want to develop an Intelligent system that can help me to fix malfunctions of my Mac computer. Will an expert system work for this problem?
Case Study 1: Diagnostic Expert System
Phone Call Rule (Firebaugh, 1988) - "Any problem that can be solved by your in-house expert in a 10-30 minute phone call can be developed in an expert system." ● Troubleshooting manuals● Series of visual inspections● Rule structure with domain knowledge
Case Study 1: Diagnostic Expert System
Taken from N. Ch9 Pg 309 & 310
Case Study 2: Classification Expert SystemNick Remish
Case Study 2: Classification E.S.
● Problem: Identify different classes of sailboat (typical classification problem)○ Handled well by both expert systems and NNs
● Collect information○ In this case, the sail plan can help identify the class
of boat.● Issues with the expert system approach
○ What if the information is incomplete or inexact? (Rough weather obscuring sails)■ Manage incrementally acquired evidence with
certainty factors.
9.3 Will a fuzzy expert system work for my problem?
● A Fuzzy Solution?○ Useful when you cannot define a set of exact rules.○ Great for inherent imprecise properties and
modeling human decision making.■ Sometimes parameters are imprecise (a doctor
dealing with a patient)○ Mainly used in engineering, but has applications in
any sector that relies on human experience or is too complex or uncertain (ex: finance)
Case Study 3: Decision-Support Fuzzy System
Decision-Support Fuzzy System
● Problem: assessing mortgage applications○ Use a Decision-Support Fuzzy System
● Steps:○ Represent the concept in fuzzy terms○ Implement the concept in a prototype○ Test and optimize
Decision-Support Fuzzy System
● Represent the concept in fuzzy terms: Triangular and trapezoidal fuzzy membership functions are used to represent knowledge.
Decision-Support Fuzzy System
● Obtain the fuzzy rules:Based off of Von Altrock'sfuzzy model and appliedto mortgages.
Decision-Support Fuzzy System
● Evaluate and analyse performance: ● Despite having 100+ rules, decision-support
fuzzy systems can be developed, tested and implemented relatively quickly.
9.4 Will a neural network work for my problem?Sean Ruck
Neural Network Overview
● Very powerful, general purpose tools● Successfully applied to prediction,
classification, and clustering problems● Quite popular due to the versatility of
neural networks
Case Study 4: Character Recognition Neural Networks
● Suppose you want to copy a document onto your computer without retyping the whole thing.○ How?
● Optical Character Recognition○ The ability of a computer to translate character
images into a text file using software○ Capture the character images by scanning the
document■ Converts the scanned document into a bit map
Choosing The Neural Network Architecture
● Architecture and size of neural newtork dependent upon complexity of the problem○ Handwritten character recognition is far more
complex than computer printed● A 3-layer network will suffice for printed
digit recognition
Determining an optimal number of hidden neurons
● More neurons leads to a more accurate network, but takes longer to train
● Too many neurons may actually prevent the network from generalising or working for anything other than training examples○ Overfitting
● How to prevent overfitting○ Choose the smallest number of neurons that give
good results and generalisation
cont'd.
● We should test out the training of the network with various numbers of hidden neurons○ Performance rated by sum of squared errors
■ The training runs that have a good enough sum of squared errors result have a number of hidden neurons to consider using
Test Examples?
● The test set should be entirely independent of the training examples○ Only use the training runs that passed the previous
test● Test examples should also contain "noise"
○ Distortion of the input● The training runs that give us a reasonable
error in recognition even with noise have a good enough number of hidden neurons to use○ Use the lowest number for practical purposes
Improving Performance
● A neural network is only as good as the examples used to train it
● Improve the network by training it with noisy examples
Case Study 5: Prediction Neural Networks
● Neural networks are useful in prediction situations such as predicting the market value of a house
● Using a neural network creates a black box around how the results were reached○ The result is more important than the how anyway
● For prediction training examples are critically important○ We need a wide array of examples to cover all
possible inputs
Determining The Size Of A Training Set
● Can be estimated with "Widrow's rule of Thumb": N = nw/e○ Where N is the number of training examples, nw is
the number of synaptic weights in the network, and e is the network error permitted
Dealing With The Data
● Neural networks work best with inputs in the 0 to 1 range, but in cases such as with determining the value of a house, our inputs are not all in that range○ Number of bedrooms, square footage, etc.
● So we need to "massage" the data to this range○ massaged value = (actual value - minimum value) /
(maximum value - minimum value)○ Good for up to a dozen possible values
cont'd
● We can also utilize 1 of N coding○ Each possible value is taken as its own input each
with a value of 0 or 1
Dealing With The Results
● To validate the results we test the network with never before seen examples, as before
● Our network is working with values between 0 and 1. We need to convert back to actual values○ We can reverse the "massaging" we did before
● To test the importance of certin inputs we can test the network's sensitivity to them: "Sensitivity Analysis"○ Set each input one at a time to its minimum and
maximum values and measure the results
Case Study 6: Classification Neural Networks With Competitive Learning
● Using a neural network we can discover significant features of input patterns and separate the data into different classes
● Using competitive learning a single layer neural network can perform clustering○ Combining similar data into groups or clusters○ Uses 1 input neuron for each input and 1
competitive neuron for each cluster
When Is The Learning Process Completed?
● In a competitive neural network, there is no obvious way to know if the network is done learning○ We do not know the desired output, so we cannot
use the sum of squared errors● Use Euclidean Distance criterion instead
○ When there has been no noticeable change in the weights of the competitive neurons, the network can be considered to have converged
How Can We Associate Neurons to Specific Classes or Clusters?
● Competitive neural networks enable us to identify clusters in input data, but does nothing to label the clusters○ We can connect a competitive neuron with a
cluster/class by analyzing its weights ● We can identify exactly which cluster is
which by feeding the network test data corresponding to one particular cluster○ The output neuron that most often is utilized is
labeled as that class
9.5 Will genetic algorithms work for my problem?Lopa Nath
Genetic Algorithm Review
● Most applicable to optimization problems○ Process of finding a better solution to a problem
■ More than one solution not of equal quality● Generates a population of competing
candidate solutions● Causes candidates to evolve through
process of natural selection○ Poor solutions die out while better solutions survive
and reproduce● Process repetition breeds an optimal
solution
Case Study 7: The Traveling Salesman Problem
● I want to develop an intelligent system that can produce an optimal itinerary. I am going to travel by car and I want to visit all major cities in Western and Central Europe and then return home. Will a genetic algorithm work for this problem?○ Known as the traveling salesman problem (TSP)
○ Given a finite number of cities, and the cost of travel (or the distance) between each pair of cities, we need to find the cheapest way (or shortest route) for visiting each city exactly once and returning to the starting point.
○ TSP naturally represented in numerous transportation and logistics applications.
■ Arranging routes, scheduling drilling of holes in a circuit board (time efficient - shortest distance)
● Although we can not be completely sure if the selected route is the best one, after several runs we can be sure that the route obtained is a good one.
How does a genetic algorithm solve the TSP?
Representation
Chromosome where order of integers represents order in which cities will be visited.
Genetic Operators in the TSP
● Genetic operators used to create new routes● Crossover Operator
○ Classical form cannot be directly applied because a simple exchange of parts between parents would contain duplicates and omissions.
Clearly classical crossover with single crossover point does not work.
How the Crossover Operator Works
Genetic Operators in the TSP Continued
● Mutation Operator○ Reciprocal Exchange
■ Simply swaps two randomly selected cities in the chromosome
○ Inversion■ Selects two random points along the
chromosome string and reverses order of cities between these points
Fitness Function in the TSP
● Evaluate total length of the route○ Fitness of each individual chromosome is
determined as the reciprocal of the route length● Shorter the route, fitter the chromosome
9.6 Will a hybrid intelligent system work for my problem?
Hybrid Intelligent Systems
● Solving complex real-world problems require an application of complex intelligent systems that combine the advantages of expert systems, fuzzy logic, neural networks, and evolutionary computation.
● Such systems can integrate human-like expertise in a specific domain with abilities to learn and adapt to a rapidly changing environment.
Case Study 8: Neuro-fuzzy decision-support systems
● I want to develop an intelligent system for diagnosing myocardial perfusion from cardiac images. I have a set of cardiac images as well as the clinical notes and physician's interpretation. Will a hybrid system work for this problem?○ Analysis of two SPECT images must be done
■ One stress image taken 10-15 minutes after injection with radioactive tracer
■ One rest image taken 2-5 hours after the injection
○ Brighter patches on image correspond to well-perfused areas while darker patches may indicate the presence of an ischemia.
○ Visual inspection is highly subjective--intelligent system can help a cardiologist diagnose.
○ One binary feature assigns an overall diagnosis--normal or abnormal
● The neuro-fuzzy system in this example has a heterogeneous structure - the neural network and fuzzy system will work as independent components but cooperate in solving the problem.
Back-Propagation Neural Network to Classify the SPECT Images into Normal and Abnormal
● Each image is divided into 22 regions, so we need 44 input neurons.
● Since SPECT images are to be classified as either normal or abnormal, we should use two output neurons.
● Good generalization in this study can be obtained with 5 to 7 neurons in the hidden layer.
Testing the Neural Network
● Testing the network, we find the network's performance is rather poor○ 25% normal are misclassified as abnormal○ Over 35% abnormal are misclassified as normal○ Indicates that the training set may lack some
important examples● Can improve this still
Neural Network Output
● Two outputs○ First - possibility that the SPECT image belongs to class
normal○ Second - possibility that the SPECT image belongs to class
abnormal
● Examples:○ NORMAL OUTPUT HIGH AND ABNORMAL OUTPUT LOW First
(normal) output is 0.92 and second (abnormal) is 0.16 - image classified as normal - risk for heart attack is low
○ NORMAL OUTPUT LOW AND ABNORMAL OUTPUT HIGH First (normal) output is 0.17 and second (abnormal) is 0.51 - image classified as abnormal 0 risk for heart attack is high
○ BOTH OUTPUTS ARE CLOSE First (normal) output is 0.51 and second (abnormal) is 0.49 - we cannot confidently classify the image.
Adding Fuzzy Logic for Decision-Making in Medical Diagnosis
● Fuzzy logic provides us with a means of modeling how the cardiologist asses the risk of a heart attack.
● Need to determine input and output variables, define fuzzy sets, and construct fuzzy rules.○ Two inputs (NN output 1 and NN output 2) and one output (the risk
of a heart attack).■ Inputs [0, 1] and output vary between 0 and 100 percent.
○ Fuzzy sets shown in Negnevitsky page 342 and 343 - Figure 9.33, Figure 9.34, and Figure 9.35
○ Fuzzy rules in Negnevitsky page 343 - Figure 9.36
■ Examples:1. If (NN-output1 is Low) and (NN_output2 is Low) then (Risk is Moderate)2. If (NN-output1 is Low) and (NN_output2 is Medium) then (Risk is High)3. If (NN-output1 is Low) and (NN_output2 is High) then (Risk is Very_High)4. If (NN-output1 is Medium) and (NN_output2 is Low) then (Risk is Low)
More Certainty
● Risk between 30 and 50 percent cannot be classified as either normal or abnormal - uncertain.
● Apply the following heuristics known by experienced cardiologists to all corresponding regions (22 in each image)
1. If perfusion inside region i at stress is higher than perfusion inside the same region at rest, then then risk of a heart attack should be decreased.
2. If perfusion inside region i is not higher than perfusion inside the same region at rest, then the risk of a heart attack should be increased.
Three Heuristics Implemented In the Diagnostic SystemStep 1 Present the neuro-fuzzy system with the cardiac case.
Step 2 If the system's output is less than 30, classify the presented case as normal and then stop. If the output
is greater than 50, classify the case as abnormal and stop. Otherwise go to step 3.
Step 3 For region 1, subtract perfusion at rest fro perfusion at stress. If the result is positive, decrease the
current risk by multiplying its value by 0.99. Otherwise, increase the risk by multiplying its value by
1.01. Repeat this procedure for all 22 regions then go to Step 4.
Step 4 If the new risk value is less than 30, classify the case as normal; if the risk is greater than 50, classify
the case as abnormal; otherwise, classify the case as uncertain.
● When we now apply the test set to the neuro-fuzzy system, we find that the accuracy of diagnosis has dramatically improved - the overall diagnostic error does not exceed 5 percent, while only 3 percent of abnormal cases are misclassified as normal.
● Although we have not improved the system's performance on normal cases (over 30 percent of normal cases are misclassified as abnormal), and up to 20 percent of the total number of cases are classified as uncertain, the neuro-fuzzy system can actually achieve even better results in classifying SPECT images than a cardiologist can.
Homogeneous Structure of Neuro-Fuzzy Systems
● A typical example of a neuro-fuzzy system with a homogeneous structure is an Adaptive Neuro-Fuzzy Inference System (ANFIS).○ It cannot be divided into two independent distinct
parts.○ An ANFIS is a multilayer neural network that
performs fuzzy inferencing.● Case Study 9: Time series prediction
○ Page 346 of Negnevitsky
Data Mining and Knowledge DiscoveryTyler Purdom
Data Mining
● Definition:○ The extraction of knowledge from data○ The exploration and analysis of large quantities of
data to to discover patterns.● Ultimate goal is to discover knowledge● Amount of data doubles every year● Important to have fast algorithms to
process data
Data Warehouses
● Definition:○ Large databases that store historical data.○ Contain millions and in some cases billions of data
records.● The data stored is time dependent and
integrated● Used to help support decision making● Query tools are used to discover
relationships in the data.
Query Tools vs. Data Mining
● Query tools are assumption-based○ User must ask the right questions to get result○ User must make assumptions○ Can select a specific variable that affects the
outcome● Data Mining tools determine the most
significant factors○ No assumptions are necessary○ Discovers patterns automatically
● The representation of data in data warehouses helps facilitate the data mining process
Data Mining Practice
● Data Mining is a new and evolving field● Very popular in the banking, finance,
marketing, and telecommunications industries
● Data Mining uses:○ Determine trends in markets○ Detect frauds○ Target people most likely to buy a product/use a
service
Data Mining Tools
● People used to use query tools and statistics to solve data mining problems○ These techniques are not very efficient for large
amounts of data○ Can only correlate a few variables at a time
● Now, tools are based off of intelligent technologies:○ Neural networks, neuro-fuzzy systems, and decision
trees● Decision trees are currently the most
popular tool used for data mining
Decision Tree
● A map of the reasoning process● These trees do not allow for the use of
noisy or incomplete data● Uses tree structure to describe the data set● Very effective in solving classification
problems● Popular because they help you visualize the
problem● Nodes are separated by predictors
○ In the book example, homeownership was used to split the tree
Decision Tree Example
Gini Coefficient
● A measure of how well the predictor separates the classes contained in the parent node
● Introduced by Corrado Gini, an Italian economist
● He used it to measure the inequality in Italy's income distribution
Calculating the Gini Coefficient
● Top curve represents the real economy
● Bottom line represents equal distribution of wealth
● Coefficient: ○ (shaded area)/ area
below bottom line
Gini Split Example
SummaryJiajuan Lin
Summary - Knowledge engineering
● What is knowledge engineering?○ Problem Assessment○ Data & knowledge acquisition○ Prototype○ Complete System○ Evaluation & Revise○ Integration & Maintain System
Summary - Assess the Problem
● Assess the Problem○ problem type
■ diagnosis, selection, prediction, classification, clustering, optimization, control
○ availability of data■ precise data? complete set input?
○ form of content of the solution■ final result only? reasoning behind the answer?
○ availability of expertise■ extra info provided? trouble to present problem
solving strategy?
Summary - Data & Knowledge acquisition
● Questions about the data○ Range? Continues? Discrete? Precise? Noise
Tolerance? Numerical? Symbolic?
● Data Mining○ analyze data, finding pattern & rules, extracting
knowledge from large quantities of data○ decision tree
■ easy to follow■ visualization of solution■ makes clear sets of rules
Summary - Prototype
● shows understanding of ○ the problem○ problem-solving strategy○ tool selected
● Test○ Throw it away if needed○ Forcing wrong tool leads to more time waste in the
later development process○ Prototype is there for discovering any
inappropriate/wrong decisions made
Summary - Complete System,Evaluation ,Revision, Integration & maintenance
● Complete System Development ○ plan, schedule, budget
● Evaluation○ no clear right/wrong○ user satisfaction = measurement
● Revision○ Modify as limitation & weaknesses discovered
● Maintenance○ Knowledge evolves over time○ keep modifying and updating to maintain efficiency
and accuracy