View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Case-Based Reasoning – through 3 Case-Based Reasoning – through 3 applications applications
Copyright Copyright ©© 2006 Reich 2006 Reich
Prof. Yoram ReichProf. Yoram Reich
Faculty of EngineeringFaculty of EngineeringTel Aviv UniversityTel Aviv University
IsraelIsrael
Copyright © 2004 Reich 2
Outline
Generic learning tasks The basic concept Past work
eCobweb, eProtos Present work
Conversational CBR (Gil Chen) CBR with weak causal knowledge (Adi Kapeliuk)
KDML – Knowledge Discovery Modeling Language
Copyright © 2004 Reich 3
Design of Cable-Stayed Bridges
Other examples of cable-stayed bridges
Copyright © 2004 Reich 4
Cable-Stayed Bridge Design: Specification Properties
8 specification properties
Copyright © 2004 Reich 5
Cable-Stayed Bridge Design: Product Description Properties
30 product description properties
Copyright © 2004 Reich 6
Cable-Stayed Bridge Design: Derived Properties
Aesthetics
Copyright © 2004 Reich 7
Cable-Stayed Bridge Design: Analysis Properties
Analysis according to American Bridge Design Code 15 analysis properties
Copyright © 2004 Reich 8
GLTs – Supporting Design Activities
Concept formation (unsupervised) – Synthesis
Concept learning (supervised) – Analysis, Redesign, Evaluation
Copyright © 2004 Reich 9
Mapping Design Tasks to GLTs and ML Programs
Copyright © 2004 Reich 10
Mapping problems into generic learning tasks
Y. Reich, Macro and micro perspectives of multistrategy learning, in Machine Learning: A Multistrategy Approach, Vol. IV (R. S. Michalski and G. Tecuci, eds.), (San Francisco, CA), pp. 379–401, Morgan Kaufmann, 1994.
Copyright © 2004 Reich 13
Bridger’s architecture
Copyright © 2004 Reich 14
eCobweb – Classification Hierarchy
Copyright © 2004 Reich 15
Example of COBWEB operators from 2nd application
Copyright © 2004 Reich 16
Case Structure & Representation
Case
ID Name DescriptionDate Version
Keyword1
Keywords
Keyword2...
KeywordN
Environment1
Environments
Environment2...
Environment4
Topic1
Topics
Topic2...
TopicM
Q/A1
Q/A
Q/A2...
Q/A J
Action1
Solution
Action2...
ActionK
Automaticallyadded by thesystem
Copyright © 2004 Reich 17
Incremental Conceptual Clustering – COBWEB
Operators
Put in Existing Class
Create a New Class
Merge
Split
C0
C1 C2 C3
C4 C5
p11
p1
p2
p3 p4 p5
p6 p7 p8 p9 p10
Copyright © 2004 Reich 18
Copyright © 2004 Reich 19
Copyright © 2004 Reich 20
Copyright © 2004 Reich 21
Copyright © 2004 Reich 22
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
Initial State:
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
Constrain sketch entities (1)Imported dimension's style (1)Import ACAD border (4)Import prof iles (5)OLE objects and bitmaps (2)
Change units of dif ferentformats (7)Wrong units and settings (9)
Missing 3D curve entitiesb (6)Missing solids (5)Missing sketch entities (7)
import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)
import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)
Drawing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16)Assembly (16)
Data Exchange (16)Settings/Defaults (16)
Part (18)Assembly (18)
Data Exchange (18)
Copyright © 2004 Reich 23
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)
import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)
Drawing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16)Assembly (16)
Data Exchange (16)Settings/Defaults (16)
Part (18)Assembly (18)
Data Exchange (18)
2.85Category
Utility =
Copyright © 2004 Reich 24
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)
import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)
Drawing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16)Assembly (16)
Data Exchange (16)Settings/Defaults (16)
Part (18)Assembly (18)
Data Exchange (18)
Category
Utility = 3.32
Copyright © 2004 Reich 25
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)
import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)
Drawing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16)Assembly (16)
Data Exchange (16)Settings/Defaults (16)
Part (18)Assembly (18)
Data Exchange (18)
Category
Utility = 3.13
Copyright © 2004 Reich 26
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)
import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)
Drawing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16)Assembly (16)
Data Exchange (16)Settings/Defaults (16)
Part (18)Assembly (18)
Data Exchange (18)
3.05
New Category )1 case(
Import (1)STEP (1)Change (1)Units (1)Control (1)
Part (1)Assembly (1)
Data Exchange (1)Settings/Defaults(1)
Category
Utility =
Copyright © 2004 Reich 27
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
2.94Category
Utility =
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS" + "MISSING DATA"(34 cases)
import (13) prof ile(3)dxf (10) part (3)dw g (10) dimension(3)sketch (6) w iref rame (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)
import (16 + 18 = 34) VDA (3 + 4 = 7) units (16)ACIS (3 + 4 = 7) control (12) scale (2)change (10) VRML (2 + 2 = 4) incorrect (7)STL (2) IGES (3 + 4 = 7) STEP (5 + 2 = 7)image (2) missing (17) curve (11)sketch (10) solid (5) geometry (4)complete (3)
Draw ing (11)Part (7)Assembly (2)
Data Exchange (13)2D Sketch (2)
Part (16 + 18 = 34)Assembly (16 + 18 = 34)
Data Exchange (16 + 18 = 34)Settings/Defaults (16)
"IMPORT"
"DXF/DWG"(13 cases)
"UNITS"(16 cases)
"MISSING DATA"(18 cases)
Constrain sketch entities (1)Imported dimension's sty le (1)Import ACAD border (4)Import prof iles (5)OLE objects and bitmaps (2)
Change units of dif ferentformats (7)Wrong units and settings (9)
Missing 3D curve entitiesb (6)Missing solids (5)Missing sketch entities (7)
Merge 2 highest
scores 2.85 3.32 3.13
Copyright © 2004 Reich 28
Incremental Conceptual Clustering – COBWEB
“Good”/”Bad” Category Utility Samples
Operators CU values
DXF/DW G Units New Merger2.6
2.8
3
3.2
3.4
Missing Data
CU
Copyright © 2004 Reich 29
eCobweb – CU – Category Utility
Y. Reich, “Constructive induction by incremental concept formation,” in Artificial Intelligence and Computer Vision (Y. A. Feldman and A. Bruckstein, eds.), pp. 191–204, Amsterdam: Elsevier Science Publishers, 1991.
Copyright © 2004 Reich 30
eCobweb characteristics
Property-value pairs that describe each category:
The probability to get an attribute value given that a case
belongs to a class
The probability to be in a class given that a case has a
particular attribute value
thresholdCVAP kiji |
thresholdP VAC ijik |
Copyright © 2004 Reich 31
eCobweb – Growth of Classification Hierarchy
Copyright © 2004 Reich 32
eCobweb – Prediction Methods
Copyright © 2004 Reich 35
CMLM – Contextualized ML Modeling
Copyright © 2004 Reich 38
Examples of Recent Systems
Conversational CBR – Help Desk (CRM) system Complex problem Requires multiple GLTs Evolution of methods Future enhancements
TeSAS (Technical Support Aiding System): CBR with weak knowledge – lesson learned system Complex problem Requires multiple GLTs Evolution of methods Future enhancements
Copyright © 2004 Reich 39
Cased-Based Reasoning
Case DataBase
NearestCases
FirstSolution
ImprovedSolution
NewCase
NewSolvedCase
2. Retrieve
5. Retain4. Revise3. Reuse
6. Store
7. MaintainNewCase
NewCase
NewCase
NewCase
NewCase
NewCase
1. Build
Copyright © 2004 Reich 40
Help Desk – CAD Software Reseller (Systematics/Solidworks)
Addressing customers questions on all aspects of a product
Manned or Automated History – Thousands heterogeneous complex cases Product evolves rapidly and continually Examples:
CAD software – Solidworks support website
Copyright © 2004 Reich 41
Solidworks Support Website
Copyright © 2004 Reich 42
Conversational CBR – Help Desk
Problem characteristics: Problems could be classified in categories.
Each category has attributes that are common to all (or most) problems of the category. There could be little overlap in attributes of problems belonging to different categories.
Problem categories could be organized in a hierarchy. Users (usually) can relate a problem to a general category or environment, and give
keywords that describe their problem. Users cannot supply all information needed to understand the exact problem.
They should be asked questions about the problem, until an accurate specification of the problem is defined.
The majority of the problems are repetitive or variants of such.
The proposed solution (based on these characteristics) includes three main steps: Automatic unsupervised category tree creation of all cases.
Derived from characteristics #1, #2, and #5. Finding in the category tree the category most similar to the new problem.
Derived from characteristics #2, #3, and #5. “Interactive dialog” with the user asking questions to reduce the number of candidate
solutions. Derived from characteristics #4 and #5.
Copyright © 2004 Reich 43
TeSAS System Design
Copyright © 2004 Reich 44
TeSAS Case Structure
Keywords: Words that describe the problem such as export, DXF, layer, or BOM. A user may define as many keywords as she likes.
Environments: There are four in SolidWorks: Part, Assembly, Drawing, and General. A case may be represented by a combination of environments. For example, a Bill Of Material problem may be defined as a Drawing related problem and/or an assembly one.
Topics: List was taken from SolidWorks Knowledge Base. A topic may be Data Exchange, Detailing, Features, Mates, Dimensions, Notes, etc. A case may have multiple topics. For example, a dimension problem of an imported DWG file may be associated to Dimensions and Data Exchange.
Copyright © 2004 Reich 45
TeSAS Category Node Structure
CategoryControl
Units
Options/Settings
What kind of anIMPORT problem is it?
Units
Keywords
Environments
Topics
Q/A
Copyright © 2004 Reich 47
TeSAS Session – # 1: Input New Case
Copyright © 2004 Reich 48
TeSAS Session – # 2: Category Selection
Copyright © 2004 Reich 49
TeSAS Session – # 2: Q&A Verification
C0
C30C44
Is it an import or export problem? --> ImportC28
C4C1
What file format is required/used? --> DXF/DWGC15
C52Is it a drawing related problem --> Yes
What kind of DXF/DWG import problem is it? --> TemplateC2 C10
Copyright © 2004 Reich 50
TeSAS Session – # 2: Q&A Verification
Copyright © 2004 Reich 51
TeSAS Session – # 3: Case Selection and Q&A Retrieval
Copyright © 2004 Reich 52
TeSAS Session – # 3: Case Selection and Q&A Retrieval
Under the hood Selecting the question that differentiates best between the
children of the present intermediate node Presenting the question and descending the hierarchy
based on answer
Copyright © 2004 Reich 53
TeSAS Session – # 4: Action Retrieval
Copyright © 2004 Reich 54
TeSAS Results – Users’ Performance
5(6+3+2) test cases Problem complexity – difficulty of characterization
Copyright © 2004 Reich 55
TeSAS Intermediate Summary
System implementation with 2 GLTs eCobweb “C4.5” – Entropy minimization
System design could be represented by a simple graph
Copyright © 2004 Reich 56
Attendance Officers Support System
Close to 9% (10.9%) of the students in Israel (US) drop out of school.
School dropout is a systemic problem with both human and social aspects.
Attendance officers, who enforce the education attendance laws, find themselves dealing with many different cases in a situation with limited resources.
Unfortunately, creative solutions that often arise in such situations, are not shared by the attendance officers’ community since there is no system that accumulates knowledge.
Develop a case management system for sharing lessons.
Copyright © 2004 Reich 57
CBR with Weak Causal Knowledge
The problem facing attendance officers has the following attributes: It is based on a many-to-many mapping. The domain is highly dimensional. The problem domain is inhomogeneous and context dependent. The solution is highly subjective. Problem solving is based on few (e.g., 3-4) characteristic case
attributes. No available decontextualized domain knowledge exists. Future auditing or quality control requires explaining solutions. Practitioners are the sole source of knowledge and they have little
motivation to spend effort beyond their usual work. Any solution devised must fit naturally into their present work practice.
There is high cost to solution failure.
Copyright © 2004 Reich 58
System Design
expanding baseline system
developing baselinequality, efficient, tested
system
structureimportance
few/nonecases
highsubjectivity
timedependency
context(place,
environment)dependency
high costfor failure
use casessolved by
practitioners
automaticprocedures
multiplesources
flexibility ofproblem
representation
matrix relationbetween
properties &solutions
problemsolving
based onpast
experience
diversecase
collection
solution'sfeedback
fast,effective,
qualitysolution
use ofsimple &
accessibletools
finding best casesfrom a cluster and
assessing theirquality
case qualitymaintenance bycommunity and
designated experts
inability toformalize
decisionrules
CBR
case matrixclustering
solution withminimal
input
augmentinput withknowledge
maintainperson in
decision loop
case qualitycontrol
reclustering
qualitymeasure
cluster'squality/index
systemexamination
need
activity
method/means
result
decomposeprocess
simplerep.
heterogenemousspace
clustering
2. usingcluster's cases
1. findingclusters
collectqualified cases
from fieldexpert
characterizeclusters
collaborativefiltering
derivedneed
poor preliminaryproblem
understanding
problemunderstanding
developappliedsystem
participatorydevelopment
literaturereview
clustering the casebase and findingthe most suitable
group or groups tonew case
continuous caseacquisition from
field practitioners
system architecture anddevelopment process
cluster correctionand reassessment
Legend
no codifiedknowledge
many fieldusers
Copyright © 2004 Reich 59
“Straight Forward” Clustering Failed
Attribute-value pair representation Tested with many clustering algorithms (traditional
statistical, NN, fuzzy) Clustering was unsuccessful
Case representation was missing critical information Search for alternative representation Influence graph
Easy to use Minimal time
Copyright © 2004 Reich 60
Influence Graph Representing Weak Knowledge
1
3
2
4
5
Output
Input
Ranking
1
M
N
1
Weight
Ranking
3
1
2
7
1
2
1
4
5
2
3
1
Output
Input
Ranking
1
M
N1
1 32 45
3
1
2
7
1
21
52
3
1
4
Copyright © 2004 Reich 61
Solution Algorithm
Stage I: Memory organization Given: n cases represented by an influence graph. Cluster cases (using E) into g clusters , (g is not specified
a priori). Characterize clusters.
Stage II: SolutionGiven: a new case described by a set of inputs and a set of g clusters,
: Identify key inputs. Retrieve b best matching clusters , . Retrieve c nearest neighbors from members of . Solve using the outputs of the c cases. Complete .
Remarks: a may be as low as 3, b would be 1 or 2 in most cases, c should be manageable, e.g., about 3 from each cluster. If b is larger than 2, c should be adjusted accordingly.
gGGG ,,1
1nC1nP
gGGG ,,1
1 nka gbest GGGG ,,1 gb
bestG
iS1nC
1nE
Copyright © 2004 Reich 62
Results – Users’ Performance
7(3+2+1) + 5(2+2+2)
Copyright © 2004 Reich 63
Results – Users’ Performance
Copyright © 2004 Reich 64
Intermediate Summary System implementation with 2 GLTs (clustering and supervised learning)
System design could be represented by an influence graph
A simple graph could be used as the basis for building decision support system
If it is good for attendance officers, it might be good for us!If it is good for attendance officers, it might be good for us! If we recorded the design of KDD systems with influence If we recorded the design of KDD systems with influence
graphs we could have used them to build a support system graphs we could have used them to build a support system that would assist us in future system development.that would assist us in future system development.
Y. Reich and A. Kapeliuk, “A framework for organizing the space of DSS with application to solving subjective, context dependent problems,” Decision Support Systems, 2004.
expanding baseline system
developing baselinequality, efficient, tested
system
structureimportance
few/nonecases
highsubjectivity
timedependency
context(place,
environment)dependency
high costfor failure
use casessolved by
practitioners
automaticprocedures
multiplesources
flexibility ofproblem
representation
matrix relationbetween
properties &solutions
problemsolving
based onpast
experience
diversecase
collection
solution'sfeedback
fast,effective,
qualitysolution
use ofsimple &
accessibletools
finding best casesfrom a cluster and
assessing theirquality
case qualitymaintenance bycommunity and
designated experts
inability toformalize
decisionrules
CBR
case matrixclustering
solution withminimal
input
augmentinput withknowledge
maintainperson in
decision loop
case qualitycontrol
reclustering
qualitymeasure
cluster'squality/index
systemexamination
need
activity
method/means
result
decomposeprocess
simplerep.
heterogenemousspace
clustering
2. usingcluster's cases
1. findingclusters
collectqualified cases
from fieldexpert
characterizeclusters
collaborativefiltering
derivedneed
poor preliminaryproblem
understanding
problemunderstanding
developappliedsystem
participatorydevelopment
literaturereview
clustering the casebase and findingthe most suitable
group or groups tonew case
continuous caseacquisition from
field practitioners
system architecture anddevelopment process
cluster correctionand reassessment
Legend
no codifiedknowledge
many fieldusers
Copyright © 2004 Reich 65
Towards better management of KDD processes
Question: What do we need to manage in order to improve KDD processes and their future understanding/reuse/accounting?
Present KDD process definitions – CRISP-DP
Integrating ideas from PDM KDD is equivalent to design or product development
Versions (of data source, software tools, needs) Linkages between different aspects of the solution Context …
KDMLKDML: Knowledge Discovery Modeling Language An evolving language describing the information used in KD processes
Copyright © 2004 Reich 66
CRISP-DM (http://www.crisp-dm.org)
CRISP-DM process overview No different than M2LTD or CMLM
Copyright © 2004 Reich 67
CRISP-DM (http://www.crisp-dm.org)
Copyright © 2004 Reich 68
CRISP-DM Use – KDnuggets Poll, July 2002
Copyright © 2004 Reich 69
Documents, DMS, …, PDM
AuthorNameCreated onLast modifiedFormat
AccessAccessrightsrights
Configuration: rules of creationLayout: presentation style (both based on user’s profile)
Check-in/Check-in/
Check-outCheck-out
Mark-Mark-upup
SharedSharedworkspaceworkspace
PersonalPersonalworkspaceworkspace
DocumentRepository
WorkflowWorkflow
Copyright © 2004 Reich 70
KDML
Y. Reich, Life cycle management of information and decisions for system analyses, Mechanical Systems and Signal Processing, vol. 15, no. 3, pp. 513–527, 2001.
Copyright © 2004 Reich 71
KDML Validation
Validation in KD processes - future Initial validation:
Own experience in developing systems Analysis of reported projects
Copyright © 2004 Reich 72
KDML – Preliminary Validation
Copyright © 2004 Reich 73
KDML – Preliminary Validation
Copyright © 2004 Reich 74
Vision
Reports of KDD processes will include influence graph representation of design decisions
A tool that realize KDML will be developed, disseminated, and evolved by the community or a vendor Improve KDD practice Increase reuse of previous processes Generate new research agenda
KDD Process reports will become as important as “theoretical” development
Copyright © 2004 Reich 75
Checkup – New Data Mining Model and Computation Tools for Extracting Knowledge from Databases and Predicting Time-Dependent Processes Goal
Design and develop a new machine learning model (Checkup Model) for the production of rules with high accuracy from databases.
The produced rules are in the form of a conjunctive list of mathematical and logical expressions.
The rules are designed for further use such as predictions, knowledge discovery, and decision-making.
Limitations of existing models Difficulty in handling unknown mathematical and logical
conjunctions correlations, especially numeric functions such as: ceil, floor, mod, etc.
Inability to handle various attribute types such as: values (nominal or numeric) and vectors (array of values) where the classification attribute could be either numeric or nominal.
Copyright © 2004 Reich 76
The Machine Learning Model
The model is unique since: It is capable of learning from datasets that other models cannot. The learning can be directed to investigating and finding relations, heuristic
correlations, and empirical functions. The error rate is equal or better compared to other machine learning algorithms
on a target set of relevant databases. The knowledge is shown as rule collections with every rule having a unique
structure of logical and mathematical functions. The produced rules are short and minimal as possible. It is possible to add limitations and known relations between attributes.
Uniqueness
Copyright © 2004 Reich 77
The Machine Learning Model
Current machine learning models can learn from datasets whose attributes are numeric (integer or float) or nominal (string). A vector is defined as a list of values where the value could be any type as float, integer, or string.
Checkup supports vector-type attributes.
Learning from datasets whose attributes have various types such as scalar or vector.
The vector attributes can have different lengths, for all attributes in all instances. Meaning that a vector which is a list of items could be in any desire list count, anywhere.
Produce rules that have logical and mathematical structure with vector functions.
Applications: vector attributes can be use in medicine follow-up for prediction the best medicine taking or decide the best process behavior during time like heat treatment, etc.
Vector support
Copyright © 2004 Reich 78
Example of DB with Vector Attributes
Copyright © 2004 Reich 79
Rule Set (example: UCI Labor Database)
IF ((duration-4*working_hours)*(-3*duration-working_hours)) < 7480.5 AND (3*duration+working_hours) < 46.5 AND
)3*Sqr(duration)+working_hours+3 < (66.5 AND )Sqr(duration)-4*Sqr(pension) < (2.5
THENclass = bad
IF ((duration-4*working_hours)*(-3*duration)-working_hours) < 6635 AND-)duration+3*working_hours < (112.5 AND
)working_hours < (39.5THENclass = good
Copyright © 2004 Reich 80
Checkup Model
Learning (Data to Rules) Iterative feedback optimization (Generation of new
attributes) Overfitting protection (Avoid learning noise and
mistakes) Optimize system settings (Parameters tuning)
Copyright © 2004 Reich 81
Checkup
InitialConditions
to Rules
Attributesto
Conditions
DecodeRules
Find Best(Conditionsand rules)
Build VirtualAttributes
ContinueLearning
OutputRules
PackRules
Discretization and Build Conditions
Overfitting and pruning
Translate rules to readable syntax rules
Initialize virtual attribute storage
Copyright © 2004 Reich 82
Attributes to Conditions
Discretization. transforms a continuous attribute’s values into a finite number of intervals and associates with each interval a
numerical, discrete value. Build conditions. build all relevant combinations of type
“Attribute Φ interval” where Φ could be greater, equal or smaller than. (ex. if a>3 then …)
Copyright © 2004 Reich 83
Conditions to Rules
Given a collection of conditions, we use it to create rules. The rules need to describe the training data part. Each rule is a set of conditions (ex. if (a>3) and (b<4) then …)
Incremental covering of positive examples Continue expansion as long as rule cover negative examples Order of conditions is based on # of covered positive examples
Rules are added incrementally to the list after excluding previously used conditions
Copyright © 2004 Reich 84
Overfitting and Pruning
The extracted rules may fit the training data, including the noise. To protect from overfitting and to reduce the number of rules in such a case so that only the most relevant rules are used, a pruning method has been implemented.
The pruning method is called a packing algorithm Packing generates various combinations of rule-sets. The best rules from the set are extracted based on their
contribution to increasing performance (reducing error). Consequently, every packing method is like pruning in a
different way. All packing rules combinations are checked, and the best
packing that produces a higher prediction score is chosen.
Copyright © 2004 Reich 85
Building Virtual Attributes
The virtual attributes are functions-of-current attributes or virtual-attributes with most-common attributes or last-most-common attributes. Three different methods for building the new attributes are present.
1) Mixing the original attribute with extracted original attributes (derived from the most common attributes).
2) Mixing the last extracted original attribute with current extracted original attributes (both derived from the most common attributes).
3) Mixing the last virtual attribute with current attributes.
Copyright © 2004 Reich 86
A sample of mathematical operations for creating virtual attributes
Y=k1*S1+k2*S2Y=S1*S2Y=min(S1,S2)Y=max(S1,S2)Y=power(S1,3)Y=abs(S1)Y=sign(S1)Y=sqr(S1)Y=sqrt(S1)Y=trunc(S1)Y=round(S1)
Y=floor(S1)Y=ceil(S1)Y=mod(S1,2)Y=mod(S1,3)Y=mod(S1,S2)Y=log(S1)Y=ln(S1)Y=sin(S1)Y=cos(S1)Y=tan(S1)
Simple operations: +, -, *, /
Si=Attribute (Simple or Vector)k=Random Constants
Copyright © 2004 Reich 87
A sample of mathematical operations for creating virtual attributes (vector)
Y=SumVec(D) {Series: +a1+a2+a3+a4+...}
Y=MinVec(D) {Series: min(a1,a2, a3, a4…)}
Y=MaxVec(D) {Series: max(a1,a2, a3, a4…)}
Y=MulVec(D) {Series: a1*a2*a3*a4*...}
Y=Ser1Vec(D) {Series: +a1-a2+a3-a4+...}
Y=Ser2Vec(D) {Series: 1*a1+2*a2+3*a3+4*a4+...}
Y=Ser3Vec(D) {Series: 1*a1+4*a2+9*a3+16*a4+...}
Y=Ser4Vec(D) {Series: a1^(1/n)+a2^(2/n)+a3^(3/n)+a4^(4/n)+... case n items}
Y=Ser5Vec(D) {Series: an-2+an-1+an; last 3 items in vector}
D=Vector Attribute
Copyright © 2004 Reich 88
Principle of Operation
data
Training data
Testing data
Train
Test
Cross validation k1-Fold
Cross validation k2-Fold
Rules Parameter
Tuning
Packing selection,
whatever needs to be selected…
Copyright © 2004 Reich 89
Principle of Operation
Parameter tuning System parameters that control learning are tuned in the process. The parameters are:
Discretization levels: find the optimal discretization splitting levels, or groups, for all attributes.
Error threshold: find the optimal value for the maximum error allowed in rules.
Search method: find the best search method, extracting rules from the best algorithm or complex algorithm.
Copyright © 2004 Reich 90
Search space Space of functions (logical and mathematical) Set of rule sets
Search strategy Depth first search (with a “front” rather than a single solution) Different reinforcement strategies for attribute creation
Principle of Operation
Copyright © 2004 Reich 94
Results and Significance DatasetC4.5Checkup (I=1)Checkup (I=10)
balance-scale78.2081.1888.27
breast-cancer72.9673.5272.47
breast-w94.5996.6096.71
bupa65.4065.9568.99
colic85.1783.7083.63
credit-a85.6284.7185.29
credit-g71.2972.4871.78
diabetes73.9475.5974.28
glass67.6568.3364.06
hayes-roth70.3670.2073.86
heart-c75.5078.3280.47
heart-h79.2179.1578.47
heart-statlog78.7278.1582.96
hepatitis79.2882.5881.78
ionosphere89.7785.6188.60
iris94.8994.5695.11
labor79.9290.3687.28
lymph76.2779.7377.54
vote96.2794.5496.32
wine93.8296.7395.04
zoo92.8693.4295.55Irvine Datasets
Copyright © 2004 Reich 95
Results and SignificanceModelScoreZeroR50
DecisionStump53DecisionTable50
HyperPipes49.5
IB157IBk57
C4.553.5RandomTree56.5
KStar51Logistic45.5
NaiveBayes44.5OneR51
SMO45VotedPerception51.5
VFI50.5NeuralNetwork52.3
LinearRegression49.5Checkup98
Checkup compared to various other machine learning models (random dataset with hidden mathematical functions) .
Copyright © 2004 Reich 96
Significance
We find that Checkup is at the top of the prediction accuracy list even in I=1
When increasing iterations only to three, the prediction score becomes much higher
Current existing models have difficulty handling the numeric functions in datasets compared to the Checkup model
The model is highly flexible enabling learning from a large collection of mathematical operation; adding newer functions and generic equations to the model (system) is simple
Copyright © 2004 Reich 97
References
Chandrasekaran, B. Generic tasks in knowledge-based reasoning: high-level building blocks for expert system design. IEEE Expert 1(3):23-30, 1986.
Y. Reich and S. J. Fenves, Integration of generic learning tasks, Tech. Rep. EDRC 12-28-89, Engineering Design Research Center, Carnegie Mellon University, Pittsburgh, PA, 1989. Available at http://www.eng.tau.ac.il/~yoram/topics/generic-learning.html.
Y. Reich, “Constructive induction by incremental concept formation,” in Artificial Intelligence and Computer Vision (Y. A. Feldman and A. Bruckstein, eds.), pp. 191–204, Amsterdam: Elsevier Science Publishers, 1991.
Y. Reich, Life cycle management of information and decisions for system analyses, Mechanical Systems and Signal Processing, vol. 15, no. 3, pp. 513–527, 2001.
Y. Reich and A. Kapeliuk, Case-based reasoning with subjective influence knowledge, Applied Artificial Intelligence, vol. 18, no. 8, pp. 735–760, 2004.
G. Chen and Y. Reich, A conversational case-based reasoning help-desk utility for complex products. Submitted, 2004.
Y. Reich and S. J. Fenves, The formation and use of abstract concepts in design, in Concept Formation: Knowledge and Experience in Unsupervised Learning (D. H. J. Fisher, M. J. Pazzani, and P. Langley, eds.), (Los Altos, CA), pp. 323–353, Morgan Kaufmann, 1991.
Y. Reich, Macro and micro perspectives of multistrategy learning, in Machine Learning: A Multistrategy Approach, Vol. IV (R. S. Michalski and G. Tecuci, eds.), (San Francisco, CA), pp. 379–401, Morgan Kaufmann, 1994.
Y. Reich, “Measuring the value of knowledge,” International Journal of Human-Computer Studies, vol. 42, no. 1, pp. 3–30, 1995.
Y. Reich and A. Kapeliuk, “A framework for organizing the space of DSS with application to solving subjective, context dependent problems,” Decision Support Systems, 2004.