Knowledge Engineering & Bioinformatics LabDepartment of Mathematics and Computer Science

Marburg University, Germany

Robin Senge & Eyke Hüllermeier

Fuzzy Pattern Trees for Regression and Fuzzy Systems Modeling

WCCI 2010, Barcelona

2

Outline

Problem Setting

Introduction to Fuzzy Pattern Trees (FPT)

Learning Fuzzy Pattern Trees from Data

Experiments

Relation to Fuzzy Rule-based Systems

Using Fuzzy Pattern Trees for Fuzzy System Modeling

3

Problem Setting

Standard setting of supervised learning:

attribute-value representation of instances

let be input domains and be the output domain

input attribute domains discretized by fuzzy sets, e.g., low, medium and high

rescale to by

model functional relationship, i.e.

4

Example: Wine Quality

aim: predicting quality of wine based on its ingredients (UCI)

input attributes: acidity, alcohol, sulfates, sulfur, ...

target (output) attribute is quality

acidity alcohol sulfates sulfur quality

7.4 9.4 0.56 11 5

7.8 10 0.46 13 3

7.8 10.5 0.80 25 6

11.2 9.3 0.91 17 3

7.4 9.8 0.55 12 5

7.3 10.6 0.53 21 4

8.9 9.4 0.66 17 8

acidity alcohol sulfates sulfur quality

low med high G(y)

7.4 0.89 0.11 0.00 0.56 11 0.50

7.8 0.03 0.97 0.00 0.46 13 0.30

7.8 0.22 0.78 0.00 0.8 25 0.60

11.2 1.00 0.00 0.00 0.91 17 0.30

7.4 0.00 0.00 1.00 0.55 12 0.50

7.3 0.00 0.81 0.19 0.53 21 0.40

8.9 0.84 0.16 0.00 0.66 17 0.80

acidity alcohol sulfates sulfur quality

low med high G(y)

0.89 0.11 0.00 0.50

0.03 0.97 0.00 0.30

0.22 0.78 0.00 0.60

1.00 0.00 0.00 0.30

0.00 0.00 1.00 0.50

0.00 0.81 0.19 0.40

0.84 0.16 0.00 0.80

5

Example Fuzzy Pattern Tree (FPT)

wine quality

alcohol high

AVG

MIN

acidity low

acidity med

MAX

sulfates med

0.8

0.8 0.2

0.20.8

0.3

0.5

10.2

6

Operators

Name T-Norm (generalized conjunction) Code

Minimum MINAlgebraic AND ALG

Lukasiewicz AND LUK

Einstein AND EIN

Name T-Conorm (generalized disjunction) Code

Maximum MAXAlgebraic OR COALG

Lukasiewicz OR COLUK

Einstein OR COEIN

Name Averaging Operator Code

Weighted Average WAOrdered Weighted Average OWA

7

Features of Fuzzy Pattern Trees

interpretability of the model class

modularity: recursive partitioning of critria into sub-criteria

flexibility without the tendency to overfit the data

monotonicity in single attributes

built-in feature selection

high wine quality

alcohol high

AVG

MIN

acidity low

acidity med

MAX

sulfates med

8

Learning Fuzzy Pattern Trees from Examples

iteratively refining = growing up trees

start with primitive pattern tree

growing tree in a top-down manner

selection based on tree performance measure

check relative performance improvement

B

AVG

A

A

A

AVG

MIN

DB

A

AVG

MIN

D

B

MAX

C

A

AVG

MIN

D

B

MAX

C

E

AVG

B

AVG

MIN

DMAX

C

E

AVG

MIN

A B

greedy beam search

(details in the paper)

9

Experiments

Are Fuzzy Pattern Trees competitive in terms of predictive accuracy?

12 data sets from UCI and STATLIB

10-fold-cross validation

root mean squared error (RMSE)

baseline algorithms

Linear Regression (LR) Multi Layer Perceptron (MLP) Support Vector Machine with

linear kernel (SMO-lin) Support Vector Machine with RBF

kernel (SMO-rbf) Fast decision tree learner with

reduced error pruning (REPtree) Fuzzy Rule Learner by Wang &

Mendel (FR)

10

Results

Dataset PT-reg LR REPtree SMO-lin MLP SMO-rbf FRauto-mpg 1 5 4 6 3 7 8concrete 2 5 1 7 3 6 8flare1M 6 1 2 5 7 3 8flare2C 4 1 2 5 7 6 8forestfires 6 4 3 2 8 1 7housing 2 5 3 6 1 7 8imports-85 5 3 7 1 2 6 8machine 2 6 7 1 8 5 4servo 2 5 3 7 1 8 6slump 3 2 7 4 1 6 8winequality-red 1 2 6 3 7 4 8winequality-white 4 2 1 3 6 5 8average rank 3.17 3.42 3.83 4.17 4.5 5.33 7.42

Ranks according to RMSE

PT-reg appears to be (at least) competitive to baseline algorithms.

11

Fuzzy Pattern Trees vs. Rule-based Fuzzy Systems

Fuzzy Pattern Trees are closely related to Fuzzy Rule-based Systems

fuzzy rules for property: low quality

IF high(acidity) AND low(alcohol) THEN quality is lowIF low(acidity) AND medium(sulfates) THEN quality is lowIF high(alcohol) AND medium(sulfur) THEN quality is low

fuzzy rules for property: low quality

Score(quality is low) = MAX { MIN {high(acidity), low(alcohol)}, MIN {low(acidity), medium(sulfates)}, MIN {high(alcohol), medium(sulfur)}}

fuzzy rules for property: low quality

IF MIN {high(acidity), low(alcohol)} THEN quality is low IF MIN {low(acidity), medium(sulfates)} THEN quality is low IF MIN {high(alcohol), medium(sulfur)} THEN quality is low

MAX

alcohollow

low quality

MIN

acidityhigh

MIN

aciditylow sulfatesmed

MIN

alcoholhigh sulfurmed

12

Fuzzy Systems Modeling

usually, not only one fuzzy set on but complete fuzzy partition

let be the fuzzy sets on

model functional relationships, i.e.

alcohol quality (three targets)

low med high low med high

0.89 0.11 0.00 0.00 0.50 0.50

0.03 0.97 0.00 0.40 0.60 0.00

0.22 0.78 0.00 0.00 0.40 0.60

1.00 0.00 0.00 1.00 0.00 0.00

0.00 0.00 1.00 0.00 0.00 0.50

0.00 0.81 0.19 0.00 0.80 0.20

0.84 0.16 0.00 0.00 0.20 0.80

F-AND

acid high sulfur low

high quality

F-OR

sulfate med

acid low

medium quality

AVG-OP

alcohol high

F-AND

acid high

sulfate low

low quality

AVG-OP

alcohol med

13

Fuzzy Systems Modeling contd.

high quality

low quality

medium quality

F-AND

acid high sulfur low

high quality

F-OR

sulfate med

acid low

medium quality

AVG-OP

alcohol high

F-AND

acid high

sulfate low

low quality

AVG-OP

alcohol med

14

Conclusions

Fuzzy Pattern Trees have been introduced as a new model class for regression and fuzzy systems design.

They do have several interesting features (interpretability , monotonicity, flexibility, feature selection).

Data-driven model construction: We can learn Fuzzy Pattern Trees from data.

Regression with Fuzzy Pattern Trees is competitive to state-of-the-art algorithms in terms of predictive accuracy.

For more information search the web for „kebi marburg“.