Lecture 11Lecture 11

Hybrid intelligent systems:Hybrid intelligent systems:Neural expert systems and neuro-fuzzy systemsNeural expert systems and neuro-fuzzy systems

IntroductionIntroduction Neural expert systemsNeural expert systems Neuro-fuzzy systemsNeuro-fuzzy systems ANFIS: Adaptive Neuro-Fuzzy Inference ANFIS: Adaptive Neuro-Fuzzy Inference

SystemSystem SummarySummary

A A hybrid intelligent systemhybrid intelligent system is one that combines at least two is one that combines at least two intelligent technologies. intelligent technologies. For example, combining a neural network with a fuzzy system For example, combining a neural network with a fuzzy system

results in a hybrid neuro-fuzzy system.results in a hybrid neuro-fuzzy system.

The combination of:The combination of: probabilistic reasoning, probabilistic reasoning, fuzzy logic, fuzzy logic, neural networks and neural networks and evolutionary computation forms the core of evolutionary computation forms the core of soft computingsoft computing, ,

Soft Computing Soft Computing is an emerging approach to building hybrid is an emerging approach to building hybrid intelligent systems capable of reasoning and learning in an intelligent systems capable of reasoning and learning in an uncertain and imprecise environment.uncertain and imprecise environment.

Hybrid SystemsHybrid Systems

Although Although words are less precise than numberswords are less precise than numbers, precision , precision carries a high cost. carries a high cost. We use words when there is a We use words when there is a tolerance for imprecisiontolerance for imprecision. . Soft computing exploits the tolerance for uncertainty and Soft computing exploits the tolerance for uncertainty and

imprecision to achieve greater tractability and imprecision to achieve greater tractability and robustness, and lower the cost of solutions.robustness, and lower the cost of solutions.

We also use words when the available data is not precise We also use words when the available data is not precise enough to use numbers. enough to use numbers. This is often the case with complex problems, and while This is often the case with complex problems, and while

“hard” computing fails to produce any solution, soft “hard” computing fails to produce any solution, soft computing is still capable of finding good solutions.computing is still capable of finding good solutions.

Using “words” rather than strict Using “words” rather than strict numbersnumbers

Lotfi Zadeh is reputed to have said that a good hybrid Lotfi Zadeh is reputed to have said that a good hybrid would be “British Police, German Mechanics, French would be “British Police, German Mechanics, French Cuisine, Swiss Banking and Italian Love”. Cuisine, Swiss Banking and Italian Love”.

But “British Cuisine, German Police, French Mechanics, But “British Cuisine, German Police, French Mechanics, Italian Banking and Swiss Love” would be a bad one. Italian Banking and Swiss Love” would be a bad one.

Likewise, a hybrid intelligent system can be good or bad Likewise, a hybrid intelligent system can be good or bad – it depends on which components constitute the hybrid. – it depends on which components constitute the hybrid.

So our goal is to select the right components for building So our goal is to select the right components for building a good hybrid system.a good hybrid system.

Comparison of Comparison of EExpert Systems, xpert Systems, FFuzzy Systems,uzzy Systems,NNeural Networks and eural Networks and GGenetic Algorithmsenetic Algorithms

Knowledge representation

Uncertainty tolerance

Imprecision tolerance

Adaptability

Learning ability

Explanation ability

Knowledge discovery and data mining

Maintainability

ES FS NN GA

* The terms used for grading are:

- bad, - rather bad, - good - rather good and

Neural Neural expert expert

systemssystems

Neural Neural expert systemsexpert systems Expert systems Expert systems rely on logical inferences and decision trees and rely on logical inferences and decision trees and

focus on modelling human reasoning. focus on modelling human reasoning. Neural networks Neural networks rely on parallel data processing and focus rely on parallel data processing and focus

on modelling a human brain.on modelling a human brain.

Expert systems Expert systems treat the brain as a black-box. treat the brain as a black-box. Neural networks look at its structure and functions, Neural networks look at its structure and functions,

particularly at its ability to learn. particularly at its ability to learn.

KnowledgeKnowledge in a in a rule-basedrule-based expert system is represented by IF- expert system is represented by IF-THEN production rules. THEN production rules. KnowledgeKnowledge in in neural networks neural networks is stored as synaptic weights is stored as synaptic weights

between neurons.between neurons.

In In expert systemsexpert systems, knowledge can be divided into , knowledge can be divided into individual rules and the user can see and individual rules and the user can see and understand the piece of knowledge applied by the understand the piece of knowledge applied by the system.system.

In In neural networksneural networks, one cannot select a single , one cannot select a single synaptic weight as a discrete piece of knowledge. synaptic weight as a discrete piece of knowledge. Here knowledge is embedded in the entire Here knowledge is embedded in the entire

network; network; it it cannot be broken into individual piecescannot be broken into individual pieces, and , and

any change of a synaptic weight may lead to any change of a synaptic weight may lead to unpredictable results. unpredictable results.

A neural network is, in fact, a A neural network is, in fact, a black-boxblack-box for its for its user.user.

Can we combine advantages Can we combine advantages of expert systems of expert systems and neural networks to create a more powerful and neural networks to create a more powerful and effective expert system? and effective expert system?

A hybrid system that combines a neural network and A hybrid system that combines a neural network and a rule-based expert system is called a a rule-based expert system is called a neural expert neural expert systemsystem (or a (or a connectionist expert systemconnectionist expert system). ).

Basic structure of a Basic structure of a neural expert systemneural expert system

Inference Engine

Neural Knowledge Base Rule Extraction

Explanation Facilities

User Interface

User

Rule: IF - THEN

Training Data

NewData

The The heart heart of a neural expert system is the of a neural expert system is the inference engineinference engine. .

The The inference engineinference engine controls the information controls the information flow in the system and initiates inference over flow in the system and initiates inference over the neural knowledge base. the neural knowledge base.

A neural inference engine also ensures A neural inference engine also ensures approximate reasoningapproximate reasoning..

The The inference engineinference engine

ApproximateApproximate reasoning reasoning In a rule-based expert In a rule-based expert system, the inference engine compares the system, the inference engine compares the

condition part of each rule with data given in the database. condition part of each rule with data given in the database. When the IF part of the rule matches the data in the When the IF part of the rule matches the data in the

database, the rule is fired and its THEN part is executed. database, the rule is fired and its THEN part is executed. The The precise matchingprecise matching is required (inference engine cannot is required (inference engine cannot

cope with noisy or incomplete data).cope with noisy or incomplete data).

Neural expert systems Neural expert systems use a trained neural network in place of use a trained neural network in place of the knowledge base. the knowledge base. The input data does not have to precisely match the data that The input data does not have to precisely match the data that

was used in network training. was used in network training. This ability is called This ability is called approximate reasoningapproximate reasoning..

Rule extraction Rule extraction

Neurons in the network are connected by links, Neurons in the network are connected by links, each of which has a each of which has a numerical weight numerical weight attached to it. attached to it.

The weights The weights in a trained neural network determine in a trained neural network determine the strength or importance of the associated neuron the strength or importance of the associated neuron inputs.inputs.

The neural knowledge baseThe neural knowledge base

-0.8

-0.2

-0.1-1.1

2.20.0

-1.0

2.8-1.6

-2.9

-1.3

Bird

Plane

Glider

+1

Wings

Tail

Beak

Feathers

Engine

+1

0

+1

+1

1

-1.6 -0.7

-1.1 1.9

1

1

Rule 1

Rule 2

Rule 3

1.0

1.0

1.0

If we set each input of the input layer to either +1 If we set each input of the input layer to either +1 (true), (true), 1 (false), or 0 (unknown), we can give a 1 (false), or 0 (unknown), we can give a semantic interpretation semantic interpretation for the activation of any for the activation of any output neuron. output neuron.

For exampleFor example, if the object has , if the object has WingsWings (+1), (+1), BeakBeak (+1) (+1) and and FeathersFeathers (+1), (+1), but does not have but does not have EngineEngine ( (1), 1), then we can conclude that this object is then we can conclude that this object is BirdBird (+1): (+1):

03.5)1.1()1(8.212.21)2.0(0)8.0(11 RuleX

11 BirdRule YY

How the system distinguishes a bird from an airplane?

We can similarly conclude that this object is not We can similarly conclude that this object is not PlanePlane::

02.49.1)1()6.1(10.01)1.0(0)7.0(12 RuleX

12 PlaneRule YY

and not and not GliderGlider::

02.4)3.1()1()9.2(1)0.1(1)1.1(0)6.0(13 RuleX

13 GliderRule YY

By By attaching a corresponding question attaching a corresponding question to to each inputeach input

neuron, neuron, we can enable the system to we can enable the system to prompt the userprompt the user

for initial values of the input variables:for initial values of the input variables:

Neuron: Neuron: WingsWingsQuestionQuestion: Does the object have wings?: Does the object have wings?

Neuron: Neuron: TailTailQuestionQuestion: Does the object have a tail?: Does the object have a tail?

NeuronNeuron: : BeakBeakQuestionQuestion: Does the object have a beak?: Does the object have a beak?

NeuronNeuron: : FeathersFeathersQuestionQuestion: Does the object have feathers?: Does the object have feathers?

NeuronNeuron: : EngineEngineQuestionQuestion: Does the object have an engine?: Does the object have an engine?

An inference can be made if the known net An inference can be made if the known net weighted input to a neuron weighted input to a neuron is greater than the is greater than the sumsum of the absolute values of the weights of of the absolute values of the weights of the unknown inputs.the unknown inputs.

n

jj

n

iii wwx

11

where where ii known, known, jj known and known and nn is the number is the number

of neuron inputs.of neuron inputs.

Example:Example:Enter initial value for the input FeathersEnter initial value for the input Feathers:: +1+1

KNOWN = 1KNOWN = 12.8 = 2.82.8 = 2.8UNKNOWN = UNKNOWN = 0.80.8 + + 0.20.2 + + 2.22.2 + + 1.11.1 = 4.3 = 4.3KNOWN KNOWN UNKNOWN UNKNOWN

Enter initial value for the input BeakEnter initial value for the input Beak:: +1+1

KNOWN = 1KNOWN = 12.8 + 12.8 + 12.2 = 5.02.2 = 5.0UNKNOWN = UNKNOWN = 0.80.8 + + 0.20.2 + + 1.11.1 = 2.1 = 2.1KNOWN KNOWN UNKNOWN UNKNOWN

CONCLUDE: CONCLUDE: Bird is TRUEBird is TRUE

An example of a An example of a multi-layermulti-layer knowledge baseknowledge base

ConjunctionLayer

InputLayer

R1

R2

R3

R4

a1

a2

a3

a4

a5 R5

b2

b1

b3

0.2

0.8

-0.1

0.9

0.6

R6

R7

R8

c1

c2

0.1

0.9

0.7

DisjunctionLayer

ConjunctionLayer

DisjunctionLayer

Rule 1: Rule 5:IF a1 AND a3 THEN b1 (0.8) IF a5 THEN b3 (0.6)

Rule 2: Rule 6:IF a1 AND a4 THEN b1 (0.2) IF b1 AND b3 THEN c1 (0.7)

Rule 3: Rule 7:IF a2 AND a5 THEN b2 (-0.1) IF b2 THEN c1 (0.1)

Rule 4: Rule 8:IF a3 AND a4 THEN b3 (0.9) IF b2 AND b3 THEN c2 (0.9)

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Neuro-Neuro-fuzzy fuzzy

systemssystems

Fuzzy logic and neural networks are Fuzzy logic and neural networks are natural natural complementary tools complementary tools in building intelligent systems. in building intelligent systems.

While While neural networks are low-level neural networks are low-level computational computational structures that perform well when dealing with raw data, structures that perform well when dealing with raw data, fuzzy logic deals with reasoning on a higher fuzzy logic deals with reasoning on a higher level, using level, using linguistic information acquired from domain experts. linguistic information acquired from domain experts.

However, fuzzy systems lack the ability to learn and However, fuzzy systems lack the ability to learn and cannot adjust themselves to a new environment. cannot adjust themselves to a new environment. On the other hand, although neural networks can On the other hand, although neural networks can

learn, they are opaque to the user. learn, they are opaque to the user.

Neuro-fuzzy systemsNeuro-fuzzy systems

Integrated neuro-fuzzy systems Integrated neuro-fuzzy systems can combine can combine the the parallel computation and learning abilities of parallel computation and learning abilities of neural networks with the human-like knowledge neural networks with the human-like knowledge representation and explanation abilities of fuzzy representation and explanation abilities of fuzzy systems. systems.

As a result, neural networks become As a result, neural networks become more more transparenttransparent, while fuzzy systems become , while fuzzy systems become capable capable of learning.of learning.

Synergy of Neural and FuzzySynergy of Neural and Fuzzy

A A neuro-fuzzy system neuro-fuzzy system is a neural network which is is a neural network which is functionally equivalent to a fuzzy inference model. functionally equivalent to a fuzzy inference model.

Neuro-Fuzzy System Neuro-Fuzzy System can be trained can be trained to develop IF-to develop IF-THEN fuzzy rules and determine membership functions THEN fuzzy rules and determine membership functions for input and output variables of the system. for input and output variables of the system.

Expert knowledge can be incorporated Expert knowledge can be incorporated into the structure into the structure of the neuro-fuzzy system. of the neuro-fuzzy system.

At the same time, the connectionist structure At the same time, the connectionist structure avoids avoids fuzzy inferencefuzzy inference, which entails a substantial , which entails a substantial computational burden.computational burden.

The structure of a neuro-fuzzy system is similar to The structure of a neuro-fuzzy system is similar to a a multi-layer neural networkmulti-layer neural network. .

In general, a neuro-fuzzy system has:In general, a neuro-fuzzy system has: input and output input and output layers, layers, and and three hidden layers three hidden layers

that represent membership functions and that represent membership functions and fuzzy rules.fuzzy rules.

Neuro-fuzzy systemNeuro-fuzzy systemLayer 2 Layer 3 Layer 4 Layer 5Layer 1

y

x1

A1

A2

A3

B1

B2x2

C1

C2

x1

x1

x1

x2

x2

x2

B1

A2

B3

C2

C1

R1

R3

R5

R6

R4

R1

R5

R4

R6

R2

R3

R2

B3

A1

wR3

wR6

wR1wR2

wR4

wR5

A3

B2

Each layer in the neuro-fuzzy system is associatedEach layer in the neuro-fuzzy system is associated

with a particular step in the fuzzy inference process.with a particular step in the fuzzy inference process.

LayerLayer 1 1 is the is the input layerinput layer. Each neuron in this layer . Each neuron in this layer transmits external crisp signals directly to the next transmits external crisp signals directly to the next layer. That is,layer. That is,

LayerLayer 2 2 is the is the fuzzification layerfuzzification layer. . • NeuronsNeurons in this layer represent in this layer represent fuzzy sets fuzzy sets used in the used in the

antecedents of fuzzy rules. antecedents of fuzzy rules. • A A fuzzification neuron fuzzification neuron receives a crisp input and receives a crisp input and

determines the degree to which this input belongs to determines the degree to which this input belongs to the neuron’s fuzzy set.the neuron’s fuzzy set.

)1()1(ii xy

Layer 2 Layer 3 Layer 4 Layer 5Layer 1

y

x1

A1

A2

A3

B1

B2x2

C1

C2

x1

x1

x1

x2

x2

x2

B1

A2

B3

C2

C1

R1

R3

R5

R6

R4

R1

R5

R4

R6

R2

R3

R2

B3

A1

wR3

wR6

wR1wR2

wR4

wR5

A3

B2

LayerLayer 1 1 is the is the input layerinput layer. Each neuron in this layer transmits . Each neuron in this layer transmits external crisp signals directly to the next layer. That is,external crisp signals directly to the next layer. That is,

LayerLayer 2 2 is the is the fuzzification layerfuzzification layer. . • NeuronsNeurons in this layer represent in this layer represent fuzzy sets fuzzy sets used in the antecedents of used in the antecedents of

fuzzy rules. fuzzy rules. • A A fuzzification neuron fuzzification neuron receives a crisp input and determines the degree receives a crisp input and determines the degree

to which this input belongs to the neuron’s fuzzy set.to which this input belongs to the neuron’s fuzzy set.

1.1. The activation function of a membership neuron is set The activation function of a membership neuron is set to the function that specifies the neuron’s fuzzy set. to the function that specifies the neuron’s fuzzy set.

2.2. We use triangular sets, and therefore, the activation We use triangular sets, and therefore, the activation functions for the neurons in functions for the neurons in LayerLayer 2 are set to the 2 are set to the triangular membership functionstriangular membership functions. .

3.3. A triangular membership function can be specified by A triangular membership function can be specified by two parameters {two parameters {aa, , bb} as follows:} as follows:

2 if ,0

22 if ,

21

2 if ,0

)2(

)2()2(

)2(

)2(

bax

bax

ba

b

ax

bax

y

i

ii

i

i

Triangular activation functionsTriangular activation functions

(b) Effect of parameter b.

0.2

0.4

0.6

0.8

1

a = 4, b =6

a = 4, b =4

0 4 6 80

1 2 3 5 7X

(a) Effect of parameter a.

0.2

0.4

0.6

0.8

1

a = 4, b =6

a = 4.5, b =6

0 4 6 80

1 2 3 5 7X

Layer 2 Layer 3 Layer 4 Layer 5Layer 1

y

x1

A1

A2

A3

B1

B2x2

C1

C2

x1

x1

x1

x2

x2

x2

B1

A2

B3

C2

C1

R1

R3

R5

R6

R4

R1

R5

R4

R6

R2

R3

R2

B3

A1

wR3

wR6

wR1wR2

wR4

wR5

A3

B2

LayerLayer 3 3 is the is the fuzzy rule layerfuzzy rule layer. .

1.1. Each neuron in this layer corresponds to a single fuzzy rule. Each neuron in this layer corresponds to a single fuzzy rule.

2.2. A fuzzy rule neuron receives inputs from the fuzzification neurons that represent A fuzzy rule neuron receives inputs from the fuzzification neurons that represent fuzzy sets in the rule antecedents. fuzzy sets in the rule antecedents.

3.3. For instance, neuron For instance, neuron RR1, which corresponds to 1, which corresponds to RuleRule 1, receives inputs from neurons 1, receives inputs from neurons AA1 and 1 and BB1.1.

LayerLayer 3 3 is the is the fuzzy rule layerfuzzy rule layer. .

1.1. Each neuron in this layer corresponds to a single Each neuron in this layer corresponds to a single fuzzy rule. fuzzy rule.

2.2. A fuzzy rule neuron receives inputs from the A fuzzy rule neuron receives inputs from the fuzzification neurons that represent fuzzy sets in fuzzification neurons that represent fuzzy sets in the rule antecedents. the rule antecedents.

3.3. For instance, neuron For instance, neuron RR1, which corresponds to 1, which corresponds to RuleRule 1, receives inputs from neurons 1, receives inputs from neurons AA1 and 1 and BB1.1.

In a neuro-fuzzy system, intersection can be In a neuro-fuzzy system, intersection can be implemented by the implemented by the product operatorproduct operator. .

Thus, the output of neuron Thus, the output of neuron ii in in LayerLayer 3 is obtained as: 3 is obtained as:

)3()3(2

)3(1

)3(kiiii xxxy 111

)3(1 RBARy

Layer 2 Layer 3 Layer 4 Layer 5Layer 1

y

x1

A1

A2

A3

B1

B2x2

C1

C2

x1

x1

x1

x2

x2

x2

B1

A2

B3

C2

C1

R1

R3

R5

R6

R4

R1

R5

R4

R6

R2

R3

R2

B3

A1

wR3

wR6

wR1wR2

wR4

wR5

A3

B2

LayerLayer 4 4 is the is the output membership layeroutput membership layer. Neurons in this layer represent . Neurons in this layer represent fuzzy sets used in the consequent of fuzzy rules. fuzzy sets used in the consequent of fuzzy rules.

An output membership neuron combines all its inputs by using the fuzzy An output membership neuron combines all its inputs by using the fuzzy operation operation unionunion. . This operation can be implemented by the This operation can be implemented by the probabilistic ORprobabilistic OR. That is,. That is,

The value of The value of CC11 represents the integrated firing strength of fuzzy rule represents the integrated firing strength of fuzzy rule

neurons neurons RR3 and 3 and RR6.6.

)4()4(2

)4(1

)4( liiii xxxy 163)4(

1 CRRCy

LayerLayer 4 4 is the is the output membership layeroutput membership layer. Neurons . Neurons in this layer represent fuzzy sets used in the in this layer represent fuzzy sets used in the consequent of fuzzy rules. consequent of fuzzy rules.

An output membership neuron combines all its An output membership neuron combines all its inputs by using the fuzzy operation inputs by using the fuzzy operation unionunion. . This operation can be implemented by the This operation can be implemented by the probabilistic ORprobabilistic OR. That is,. That is,

The value of The value of CC11 represents the integrated firing represents the integrated firing

strength of fuzzy rule neurons strength of fuzzy rule neurons RR3 and 3 and RR6.6.

)4()4(2

)4(1

)4( liiii xxxy 163)4(

1 CRRCy

Layer 2 Layer 3 Layer 4 Layer 5Layer 1

y

x1

A1

A2

A3

B1

B2x2

C1

C2

x1

x1

x1

x2

x2

x2

B1

A2

B3

C2

C1

R1

R3

R5

R6

R4

R1

R5

R4

R6

R2

R3

R2

B3

A1

wR3

wR6

wR1wR2

wR4

wR5

A3

B2

LayerLayer 5 5 is the is the defuzzification layerdefuzzification layer. Each neuron in this layer represents a single . Each neuron in this layer represents a single output of the neuro-fuzzy system. It takes the output fuzzy sets clipped by the output of the neuro-fuzzy system. It takes the output fuzzy sets clipped by the respective integrated firing strengths and combines them into a single fuzzy respective integrated firing strengths and combines them into a single fuzzy set.set.

Neuro-fuzzy systems can apply standard defuzzification methods, including Neuro-fuzzy systems can apply standard defuzzification methods, including the centroid technique. the centroid technique.

We will use the We will use the sum-product compositionsum-product composition method. method.

LayerLayer 5 5 is the is the defuzzification layerdefuzzification layer. Each neuron . Each neuron in this layer represents a single output of the in this layer represents a single output of the neuro-fuzzy system. It takes the output fuzzy sets neuro-fuzzy system. It takes the output fuzzy sets clipped by the respective integrated firing clipped by the respective integrated firing strengths and combines them into a single fuzzy strengths and combines them into a single fuzzy set.set.

Neuro-fuzzy systems can apply standard Neuro-fuzzy systems can apply standard defuzzification methods, including the centroid defuzzification methods, including the centroid technique. technique.

We will use the We will use the sum-product compositionsum-product composition method.method.

The sum-product composition calculates the crisp The sum-product composition calculates the crisp output as the weighted average of the centroids of output as the weighted average of the centroids of all output membership functions. all output membership functions.

For example, the weighted average of the centroids of For example, the weighted average of the centroids of the clipped fuzzy sets the clipped fuzzy sets CC1 and 1 and CC2 is calculated as,2 is calculated as,

2211

222111

CCCC

CCCCCC

bb

babay

How does a neuro-fuzzy system learn?How does a neuro-fuzzy system learn?

A neuro-fuzzy system is essentially a A neuro-fuzzy system is essentially a multi-layer multi-layer neural networkneural network, ,

and thus it can apply and thus it can apply standard learning algorithms standard learning algorithms developed for neural networks, developed for neural networks,

including the including the back-propagationback-propagation algorithm. algorithm.

When a training input-output example is presented to the system, the When a training input-output example is presented to the system, the back-propagation algorithm computes the system output and compares it back-propagation algorithm computes the system output and compares it with the desired output of the training example. with the desired output of the training example.

The error is propagated backwards through the network from the The error is propagated backwards through the network from the output layer to the input layer. output layer to the input layer.

The neuron activation functions are modified as the error is The neuron activation functions are modified as the error is propagated. propagated.

To determine the necessary modifications, the back-propagation To determine the necessary modifications, the back-propagation algorithm differentiates the activation functions of the neurons.algorithm differentiates the activation functions of the neurons.

Let us demonstrate Let us demonstrate how a neuro-fuzzy system works on a simple how a neuro-fuzzy system works on a simple example.example.

Training patternsTraining patterns

0

1

0

Y

1

1

0

The data set is used for training the five-rule neuro-The data set is used for training the five-rule neuro-fuzzy system shown below.fuzzy system shown below.

(a) Five-rule system.

x2

L

S

x2

L

S

0 10 20 30 40 500

0.2

0.4

0.6

0.8

1

y

1

2

4

5

3L

S

EpochW

eigh

t

wR2

0.99

0

0.72

0.61

0.79

(b) Training for 50 epochs.

wR3 wR4

wR5

wR1

Five-rule neuro-fuzzy systemFive-rule neuro-fuzzy system

Suppose that fuzzy IF-THEN rules incorporated into the Suppose that fuzzy IF-THEN rules incorporated into the system structure are supplied by a system structure are supplied by a domain expertdomain expert. . PriorPrior or existing knowledge or existing knowledge can dramatically expedite can dramatically expedite

the system training.the system training.

Besides, if the quality of training data is poor, Besides, if the quality of training data is poor, the expert the expert knowledge may be the only way knowledge may be the only way to come to a solution at all. to come to a solution at all. However, However, experts do occasionally make mistakesexperts do occasionally make mistakes, and , and

thus some rules used in a neuro-fuzzy system may be thus some rules used in a neuro-fuzzy system may be false or redundant. false or redundant.

Therefore, a neuro-fuzzy system should also be capable Therefore, a neuro-fuzzy system should also be capable of of identifying bad rulesidentifying bad rules..

Good and Bad Rules from Experts in systemsGood and Bad Rules from Experts in systems

Given input and output linguistic values, a neuro-fuzzy system Given input and output linguistic values, a neuro-fuzzy system can automatically generate a complete set of fuzzy IF-THEN can automatically generate a complete set of fuzzy IF-THEN rules.rules.

Let us create the system for the XOR example. Let us create the system for the XOR example. This system consists of 2This system consists of 222 2 = 8 rules. 2 = 8 rules. Because expert knowledge is not embodied in the system Because expert knowledge is not embodied in the system

this time, we set all initial weights between this time, we set all initial weights between LayerLayer 3 and 3 and LayerLayer 4 to 0.5. 4 to 0.5.

After training we can eliminate all rules whose certainty After training we can eliminate all rules whose certainty factors are less than some sufficiently small number, say 0.1. factors are less than some sufficiently small number, say 0.1. As a result, we obtain the same set of four fuzzy IF-THEN As a result, we obtain the same set of four fuzzy IF-THEN

rules that represents the XOR operation.rules that represents the XOR operation.

Example: A neuro-fuzzy system for XORExample: A neuro-fuzzy system for XOR

Eight-rule neuro-fuzzy Eight-rule neuro-fuzzy system for XORsystem for XOR

(a) Eight-rule system.

Sx1

L

x2

y

L

S

L

S

1

2

3

4

5

6

7

80 10 20 30 40 50

Epoch

wR1

(b) Training for 50 epochs.

wR4

0

0.78

0.69

0.62

00

0.80

0

wR2 wR8

wR5wR3

wR6 & wR7

0

0.2

0.3

0.5

0.7

0.1

0.4

0.6

0.8

Neuro-fuzzy systems: Neuro-fuzzy systems: summarysummary The combination of fuzzy logic and neural networks constitutes a The combination of fuzzy logic and neural networks constitutes a

powerful means powerful means for designing intelligent systems. for designing intelligent systems.

Domain knowledge Domain knowledge can be put into a neuro-fuzzy system by can be put into a neuro-fuzzy system by human experts in the form of human experts in the form of linguistic variables and fuzzy ruleslinguistic variables and fuzzy rules..

When a representative set of examples is available, a neuro-fuzzy When a representative set of examples is available, a neuro-fuzzy system system can automatically transform it into a robust set of fuzzy can automatically transform it into a robust set of fuzzy IF-THEN rulesIF-THEN rules, and thereby , and thereby reduce our dependence reduce our dependence on expert on expert knowledge when building intelligent systems.knowledge when building intelligent systems.

ANFIS: ANFIS: Adaptive Neuro-Adaptive Neuro-Fuzzy Inference Fuzzy Inference

SystemSystem

ANFIS: ANFIS: Adaptive Neuro-Fuzzy Inference SystemAdaptive Neuro-Fuzzy Inference System

The The Sugeno fuzzy model Sugeno fuzzy model was proposed for generating was proposed for generating fuzzy rules from a given input-output data set. fuzzy rules from a given input-output data set.

A typical Sugeno fuzzy rule is expressed in the following A typical Sugeno fuzzy rule is expressed in the following form:form:

IFIF xx11 is is AA11

ANDAND xx22 is is AA22

. . . . .. . . . .

ANDAND xxmm is is AAmm

THEN THEN yy = = ff ( (xx11, , xx22, . . . , , . . . , xxmm))

where where xx11, , xx22, . . . , , . . . , xxmm are input variables are input variables; ; AA11, , AA22, . . . , , . . . , AAmm

are fuzzy sets. are fuzzy sets.

When When yy is a constant is a constant, we obtain a , we obtain a zero-orderzero-order Sugeno fuzzy modelSugeno fuzzy model in which the consequent of in which the consequent of a rule is specified by a singleton. a rule is specified by a singleton.

When When yy is a first-order polynomial, i.e. is a first-order polynomial, i.e.

yy = = kk00 + + kk11 xx11 + + k k22 xx22 + . . . + + . . . + kkmm xxmm

we obtain a we obtain a first-orderfirst-order Sugeno fuzzy model Sugeno fuzzy model..

Orders of Orders of Sugeno fuzzy model Sugeno fuzzy model

N1

y

N3

N2

N4

1

x2

x1

Layer 2 Layer 5Layer 3 Layer 4 Layer 6Layer 1

A1

A2

B1

B2

x1 x2

1

2

3

4

2

3

4

Adaptive Neuro-Fuzzy Inference Adaptive Neuro-Fuzzy Inference SystemSystem

LayerLayer 1 1 is the is the input layerinput layer. Neurons in this layer simply pass external crisp . Neurons in this layer simply pass external crisp signals to signals to LayerLayer 2. 2.

LayerLayer 2 2 is the is the fuzzification layerfuzzification layer. Neurons in this layer perform . Neurons in this layer perform fuzzification. In Jang’s model, fuzzification neurons have a fuzzification. In Jang’s model, fuzzification neurons have a bell bell activation functionactivation function..

N1

y

N3

N2

N4

1

x2

x1

Layer 2 Layer 5Layer 3 Layer 4 Layer 6Layer 1

A1

A2

B1

B2

x1 x2

1

2

3

4

2

3

4

Adaptive Neuro-Fuzzy Inference Adaptive Neuro-Fuzzy Inference SystemSystem

LayerLayer 3 3 is the is the rule layerrule layer. . Each neuron in this layer corresponds to a single Sugeno-type fuzzy rule. Each neuron in this layer corresponds to a single Sugeno-type fuzzy rule. A rule neuron receives inputs from the respective fuzzification neurons and A rule neuron receives inputs from the respective fuzzification neurons and

calculates the firing strength of the rule it represents. calculates the firing strength of the rule it represents. In an ANFIS, the conjunction of the rule antecedents is evaluated by the In an ANFIS, the conjunction of the rule antecedents is evaluated by the

operator operator productproduct. .

LayerLayer 3 3 is the is the rule layerrule layer. Each neuron in this layer . Each neuron in this layer corresponds to a single Sugeno-type fuzzy rule. A corresponds to a single Sugeno-type fuzzy rule. A rule neuron receives inputs from the respective rule neuron receives inputs from the respective fuzzification neurons and calculates the firing fuzzification neurons and calculates the firing strength of the rule it represents. strength of the rule it represents.

In an ANFIS, the conjunction of the rule antecedents is In an ANFIS, the conjunction of the rule antecedents is evaluated by the operator evaluated by the operator productproduct. Thus, the output . Thus, the output of neuron of neuron ii in in LayerLayer 3 is obtained as, 3 is obtained as,

where the value of where the value of 11 represents the firing strength, represents the firing strength,

or the truth value, of or the truth value, of RuleRule 1. 1.

k

jjii xy

1

)3()3( y(3) = A1

B1 = 1,1

N1

y

N3

N2

N4

1

x2

x1

Layer 2 Layer 5Layer 3 Layer 4 Layer 6Layer 1

A1

A2

B1

B2

x1 x2

1

2

3

4

2

3

4

Adaptive Neuro-Fuzzy Inference Adaptive Neuro-Fuzzy Inference SystemSystem

LayerLayer 4 4 is the is the normalisation layernormalisation layer. Each neuron in . Each neuron in this layer receives inputs from all neurons in the this layer receives inputs from all neurons in the rule layer, and calculates the rule layer, and calculates the normalised firing normalised firing strengthstrength of a given rule. of a given rule.

LayerLayer 4 4 is the is the normalisation layernormalisation layer. Each neuron in . Each neuron in this layer receives inputs from all neurons in the this layer receives inputs from all neurons in the rule layer, and calculates the rule layer, and calculates the normalised firing normalised firing strengthstrength of a given rule. of a given rule.

The normalised firing strength is the ratio of the The normalised firing strength is the ratio of the firing strength of a given rule to the sum of firing firing strength of a given rule to the sum of firing strengths of all rules. It represents the contribution strengths of all rules. It represents the contribution of a given rule to the final result. of a given rule to the final result. Thus, the Thus, the output of neuron output of neuron ii in in LayerLayer 4 is determined as, 4 is determined as,

in

jj

in

jji

iii

x

xy

11

)4(

)4()4(

14321

1)4(1N

y

LayerLayer 5 5 is the is the defuzzification layerdefuzzification layer. Each neuron . Each neuron in this layer is connected to the respective in this layer is connected to the respective normalisation neuron, and also receives initial normalisation neuron, and also receives initial inputs, inputs, xx11 and and xx22. A defuzzification neuron . A defuzzification neuron

calculates the weighted consequent value of a calculates the weighted consequent value of a given rule as,given rule as,

where is the input and is the output of where is the input and is the output of defuzzification neuron defuzzification neuron ii in in LayerLayer 5, and 5, and kkii00, , kkii11 and and

kkii22 is a set of consequent parameters of rule is a set of consequent parameters of rule ii..

21 21 2 10 2 10)5()5( xkxkkxkxkkxy iiiiiiiii )5(

ix )5(iy

LayerLayer 6 6 is represented by a single is represented by a single summation summation neuronneuron. .

This neuron calculates the sum of outputs of all This neuron calculates the sum of outputs of all defuzzification neurons and produces the overall defuzzification neurons and produces the overall ANFIS output, ANFIS output, yy,,

n

iiiii

n

ii xkxkkxy

1210

1

)6( 21

Can an ANFIS deal with problems Can an ANFIS deal with problems where we where we do not have any prior knowledge of the rule do not have any prior knowledge of the rule consequent parameters? consequent parameters?

It is not necessary to have any prior It is not necessary to have any prior knowledge of rule consequent knowledge of rule consequent parameters. parameters.

An ANFIS learns these parameters An ANFIS learns these parameters and tunes membership functions.and tunes membership functions.

Learning in the ANFIS model Learning in the ANFIS model An ANFIS uses a hybrid learning algorithm that An ANFIS uses a hybrid learning algorithm that

combines the least-squares estimator and the gradient combines the least-squares estimator and the gradient descent method. descent method.

In the ANFIS training algorithm, each epoch is In the ANFIS training algorithm, each epoch is composed from a forward pass and a backward pass. composed from a forward pass and a backward pass. In the forward pass, a training set of input patterns In the forward pass, a training set of input patterns

(an input vector) is presented to the ANFIS, neuron (an input vector) is presented to the ANFIS, neuron outputs are calculated on the layer-by-layer basis, outputs are calculated on the layer-by-layer basis, and rule consequent parameters are identified.and rule consequent parameters are identified.

The rule consequent parameters are identified by the The rule consequent parameters are identified by the least-squares estimatorleast-squares estimator. .

In the Sugeno-style fuzzy inference, an output, In the Sugeno-style fuzzy inference, an output, yy, is a , is a linear functionlinear function. .

Thus, given the values of the membership parameters and Thus, given the values of the membership parameters and a training set of a training set of PP input-output patterns, input-output patterns, we can form we can form PP linear equations linear equations in terms of the consequent parameters as:in terms of the consequent parameters as:

(1) f(1)yd (1) (1) f(1)

(2) f(2)yd (2) (2) f(2)

(p) f(p)yd (p) (p) f(p)

(P) f(P)yd (P) (P) f(P)

n(1) fn(1)

n(2) fn(2)

n(p) fn(p)

n(P) fn(P)

the the least-squares estimatorleast-squares estimator. .

In the matrix notation, we haveIn the matrix notation, we have

yydd = = AA kk,,

where where yydd is a is a PP 1 desired output vector, 1 desired output vector,

and and kk is an is an n n (1 + (1 + mm) ) 1 vector of unknown consequent 1 vector of unknown consequent parameters,parameters,

kk = [ = [kk1010 kk1111 kk1212 … … kk11mm kk2020 kk2121 kk2222 … … kk22mm … … kknn00 kknn11 kknn22 … … kkn mn m]]TT

yd (1)

yd (2)

yd (p)

yd

yd (P)

A

(2) (1) x(2) (2) xm(2)

(1) (1) x(1) (1) xm(1) n(1)

n (1) x(1) n (1) xm(1)

n(2)

n (2) x(2) n (2) xm(2)

(p) (p) x(p) (p) xm(p)

n(p) n (p) x(p)

n (p) xm(p)

(P) (P) x(P) (P) xm(P) n(P)

n (P) x(P) n (P) xm(P)

As soon as the rule consequent parameters are As soon as the rule consequent parameters are established, we compute an actual network output established, we compute an actual network output vector, vector, yy, and determine the error vector, , and determine the error vector, ee

ee = = yydd yy

In the backward pass, the back-propagation algorithm In the backward pass, the back-propagation algorithm is applied. is applied. The error signals are propagated back, and the The error signals are propagated back, and the

antecedent parameters are updated according to antecedent parameters are updated according to the chain rule.the chain rule.

back-propagation algorithm back-propagation algorithm

In the ANFIS training algorithm suggested by Jang, In the ANFIS training algorithm suggested by Jang, both antecedent parameters and consequent both antecedent parameters and consequent parameters are optimised. parameters are optimised.

In the forward pass, the consequent parameters are In the forward pass, the consequent parameters are adjusted while the antecedent parameters remain adjusted while the antecedent parameters remain fixed. fixed.

In the backward pass, the antecedent parameters are In the backward pass, the antecedent parameters are tuned while the consequent parameters are kept fixed.tuned while the consequent parameters are kept fixed.

ANFIS training algorithm of JangANFIS training algorithm of Jang

Function approximation Function approximation using the ANFIS modelusing the ANFIS model

In this example, an ANFIS is used to follow a In this example, an ANFIS is used to follow a trajectory of the non-linear function defined by trajectory of the non-linear function defined by the equationthe equation

First, we choose an appropriate architecture for First, we choose an appropriate architecture for the ANFIS. An ANFIS must have two inputs – the ANFIS. An ANFIS must have two inputs – xx1 and 1 and xx2 – and one output – 2 – and one output – yy..

Thus, in our example, the ANFIS is defined by Thus, in our example, the ANFIS is defined by four rules, and has the structure shown below.four rules, and has the structure shown below.

2)1 2cos(

xe

xy

An ANFIS model with An ANFIS model with four rulesfour rules

N1

y

N3

N2

N4

1

x2

x1

Layer 2 Layer 5Layer 3 Layer 4 Layer 6Layer 1

A1

A2

B1

B2

x1 x2

1

2

3

4

2

3

4

The The ANFIS training data ANFIS training data includes 101 training includes 101 training samples. samples. They are represented by a 101 They are represented by a 101 3 matrix [ 3 matrix [xx11 xx22

yydd], where ], where xx11 and and xx22 are input vectors, and are input vectors, and yydd is a is a

desired output vector. desired output vector.

The first input vector, The first input vector, xx11, starts at 0, increments by 0.1 , starts at 0, increments by 0.1 and ends at 10. and ends at 10.

The second input vector, The second input vector, xx22, is created by taking sin , is created by taking sin from each element of vector from each element of vector xx11, with elements of the , with elements of the desired output vector, desired output vector, yydd, determined by the function , determined by the function

equation.equation.

Learning in an ANFIS with two membership Learning in an ANFIS with two membership functions assigned to each input (one epoch)functions assigned to each input (one epoch)

1-3

-2

-1

0

1

2

-1

-0.5

0

0.5

02

46

810

y

x1x2

Training DataANFIS Output

Learning in an ANFIS with two membership Learning in an ANFIS with two membership functions assigned to each input (100 epochs)functions assigned to each input (100 epochs)

02

46

810

-1

-0.5

0

0.5

1-3

-2

-1

0

1

2

x1x2

y

Training DataANFIS Output

We can achieve some improvement, but much We can achieve some improvement, but much better results are obtained when we assign three better results are obtained when we assign three membership functions to each input variable. membership functions to each input variable.

In this case, the ANFIS model will In this case, the ANFIS model will have nine have nine rulesrules, as shown in figure below., as shown in figure below.

y

x1 x2

11

2

3

4

5

6

7

8

9

A3

A1

x1A2

B3

B1

B2

N1

N2

N3

N4

N6

N5

N7

N8

N9

2

3

4

5

6

7

8

9

x2

An ANFIS model with An ANFIS model with ninenine rules rules

Learning in an ANFIS with Learning in an ANFIS with three three membership membership functions assigned to each input (one epoch)functions assigned to each input (one epoch)

1-3

-2

-1

0

1

2

-1

-0.5

0

0.5

02

46

810

y

x1x2

Training DataANFIS Output

Learning in an ANFIS with three membership Learning in an ANFIS with three membership functions assigned to each input (functions assigned to each input (100 epochs100 epochs))

02

46

810

-1

-0.5

0

0.5

1-3

-2

-1

0

1

2

x1x2

y

Training DataANFIS Output

Initial and final membership functions of the ANFISInitial and final membership functions of the ANFIS