Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
2
Contents
Introduction
Study of Artificial Neural
Network (ANN)
Study of Fuzzy Logic
Neuro-Fuzzy
Hybridization
Conclusion
Solution!
Problems!
Literature Survey
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 20
2.1 Introduction
The objective of the study is to design a generic architecture platform for
development of Neuro-Fuzzy system using extended functionalities of type 2 fuzzy
logic. The base for developing the architecture under study lies on two major
technologies of soft computing artificial neural network and fuzzy logic. The fuzzy
logic was later extended to type 2 fuzzy logic to enhance the decision support and
easy attachment while hybridizing with artificial neural network. Figure 2.1 describes
the major areas in which literature survey is done.
Figure 2.1: Areas of Literature Survey
Artificial Neural Network
To explore various existing artificial neural networks like Feed Forward, Radial Basis, SOM, Learning Vector Quantization, Recurrent, etc.
To study and analyze various algorithms, methods & properties (static & dynamic) for the considered artificial neural networks.
To clearly understand advantages and disadvantages of various standard methods for respective domain applicability.
To study the improvement and research done by other researchers in the field, and also find the future scope for the practical applicability of subject under research.
Fuzzy Logic (Type2)
To study the mechanism of fuzzy inference engine to understand working of fuzzy systems.
To analyze various existing fuzzy inference models like Mamdani, Sugeno and Tsukamoto.
To analyze the advantages offered by type 2 fuzzy logic and its practical applicability.
To study various types of membership functions for fuzzy inference systems.
To study defuzzification methods and their practical applicability.
To study how to generate a knowledge base for given domain area.
Neuro-Fuzzy Approach
To analyze various hybridization methods with their parameter for hybridizing artificial neural network with fuzzy logic.
To bridge the gap between crisp data of artificial neural network and fuzzy logic to generate an efficient interface for generating various neuro-fuzzy advisory systems under research.
To study existing neuro-fuzzy systems with their practical applicability and development time required.
Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 21
2.2 Study of Artificial Neural Network (ANN)
An artificial neural network (ANN) is an interconnected group of artificial neurons that
uses a computational model for processing information based on a connectionist
approach to computation. An artificial neural network is a parallel system, capable of
resolving paradigms that linear computing cannot. ANN is powerful tool for modeling
intelligent systems when underlying relationship is unknown. ANN has ability to
identify and learn the correlation between the input values and the required output
target values. For this purpose ANN have to be trained through various existing
learninig approaches so that they can predict correct output as required by the
parallel systems. The most important feature of ANN is that they learn by example
(training) which replaces traditional programming approach towards problem solving.
Due to this feature, ANN are increasingly being used in intelligent systems where
one has little or incomplete understanding of the problem under research but training
data of domain expert is readily available.
In 1943 McCulloch – Pitts[14] proposed a model for computing elements which
performed weighted sum of inputs on computing elements and then applied
threshold logic on them. This model has drawback, as the weights were fixed, it was
not able to learn by example. In 1949, Hebbian[6] proposed a model for learning
scheme by adjusting the weight of the connection on pre and post synaptic values of
the variables. Later this became the famous Hebbs law for learning rule in neural
network. In 1958 Rosenblatt[19] proposed a model based on perceptron which also
had adjustable weights based on percptron learning rule. Widrows and Hoff in
1960[27] proposed adaptive linear element for computing elements and least mean
square algorithm for learning. In 1982 Hopefield[7] made first move to develop
feedback neural networks, in his model with symmetrical weights the analysis has
shown stable equilibrium states. Taking a step further to this study in 1986
Rumelhart et al.[20] showed that it possible to adjust weights of neuron in feed
forward multi layer neural network in a systematic manner to map implicit learning of
input and output patterns in form of pairs using training sets. The method was called
as generalized delta rule or error back propagation method. From here onwards the
real life applicability of artificial neural network started. In 1998 Zhang et al [30]
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 22
provided summary of working of ANN in forecasting, further provided guidelines for
forecasting, paradigms and other issues of ANN.
Developing Artificial Neural Networks
Artificial neural networks are constructed with layers of neurons; hence the term
„Multilayer‟ is used with such artificial neural networks. A Multilayer consists of three
kind of layer namely input layer, output layer and hidden layer. There can be more
than one hidden layer. A multi layered neural network is shown in Figure 2.2. The
process of mapping of input training pattern to that of output training pattern is
actually carried out by back propagating error in hidden layers, which in turn adjust
the weights of neurons to match the required output pattern. Generally, multiple
hidden layers are designed in ANN as per requirement of the problem domain. The
smoothness of learning increases with more number of hidden layers but the
computational complexity also rises. In ANN each neuron of the preceding layer is
connected to every other neurons of the following layer. There are two functions that
steer the behavior of a neuron in particular layer, which is also applied to every other
neuron of the same neural network. They are Input function and the output function
(Activation function). Input into a node is weighted sum of output connected to it. The
threshold sign function is defined as � = 1, ℎ 0 � � = 0, ℎ < 0
And its differentiable form is represented with the help of sigmoid function
�� � =1
1 + −
The activation functions are applied under particular learning algorithm. One of the
common learning algorithms for artificial neural network is back propagation. Beside
these other algorithms are radial base, self organizing map, learning vector
quantization, and reinforcement.
Back Propagation Algorithm
Back propagation algorithm was initially introduced by Werbos in 1974 [26] and further
developed by Rumelhart and McClelland in 1986 [20]. Multi Layer Perceptrons (MLP)
networks are neural networks with one or more hidden layers. Cybenko in 1989 [5] and
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 23
Funahashi in 1989 [10] have proved that the MLP network is a general function approximator
and that one hidden layer networks will always be sufficient to approximate any continuous
function up to certain accuracy. A MLP network with two hidden layers is shown in Figure
2.2. The input layer acts as an input data holder that distributes the input to the first hidden
layer. The outputs from the first hidden layer then become the inputs to the second layer and
so on. The last layer acts as the network output layer.
A hidden neuron performs two functions that are the combining function and the
activation function. The output of the j-th neuron of the k-th hidden layer is given by
1
1
1kn
i
k
j
k
i
k
ij
k
j btvwFtv ; for 0 knj (1)
and if the m-th layer is the output layer then the output of the l-th neuron yl of the
output layer is given by
1
1
1ˆmn
i
m
i
m
ijl tvwty ; for 0 onl (2)
where nk, no w‟s, b‟s and F(.) are the number of neurons in k-th layer, number of
neurons in output layer, weights, thresholds and an activation function respectively.
Figure 2.2 represent multi layered perceptron network.
Figure 2.2: Multilayered Perceptron Network
Consider a network with a single output node in output layer and a single hidden
layer is used, i.e m = 2 and no = 1. With these simplifications the network output is:
rn
j
jjij
n
i
ii
n
i
i btvwFwtvwty1
11
1
21
1
211
ˆ (3)
Where nr is the number of nodes in the input layer.
V1
V2
Vn
y1
Yn
Hidden Layers
Input Layer Output Layer
Wi Weight
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 24
The activation function F(.) is selected to be
tve
tvF
1
1 (4)
The weights wi and threshold bj are unknown and should be selected to minimise the
prediction errors defined as
tytyt ˆ (5)
Where y (t) is the actual output and y t is the network output.
Back propagation is the steepest decent type algorithm where the weight connection
between the j-th neuron of the (k-1)-th layer and the i-th neuron of the k-th layer are
respectively updated according to
tbtbtb
twtwtw
k
i
k
i
k
i
k
ij
k
ij
k
ij
1
1 (6)
with the increment w tijk and b ti
k given by
1
11
tbttb
twtvttw
k
ib
k
ib
k
i
k
ijw
k
j
k
iw
k
ij
(7)
where the subscripts w and b represent the weight and threshold respectively, w
and b are momentum constants which determine the influence of the past
parameter changes on the current direction of movement in the parameter space, w
and b represent the learning rates and ik
t is the error signal of the i-th neuron of
the k-th layer which is back propagated in the network.
Since the activation function of the output neuron is linear, the error signal at the
output node is
tytytm ˆ (8)
and for the neurons in the hidden layer
j
k
ji
k
j
k
i
k
i twttvFt 111 k = m-1, ... , 2, 1 (9)
where F v tik is the first derivative of F v ti
k with respect to v tik .
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 25
Since back propagation algorithm is a steepest decent type algorithm, the algorithm
suffers from a slow convergence rate. The search for the global minima may become
trapped at local minima and the algorithm can be sensitive to the user selectable
parameters.
Topologies for Artificial Neural Network
Feed Forward- where the data flow from input to output units is strictly
feedforward. The data processing can extend over multiple (layers of) units,
but no feedback connections are present, that is, connections extending from
outputs of units to inputs of units in the same layer or previous layers.
Recurrent- that do contain feedback connections. Contrary to feed-forward
networks, the dynamical properties of the network are important. In some
cases, the activation values of the units undergo a relaxation process such
that the neural network will evolve to a stable state in which these activations
do not change anymore. In other applications, the changes of the activation
values of the output neurons are significant, such that the dynamical behavior
constitutes the output of the neural network as shown by Pearlmutter [17].
Radial Basis- is an artificial neural network that uses radial basis functions as
activation functions. It is a linear combination of radial basis functions. They
are used in function approximation, time series prediction, and control. Radial
basis function (RBF) networks typically have three layers: an input layer, a
hidden layer with a non-linear RBF activation function and a linear output
layer. In the basic form all inputs are connected to each hidden neuron. The
norm is typically taken to be the Euclidean distance (though distance appears
to perform better in general) and the basis function is taken to be gaussian
function .i.e. changing parameters of one neuron has only a small effect for
input values that are far away from the center of that neuron. RBF networks
are universal approximators. This means that a RBF network with enough
hidden neurons can approximate any continuous function with arbitrary
precision.
Kohenen Self Organizing Map (SOM)- is a type of artificial neural network
that is trained using unsupervised learning to produce a low-dimensional
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 26
(typically two-dimensional), discrete representation of the input space of the
training samples, called a map. Self-organizing maps are different from other
artificial neural networks in the sense that they use a neighborhood function to
preserve the topological properties of the input space. This makes SOMs
useful for visualizing low-dimensional views of high-dimensional data.
A self-organizing map consists of components called nodes or neurons.
Associated with each node is a weight vector of the same dimension as the
input data vectors and a position in the map space. The usual arrangement of
nodes is a regular spacing in a hexagonal or rectangular grid. The self-
organizing map describes a mapping from a higher dimensional input space
to a lower dimensional map space. The procedure for placing a vector from
data space onto the map is to first find the node with the closest weight vector
to the vector taken from data space. Once the closest node is located it is
assigned the values from the vector taken from the data space.
While it is typical to consider this type of network structure as related to feed
forward networks where the nodes are visualized as being attached, this type
of architecture is fundamentally different in arrangement and motivation.
Large SOMs display properties which are emergent. In maps consisting of
thousands of nodes, it is possible to perform cluster operations on the map
itself.
Learning Vector Quantization- LVQ was invented by Teuvo Kohonen [23]
and it can be understood as a special case of an artificial neural network,
more precisely, it uses winner-take-all Hebbian learning-based approach. An
LVQ system is represented by prototypes W= (w(i),..., w(n)) which are defined
in the feature space of observed data. In winner-take-all training algorithms
one determines, for each data point, the prototype which is closest to the input
according to a given distance measure. The position of this so-called winner
prototype is then adapted, i.e. the winner is moved closer if it correctly
classifies the data point or moved away if it classifies the data point
incorrectly. Advantage of LVQ is that it creates prototypes that are easy to
interpret for experts in the respective application domain. LVQ systems can
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 27
be applied to multi-class classification problems in a natural way. It is used in
a variety of practical applications.
A key issue in LVQ is the choice of an appropriate measure of distance or
similarity for training and classification. Recently, techniques have been
developed which adapt a parameterized distance measure in the course of
training the system as shown by Schneider, Biehl, and Hammer [16].
Modular Networks- is a neural network characterized by a series of
independent neural networks moderated by some intermediary. Each
independent neural network serves as a module and operates on separate
inputs to accomplish some subtask of the task that the network hopes to
perform. The intermediary takes the outputs of each module and processes
them to produce the output of the network as a whole. The intermediary only
accepts the modules‟ outputs; the modules do not interact with each other.
Training of Artificial Neural Network
Once a network has been structured for a particular application, that network is
ready to be trained. To start this process the initial weights are chosen randomly.
Then, the training, or learning, begins. There are two basic approaches to training -
supervised and unsupervised. Supervised training involves a mechanism of
providing the network with the desired output either by manually "grading" the
network's performance or by providing the desired outputs with the inputs.
Unsupervised training is where the network has to make sense of the inputs without
outside help. The vast bulk of networks utilize supervised training. Unsupervised
training is used to perform some initial characterization on inputs. The third approach
for training ANN is reinforcement learning where a punishment strategy is used to
train the neural network.
Supervised Learning- we are given a set of example pairs and
the aim is to find a function in the allowed class of functions that
matches the examples. In other words, we wish to infer the mapping implied
by the data; the cost function is related to the mismatch between our mapping
and the data and it implicitly contains prior knowledge about the problem
domain. A commonly used cost is the mean-squared error, which tries to
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 28
minimize the average squared error between the network's output, f(x), and
the target value y over all the example pairs.
When one tries to minimize this cost using gradient descent for the class of
neural networks called multilayer perceptrons, one obtains the common and
well-known back propagation algorithm for training neural networks. Some
popular tasks that fall within the paradigm of supervised learning are pattern
recognition (also known as classification) and regression (also known as
function approximation). The supervised learning paradigm is also applicable
to sequential data (e.g., for speech and gesture recognition).
Un-supervised Learning- some data is given and a cost function is to be
minimized, that can be any function of the data and the network's output, .
The cost function is dependent on the task (what we are trying to model, the
implicit properties of our model, its parameters and the observed variables).
As a trivial example, consider the model , where is a constant and
the cost . Minimizing this cost will give us a value of that is
equal to the mean of the data. The cost function may be much more
complicated depending on the application: for example, in compression it
could be related to the mutual information between and , whereas in
statistical modeling, it could be related to the posterior probability of the model
given the data.
Reinforcement Learning- Here data are usually not given, but generated
by an agent's interactions with the environment. At each point in time t, the
agent performs an action and the environment generates an observation
and an instantaneous cost , according to some (usually unknown)
dynamics. The aim is to discover a policy for selecting actions that minimizes
some measure of a long-term cost; i.e., the expected cumulative cost. The
environment's dynamics and the long-term cost for each policy are usually
unknown, but can be estimated. More formally, the environment is modeled
as a Markov decision process (MDP). Dynamic programming has been
coupled with ANNs (Neuro dynamic programming) by Bertsekas and
Tsitsiklis[3] and applied to multi-dimensional nonlinear problems such as
vehicle routing and natural resources management.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 29
2.3 Study of Fuzzy Logic
In 1965 prof. lofti zadeh[29] introduced fuzzy logic. A fuzzy subset A of a (crisp)
set X is characterized by assigning to each element x of X the degree of
membership of x in A (e.g., X is a group of people, A is the fuzzy set of old people
in X). Now if X is a set of propositions then its elements may be assigned
their degree of truth, which may be “absolutely true,” “absolutely false” or some
intermediate truth degree: a proposition may be more appropriate than another
proposition. Fuzzy variables can be used in IF-THEN-ELSE logic. They may be
understood as partial imprecise knowledge on some crisp function and have (in the
simplest case) the form IF x is Ai THEN y is Bi. In a wider sense fuzzy logic is
synonymous with the theory of fuzzy sets, a theory which relates to classes of
objects with smooth boundaries in which membership is a matter of degree. There
exists various membership functions used to classify the applicability of fuzzy
linguistic variable. A fuzzy linguistic variable are non numeric data that represents
rules or fact. A membership function can be designed in three ways: (1) through the
knowledge acquisition process like interview from experts, those who are familiar
with the underlying concept and later adjust it based on a tuning strategy; (2)
construct it automatically from data; and (3) learn it based on feedback from the
system performance. The first method was commonly used by researchers until the
late 1980s and is still a useful way if there is sufficient a priori knowledge about the
control system. Because of the poor systematic tuning strategy, most fuzzy systems
are tuned using a trial and error process. This has become one of the points of
criticism in fuzzy logic technology. Fortunately, some techniques have become
available for developing the second two methods since the late 80s, for example the
statistical techniques. The most commonly used membership functions are linear,
triangular and trapezoidal. Other membership functions like Gaussian, S-Shaped,
Generalized bell shaped etc are application specific and many other membership
functions have been developed by researcher for their application specific use.
To construct a fuzzy logic based systems it is necessary to obtain knowledge from
human domain experts in form of fuzzy IF-THEN rule analysis. These rules are
combined together and applied on the given problem domain. The fuzzy logic
systems constitutes of four basic components, i.e. Rule Base, Inference Engine,
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 30
Fuzzifier and De-Fuzzifier. Type 2 fuzzy system uses membership functions that
constitutes of type 2 fuzzy sets. For type 2 fuzzy logic same components are used
along with additional type reducer that helps in type reduction from type 2 fuzzy logic
to simple fuzzy logic. The type reducer generalizes type 2 fuzzy sets to type 1 fuzzy
sets using type 2 membership functions. Figure 2.3 represents the flow of type 2
fuzzy logic system. The detailed mathematical equation for fuzzy logic components
can be found in paper written by klir & Yuan in 1995[12] and interval type method
was proposed by Qilian Liang & Jerry Mendel in 2000 [18] for type 2 fuzzy logic.
Figure 2.3: Flow of Fuzzy Logic (Type 2) Systems
Inference Engine
The basic function of the inference engine is to compute level(s) of belief in output
fuzzy sets from the levels of belief in the input fuzzy sets. The output is a single
belief value for each output fuzzy set. In this stage, the fuzzy operator is applied in
order to gain a single number that represents the result of the antecedent for that
rule. The inference engine is mainly based on “rules”. Rules determine the closed-
loop behavior of the system. The rules are based on expert opinion, operator
experience, and system knowledge. The basic function of the rule base is to
represent in a structured way the control policy of an experienced process operator
and/or control engineer in the form of a set of production rules such as if (process
state) and then (control output).
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 31
Fuzzy Inference Operators
The process of mapping the given input to output using fuzzy logic is carried out
using fuzzy inference operators. The fuzzy inference operators evaluates fuzzy
membership functions using specific output. The following are few of the fuzzy
inference operators used with a specified method,
Dienes – Rescher Implication – μR(x, y) = max (1 − μA(x), μB(y)) (1)
Zadeh Implication – μR(x, y) = max ( min ( μA(x), μB(y) ), 1 − μA(x)) (2)
Lukasiewicz Implication – μR(x, y) = min (1, 1 − μA(x) + μB(y)) (3)
Godel Implication – μR(x, y) = � �� � � ( ) � � � � (4)
Minimum Implication – μR(x, y) = min (μA(x), μB(y)) (5)
Product Implication – μR(x, y) = μA(x) ・ μB(y) (6)
All the operators mentioned above are individual rule based inference.
Composition Based Inference
The ways in which rules are combined depend on the interpretation or meaning of
the rule in general sense. When rules are viewed as independent conditional
statements, then a reasonable mechanism for aggregating nR individual rules Ri
(fuzzy relation) is the union:
≐ ��=1
= � 1 �, ,… , � �,
(8)
On the other hand, if rules are seen as strongly coupled conditional statements, their
combination should also have facility of intersection operator:
≐ ��=1
= � 1 �, , … , � �, (9)
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 32
Based on the fuzzy inference operator mentioned in equations (1) to (6) and their
generalized form as shown in equation (8) and (9) on applying composition based
inference respective inference engines namely Dienes – Rescher, Zadeh,
Lukasiewicz, Godel, Minimum, Product, etc. are obtained.
Fuzzy Inference Models
To deduce fuzzy inference from mathematical equation of fuzzy logic there was a
need of generalized fuzzy model. Following are the major model for inference in
fuzzy inference systems (FIS).
Mamdani Model: Mamdani Fuzzy Inference System (FIS) [21] was initially proposed
for controlling a steam engine and boiler combination by synthesizing a set of
linguistic control rules obtained from experienced human operators. Following steps
are used to compute output based on Mamdani inference logic.
Determine a set of fuzzy rules.
Fuzzify the inputs using the input membership functions.
Combine these fuzzified inputs with fuzzy rules to compute strength of fuzzy
rule.
Obtain the consequence of the rule by hybridizing the rule strength with
output membership functions.
Combine the obtained consequence to get output distribution.
Defuzzify these outputs (This step is required only when the result is required
in crisp data form).
Sugeno Model: The Sugeno fuzzy model also known as TSK fuzzy model was
proposed by Takagi, Sugeno and Kang [24], [21] to develop a systematic approach
to generating fuzzy rules from specified input-output data set. The main difference
between Mamdani and Sugeno model is that the output consequence is not
computed by clipping an output membership function at the strength of the specified
rule. Actually Sugeno model does not use output membership function, instead the
output is a crisp number computed by multiplying each input by a constant and then
adding up the result. “Rule Strength” is reoffered as “degree of applicability” and the
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 33
outputs are called as actions. Also there is no output distribution. There is only a
“resulting action” which is a mathematical combination of rule strengths and outputs.
Tsukamoto Model: In this model the consequent of each fuzzy IF-THEN rule is
represented by a fuzzy set with a monotonical membership function, hence as a
result the inferred output of every rule is defined as a crisp value induced by the
rules firing strength [21]. A output of each rule is a crisp value, the Tsukamoto fuzzy
model aggregates the output of each rule by weighted average and thus saves the
time consumed by the process of Defuzzification. However this model is not
practically feasible as it is not as transparent compared to Mamdani and Sugeno
model.
Type 2 Fuzzy Based Inference System
The last decade saw how type 2 fuzzy logic based system replaced traditional fuzzy
inference systems. A group of researchers led by Jerry Mendel made way to
practically implement the idea of type 2 fuzzy logic which was initially proposed by
prof. Zadeh. The basis of type 2 fuzzy logic is type 2 fuzzy sets. Type 2 fuzzy sets
incorporate uncertainty as extra third dimension which gives much clear and logical
information about the problem under research. A type 2 fuzzy sets À is characterized
by type 2 fuzzy membership function:
µÀ (x,u) where x ∈ X and u ∈ Jx ⊆ [0,1]
À = , � , �À , � ∀ ∈ �, ∀ � ∈ � ⊆ 0,1 where, 0 �À , � 1.
Also computation of the system is significantly reduced when, �À , � = 1.
Example of type 2 fuzzy membership functions are Gaussian membership functions
with uncertain mean or uncertain standard deviation. The membership function of
type 2 fuzzy sets is also a three dimensional, where the third dimension is the value
of the membership function at each point on its two dimension domain; this is called
as footprint of uncertainty (FOU). The footprint of uncertainty actually represents the
vagueness of type 1 membership functions and it is completely described by its two
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 34
bounding functions lower membership function (LMF) and upper membership
function (UMF) both of which are basically of the type 1 fuzzy sets. Hence it is much
easier to shift from type 1 fuzzy set to type 2 fuzzy sets without wasting much time in
understanding it. Prof. Mendel and his students introduced interval type 2 fuzzy
systems which actually help to represent footprint of uncertainty in more precise
manner. This in turn led to added advantage of representing unclear ideas or the
degree of relativity in much better and efficient way. The differential functions offered
by type 2 fuzzy logic are much easy to interpret, as they are non-zero equations.
The Defuzzification Methodologies
The defuzzificatation is the process of conversion of fuzzy logic values, obtained
from fuzzy inference engine by applying fuzzy membership functions, back to crisp
values. The process of defuzzification is required only when output is required in
crisp values. There are different defuzzification methods which enables to convert
inferred fuzzy rules back to crisp values, some of the standard defuzzification
methods are;
Centroid – It returns the center of area under the curve, when thinking of a
plain surface, the centroid can be defined as the point along the x – axis,
about which this shape would balance.
Bisector – is a vertical line that divides a region into two sub region of equal
area, gives values approximately similar to that of Centroid function.
Middle, Smallest & Largest of Maximum – these are three different methods
with a similar approach, these three methods key off the maximum values
assumed by the aggregate membership functions, If the aggregate function
has unique maximum value, then all the three i.e. Middle, Smallest & Largest
of Maximum take the same value.
Generally the centroid method is the most commonly used method when the
situation is not known, however at a later stage it is possible to change
defuzzification method to select the most appropriate methods. Some other methods
used for defuzzification also includes AI (adaptive integration), BADD (basic
defuzzification distributions), CDD (constraint decision defuzzification),
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 35
COA (center of area), COG (center of gravity), ECOA (extended center of area),
EQM (extended quality method), FCD (fuzzy clustering defuzzification), FM (fuzzy
mean), FOM (first of maximum), GLSD (generalized level set defuzzification), ICOG
(indexed center of gravity), QM (quality method), RCOM (random choice of
maximum), SLIDE (semi-linear defuzzification), and WFM (weighted fuzzy mean),
however these methods are application specific. Wu and Mendal[28] have shown
working of these methods for interval type 2 fuzzy systems.
Generation of Knowledge Base
When a database of rules is processed by inference engine knowledge base is
generated. The knowledge base generated through fuzzy inference is domain
specific according to rules fired by the inference engines which are stored inside
database.
Figure 2.4: Knowledge Base
The raw data regarding a specific domain under study is stored inside database in
an organized manner. The data is retrieved as per operational requirement of the
data. On the selected set of data a set of rules are to be applied. These rules are in
the form of logical condition‟s like IF-THEN ELSE statements and structured in
formed of decision trees, which decides the appropriate rule to be fired on
encountering certain conditions, and finally rule applicability is inferred by judging the
degree of truthiness of the rule under execution. Figure 2.4 represents knowledge
base with its parameters/entities. Table 2.1 describes these parameters/entities in
brief.
Operating Parameters (knowledge Tuning)
Connecting Parameters (Behavior Learning)
Structural Parameters (Structure Learning)
Logical Parameters (System Design)
Database
Rule base
Inference
Engine
Fuzzifier De-Fuzzifier
Knowledge Base
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 36
Class Parameters/Entities Components
Logic
Reasoning Mechanism Inference Engine
Fuzzy Operators
Type of Membership Functions Fuzzifier and
Defuzzifier
Defuzzification Method Defuzzifier
Structure
Relevant Variables
Knowledge Base Number of membership functions
Number of Rules
Connection
Antecedent of Rule
Rule base Consequent of Rule
Rule Weight
Operation Membership function value Database
Table 2.1: Parameters of Fuzzy Systems
Hence the fuzzy system goes under various phases of execution; the final decision
obtained will represent the knowledge base for the fuzzy system in a given domain.
Knowledge base serves as a backbone for fuzzy inference based expert systems.
2.4 Neuro-Fuzzy Hybridization
On the basis of the study made on artificial neural network and fuzzy logic, it is
obvious that both artificial neural network and fuzzy logic have limitations. To briefly
summarize, following are the advantages and limitations for artificial neural network
and fuzzy logic:
Advantages of Fuzzy Logic
Mimic human decision making to handle vague or uncertain condition.
Rapid computation due to parallel processing attribute.
Ability to deal with imprecise or imperfect information.
Resolving conflicts by applying underlying information of linguistic variable
with help of collaboration, aggregation and propagation.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 37
Improved knowledge representation and modeling of complex, non-linear
problems.
It also helps in documentation of knowledge in rule form.
Natural language processing capability with the use of linguistic variable for
better reasoning.
Limitations of Fuzzy Logic
No self learning capability.
Needs help of domain experts to identify rule for data relationship.
It is abstract form of crisp logic and presents heuristic information.
Advantages of Artificial Neural Network
Mimics the working of biological neural network (Human brain).
No need to know about data relationship as the network can be self
organized.
It has self learning capability.
It has self tune itself adaptively.
Applicable to different models on various systems.
Limitations of Artificial Neural Network
Unable to process linguistic information.
Unable to manage imprecise or vague information, hence it is unable to
resolve conflicts based on numeric data combined with linguistic and logical
data.
Rely on trial and error mechanism to determine hidden layers and nodes.
Hence the objective is to remove the limitations and use the advantages offered by
both the fields - artificial neural network and fuzzy logic. Hybridizing these two
paradigms will remove their limitations and create better decision support and import,
maximum degree of intelligence.
In 2007 Ajith Abraham summarized state of art modeling techniques for neuro-fuzzy
systems [1], [2]. The architectures are categorized in following different categories:
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 38
Cooperative Model: In cooperative model artificial neural network processes data
before it is applied to fuzzy inference system. Here a pre-processing phase is carried
out, where artificial neural network first determines some components of the fuzzy
system. The output of the artificial neural network determines the membership
function or in simpler words fuzzy rules for inference engine of fuzzy inference
system. Once the input parameter for fuzzy inference systems are made available
the artificial neural network goes in background and lets membership function
decides the output for the given neuro-fuzzy system.
Concurrent Model: In the concurrent systems the neural network and the fuzzy
system work together continuously. Hence, the neural networks pre-process the
inputs or post-process the outputs of fuzzy system. The major difference between
the cooperative model and concurrent model is that, the artificial neural network
keeps on churning the data for fuzzy inference system continuously. This helps in
situation where input variables of fuzzy inference system cannot be controlled
directly. Hence sometimes it might happen that the output of fuzzy inference system
might be redirected to artificial neural network for further processing to determine the
fuzzy input control variable.
Hybrid (Fused) Model: In hybrid neuro-fuzzy architecture the artificial neural
network determines the variables for fuzzy inference systems, which enables them to
share data structure and knowledge representations. This process is carried out in
iterative fashion. The easiest way to apply learning algorithm to fuzzy logic is to
generate a special ANN like hybrid structures. However the theory does not fit to
practical applicability as conventional learning algorithms are gradient decent and
the function used for fuzzy inference are non-differentiable. Hence to overcome this
problem either standard learning algorithm for artificial neural network should be
followed or differentiable function should be used to deduce fuzzy inference. The
significant work in this area are GARIC[4], FALCON[13], ANFIS[8], NEFCON[15],
FUN[22], SOFIN[9], FINEST[25], EFuNN[11], dmEFuNN[11], evolutionary design of
neuro-fuzzy systems[2] and many other researcher have contributed in this area.
(i) FALCON: Fuzzy Adaptive Learning Control Network (FALCON)[13] uses a
five layered architecture, two linguistic nodes are determined for each output
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 39
variable. One of them is for training the data (or desired output) and the
second one represents the actual output. Fuzzification of each input variable
is carried out by the first hidden layer; the second hidden layer specifies the
conditions that are to be followed by the consequent of the rule in third hidden
layer. An unsupervised hybrid learning algorithm is used to locate
membership functions or rule base and a gradient decent approach is
followed to optimally adjust the parameter of membership function to produce
desired output.
(ii) ANFIS: Adaptive Neuro-Fuzzy Inference System (ANFIS)[8] implements
Takagi-Sugeno based fuzzy inference system. It also has a five layered
architecture. The first hidden layer is for Fuzzification of input variables,
second hidden layer uses T-norm operator to compute the antecedent part,
the third hidden layer normalizes the rule strength followed by the fourth
hidden layer which determines the consequent part of the rule. The learning
procedure is carried out in two parts; the first part propagates the input pattern
and the optimal consequent which is estimated by the least mean square
method, here the premise parameter are fixed for the current cycle through
the training set, in the second part the patterns are propagated again using
the backpropagation algorithm which in turn modifies the premise parameter
while the consequent parameters are fixed. This procedure is then iterated.
(iii) GARIC: Generalized Approximated Reasoning based Intelligent Control
(GARIC)[4] generates a neuro-fuzzy controller using two artificial neural
networks, the Action Selection Network (ASN) and Action State Evaluation
Network (AEN). The AEN is an adaptive network that evaluates the actions
generated by ASN. ASN used in GARIC is a feed forward network with five
layers, here the connections are not weighted connection. The first hidden
layer stores linguistic values of all input variables, the second layer represents
fuzzy rule nodes, the third hidden layer represents the linguistic values of the
output control variable. GARIC uses a mixture of gradient descent and
reinforcement learning to fine tune the node parameters.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 40
(iv) NEFCON: Neuro-fuzzy Control (NEFCON)[15] was designed to implement
the Mamdani fuzzy inference system. Connection in NEFCON are weighted
with fuzzy sets and rules, the same antecedent used shared weight, which are
represented by ellipse drawn around the connections. They ensure the
integrity of the rule base. The fuzzification is carried out through input unit; the
propagation function represents the inference logic and the output unit is the
defuzzification interface. The learning process is a mixture of back
propagation and reinforcement learning. It can be used to learn initial rule
base where no prior knowledge is available.
(v) FINEST: Fuzzy Inference and Neural Network in Fuzzy Inference Software
(FINEST)[25] is capable of two kind of tuning processes, fuzzy predicate
tuning and combination of functions and tuning of implication functions. It uses
backpropagation algorithm to fine tune its parameters and thus provides a
framework to tune any parameter, which appears in the node of the network
representing the calculation processes of the fuzzy data, if the derivative
function with respect to its parameter is given.
(vi) FUN: Fuzzy Net (FUN)[22] performs fuzzification in first hidden layer with the
help of membership functions and in second hidden layer it performs the
conjunction operations. Membership functions of the output variable are
stored in third hidden layer. The network is initialized with fuzzy rule base and
it uses stochastic learning techniques that randomly change the parameters of
the membership function and connection within the network. The learning
process is driven by cost function and if the modified results are better than
previous result they are retained otherwise the process is repeated.
(vii) EFuNN: Evolving Fuzzy Neural Network (EFuNN)[11] creates nodes
dynamically during learning process. The input layer passes data to the
second layer; the second layer calculates the fuzzy membership degree to
which input values belong to predefined fuzzy membership functions. The
third layer consists of nodes containing fuzzy rules that determine the
input/output data. Every rule node is defined by 2 vectors of connection
weights, which are adjusted through hybrid learning techniques. The fourth
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 41
layer calculates the applicability of the degree of membership function and the
fifth layer performs Defuzzification. A variant of EFuNN is Dynamic EFuNN
(DEFuNN)[11] was proposed with idea that not only the wining rule node to be
propagated but a group of rule node is dynamically selected with every new
input vector and their activation values are used to calculate dynamic
parameters of the output function. Also the major difference is that EFuNN
implements Mamdani model while DEFuNN implements Takagi-Sugeno
model.
(viii) SONFIN: Self Constructing Neural Fuzzy Inference Network (SONFIN)[9]
implements Takagi-Sugeno model for fuzzy inference system with
modification. In the phase of structure identification the input space is
portioned in a flexible way according to aligned cluster based algorithm. With
the help of projection based inference measure some selected additional
variable are added to consequent part incrementally as learning proceeds. To
identify parameter, least mean square or recursive least square method is
used to fine tune consequent parameter and to fine tune precondition
parameter backpropagation algorithm is used.
(ix) Evolutionary Design of Neuro-Fuzzy System: A five-tier hierarchical
evolutionary search procedure is used. The evolving neuro-fuzzy system can
adapt to Mamdani or Takagi-Sugeno based fuzzy inference system. The basic
architecture defines only the layers not procedure. The evolutionary search
process actually decides the optimal type and quantity of nodes and
connection between layers. The Fuzzification layer and the rule antecedent
layer behaves similarly to any other neuro-fuzzy model and the consequent
part of the rule will be determined according to the inference system
depending upon the type of the problem, which will be adapted accordingly to
evolutionary search mechanism. Defuzzification, aggregation operators will
also be adapted according to the fuzzy inference system chosen by the
evolutionary algorithm.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 42
Besides the above models, there are dedicated software systems available for
artificial neural network and fuzzy logic. Some example of these systems are as
follows.
For development of artificial neural network based systems various commercial tools
like Forecaster/XL[http://www.alyuda.com/neural-network-software.htm], Neuro
Intelligence[http://www.alyuda.com/neural-network-software.htm],
Neurosignal/XL[http://www.alyuda.com/neural-network-software.htm],
predictor[http://www.alyuda.com/neural-network-software.htm], Neural Planner
Plus[http://www.easynn.com/], Easy NN plus[http://www.easynn.com/], Neural
Power[http://www.neuroshell.com/], Neural Shell
Predictor[http://www.neuroshell.com/], and many more. Public domain tools like
brainbox[http://brainbox.sourceforge.net/], tiberirusXL[www.tiberius.biz/],
JNNS[http://www.ra.cs.uni-tuebingen.de/downloads/JavaNNS/], and
SNNS[http://www.ra.cs.uni-tuebingen.de/SNNS/] are available but either, they are in
terms with license or the output generated is not modifiable.
For development of fuzzy logic based systems Various fuzzy editors or tools or
programs like
FuzzyCOPE[https://www.aut.ac.nz/resources/research/research_institutes/kedri/dow
nloads/pdf/fuzzycope.pdf], Fuzzy Logic Inference Engine, Fuzzy
Clips[http://www.graco.unb.br/alvares/DOUTORADO/omega.enm.unb.br/pub/doutor
ado/disco2/ai.iit.nrc.ca/IR_public/fuzzy/fuzzyClips/fuzzyCLIPSFiles.html],
NEFCLASS[http://fuzzy.cs.uni-magdeburg.de/nefclass/], Fuzzy logic development
environment for embedded systems, fuzzy logic inferencing
toolkit[http://www.mathworks.in/products/fuzzy-
logic/index.html;jsessionid=e17603dbc9ea8bab686ae8b65030],
FuzzyTECH[http://www.fuzzytech.com/] are available on the internet with a limited
functionalities which includes terms and conditions.
Much later software packages like ANFIS and DENFIS were introduced in
MATLAB[http://www.mathworks.in] software. Further these software‟s had fixed set
of activation function and fixed methods of learning, hence to generate new
activation function or learning methods coding had to be done from the beginning.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 43
Moreover the MATLAB files are not easy to configure with other programming
languages or to deploy them on web.
The aforementioned models, techniques and software systems are application
specific and not generic. Hence at this point of time, there is no generalized
model/tool available for neuro-fuzzy hybridization. The developer of the application
has to choose which model will best fit in the situation of given problem under study.
2.5 Conclusion
The chapter provides a literature review on various neuro-fuzzy modeling
techniques. Also according to situations different neuro-fuzzy modeling techniques
can be used. Through the in-depth literature survey made here it was possible to
study and design the most prominent neuro-fuzzy modeling techniques in the
research work. Furthermore advantages of using neuro-fuzzy system are also listed
which overcomes the limitations of artificial neural network and fuzzy logic when
used separately. Thus designing of library will become simpler based on the
literature survey and will aid in incorporating the beneficial features of artificial neural
network along with type 2 fuzzy logic in order to generate a generic library for the
neuro-fuzzy system. Further it is observed that, the neuro-fuzzy model helps in
extending interpretability. Thus we can conclude that by combining these two major
components soft computing, i.e. artificial neural network and fuzzy logic one can
generate hybrid intelligent system that takes advantages of both the components
and overcomes their limitations to generate intelligent application which will be most
needed in the coming future to meet daily requirements and expectation of a
common man of modern world.
References
[1] Abraham, Adaptation Neuro Fuzzy Systems: State-of-the-art Modeling
Techniques, http://arxiv.org/ftp/cs/papers/0405/0405011.pdf, retrieved on 26th
December 2012.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 44
[2] Abraham, Adaptation of Fuzzy Inference System Using Neural Learning,
http://www.softcomputing.net/nf_chapter.pdf, retrieved on 26th december
2012.
[3] Bertsekas, D.P., Tsitsiklis, J.N. , Neuro-dynamic programming. Athena
Scientific. pp. 512.,1996.
[4] Bherenji H R and Khedkar P, Learning and Tuning Fuzzy Logic Controllers
through Reinforcements, IEEE Transactions on Neural Networks, Vol .3, pp.
724-740, 1992.
[5] G. Cybenko. Approximation by superposition of a sigmoidal function. Math.
Control Systems Signals, Vol.2, pp.303-314, 1989.
[6] Hebb, D.O. Organization of Behavior: A Neuropsychological Theory (New
York: John Wiley and Sons).,1949.
[7] J. J. Hopfield, "Neural networks and physical systems with emergent
collective computational abilities", Proceedings of the National Academy of
Sciences of the USA, Vol. 79 no. 8 pp. 2554–2558, April 1982.
[8] Jang R, Neuro-Fuzzy Modeling: Architectures, Analyses and Applications,
PhD Thesis, University of California, Berkeley, July 1992.
[9] Juang Chia Feng, Lin Chin Teng, An Online Self Constructing Neural Fuzzy
Inference Network and its Applications, IEEE Transactions on Fuzzy Systems,
Vol 6, No.1, pp.12-32, 1998.
[10] K. Funahashi. On the approximate realization of continuous mappings by
neural networks. Neural Networks, Vol.2, pp-183-192, 1989.
[11] Kasabov N and Qun Song, Dynamic Evolving Fuzzy Neural Networks with 'm-
out-of-n' Activation Nodes for On-line Adaptive Systems, Technical Report
TR99/04, Department of information science, University of Otago, 1999.
[12] Klir, G. and Yuan, B. Fuzzy Sets and Fuzzy Logic, Theory and Applications,
chapter 3, pages 50-96. Prentice Hall, Upper Saddle River, New Jersey,
USA.,1995.
[13] Lin C T & Lee C S G, Neural Network based Fuzzy Logic Control and
Decision System, IEEE Transactions on Comput. Vol.40 No.12, pp.1320-
1336, 1991.
[14] McCulloch, W. and Pitts, W. A logical calculus of the ideas immanent in
nervous activity. Bulletin of Mathematical Biophysics, Vol.7, pp.115 – 133,
1943.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 45
[15] Nauck D, Kruse R, Neuro-Fuzzy Systems for Function Approximation, 4th
International Workshop Fuzzy-Neuro Systems, 1997.
[16] P. Schneider, B. Hammer, and M. Biehl. Adaptive Relevance Matrices in
Learning Vector Quantization.Neural Computation 21: 3532–3561, 2009.
[17] Pearlmutter B.A., Dynamic Recurrent Neural Network, CMU,1990.
[18] Q. Liang and J. Mendel, Interval Type-2 Fuzzy Logic Systems: Theory and
Design. IEEE Trans. On Fuzzy Systems, vol. 8, no.5, pp. 535-550, 2000.
[19] Rosenblatt, Frank., The Perceptron: A Probabilistic Model for Information
Storage and Organization in the Brain, Cornell Aeronautical Laboratory,
Psychological Review, v65, No. 6, pp. 386–408.,1958
[20] Rumelhart, D. E., McClelland, J. L., & the PDP research group. Parallel
distributed processing: Explorations in the microstructure of cognition. Vol.1,
MIT Press, Chapter 8, pp.318-362, 1986.
[21] S.N.Sivanandam & S.N.Deepa ,Principles Of Soft Computing, pp.697-710.,
2008.
[22] Sulzberger SM, Tschicholg-Gurman NN, Vestli SJ, FUN: Optimization of
Fuzzy Rule Based Systems Using Neural Networks, In Proceedings of IEEE
Conference on Neural Networks, San Francisco, pp. 312-316, March 1993.
[23] T. Kohonen. Self-Organizing Maps. Springer, Berlin, 1997.
[24] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications
to modeling and control,” IEEE Trans. on Syst., Man, Cybern., vol. 15, pp.
116-132, 1985.
[25] Tano S, Oyama T, Arnould T, Deep combination of Fuzzy Inference and
Neural Network in Fuzzy Inference, Fuzzy Sets and Systems, vol.82 no.2 pp.
151-160, 1996.
[26] Werbos, P.J. Beyond Regression: New Tools for Prediction and Analysis in
the Behavioral Sciences. Ph.D. Thesis, Committee on Applied Mathematics,
Harvard U., 1974. Reprinted in its entirety in Werbos, P.J.: The Roots of
Backpropagation: From Ordered Derivatives to Neural Networks and Political
Forecasting. Wiley, New York (1994). Originally published in 1974.
[27] Widrow, B. and Hoff, M.E., Adaptive switching circuits. IREWESCON
Convention Record Part IV pp. 96-104., 1960.
Chapter 2 : Literature Survey
Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 46
[28] WU, H. AND MENDAL, J.M., Uncertainty bounds and their use in the design
of interval type-2 fuzzy logic system. IEEE Transactions on fuzzy systems,
Vol.10, no.5, pp. 622-639., 2002.
[29] Zadeh, L.A. Fuzzy sets, Information and Control, Vol. 8, 338-353. 1965.
[30] Zhang, G., Patuwo, B.E. and Hu, M.Y., Forecasting with artificial neural
networks: The state of the art. International Journal of Forecasting Vol.14,
pp.35–62., 1998