UNIT-IINTRODUCTION TO

ARTIFICIAL NEURAL NETWORK

EC0054 NEURAL NETWORK AND FUZZY LOGIC

ELEMENTARY NEURO- PHYSIOLOGY

Neuron:A neuron nerve cell is an electricallyexcitable cell that processes and transmits information by electrical and chemical signaling. Chemical signaling occurs via synapses, specialized connections with other cells. Neurons connect to each other to form networks.

PARTS OF THE NEURON Cell Body

Contains the nucleus Dendrites

Receptive regions; transmit impulse to cell body

Short, often highly branched May be modified to form receptors

Axons Transmit impulses away from cell body Axon hillock; trigger zone

Where action potentials first develop Presynaptic terminals (terminal

boutons) Contain neurotransmitter substance (NT) Release of NT stimulates impulse in

next neuronBundles of axons form nerves

ELECTRICAL SIGNALS Neurons produce electrical signals called

action potentials ( = nerve impulse) Nerve impulses transfer information from

one part of body to anothere.g., receptor to CNS or CNS to effector

Electrical properties result from ionic concentration differences across plasma

membrane permeability of membrane

Single Neuron Physiology

Resting Potential

Inhibitory & Exitatory Action Potential

Synapse or synaptic junction

axon of presynapticneuron

dendrite ofpostsynapticneuron

bipolar.about.com/library

Neural circuits and computation Networks either converge or diverge. Divergences is sending nerve impulse,

converging is receiving inputs. Some have feedback – excitatory is positive

feedback and inhibitory is negative feedback.

The working and interconnection of these simple networks, depicts the function the complex human brain – was modeled as a computational unit by McColloch and Pitts in 1943.

Assumptions of the McColloch – Pitts model –

1. The activity of a neuron is all or none process.2. certain number of synapses (>1) must be

excited within a latent period , for a neuron to be excited.

3. The only delay is the synaptic delay.4. An active inhibitory synapse prevents

excitation.5. The structure of the network does not change

over time.

Mc-Pitts Model of Neural Networks

Fig. A and Fig. B. are concepts of divergence and convergence.

Fig. B, C and D illustrate concepts of feedback.

According to assumptions of McColloch-Pitts, a neuron has binary behaviour (activity is all or none) – either ON or OFF.

This can be represented in propositional logic as - a predicate Ni(t) that denotes the assertion that ith neuron fires at time t.

Conversely, ~Ni(t) denotes that ith neuron does not fire at time t.

MCCULLOCH-PITTS NEURON (1943) Inputs - (+1) or (-1) Activation function

Multiply inputs with corresponding weights & sum it Output – (+1) if output is positive else (-1)

EA C461 Artificial Intelligence 16

MCCULLOCH-PITTS NEURON (1943)

FEATURES OF MCCULLOCH-PITTS MODEL Allows binary 0,1 states only Operates under a discrete-time

assumption Weights and the neurons’ thresholds are

fixed in the model and no interaction among network neurons

Just a primitive model NOT ACCURATE.

HEBBIAN LEARNING Biological neurons – learn over time –

supervised learning. Hebb – in his theory of Organizational

behaviour – states that – when an axon of a cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both of the cells such that A’s efficiency (as one of the cells firing B) is increased.

Based on behavior of dogs during lunch timing.

Dogs were initially trained to see the food to cause salivation.

Later bell was rung along with the sight of food to cause salivation.

Finally, sight of food was removed and only bell was rung – dogs still salivated…

So sound sense (second trigger) overruled sight sense (first trgieer) to cause salivation..

How does neuron learn ?

ESSENCE OF Hebbian learning (1949) Speculated that the learning occur by the

modification of synapsesRepeated firing across a synapse increase it’s

sensitivity & hence the future likelihood of firing If a particular stimulus repeatedly causes activity in

a group of cells, those become strongly associated In future similar stimuli would tend to excite the same

neural pathways recognition of the stimuli

Feedforward unsupervised learning

FROM NEURONS TO ARTIFICIAL NEURAL

NETWORKS (ANS)

Structure of Processing Element

WEIGHTS Each neuron is connected to every other

neuron by means of directed links Links are associated with weights Weights contain information about the

input signal and is represented as a matrix

Weight matrix also called connection matrix

WEIGHT MATRIXW=

1

2

3

.

.

.

.

.

T

T

T

T

n

www

w

=

11 12 13 1

21 22 23 2

1 2 3

...

...

..................

...................

...

m

m

n n n nm

w w w ww w w w

w w w w

The output is a function of activation value.

xi = f(ai) The activation value is fn. of

activation value at previous time instant and the net of the PE at that instant.

ai = F(ai(t-1), neti) =~ F(neti) So, xi = F(neti)

Is referred to activation function. Domain is set of activation values net.

Scalar product of weight and input vector

Neuron as a processing node performs the operation of summation of its weighted input – as a function, which is the learning law.

TYPES OF LEARNING LAWS OR LEARNING RULES

Perceptron Learning rule. Delta Learning Rule Widrow-Hoff Winner-Take-All

THE PERCEPTRON - PART 1 The first ANS developed by Frank

Rosenblatt in the late 1950s. to "illustrate some of the fundamental

properties of intelligent systems”. The photoperceptron is a device

that responds to optical patterns. Has - Sensory (S) area , Association (A)

area and Response (R) area

light impinges on the sensory (S) points of the retina structure.

Each S point responds in an all-or-nothing manner to the incoming light.

connections may be either excitatory or inhibitory.

Impulses generated by the S points are transmitted to the associator (A) units in the association layer.

an A unit becomes active if the sum of its inputs exceeds some threshold value.

It then produces an output.

Associator units are connected to response (R) units in the response layer.

The pattern of connectivity is again random.

R unit inhibits the A units in the complement.

each R unit inhibits the other. These factors aid in the establishment of a

single, winning R unit for each stimulus pattern appearing on the retina.

The R units give an output value of +1; otherwise, the output is —1.

Such a system classifies patterns appearing on the retina into categories, according to the number of response units in the system.

Patterns that are sufficiently similar should excite the same R unit.

So, a perceptron has to successfully distinguish between different pattern classes.

The perceptron was a learning device.

A pattern was applied to the retina, and the stimulus was propagated through the layers until a response unit was activated.

If the correct response unit was active, the output of the contributing A units was increased.

If the incorrect R unit was active, the output of the contributing A units was decreased.

Rosenblatt claims that the perceptron could classify patterns successfully in what he termed a differentiated environment.

PERCEPTRON - PART 2: by Marvin Minsky and Seymour Papert. Analysis of the perceptron in terms of its

capabilities and limitations. Perceptrons can differentiate patterns

only if the patterns are linearly separable.

Instead of probabilistic approach by Rosenblatt,they used predicate calculus in their analysis.

PERCEPTRON 2.

Echo suppression in Telephone networks

Frequency response characteristics of different Filters

NEURAL NETWORK ARCHITECTURES• Several NN have been proposed & investigated in

recent years

• Supervised versus unsupervised• Architectures (feedforward vs. recurrent)• Implementation (software vs. hardware)• Operations (biologically inspired vs. psychologically

inspired)

• In this chapter, we will focus on modeling problems with desired input-output data set, so the resulting networks must have adjustable parameters that are updated by a supervised learning rule

SAMPLE FEED FORWARD NETWORK (NO LOOPS) FFN

WeightsWeights

Weights

WjiVik

F( S wji xj

LMS LEARNING RULE 1. Apply input to Adaline input 2. Find the square error of current input Errsq(k) = (d(k) - W x(k))**2 3. Approximate Grad(ErrorSquare) by differentiating Errsq approximating average Errsq by Errsq(k) obtain -2Errsq(k)x(k) Update W: W(new) = W(old) +

2mErrsq(k)X(k) Repeat steps 1 to 4.

Structure of ADALINE

Structure of ALC(Adaptive Linear Combiner)

ALC as a transversal Filter

Use of ADALINE in solving XOR problems

MDALINE Architecture

BPN Architecture

IMAGE TO ASCII CONVERSION USING NEURAL NETWORK

Image to ASCII Conversion using Neural Network

Image to ASCII Conversion using Neural Network (Cont.d)

REVIEW QUESTIONS

What is Processing Element. How would you relate the PEs with real neurons

Define Resting Potential. What is the average refractory period of a neuron. Is it limited to a particular value. If Yes mention How?

Differentiate Resting potential and action potential

State Hebbs Learning Rule. Draw a sample memory mapping diagram by your own.

How would you factor out the weight vector from the exception value terms

What is the use of signal processing techniques in neural networks

REVIEW QUESTIONS (CONTD..)

REFERENCES

J. A. Freeman and D. M. Skapura, Neural Networks- Algorithms, Applications and Programming Techniques, Pearson Education( singapore) Pvt. Ltd., 1991.

(Chapters 1 &2) psychology.about.com/od/biopsychology/f/

neuron01.htm www.cell.com/neuron www.neurophys.com faculty.washington.edu/chudler/chnt1.html