Introduction to Neural Networks

John Paxton

Montana State University

Summer 2003

Textbook

Fundamentals of Neural Networks:

Architectures, Algorithms, and Applications

Laurene Fausett

Prentice-Hall

1994

Chapter 1: Introduction

• Why Neural Networks?

Training techniques exist.

High speed digital computers.

Specialized hardware.

Better capture biological neural systems.

Who is interested?

• Electrical Engineers – signal processing, control theory

• Computer Engineers – robotics

• Computer Scientists – artificial intelligence, pattern recognition

• Mathematicians – modelling tool when explicit relationships are unknown

Characterizations

• Architecture – a pattern of connections between neurons

• Learning Algorithm – a method of determining the connection weights

• Activation Function

Problem Domains

• Storing and recalling patterns

• Classifying patterns

• Mapping inputs onto outputs

• Grouping similar patterns

• Finding solutions to constrained optimization problems

A Simple Neural Network

x2

y

w1

w2

x1

yin = x1w1 + x2w2

Activation is f(yin)

Biological Neuron

• Dendrites receive electrical signals affected by chemical process

• Soma fires at differing frequencies

somadendrite

axon

Observations

• A neuron can receive many inputs

• Inputs may be modified by weights at the receiving dendrites

• A neuron sums its weighted inputs

• A neuron can transmit an output signal

• The output can go to many other neurons

Features

• Information processing is local

• Memory is distributed (short term = signals, long term = dendrite weights)

• The dendrite weights learn through experience

• The weights may be inhibatory or excitatory

Features

• Neurons can generalize novel input stimuli

• Neurons are fault tolerant and can sustain damage

Applications

• Signal processing, e.g. suppress noise on a phone line.

• Control, e.g. backing up a truck with a trailer.

• Pattern recognition, e.g. handwritten characters or face sex identification.

• Diagnosis, e.g. aryhthmia classification or mapping symptoms to a medical case.

Applications

• Speech production, e.g. NET Talk. Sejnowski and Rosenberg 1986.

• Speech recognition.

• Business, e.g. mortgage underwriting. Collins et. Al. 1988.

• Unsupervised, e.g. TD-Gammon.

Single Layer Feedforward NN

x1

xn

y1

ym

w11

w1m

wn1

wnm

Multilayer Neural Network

• More powerful

• Harder to train

x1

xn zp

z1

ym

y1

Setting the Weight

• Supervised

• Unsupervised

• Fixed weight nets

Activation Functions

• Identity f(x) = x

• Binary step f(x) = 1 if x >= f(x) = 0 otherwise

• Binary sigmoidf(x) = 1 / (1 + e-x)

Activation Functions

• Bipolar sigmoidf(x) = -1 + 2 / (1 + x)

• Hyperbolic tangentf(x) = (ex – e-x) / (ex + e-x)

History

• 1943 McCulloch-Pitts neurons

• 1949 Hebb’s law

• 1958 Perceptron (Rosenblatt)

• 1960 Adaline, better learning rule (Widrow, Huff)

• 1969 Limitations (Minsky, Papert)

• 1972 Kohonen nets, associative memory

History

• 1977 Brain State in a Box (Anderson)

• 1982 Hopfield net, constraint satisfaction

• 1985 ART (Carpenter, Grossfield)

• 1986 Backpropagation (Rumelhart, Hinton, McClelland)

• 1988 Neocognitron, character recognition (Fukushima)

McCulloch-Pitts Neuron

x1

x2

x3

y

f(yin) = 1 if yin >=

Exercises

• 2 input AND

• 2 input OR

• 3 input OR

• 2 input XOR