IE 585

Introduction to Neural Networks

2

Modeling Continuum

UnarticulatedWisdom

ArticulatedQualitativeModels

Theoretic(First Principles)Models

EmpiricalCategoricalModels

EmpiricalContinuousModels

Low Cost /High Error

High Cost /Low Error

3

Rise of Empirical Models

• Sensoring - lots of data• Fast computing• Computing available on site• More complicated systems - do not

adhere to simple models• Easy to use software

4

Typical Empirical Models

• linear regression• splines• nearest neighbor clustering• neural networks

5

What is a Neural Net?

• An NN is a network of many simple processors (“units, neurons”), each possibly having a small amount of local memory. The units are connected by communication channels (“connections”) which usually carry numeric data, encoded by any of various means. The units operate only on their local data and on the inputs they receive via the connections.

Usenet newsgroup comp.ai.neural-nets

6

What is a Neural Net?

• An NN is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects:

1. Knowledge is acquired by the network through a learning process.

2. Interneuron connection strengths known as synaptic weights are used to store the knowledge.

Haykin (1994)

7

Objectives of Neural Nets

• High Computing Speed

• Large Memory Capacity

• Adaptive Learning

• Fault Tolerance

8

Neural Network Predictive Models - Advantages

• Can accommodate non-linear relationships with interactions among variables

• Generalize well even for noisy and imprecise data

• No assumption of analytical function or theoretic relation needed

• User friendly software available• Computationally very fast, once built

9

Neural Network Predictive Models - Disadvantages

• Strongly data dependent• No statistical interpretation of significance or

confidence• Difficult to build and validate properly - too

many choices, too little general guidance, misleading validation results

10

How Do Neural Networks Work?

• Inspired by the biological brain• Consist of small, but numerous, parallel

elements - weighted connections (synapses) and summing nodes (neurons)

• “Learn” relationships through repeated calculations called “training”

• Remain fixed after training to be applied to new data

11

Biological Neuron

12

How are Signals Transmitted?

13

Elements of Neural Networksw1

w2

wn

x2

xn

yy = f ( wixi )i...

Simple SummingNode (Neuron)

x1

1

wi

xi

y

0

0.5 Non-linearTransferFunction

14

Typical Neural Network

HiddenLayer

OutputLayer

InputLayer

Error Feedback

Weighted Synapses During Training

NeuralNetworkOutput

I

N

P

U

T

S

15

Terminology• Neurons / nodes / units / cells / processing elements

(PEs)• Transfer / activation function• Connections / links / synapses• Weights / bias (fixed input of 1)• Feedforward / feedback• Input / output vectors / patterns• Self organizing (unsupervised) / supervised• Training / testing data sets

16

Biological vs Artificial Neural Networks

• Biological neurons are all excitatory (positive) or inhibitory (negative) - ANN neurons can be mixed

• Biological neurons operate asynchronously - ANN neurons usually synchronize by layer

• Biological neurons transmit signals at varying rates but ANN use a single rate

17

Biological vs Artificial Neural Networks

• There are many specialized biological neurons - ANN neurons tend to be generic

• Biological neurons work through chemical / electrical transmission (“wet ware”)

• Biological neurons are much slower but there are many, many more of them (~ 1011 neurons with 104 synapses per neuron!)

18

Types of Neural Nets

• Supervised • Unsupervised• Associate• Optimization

19

Common Neural Net Applications

• Pattern classification / recall– medical– defense– manufacturing quality– machine vision / postal– speech recognition– security detection– noise removal

20

Common Neural Net Applications

• Clustering / compression– data mining– signal processing– space exploration applications– speech recognition

21

Common Neural Net Applications

• Prediction / simulation– financial / stock market– music composition– utility usage– fault / degradation detection– sunspots

22

Common Neural Net Applications

• Control - real time / on line– robots– vehicles– manufacturing

• Control - off line– batch manufacturing– process optimization

23

Common Neural Net Applications

• Optimization– traveling salesman– routing– scheduling– facility location

24

Cool Neural Net Web Sites

http://www.csse.monash.edu.au/~app/CSE5301/index.html

http://www.geocities.com/CapeCanaveral/1624/

Detailed class notes and some matlab code.

C source code for lots of neural nets.