Internet EngineeringJacek Mazurkiewicz, PhD

Softcomputing

Part 1: Introduction, Elementary ANNs

Formal Introduction• contact hours, room No. 225 building C-3:

Monday: 13:00 – 15:00Thursday: 11:00 – 13:00

• slides: www.zsk.ict.pwr.wroc.pl

• „Professor Wiktor Zin”

• test: 27.01.2020 during lecture time

- softcomputing:

- lecture + project

- project mark – 20% of final mark, bonus question!

Softcomputing Lecture Schedule• Lecture 01 – 07th October 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 02 – 21st October 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 03 – 28th October 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 04 – 04th November 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 05 – 05th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3

• Lecture 06 – 12th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3

• Lecture 07 – 13th November 2019 – Wednesday, 11:15 - 13:00, room 21 C-3

• Lecture 08 – 18th November 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 09 – 19th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3

• Lecture 10 – 25th November 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 11 – 26th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3

• Lecture 12 – 02nd December 2019 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 13 – 03rd December 2019 – Tuesday, 11:15 - 13:00, room 21 C-3

• Lecture 14 – 13th January 2020 – Monday, 11:15 - 13:00, room 21 C-3

• Lecture 15 – 27th January 2020 – Monday, 11:15 - 13:00, room 21 C-3

Program• Idea of intelligent processing

• Fuzzy sets and approximate reasoning

• Expert systems - knowledge base organization

• Expert systems - reasoning rules creation

• Expert systems: typical organization and applications

• Artificial neural networks: learning and retrieving algorithms

• Multilayer percetpron, RBF

• Kohonen neural network, CNN

• Hopfield neural network

• Hamming neural network

• Artificial neural networks: applications

• Genetic algorithms: description and classification

• Genetic algorithms: basic mechanisms and solutions

SUBJECT OBJECTIVES

C1. Knowledge of artificial neural networks in pattern recognition, digital signals

and data processing: topology of networks, influence of parameters for network behavior.

C2. Knowledge of genetic algorithms used for data pre- and postprocessing.

C3. Knowledge of expert systems – reasoning rules and knowledge base creation for different tasks.

C4. Skills of special environment usage for project phase, modeling and simulation

of softcomputing systems in case of different scientific problems.

SUBJECT EDUCATIONAL EFFECTS

relating to knowledge:

PEK_W01 – knows the rules and the idea of intelligent processing.

PEK_W02 – defines the fuzzy sets and understands the idea of approximate reasoning.

PEK_W03 – defines the knowledge base and reasoning rules, knows the expert systems construction.

PEK_W04 – knows the architecture of typical artificial neural networks structures, learning and retrieving algorithms,

applications.

PEK_W05 – knows the description, classification, examples of applications of genetic algorithms

relating to skills:

PEK_U01 – can use the environments for project phase, modeling and simulation of artificial neural networks

as well as genetic algorithms in different tasks about pattern digital signals recognition.

PEK_U02 – can use the environments for project phase, modeling and implementation of expert systems

to dedicated fields of knowledge.

PEK_U03 – can use the environments for project phase, modeling and implementation of fuzzy sets and fuzzy reasoning

to dedicated fields of knowledge.

Literature

• B. Bouchon Meunier, Fuzzy Logic and Soft Computing

• O. Castilo, A. Bonarini, Soft Computing Applications

• M. Caudill, Ch. Butler, Understanding Neural Networks

• E. Damiani, Soft Computing in Software Engineering

• R. Hecht-Nielsen, Neurocomputing

• S. Y. Kung, Digital Neural Networks

• D. K. Pratihar, Soft Computing

• S. N. Sivanandam, S. N. Deepa, Principles of Soft Computing

• A. K. Srivastava, Soft Computing

• D. A. Waterman, A Guide to Expert Systems

• D. Zhang, Parallel VLSI Neural System Design

Why Neural Networks and Company?

Still in active use

No chance to solve some problems in other way

Human ability vs. classical programs

Works as primitive human’s brain

Artificial intelligence has power!

ANN + Fuzzy Logic + Expert Systems + Rough Sets + Ant Algorithms

= SoftComputing

The Story1943 – McCulloch & Pitts

– model of artificial neuron

1949 – Hebb– information stored by biological neural nets

1958 – Rosenblatt– perceptron model

1960 – Widrow & Hoff– first neurocomputer - Madaline

1969 – Minsky & Papert– XOR problem – single-layer perceptron limitations

1986 – McCleland & Rumelhart– backpropagation algorithm

Where Softcomputing is in Use?

Letters, signs, characters, digits recognition

Recognition of ship types – data from sonar

Electric power prediction

Different kinds of simulators and computer games

Engine diagnostic – in planes, vehicles

Rock-type identification

Bomb searching devices

Neural Networks Realisation

Set of connected identical neurons

Artificial neuron based on a biological neuron

Hardware realisation – digital device

Software realisation – simulators

Artificial neural network – idea, algorithm, mathematical formulas

Works in parallel

No programming – learning process necessary

Learning

With a Teacher

Without a Teacher

Klasyfikator

Wektor cech

(dane nauki)

Wynik

klasyfikacji

NauczycielTeacher

Learningvector

Parameters Weights

Result oflearning

Klasyfikator

Wektor cech

(dane testowe)

Wynik

klasyfikacjiLearning

vectorResult oflearning

Parameters Weights

Softcomputing vs. Classical Computer

Different limitations of softcomputing methods

No softcomputing:

– operations based on symbols: editors, algebraic equations

– calculations with a high level of precision

Softcomputing is very nice, but not as universal as computer

Anatomy Foundations (1)Nervous System – 2-ways, symmetrical

set of structures, divided into 4 parts:

Spinal Cord– receiving and transmission of data

Prolonged Cord– breathing, blood system, digestion

Cerebellum– movement control

Brain (ca. 1.3 kg) – 2 hemispheres– feeling, thinking, movement

brain

brain sterncerebellum

prolonged cord

spinal cord

nervous system

Anatomy Foundations (2)

Anatomy Foundations (3)

Cerebral cortex – thickness: 2 mm, area: ca. 1.5 m2

Cerebral cortex divided into 4 part – lobes

Each lobe is corrugated

Each hemisphere is responsible for half part of body:right for left part, left for right part

Hemispheres are identical in case of a structure, buttheir functions are different

Anatomy Foundations (4)

Brain composed by fibres with large number of branches

Two types of cells in nervous tissue: neurons and gley cells

There are more gley cells:– no data transfer among neurons– catering functions

Ca. 20 milliard neurons in cerebral cortex

Ca. 100 milliard neurons in whole brain

Neuron: dendrites – inputs, axon – output, body of neuron

Neuron: thousands of synapses – connections to other neurons

Anatomy Foundations (5)

Neurons in work:

• chemical-electrical signal transferring

• cell generates electrical signals

• electric pulse is changed into a chemical signal at the end of axon

• chemical info passed by neurotransmitters

• 50 different types of neurons

• neurons driven by a frequency of hundreds of Hz

• neurons are rather low devices!

Anatomy Foundations (6)

Biological and Artificial Neural Nets

Artificial neural networks are a good solution for:– testing already identified biological systems– pattern recognition– alternative configurations to find the basic features of them

Artificial neural networks are primitive brothers of biological nets

Biological nets have sophisticated internal features important for their normal work

Biological nets have sophisticated time dependences ignored in most artificial networks

Biological connections among neurons are different and complicated

Most architectures of artificial nets are unrealistic from the biology point of view

Most learning rules for artificial networks are unreal in biology point of view

Most biological nets we can compare to already learned artificial nets to realise function described in a very detailed way

Linear ANN - ADALINE (ADAive Linear Neuron)single neuron’s answer:

+...

x

x

x

1

2

M

w

w

w

1

2

M

w0

y

1

=

+=M

j

jj wxwy1

0

M – number of input neurons

K – number of output neurons

=

==M

j

jj xwy0

~)~( xwxT

),...,,(~10 Mxxxcol=x

10 =x

),...,,( 10 Mwwwcol=w

scalar description vector description

multi-output net:

Single-Layer Multi-Output Network

1 x x x1 2 M

y y y1 2 K

w w w10 20 K0

w w w11 12 1K

w w w21 22 2Kw w wM1 M2 MK

Wkj

Outputneuron

Inputneuron

k-neuron’s answer:

=

=M

j

jkjK xwy0

)(x

column= xwy(x)T WXy(X) =

=

KMKK

M

M

www

www

www

10

22120

11110

W

Learning Procedureexperimental data: N - series

Nxxx ,...,, 21

N

KKK ttt ,...,, 21

– learning data

– required answers

,N

K

N tx → – function implemented by net

error function – mean-square error:

( )( )= =

−=N

n

K

k

n

kk tyWE1 1

2

2

1)( w

= = =

−=

N

n

K

k

M

j

n

k

n

jjk txwWE1 1

2

02

1)(

looking for a minimum of E(W) function:

0)(

,

=

kjjk w

WE

Pseudoinverse Algorithm

= =

=

−=

N

n

nj

M

j

nk

njkj

kj

xtxww

WE

1 0'' 02

2

1)( = = =

= N

n

M

j

N

n

nj

nk

nj

njkj xtxxw

jk 1 0' 1'

,

where:

=

N

M

N

M

M

xx

xx

xx

1

22

1

11

1

1

1

1

X

=

N

K

NN

K

K

ttt

ttt

ttt

21

22

2

2

1

11

2

1

1

T

=

KMKK

M

M

www

www

www

10

22120

11110

W

finally:

( ) TXWXXTTT = TXWT = TXX)(XW T1TT −=

rsepseudoinve−

= −

τ

T,XWτT

Gradient-Type Algorithms (1)y

x

iterative approach:

xx

yxx

−=+1

steps:

– random weight vector

– new weight vector following: -w E

– repeat process generating the sequence of weights vectors:

– components of weight vectors calculated by

)(w

w

w

kj

kjkjw

Eww

−=

+ )()1(

error function: =n

nEE )()( ww = =

−=

K

k

M

j

n

k

n

jkj

n txwE1

2

02

1)(w

Gradient-Type Algorithms (2)sequential approach:

kj

n

kjkjw

Eww

−=

+

)1( n

j

n

k

n

j

n

k

n

k

kj

n

xxtyw

E**)( =−=

x

error – delta rule: Widrow-Hoff rule:

n

k

n

k

n

k ty −= )(xn

j

n

kjkkj xww −=

+ )1(

algorithm: 1. set start values – by a random way for example

2. calculate a net answer for available xn

3. calculate an error value kn

4. calculate a new weight vector wkj(+1) according to the delta rule

5. repeat steps 2. – 4. until E less than required value

Perceptron (1)x1

x2

xM

w0

wiM

wi2

wi1

yi

1

the story:

– Rosenblatt (1962)

– classification task

– Widrow & Hoff (1960) - ADALINE

answer: )()(0

xwTgxwgXy

M

j

jj =

=

=

w0 – threshold value

activation function:

−=

01

01)(

afor

aforag

=

00

01)(

afor

aforag

bipolar unipolar

=

=M

j

jj xwa0

Perceptron (2)error function:

jkw

E

– does not exist, because g(a) is not differentiable

perceptron criterions:

compare the actual value of yi and the required output value di and:

– if yi = di the weights values of Wij and w0 are unchanged

– if yi = 0 and the required value di =1 update the weights as follow:

where: t – previous cycle, t+1 – actual cycle

– if yi = 1 and di = 0 update the weights according to:

where: bi – polarity, di – required neuron’s output signal

,)()1( jijij xtWtW +=+ ,1)()1( +=+ tbtb ii

,)()1( jijij xtWtW −=+ 1)()1( −=+ tbtb ii

Perceptron (3)

=

−=p

k

k

i

k

i dyE1

2)()( ,)(

summarising:

1. look-up the input learning vectors

2. if classification is correct weights are not changed

3. if classification is wrong:

– if tn = +1 add xn to the weight values

– else subtract xn from the weight values

– value of is not important – can be set to 1, it only scales wi

Perceptron – Problems (1)

linear separability: XOR problem – Minsky & Papert (1969):

y(x)=0C2

X1

X2

C1

In1 In2

Out

XOR

0 0 00 1 11 0 11 1 0

In In Out1 2

– non-linear separable problem

– solution: multilayer net

Perceptron – Problems (2)

multilayer network for XOR problem solution:

w ww w

-2w w

= w =− w

=w

s1 s2

S