Brief Introduction to Deep Learning + Solving XOR using ANNs

Brief Introduction to Deep Learning + Solving XOR using ANN

MENOUFIA UNIVERSITYFACULTY OF COMPUTERS AND INFORMATION

جامعة المنوفية

والمعلوماتكلية الحاسبات

جامعة المنوفية

Ahmed Fawzy Gad

ahmed.fawzy@ci.menofia.edu.eg

Classification Example

BA

011

10

000

11

Neural Networks

Input Hidden Output

BA

011

10

000

11

Neural Networks

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

=

+0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Network Architecture!!

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

=

+0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

Input OutputHidden

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

Input OutputHidden

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

Input OutputHidden

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

A

B

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

A

B

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011 10

000 11 Input Output

A

B

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

A

B

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

BA

011

10

000

11 Input Output

A

B

1/0

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

Input Output

A

B

1/0

𝑾𝟏

𝑾𝟐

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

=

+0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

Input Output

A

B

1/0

𝑾𝟑

𝑾𝟒

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

A

B

1/0

𝑾𝟑

𝑾𝟒

A

B

1/0

𝑾𝟏

𝑾𝟐

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

A

B

𝑾𝟑

𝑾𝟒

A

B

𝑾𝟏

𝑾𝟐

A

B

𝑾𝟑

𝑾𝟒

𝑾𝟓

𝑾𝟔

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

A

B

𝑾𝟑

𝑾𝟒

𝑾𝟓

𝑾𝟔

A 𝑾𝟏

𝑾𝟐

B

𝑾𝟑

𝑾𝟒

𝑾𝟓

𝑾𝟔

A 𝑾𝟏

𝑾𝟐

B

𝑾𝟑

𝑾𝟒

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Input Output

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Hidden

Weights=𝑾𝒊

𝑾𝟓

𝑾𝟔

1/0

𝒀𝒋

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation Function

Output

𝒀𝒋

BA

011

10

000

11

Input Hidden

1/0

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation Function

Output

𝒀𝒋

BA

011

10

000

11

Input Hidden

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation Function

Output

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation FunctionComponents

Output

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation FunctionInputs

Output

s

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation FunctionInputs

Output

s

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

s=SOP(𝑿𝒊,𝑾𝒊)

Activation FunctionInputs

Output

s

𝑿𝒊=Inputs 𝑾𝒊=Weights

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

s=SOP(𝑿𝒊,𝑾𝒊)

Activation FunctionInputs

Output

s

𝑿𝒊=Inputs 𝑾𝒊=Weights

𝑿𝟏

𝑿𝟐

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

s=SOP(𝑿𝒊,𝑾𝒊)

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation FunctionInputs

Output

s

𝑿𝒊=Inputs 𝑾𝒊=Weights

𝑿𝟏

𝑿𝟐

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

S= 𝟏𝒎𝑿𝒊𝑾𝒊

s=SOP(𝑿𝒊,𝑾𝒊)

1/0

Activation FunctionInputs

Output

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

S= 𝟏𝒎𝑿𝒊𝑾𝒊

1/0

Each Hidden/Output Layer Neuron has its SOP.

Activation FunctionInputs

Output

s

𝑿𝟏

𝑿𝟐

𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

S= 𝟏𝒎𝑿𝒊𝑾𝒊

1/0

Activation FunctionInputs

Output

s

𝑿𝟏

𝑿𝟐

𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

S= 𝟏𝒎𝑿𝒊𝑾𝒊

1/0

Activation FunctionInputs

Output

s

𝑿𝟏

𝑿𝟐

𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

S= 𝟏𝒎𝑿𝒊𝑾𝒊

1/0

Activation FunctionOutputs

Output

F(s)s

𝑿𝟏

𝑿𝟐

Class Label

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation FunctionOutputs

Output

F(s)s

𝑿𝟏

𝑿𝟐

Class Label

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input Hidden

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Activation Functions

Piecewise Linear Sigmoid Binary

Activation Functions

Which activation function to use?

OutputsClass

Labels

Activation Function

TWO Class Labels

TWOOutputs

One that gives two outputs.Which activation function to use?

𝑪𝒋𝒀𝒋

BA

011

10

000

11

BA

011 10

000 11

Activation Functions

Piecewise Linear Sigmoid BinaryBinary

Activation Function

Output

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

BA

011

10

000

11

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

1/0

Input Hidden

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Bias

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

BiasHidden Layer Neurons

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

+1

𝒃𝟏

+1𝒃𝟐

1/0

BiasOutput Layer Neurons

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟑1/0

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

All Bias Values

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

1/0

+1

𝒃𝟑

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

BiasAdd Bias to SOP

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)

𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)

𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)

1/0

+1

𝒃𝟑

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

BiasAdd Bias to SOP

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)

1/0

+1

𝒃𝟑

𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

BiasAdd Bias to SOP

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)

1/0

+1

𝒃𝟑

𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

BiasAdd Bias to SOP

Input Output

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)

1/0

+1

𝒃𝟑

𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Learning Rate

𝟎 ≤ η ≤ 𝟏

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

1/0

+1

𝒃𝟑

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

Other ParametersStep n

𝒏 = 𝟎, 𝟏, 𝟐, …

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

1/0

+1

𝒃𝟑

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

s=(𝑿𝟎𝑾𝟎+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟐+…)

𝟎 ≤ η ≤ 𝟏

𝑿(𝒏)=(𝑿𝟎, 𝑿𝟏,𝑿𝟐, …)

W(𝒏)=(𝑾𝟎,𝑾𝟏,𝑾𝟐, …)

Other ParametersDesired Output 𝒅𝒋

𝒏 = 𝟎, 𝟏, 𝟐, …

𝒅 𝒏 = 𝟏, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟏 (𝟏)

𝟎, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟐 (𝟎)

BA

011

10

000

11

F(s)s

𝑿𝟏

𝑿𝟐

bin

𝒀𝒋

+1

𝒃𝟏

+1𝒃𝟐

1/0

+1

𝒃𝟑

𝑾𝟓

𝑾𝟔

A

B

𝑾𝟏

𝑾𝟐

𝑾𝟑

𝑾𝟒

s=(𝑿𝟎𝑾𝟎+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟐+…)

𝟎 ≤ η ≤ 𝟏

𝑿(𝒏)=(𝑿𝟎, 𝑿𝟏,𝑿𝟐, …)

W(𝒏)=(𝑾𝟎,𝑾𝟏,𝑾𝟐, …)

Neural Networks Training Steps

Weights Initialization

Inputs Application

Sum of Inputs-Weights Products

Activation Function Response Calculation

Weights Adaptation

Back to Step 2

1

2

3

4

5

6

Regarding 5th Step: Weights Adaptation

• If the predicted output Y is not the same as the desired output d,then weights are to be adapted according to the following equation:

𝑾 𝒏+ 𝟏 = 𝑾 𝒏 + η 𝒅 𝒏 − 𝒀 𝒏 𝑿(𝒏)

Where𝑾 𝒏 = [𝒃 𝒏 ,𝑾𝟏(𝒏),𝑾𝟐(𝒏),𝑾𝟑(𝒏), … ,𝑾𝒎(𝒏)]

Neural NetworksTraining ExampleStep n=0• In each step in the solution, the parameters of the neural network

must be known.

• Parameters of step n=0:η = .001

𝑋 𝑛 = 𝑋 0 = +1,+1,+1,1, 0𝑊 𝑛 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6

= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 0 = 1

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=0

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟏

𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+1*1+0*1

=-.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟏

𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 −. 𝟓

= 𝟎

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟐

𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+1*1+0*1

=.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟐

𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 . 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟑

𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+1*1

=.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟑

𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 . 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0 - Output

𝒀 𝒏 = 𝒀 𝟎 = 𝒀 𝑺3= 1

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=0Predicted Vs. Desired

𝒀 𝒏 = 𝒀 𝟎 = 1𝐝 𝒏 = 𝒅 𝟎 = 1

∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.

No Adaptation

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1• In each step in the solution, the parameters of the neural network

must be known.

• Parameters of step n=1:η = .001

𝑋 𝑛 = 𝑋 1 = +1,+1,+1,0, 1𝑊 𝑛 = 𝑊 1 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6

= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 1 = +1

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=1

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟏

𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+0*1+1*1

=-.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟏

𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 −. 𝟓

= 𝟎

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟐

𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+0*1+1*1

=.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟐

𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 . 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟑

𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+1*1

=.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟑

𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 . 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1 - Output

𝒀 𝒏 = 𝒀 𝟏 = 𝒀 𝑺3= 1

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=1Predicted Vs. Desired

𝒀 𝒏 = 𝒀 𝟏 = 1𝐝 𝒏 = 𝒅 𝟏 = 1

∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.

No Adaptation

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−2

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2• In each step in the solution, the parameters of the neural network

must be known.

• Parameters of step n=2:η = .001

𝑋 𝑛 = 𝑋 2 = +1,+1,+1,0, 0𝑊 𝑛 = 𝑊 2 = 𝑊 1 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6

= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 2 = 0

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=2

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟏

𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+0*1+0*1

=-1.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟏

𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏

= 𝑩𝑰𝑵 −𝟏. 𝟓= 𝟎

𝒃𝒊n 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟐

𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+0*1+0*1

=-.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟐

𝒀 𝑺2 == 𝑺𝑮𝑵 𝑺2= 𝑺𝑮𝑵 −. 𝟓

=0

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟑

𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+0*1

=-.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟑

𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 −. 𝟓

= 𝟎

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2 - Output

𝒀 𝒏 = 𝒀 𝟐 = 𝒀 𝑺3= 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=2Predicted Vs. Desired

𝒀 𝒏 = 𝒀 𝟐 = 𝟎𝐝 𝒏 = 𝒅 𝟐 = 𝟎

∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.

No Adaptation

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3• In each step in the solution, the parameters of the neural network

must be known.

• Parameters of step n=3:η = .001

𝑋 𝑛 = 𝑋 3 = +1,+1,+1,1, 1𝑊 𝑛 = 𝑊 3 = 𝑊 2 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6

= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 3 = 0

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=3

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

BA

011 => 1

10

000 => 0

11

Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟏

𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+1*1+1*1

=.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟏

𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 . 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟐

𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+1*1+1*1

=1.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟐

𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 𝟏. 𝟓

= 1

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟑

𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+1*-2+1*1

=-1.5

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟑

𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3

= 𝑩𝑰𝑵 −𝟏. 𝟓= 𝟎

𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3 - Output

𝒀 𝒏 = 𝒀 𝟑 = 𝒀 𝑺3= 𝟎

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Neural NetworksTraining ExampleStep n=3Predicted Vs. Desired

𝒀 𝒏 = 𝒀 𝟑 = 𝟎𝐝 𝒏 = 𝒅 𝟑 = 𝟎

∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.

No Adaptation

BA

011 => 1

10

000 => 0

11

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Final Weights

s

𝑿𝟏

𝑿𝟐

𝒀𝒋

+1

−𝟏. 𝟓

+1−. 𝟓

1/0

+1

−. 𝟓−𝟐

+𝟏

A

B

+𝟏+𝟏

+𝟏+𝟏

bin

Current weights predictedthe desired outputs.