Upload
ahmed-gad
View
273
Download
1
Embed Size (px)
Citation preview
Brief Introduction to Deep Learning + Solving XOR using ANN
MENOUFIA UNIVERSITYFACULTY OF COMPUTERS AND INFORMATION
جامعة المنوفية
والمعلوماتكلية الحاسبات
جامعة المنوفية
Ahmed Fawzy Gad
Classification Example
BA
011
10
000
11
Neural Networks
Input Hidden Output
BA
011
10
000
11
Neural Networks
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
=
+0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Network Architecture!!
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
=
+0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
Input OutputHidden
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
Input OutputHidden
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
Input OutputHidden
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
A
B
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
A
B
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011 10
000 11 Input Output
A
B
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
A
B
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
BA
011
10
000
11 Input Output
A
B
1/0
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
Input Output
A
B
1/0
𝑾𝟏
𝑾𝟐
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
=
+0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
Input Output
A
B
1/0
𝑾𝟑
𝑾𝟒
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
A
B
1/0
𝑾𝟑
𝑾𝟒
A
B
1/0
𝑾𝟏
𝑾𝟐
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
0 0.5 1 1.5
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
A
B
𝑾𝟑
𝑾𝟒
A
B
𝑾𝟏
𝑾𝟐
A
B
𝑾𝟑
𝑾𝟒
𝑾𝟓
𝑾𝟔
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
A
B
𝑾𝟑
𝑾𝟒
𝑾𝟓
𝑾𝟔
A 𝑾𝟏
𝑾𝟐
B
𝑾𝟑
𝑾𝟒
𝑾𝟓
𝑾𝟔
A 𝑾𝟏
𝑾𝟐
B
𝑾𝟑
𝑾𝟒
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Input Output
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
Weights=𝑾𝒊
𝑾𝟓
𝑾𝟔
1/0
𝒀𝒋
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation Function
Output
𝒀𝒋
BA
011
10
000
11
Input Hidden
1/0
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation Function
Output
𝒀𝒋
BA
011
10
000
11
Input Hidden
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation Function
Output
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation FunctionComponents
Output
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation FunctionInputs
Output
s
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation FunctionInputs
Output
s
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
s=SOP(𝑿𝒊,𝑾𝒊)
Activation FunctionInputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
s=SOP(𝑿𝒊,𝑾𝒊)
Activation FunctionInputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝑿𝟏
𝑿𝟐
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
s=SOP(𝑿𝒊,𝑾𝒊)
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation FunctionInputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝑿𝟏
𝑿𝟐
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
S= 𝟏𝒎𝑿𝒊𝑾𝒊
s=SOP(𝑿𝒊,𝑾𝒊)
1/0
Activation FunctionInputs
Output
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
S= 𝟏𝒎𝑿𝒊𝑾𝒊
1/0
Each Hidden/Output Layer Neuron has its SOP.
Activation FunctionInputs
Output
s
𝑿𝟏
𝑿𝟐
𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
S= 𝟏𝒎𝑿𝒊𝑾𝒊
1/0
Activation FunctionInputs
Output
s
𝑿𝟏
𝑿𝟐
𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
S= 𝟏𝒎𝑿𝒊𝑾𝒊
1/0
Activation FunctionInputs
Output
s
𝑿𝟏
𝑿𝟐
𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
S= 𝟏𝒎𝑿𝒊𝑾𝒊
1/0
Activation FunctionOutputs
Output
F(s)s
𝑿𝟏
𝑿𝟐
Class Label
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation FunctionOutputs
Output
F(s)s
𝑿𝟏
𝑿𝟐
Class Label
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Activation Functions
Piecewise Linear Sigmoid Binary
Activation Functions
Which activation function to use?
OutputsClass
Labels
Activation Function
TWO Class Labels
TWOOutputs
One that gives two outputs.Which activation function to use?
𝑪𝒋𝒀𝒋
BA
011
10
000
11
BA
011 10
000 11
Activation Functions
Piecewise Linear Sigmoid BinaryBinary
Activation Function
Output
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
BA
011
10
000
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
1/0
Input Hidden
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Bias
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
BiasHidden Layer Neurons
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
+1
𝒃𝟏
+1𝒃𝟐
1/0
BiasOutput Layer Neurons
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟑1/0
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
All Bias Values
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
1/0
+1
𝒃𝟑
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
BiasAdd Bias to SOP
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)
𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)
𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)
1/0
+1
𝒃𝟑
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
BiasAdd Bias to SOP
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
𝑺𝟏=(𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)
1/0
+1
𝒃𝟑
𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
BiasAdd Bias to SOP
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
𝑺𝟐=(𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)
1/0
+1
𝒃𝟑
𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
BiasAdd Bias to SOP
Input Output
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
𝑺𝟑=(𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)
1/0
+1
𝒃𝟑
𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Learning Rate
𝟎 ≤ η ≤ 𝟏
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
1/0
+1
𝒃𝟑
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
Other ParametersStep n
𝒏 = 𝟎, 𝟏, 𝟐, …
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
1/0
+1
𝒃𝟑
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
s=(𝑿𝟎𝑾𝟎+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿𝟎, 𝑿𝟏,𝑿𝟐, …)
W(𝒏)=(𝑾𝟎,𝑾𝟏,𝑾𝟐, …)
Other ParametersDesired Output 𝒅𝒋
𝒏 = 𝟎, 𝟏, 𝟐, …
𝒅 𝒏 = 𝟏, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟏 (𝟏)
𝟎, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟐 (𝟎)
BA
011
10
000
11
F(s)s
𝑿𝟏
𝑿𝟐
bin
𝒀𝒋
+1
𝒃𝟏
+1𝒃𝟐
1/0
+1
𝒃𝟑
𝑾𝟓
𝑾𝟔
A
B
𝑾𝟏
𝑾𝟐
𝑾𝟑
𝑾𝟒
s=(𝑿𝟎𝑾𝟎+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿𝟎, 𝑿𝟏,𝑿𝟐, …)
W(𝒏)=(𝑾𝟎,𝑾𝟏,𝑾𝟐, …)
Neural Networks Training Steps
Weights Initialization
Inputs Application
Sum of Inputs-Weights Products
Activation Function Response Calculation
Weights Adaptation
Back to Step 2
1
2
3
4
5
6
Regarding 5th Step: Weights Adaptation
• If the predicted output Y is not the same as the desired output d,then weights are to be adapted according to the following equation:
𝑾 𝒏+ 𝟏 = 𝑾 𝒏 + η 𝒅 𝒏 − 𝒀 𝒏 𝑿(𝒏)
Where𝑾 𝒏 = [𝒃 𝒏 ,𝑾𝟏(𝒏),𝑾𝟐(𝒏),𝑾𝟑(𝒏), … ,𝑾𝒎(𝒏)]
Neural NetworksTraining ExampleStep n=0• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=0:η = .001
𝑋 𝑛 = 𝑋 0 = +1,+1,+1,1, 0𝑊 𝑛 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 0 = 1
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=0
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟏
𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+1*1+0*1
=-.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟏
𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟐
𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+1*1+0*1
=.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟐
𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 – SOP – 𝑺𝟑
𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+1*1
=.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 – Output – 𝑺𝟑
𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0 - Output
𝒀 𝒏 = 𝒀 𝟎 = 𝒀 𝑺3= 1
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=0Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟎 = 1𝐝 𝒏 = 𝒅 𝟎 = 1
∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.
No Adaptation
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=1:η = .001
𝑋 𝑛 = 𝑋 1 = +1,+1,+1,0, 1𝑊 𝑛 = 𝑊 1 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 1 = +1
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=1
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟏
𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+0*1+1*1
=-.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟏
𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟐
𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+0*1+1*1
=.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟐
𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 – SOP – 𝑺𝟑
𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+1*1
=.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 – Output – 𝑺𝟑
𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1 - Output
𝒀 𝒏 = 𝒀 𝟏 = 𝒀 𝑺3= 1
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=1Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟏 = 1𝐝 𝒏 = 𝒅 𝟏 = 1
∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.
No Adaptation
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−2
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=2:η = .001
𝑋 𝑛 = 𝑋 2 = +1,+1,+1,0, 0𝑊 𝑛 = 𝑊 2 = 𝑊 1 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 2 = 0
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=2
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟏
𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+0*1+0*1
=-1.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟏
𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏
= 𝑩𝑰𝑵 −𝟏. 𝟓= 𝟎
𝒃𝒊n 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟐
𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+0*1+0*1
=-.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟐
𝒀 𝑺2 == 𝑺𝑮𝑵 𝑺2= 𝑺𝑮𝑵 −. 𝟓
=0
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 – SOP – 𝑺𝟑
𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+0*-2+0*1
=-.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 – Output – 𝑺𝟑
𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2 - Output
𝒀 𝒏 = 𝒀 𝟐 = 𝒀 𝑺3= 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=2Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟐 = 𝟎𝐝 𝒏 = 𝒅 𝟐 = 𝟎
∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.
No Adaptation
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=3:η = .001
𝑋 𝑛 = 𝑋 3 = +1,+1,+1,1, 1𝑊 𝑛 = 𝑊 3 = 𝑊 2 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1𝑑 𝑛 = 𝑑 3 = 0
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=3
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
BA
011 => 1
10
000 => 0
11
Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟏
𝑺𝟏=(+𝟏𝒃𝟏+𝑿𝟏𝑾𝟏+𝑿𝟐𝑾𝟑)=+1*-1.5+1*1+1*1
=.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟏
𝒀 𝑺𝟏 == 𝑩𝑰𝑵 𝑺𝟏= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟐
𝑺𝟐=(+𝟏𝒃𝟐+𝑿𝟏𝑾𝟐+𝑿𝟐𝑾𝟒)=+1*-.5+1*1+1*1
=1.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟐
𝒀 𝑺2 == 𝑩𝑰𝑵 𝑺2= 𝑩𝑰𝑵 𝟏. 𝟓
= 1
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎−𝟏, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 – SOP – 𝑺𝟑
𝑺𝟑=(+𝟏𝒃𝟑+𝑺𝟏𝑾𝟓+𝑺𝟐𝑾𝟔)=+1*-.5+1*-2+1*1
=-1.5
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 – Output – 𝑺𝟑
𝒀 𝑺3 == 𝑩𝑰𝑵 𝑺3
= 𝑩𝑰𝑵 −𝟏. 𝟓= 𝟎
𝒃𝒊𝒏 𝒔 = +𝟏, 𝒔 ≥ 𝟎𝟎, 𝒔 < 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3 - Output
𝒀 𝒏 = 𝒀 𝟑 = 𝒀 𝑺3= 𝟎
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Neural NetworksTraining ExampleStep n=3Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟑 = 𝟎𝐝 𝒏 = 𝒅 𝟑 = 𝟎
∵𝒀 𝒏 = 𝒅 𝒏∴ Weights are Correct.
No Adaptation
BA
011 => 1
10
000 => 0
11
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Final Weights
s
𝑿𝟏
𝑿𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1−. 𝟓
1/0
+1
−. 𝟓−𝟐
+𝟏
A
B
+𝟏+𝟏
+𝟏+𝟏
bin
Current weights predictedthe desired outputs.