A Brief History of the Development of ArtificialNeural Networks
Prof. Bernard Widrow
Department of Electrical Engineering
Stanford University
Baidu • July 18, 2018
Prof. Widrow @ Berkeley A Brief History of Neural Networks 1 / 54
Adaptive linear neuron (Adaline)
Figure 1: Adaline. An adaptive linear neuron
Widrow talk The Hebbian-LMS Algorithm
LMS algorithm
𝑊𝑘+1 = 𝑊𝑘 + 2𝜇𝜖𝑘𝑋𝑘𝜖𝑘 = 𝑑𝑘 − 𝑋𝑘𝑇𝑊𝑘
Prof. Widrow @ Berkeley A Brief History of Neural Networks 2 / 54
Knobby Adaline, 1959
FIGURE 2 Knobby adaline
Widrow talk The Hebbian-LMS Algorithm
Prof. Widrow @ Berkeley A Brief History of Neural Networks 3 / 54
Prof. Widrow @ Berkeley A Brief History of Neural Networks 4 / 54
An adaptive linear neuron(ADALINE)
Prof. Widrow @ Berkeley A Brief History of Neural Networks 5 / 54
A Two-Input Adaline
Prof. Widrow @ Berkeley A Brief History of Neural Networks 6 / 54
Separating line in pattern space
Prof. Widrow @ Berkeley A Brief History of Neural Networks 7 / 54
Adaline Capacity
Prof. Widrow @ Berkeley A Brief History of Neural Networks 8 / 54
Adaline with polynomial preprocessor
Prof. Widrow @ Berkeley A Brief History of Neural Networks 9 / 54
An elliptical separating boundary for the Exclusive
NOR Function
Prof. Widrow @ Berkeley A Brief History of Neural Networks 10 / 54
A two-Adaline form of Madaline I
Prof. Widrow @ Berkeley A Brief History of Neural Networks 11 / 54
Separating lines for the two-element Madaline
Prof. Widrow @ Berkeley A Brief History of Neural Networks 12 / 54
A neuronal implementation of AND, OR, and MAJ
logic function
Prof. Widrow @ Berkeley A Brief History of Neural Networks 13 / 54
A three-layer adaptive neural network
Prof. Widrow @ Berkeley A Brief History of Neural Networks 14 / 54
Principal of Minimal Disturbance
Prof. Widrow @ Berkeley A Brief History of Neural Networks 15 / 54
Error-Correction Algorithms for the Single Element
Prof. Widrow @ Berkeley A Brief History of Neural Networks 16 / 54
α-LMS Algorithm
Prof. Widrow @ Berkeley A Brief History of Neural Networks 17 / 54
Weight correction by the LMS rule
Prof. Widrow @ Berkeley A Brief History of Neural Networks 18 / 54
Rosenblatt’s Perceptron
Prof. Widrow @ Berkeley A Brief History of Neural Networks 19 / 54
Adaptive Threshold Element in the Perceptron
Prof. Widrow @ Berkeley A Brief History of Neural Networks 20 / 54
Perceptron Rule
Prof. Widrow @ Berkeley A Brief History of Neural Networks 21 / 54
May’s Rule
Prof. Widrow @ Berkeley A Brief History of Neural Networks 22 / 54
May’s Increment Adaptation Rule
Prof. Widrow @ Berkeley A Brief History of Neural Networks 23 / 54
May’s Modified Relaxation Rule
Prof. Widrow @ Berkeley A Brief History of Neural Networks 24 / 54
Error-Correction Algorithms for Multi-Element
Networks
Prof. Widrow @ Berkeley A Brief History of Neural Networks 25 / 54
A five-Adaline example of the MR I Architecture
Prof. Widrow @ Berkeley A Brief History of Neural Networks 26 / 54
MR II of B. Widrow and R. Winter
Prof. Widrow @ Berkeley A Brief History of Neural Networks 27 / 54
Steepest-Descent Algorithms for the Single Element
Prof. Widrow @ Berkeley A Brief History of Neural Networks 28 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 29 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 30 / 54
Conventional LMS
Prof. Widrow @ Berkeley A Brief History of Neural Networks 31 / 54
Implementation of Conventional LMS
Prof. Widrow @ Berkeley A Brief History of Neural Networks 32 / 54
“Sigmoid” LMS(Back-Prop)
Prof. Widrow @ Berkeley A Brief History of Neural Networks 33 / 54
Implementation of “Sigmoid” LMS(Back-Prop)
Prof. Widrow @ Berkeley A Brief History of Neural Networks 34 / 54
“Sigmoid” LMS(MR III)
Prof. Widrow @ Berkeley A Brief History of Neural Networks 35 / 54
Implementation of Sigmoid LMS-MRIII
Prof. Widrow @ Berkeley A Brief History of Neural Networks 36 / 54
Error Surfaces for the Single Element
Prof. Widrow @ Berkeley A Brief History of Neural Networks 37 / 54
Linear MSE, Sigmoid MSE, and Signum MSE
Prof. Widrow @ Berkeley A Brief History of Neural Networks 38 / 54
Linear Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 39 / 54
Sigmoid Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 40 / 54
Signum Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 41 / 54
Linear Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 42 / 54
Sigmoid Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 43 / 54
Signum Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 44 / 54
Steepest-Descent Algorithms for Multi-Element
Networks
Prof. Widrow @ Berkeley A Brief History of Neural Networks 45 / 54
Backpropagation Network
Prof. Widrow @ Berkeley A Brief History of Neural Networks 46 / 54
Detail of Backpropagation Node
Prof. Widrow @ Berkeley A Brief History of Neural Networks 47 / 54
MRIII of David Andes
Prof. Widrow @ Berkeley A Brief History of Neural Networks 48 / 54
Prof. Widrow @ Berkeley A Brief History of Neural Networks 49 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 50 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 51 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 52 / 54
Mean Square Error Surface
Prof. Widrow @ Berkeley A Brief History of Neural Networks 53 / 54
Learning Rules
Prof. Widrow @ Berkeley A Brief History of Neural Networks 54 / 54