A Brief History of the Development of ArtificialNeural Networks

Prof. Bernard Widrow

Department of Electrical Engineering

Stanford University

Baidu • July 18, 2018

Prof. Widrow @ Berkeley A Brief History of Neural Networks 1 / 54

Adaptive linear neuron (Adaline)

Figure 1: Adaline. An adaptive linear neuron

Widrow talk The Hebbian-LMS Algorithm

LMS algorithm

𝑊𝑘+1 = 𝑊𝑘 + 2𝜇𝜖𝑘𝑋𝑘𝜖𝑘 = 𝑑𝑘 − 𝑋𝑘𝑇𝑊𝑘

Prof. Widrow @ Berkeley A Brief History of Neural Networks 2 / 54

Knobby Adaline, 1959

FIGURE 2 Knobby adaline

Widrow talk The Hebbian-LMS Algorithm

Prof. Widrow @ Berkeley A Brief History of Neural Networks 3 / 54

Prof. Widrow @ Berkeley A Brief History of Neural Networks 4 / 54

An adaptive linear neuron(ADALINE)

Prof. Widrow @ Berkeley A Brief History of Neural Networks 5 / 54

A Two-Input Adaline

Prof. Widrow @ Berkeley A Brief History of Neural Networks 6 / 54

Separating line in pattern space

Prof. Widrow @ Berkeley A Brief History of Neural Networks 7 / 54

Adaline Capacity

Prof. Widrow @ Berkeley A Brief History of Neural Networks 8 / 54

Adaline with polynomial preprocessor

Prof. Widrow @ Berkeley A Brief History of Neural Networks 9 / 54

An elliptical separating boundary for the Exclusive

NOR Function

Prof. Widrow @ Berkeley A Brief History of Neural Networks 10 / 54

A two-Adaline form of Madaline I

Prof. Widrow @ Berkeley A Brief History of Neural Networks 11 / 54

Separating lines for the two-element Madaline

Prof. Widrow @ Berkeley A Brief History of Neural Networks 12 / 54

A neuronal implementation of AND, OR, and MAJ

logic function

Prof. Widrow @ Berkeley A Brief History of Neural Networks 13 / 54

A three-layer adaptive neural network

Prof. Widrow @ Berkeley A Brief History of Neural Networks 14 / 54

Principal of Minimal Disturbance

Prof. Widrow @ Berkeley A Brief History of Neural Networks 15 / 54

Error-Correction Algorithms for the Single Element

Prof. Widrow @ Berkeley A Brief History of Neural Networks 16 / 54

α-LMS Algorithm

Prof. Widrow @ Berkeley A Brief History of Neural Networks 17 / 54

Weight correction by the LMS rule

Prof. Widrow @ Berkeley A Brief History of Neural Networks 18 / 54

Rosenblatt’s Perceptron

Prof. Widrow @ Berkeley A Brief History of Neural Networks 19 / 54

Adaptive Threshold Element in the Perceptron

Prof. Widrow @ Berkeley A Brief History of Neural Networks 20 / 54

Perceptron Rule

Prof. Widrow @ Berkeley A Brief History of Neural Networks 21 / 54

May’s Rule

Prof. Widrow @ Berkeley A Brief History of Neural Networks 22 / 54

May’s Increment Adaptation Rule

Prof. Widrow @ Berkeley A Brief History of Neural Networks 23 / 54

May’s Modified Relaxation Rule

Prof. Widrow @ Berkeley A Brief History of Neural Networks 24 / 54

Error-Correction Algorithms for Multi-Element

Networks

Prof. Widrow @ Berkeley A Brief History of Neural Networks 25 / 54

A five-Adaline example of the MR I Architecture

Prof. Widrow @ Berkeley A Brief History of Neural Networks 26 / 54

MR II of B. Widrow and R. Winter

Prof. Widrow @ Berkeley A Brief History of Neural Networks 27 / 54

Steepest-Descent Algorithms for the Single Element

Prof. Widrow @ Berkeley A Brief History of Neural Networks 28 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 29 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 30 / 54

Conventional LMS

Prof. Widrow @ Berkeley A Brief History of Neural Networks 31 / 54

Implementation of Conventional LMS

Prof. Widrow @ Berkeley A Brief History of Neural Networks 32 / 54

“Sigmoid” LMS(Back-Prop)

Prof. Widrow @ Berkeley A Brief History of Neural Networks 33 / 54

Implementation of “Sigmoid” LMS(Back-Prop)

Prof. Widrow @ Berkeley A Brief History of Neural Networks 34 / 54

“Sigmoid” LMS(MR III)

Prof. Widrow @ Berkeley A Brief History of Neural Networks 35 / 54

Implementation of Sigmoid LMS-MRIII

Prof. Widrow @ Berkeley A Brief History of Neural Networks 36 / 54

Error Surfaces for the Single Element

Prof. Widrow @ Berkeley A Brief History of Neural Networks 37 / 54

Linear MSE, Sigmoid MSE, and Signum MSE

Prof. Widrow @ Berkeley A Brief History of Neural Networks 38 / 54

Linear Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 39 / 54

Sigmoid Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 40 / 54

Signum Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 41 / 54

Linear Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 42 / 54

Sigmoid Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 43 / 54

Signum Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 44 / 54

Steepest-Descent Algorithms for Multi-Element

Networks

Prof. Widrow @ Berkeley A Brief History of Neural Networks 45 / 54

Backpropagation Network

Prof. Widrow @ Berkeley A Brief History of Neural Networks 46 / 54

Detail of Backpropagation Node

Prof. Widrow @ Berkeley A Brief History of Neural Networks 47 / 54

MRIII of David Andes

Prof. Widrow @ Berkeley A Brief History of Neural Networks 48 / 54

Prof. Widrow @ Berkeley A Brief History of Neural Networks 49 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 50 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 51 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 52 / 54

Mean Square Error Surface

Prof. Widrow @ Berkeley A Brief History of Neural Networks 53 / 54

Learning Rules

Prof. Widrow @ Berkeley A Brief History of Neural Networks 54 / 54