The History of Neural Networks: Unveiling the Legacy of Frank Rosenblatt's Perceptron

At a basic level the perceptron is a single-layer neural network. The perceptron is worth looking at because it sheds light on how individual neurons within a neural network function. If you know how a perceptron functions, you know how an artificial neuron functions.

A Perceptron's Structure and Function: From Input to Output

A perceptron consists of five components:

• Inputs
• Weights
• Weighted sum
• Linear/binary activation function
• Bias

Here's how a perceptron works:

1. Inputs are multiplied by weights. Weights enable the perceptron to assign more importance to some inputs than others.
2. The weighted values are totaled to create the weighted sum.
3. Bias is added, if necessary, as another adjustment to ensure accurate output.
4. Based on the weighted sum (and any bias), the activation function delivers the perceptron's output, which is linear or binary (yes or no, 1 or 0, cat or dog, etc.)

Weights and Bias in Machine learning algorithm

Weights and bias are primarily responsible for enabling machine learning in a neural network. The neural network can adjust the weights of the various inputs and the bias to improve the accuracy of its binary classification system.

For example, the figure below illustrates how the output function of a perceptron might draw a line to distinguish between pictures of cats and dogs. If one or more dog pictures ended up on the line or slightly below the line, bias could be used to adjust the position of the line so it more precisely separated the two groups.

The Birth of the Perceptron: Frank Rosenblatt

Frank Rosenblatt invented the perceptron in 1958 while working as a professor at Cornell University. He then used it to build a machine, called the Mark 1 Perceptron, which was designed for image recognition. The machine had an array of photocells connected randomly to neurons. Potentiometers were used to determine weights, and electric motors were used to update the weights during the learning phase.

Rosenblatt's goal was to train the machine to distinguish between two images. Unfortunately, it took thousands of tries, and even then the Mark I struggled to distinguish between distinctly different images.

The Fall and Rise of the Perceptron

While Rosenblatt was working on his Mark I Perceptron, MIT professor Marvin Minsky was pushing hard for a symbolic approach. Minsky and Rosenblatt debated passionately about which was the best approach to AI. The debates were almost like family arguments. They had attended the same high school and knew each other for decades.

In 1969 Minsky co-authored a book called Perceptrons: An Introduction to Computational Geometry with Seymour Papert. In it they argued decisively against the perceptron, showing that it would only ever be able to solve linearly separable functions and thus be able to distinguish between only two classes. Minsky and Papert also, mistakenly, claimed that the research being done on the perceptron was doomed to fail because of the perceptron's limitations.

Sadly, two years after the book was published, Rosenblatt died in a boating accident. Without Rosenblatt to defend perceptrons and with many experts in the field believing that research into the perceptron would be unproductive, funding for and interest in Rosenblatt's perceptron dried up for over a decade.

Not until the early 1980s did interest in the perceptron experience a resurgence, with the addition of a hidden layer in neural networks that enables these multi-layer neural networks to solve more complex problems.

Frequently Asked Questions

What is the significance of Rosenblatt's Perceptron in the history of artificial intelligence?

Rosenblatt's Perceptron is considered a foundational model in the history of artificial intelligence. It was the first algorithm capable of supervised learning of binary classifiers, laying the groundwork for modern neural networks and deep learning.

How does a single-layer perceptron function in a neural network?

A single-layer perceptron functions by taking an input vector, applying weights, and summing these to produce an output. This output is then passed through an activation function, typically yielding a binary decision of 0 or 1, enabling basic pattern classification.

What role do the activation function and summation play in a perceptron model?

In a perceptron model, the summation process combines the weighted inputs, and the activation function then determines the output based on this sum. The activation function decides whether the neuron fires, leading to a binary output of either 0 or 1.

Can you explain the perceptron learning rule?

The perceptron learning rule is a technique used to update the weights in the perceptron algorithm. It adjusts the weights based on the error in the prediction, aiming to minimize this error over time to improve the model's accuracy.

What limitations were identified by Minsky and Papert in perceptrons?

Minsky and Papert demonstrated that single-layer perceptrons could not solve non-linear problems like the XOR problem. This limitation highlighted the need for multilayer perceptrons and more sophisticated neural network architectures in deep learning.

Who were Warren McCulloch and Walter Pitts, and what was their contribution to neural networks?

Warren McCulloch and Walter Pitts were pioneering researchers in artificial intelligence. They developed the first mathematical model of a neuron, known as the McCulloch-Pitts neuron, which inspired later work on artificial neurons and neural networks, including the perceptron model.

How does the learning rate influence the perceptron learning algorithm?

The learning rate in the perceptron learning algorithm determines how quickly the model updates its weights in response to errors. A high learning rate may lead to faster convergence but can overshoot optimal solutions, whereas a low rate provides more precision but slows down the learning process.

What is the difference between a single-layer perceptron and a multi-layer perceptron?

A single-layer perceptron has one layer of weights directly connecting the input to the output, suitable for linear classification. In contrast, a multi-layer perceptron includes one or more hidden layers, enabling it to capture complex, non-linear relationships in the data.

How does the history of the perceptron influence modern deep learning practices?

The perceptron laid the foundational concepts for neural networks and deep learning. Despite its initial limitations, it inspired the development of more advanced architectures like multilayer perceptrons, leveraging the idea that artificial neurons can be capable of learning complex patterns and functions.

This is my weekly newsletter that I call The Deep End because I want to go deeper than results you’ll see from searches or AI, incorporating insights from the history of data and data science. Each week I’ll go deep to explain a topic that’s relevant to people who work with technology. I’ll be posting about artificial intelligence, data science, and data ethics.?

This newsletter is 100% human written ?? (* aside from a quick run through grammar and spell check).

More sources

1. https://news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon
2. https://www.simplilearn.com/tutorials/deep-learning-tutorial/perceptron
3. https://proyectoidis.org/mark-i-perceptron/
4. https://en.wikipedia.org/wiki/Frank_Rosenblatt
5. https://www.javatpoint.com/perceptron-in-machine-learning
6. https://en.wikipedia.org/wiki/Perceptron
7. https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53
8. https://americanhistory.si.edu/collections/nmah_334414
9. https://www.neuroelectrics.com/blog/2023/10/03/rosenblatt-perceptron-the-origin-of-artificial-neural-networks/
10. https://visionarymarketing.com/en/2024/04/22/perceptron-a-history-of-neural-networks/
11. https://www.analytixlabs.co.in/blog/what-is-perceptron/
12. https://lamaquinaoraculo.com/deep-learning/el-perceptron-de-rosenblatt/