Foundations of AI The Revolutionary Contributions of Hopfield and Hinton in Deep Learning
Recently, The Royal Swedish Academy of Sciences has awarded the Nobel Prize in Physics 2024 to John J. Hopfield and Geoffrey E. Hinton for their notable foundational discoveries in the field of AI and machine learning.John Hopfield developed a framework capable of retaining and retrieving data. Geoffrey Hinton created a technique that is able to identify features within data on its own, which has become crucial for the widespread artificial neural networks utilized today.
What is an Artificial Neural Network?
Computers functions based on the instructions provided by the humans to perform a specific task. To automate those tasks, machine learning concept came in to picture where the machine learns the patterns and results the output. With later advancements, Deep learning idea blossomed where the machine was capable of extracting features on its own with the help of neural network like structure.
So what exactly is a “neural” network? It is inspired by the idea in which our human? brain functions. Neurons ,the elementary nerve cell, is the fundamental building block of the biological neural network. The structure of neuron involves axon, dendrites and synapses.
Dendrites are responsible for getting incoming signals from outside, axon then ensures that the processed signals are transferred from neuron to relevant cells and synapse act as a connection between axon and other neuron dendrites. Similarly, Artificial Neural Network structure mimics the behavior of neuron where nodes are neurons , inputs are the dendrites, outputs are the axon and weights are the synapses. ANNs learn by adjusting the weights (synapses) to improve signal transmission for better decision-making and pattern recognition.
?In the 1980s,Hopfield was fascinated by ANN and started to think about the dynamics of simple neural networks. With strong? foundation in physics,? he found inspiration for his understanding of how systems with many small components that work together can give rise to new and interesting phenomena. From this idea, he was able to make a model network named Hopfield network.
?Hopfield Network
Hopfield Network is described as the overall state of the network with a property that is equivalent to the energy in the spin system found in physics; the energy is calculated using a formula that uses all the values of the nodes and all the strengths of the connections between them.
It works by storing images as patterns of black (0) and white (1) pixels. It adjusts connections between nodes using an energy formula so that the saved image has the lowest energy. When a new pattern is given, the network checks each node to see if changing its value reduces the overall energy. If it does, the pixel changes color. This process continues until no more changes lower the energy, often resulting in the network recreating the original image it was trained on. What makes Hopfield networks unique is their ability to store and recognize multiple images.
?It can be visualised like a landscape with hills and valleys. Imagine dropping a ball at the top of a hill—it will naturally roll down into the nearest valley, where it will come to rest. Similarly, the Hopfield network adjusts its nodes (representing pixels) to move towards lower energy states, like the ball rolling downhill. The network’s goal is to recreate a pattern it has learned by finding the "valley" that corresponds to the lowest energy, which represents the saved image. When the energy level matches that of the saved pattern, the network stops adjusting, confirming that it has recreated the closest version of the original image.
?Expansion of Hopfield Network
领英推荐
When Hopfield published his article on associative memory, Geoffrey Hinton with an experience in? psychology and artificial intelligence wondered whether machines could learn to process patterns in a similar way to humans, finding their own categories for sorting and interpreting information. He decided to expand Hopfield network to something new using statistical physics.
Statistical physics refers to the states in which the individual components can jointly exist can be analyzed and the probability of their occurrence is calculated. The probability of states depends on the amount of available energy, which is described in an equation by the nineteenth-century physicist Ludwig Boltzmann. Hinton’s network utilized that equation from Boltzmann machine.
?A Boltzmann machine. works by updating its nodes one by one until the overall pattern stabilizes, with each pattern's probability determined by its energy based on Boltzmann’s equation. This makes it an early generative model. However, training large networks with Boltzmann machines was slow and inefficient. In 2006, Hinton and his colleagues improved this by introducing a method for pretraining the network in layers, using multiple Boltzmann machines stacked on top of each other. This pretraining provided better initial connections, which optimized the network's ability to recognize elements in images, making training more effective.
Hinton and Hopfield's groundbreaking contributions have shaped the field of AI and deep learning. Hopfield's network introduced a way to store and retrieve patterns, while Hinton's pre training method revolutionised how neural networks could be trained more efficiently. Their innovations opened doors to future advancements in AI, from image recognition to generative models, driving significant progress in fields like computer vision, natural language processing, and beyond. Their work continues to inspire and enable the development of more sophisticated AI systems.
?
?
References:
?