Where did AI come from?

Here I provide a simple potted history to the rise of modern #AI. The #brain is at the heart of the inspiration for thinking machines: it's where AI comes from, and it still has much to teach us as we improve technology.

Artificial Intelligence (AI) seems to have taken over the world, and media stories are dominated by AI's promises and/or perils.

The idea of making machines intelligent isn't new: Alan Turing and John von Neumann, fathers of modern computing, both wrote about using densely interconnected arrays to perform tasks like the human brain. But, there have been a number of false dawns in realising the potential of these artificial architectures to solve complex problems.

The long road to modern AI

Artificial neurons are the building blocks of AI - a mathematical simplification of the billions of brain cells that sit between our ears. In 1943, Warren McCulloch and Walter Pitts developed a simple (On vs. Off) mathematical model of a brain cell. Their logic was compelling: the brain is made up of many simple cells. Somehow human intelligence emerges by connecting many of these simple units together. McCulloch and Pitts connected multiple simple “neurons” to create what they called a “nerve net”. This gave them a tractable computational model of the brain (albeit a very simple one).

In 1957, the psychologist Frank Rosenblatt built on this idea to create a "perceptron", the first neural network that could learn through trial and error. This development was critical: rather than requiring a human to specify the exact way that each neuron is connected (a bit like the job of an old telephone exchange operator), the network could adapt and learn by changing the connections between each of its neurons. This process is described as reweighting – where little used connections are down-weighted and more important connections are enhanced.

While this development remains the cornerstone of modern AI, Rosenblatt’s bold statements about what these networks would be able to do (e.g., walk, talk and see) generated controversy. This now looks precedent, but at the time it seemed hyperbolic, and frustrations with the approach grew due to slow progress and a failure to crack complex problems.

During the intervening years, enthusiasm for the idea of artificial intelligence has waxed and waned. There was a notable resurgence in the 1980s, but that died away and by the early 2000s the idea of building intelligent machines from simple neural nets was something of an embarrassing relation in computer science.

I worked at the Max Planck Institute for Biological Cybernetics at the time, and remember discussions about how quaint the thinking from the generations before had been. The solution to problems that were really hard—like recognising an object in a cluttered scene—surely lay in exploiting human intelligence to hand-craft new clever solutions and approaches rather than dumb neural nets?

This thinking by my earlier self now looks quaint, and speaks to a tension at the heart of the machine intelligence that is quasi-philosophical: should solutions be based on a mathematical forward-view of how the system should work, or should we just let algorithms ‘get on with it’ even though we might have no real insight into what they are doing?

The PhD that changed the world

In December 2012, a PhD student called Alex Krizhevsky gave a conference talk at Lake Tahoe that changed the world. His talk addressed the problem of how to get a computer to recognise a range of different objects e.g., people, dogs, cars. This is still a difficult problem because real objects of a given type (e.g. dogs) differ in many ways, and one object is often hidden behind another in a photograph. Krizhevsky and his supervisor Geoff Hinton, smashed all previous records at getting computers to do this – overnight increasing accuracy on a standard set of photographs from 75% to 85% correct.

This far outstripped all progress on this task over the preceding decade. How had they made so much progress in cracking the problem? Their approach used neural networks to solve the problem – at that point very untrendy and old fashioned. Their innovation was to stack multiple neural networks on top of each other to create a “deep” neural network. Making this work required a subtle change in the maths, but their architecture is still very similar to Rosenblatt’s brain-inspired approach from the 1950s. Their other key ingredient was data. And lots of it.

Neural networks need training so that the connection weights between each neuron become optimised. The amount of training depends on the number of neurons and connections in the network. Adding more neurons (e.g., another layer in a deep neural network) can lead to a massive increase the number of connections: e.g., 2 neurons have 1 connection, 3 neurons have 3 connections, 5 have 10, 10 have 45, 100 have 4950 etc. Using their deep neural network therefore depended on finding a lot of images of real objects with which to tune the network’s parameters...

Let me google that for you

Changes in digital camera technology and search engines meant that Krizhevksy and colleagues had lots of data at their disposal. Namely, real world photographs whose content had helpfully been labelled - ‘cat’, ‘dog’, ‘car’, etc, by many millions of internet users. This resource allowed them to train the many thousands of connection weights in their neural network to produce a step change in object recognition performance by a computer.

Within three years, their approach had reshaped much of modern computer science and caused an explosion in the potential uses of AI and deep neural network technology. Object recognition performance on the standard test can now be solved by a computer as well as it can by a human viewer, and large language models (e.g. ChatGPT) are now tackling the complexities of linguistics that until recently felt beyond reach.

You are what you eat

Despite the potential for AI technologies, there are a number of challenges ahead. The current enthusiasm for the approach often involves magical thinking about what machine learning can do.

A major misconception is that AI can do more-or-less anything with data, so that the quality of the data itself doesn’t really matter. For instance, there is lots of hype about using wearables and/or mobile phones to diagnose health conditions based on very simple metrics about usage patterns. But, simply providing a large quantity of poor quality data is unlikely to be successful (Computer Scientists have long talked about #GIGO: garbage in, garbage out).

The key to identifying markers of health will be collaboration with #biomedical scientists to extract biologically-meaningful features from complex data.?And we need to be mindful in thinking about which data we use and learn from. The issue of bias in AI (and the potential to further reinforce existing stereotypes) has been discussed for some time - for example, Google translate's original implementation was fed on a large corpus of literature in which the doctors are men and nurses are women, so it reinforced gender stereotypes in its translations.

Power hungry

Another important consideration as neural networks become deeper is the amount of energy that is needed to train and run them. Computer chips have improved their efficiency, but it can still take many hours of supercomputing (and a huge amount of electricity) to optimise a network (Deepmind’s AlphaGo took months of training). Given the significant opportunities of embedding machine intelligence into a range of everyday settings, how could we manage these energy requirements?

As we look to the future, we'll need to consider how we can make AI more #green - to reduce the environmental impact of vast data servers and supercomputers. We could do worse than go back to the original inspiration for AI: the brain. The human brain is capable of remarkable feats of memory storage and computation, especially when we consider that it does so on the energy budget of a lightbulb. As AI technologies develop, expect some valuable insights into how to compute in a low energy way to come from the delicate balance of chemical energy used inside our own heads.