Artificial Intelligence #36 : Is the future of AI = future of Deep Learning?
Introduction
Welcome to Artificial Intelligence #36 – the last edition for this year.
For many of us, 2021 has been a challenging year.
For me, we did well workwise and also for teaching at #universityofoxford. On a personal note, I lost my father this year and I was also confirmed to be on the autism spectrum as an adult. ?
But also, this newsletter reached 37,000 subscribers in about 9 months – so thanks again for your support.
In this last edition for the year, I will discuss the evolution of Deep learning and AI – with the question: Is the future of AI = future of Deep Learning?
I have covered this subject before in a few ways, but I plan to consolidate these ideas together.
Background
The term artificial intelligence refers to (ideally) achieving human level intelligence and creative problem-solving ability through machines. Today, Deep learning neural networks are the primary mechanism to implement systems that have some characteristics of AI. Deep learning networks have been successful in many areas but are still far from modelling the many complex aspects of intelligence. ?
To recap, Deep learning is a technique for classifying information through layered (deep) neural networks. Although the goal of deep learning is to mimic the working of the brain, the layered structure of a neural network is a restricted mechanism to achieve intelligence. ?The power of deep neural networks lies in it’s depth i.e. in the ability to achieve representation learning. Representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data (as opposed to manually doing so). Despite their amazing capabilities in representation learning, deep learning networks are far from achieving true intelligence.
Consider the case of a curious (but real) 120 sided dice(which no one quite knows what to do with). If you had a normal dice(6 sided), your sample space becomes 6 because once a dice has been rolled, there are 6 possible outcomes/events. With 120 sided dice, the sample space is much larger (120). If you extrapolate this example to features in machine learning, then you end up with a ‘sparse matrix’ if the number of features are very large. A sparse matrix is a situation where the matrix has more zeros than numbers. This is one example of a situation where you may not have enough data to model that situation.
The limits and challenges of deep learning
We can summarise the limitations of deep learning as follows:
Humans need a very small number of samples to learn about a problem. Deep learning, in contrast, needs to be trained with a very large amount of labelled data – which is not often available. The need for data also creates a quest for hoarding data (Data is the new oil etc) and also to the need for more compute. Ironically, the need for labelled data creates a requirement for manual processes of labelling data – leading to a whole new class of data annotation companies. Because data and compute are not infinite, often there is a lack of sufficient training data. In this case, the algorithm can fail and also can be easily deceived.
Also, deep learning algorithms do not understand the context behind a problem. Hence, deep learning will continue to be successful in classification problems where we have data to train and also the training data matches the test data i.e. satisfies the iid condition (Independent and identically distributed data). So, currently for deep learning, supervised learning needs a lot of labelled data and reinforcement learning needs a very large number of interactions. ?
To provide some context, there are two paradigms of AI: logic inspired, and brain inspired. In the logic inspired AI (symbolic AI), rules or symbolic representations are encoded explicitly ?which capture the knowledge. The brain inspired paradigm learns representation from data without the need to create explicit symbolic knowledge. Deep learning is an example of brain inspired paradigm. In this case, the knowledge is captured as vectors, weights and biases in the neural network. But the algorithm has to learn the entire sample space from scratch – even if the information it seeks is available previously (more on this later).
The future of deep learning, according to its pioneers
In 2021, the three pioneers of AI ie Hinton, Le Cun and Bengio addressed these concerns in a paper and a video (links below). In their view, we do not need to resort to symbolic (or even hybrid) techniques to overcome the limitations of the current deep learning. Instead, they propose a number of new techniques that they believe will overcome the challenges of deep learning without considering rules/symbols.
These techniques are
Transformers which are already making a huge impact through GPT-3 and other models
领英推荐
My views
I think much of the views of symbolic or not is a bit dated and comes from a dichotomy ignoring options that lie in between the two extremes.
Let’s go back to the example of the dice. If you provided extra information (ex that the outcome is an even number) then you have suddenly reduced your sample space in half.
So, if you could provide information in some means to complement the neural approach then many more problem domains could be addressed. Note that these are not traditional ‘rules’ in the symbolic sense.
The obvious example if hybrid models such as Neuro-Symbolic Concept Learner – I have been working on a hybrid model called CLARION
Combining Bayesian strategies with deep learning is another way see this Neurips workshop on Bayesian deep learning strategies
So, I believe that deep learning will evolve but also some of the strategies listed above will also come into play
We have currently a group think/ consensus in the power of data.
People who have the largest amount of data – also are the biggest proponents of techniques that need a large amount of data – downplaying the shortcomings of data.
Pithy slogans like ‘data is the new oil’ – hide the complexity of many use cases where no data exists
A similar story plays out in compute for deep learning
More data needs more compute – but that still solves only a specific set of problems – adding to environmental ones
Like Churchill’s famous quote, we should not be like the Generals who are prepared to fight the last war.
This post took a long time to write because it is a complex topic - but i also wrote it for my students.
Comments welcome
Happy new year! Stay safe!
References
Bengio – Le Cun – Hinton paper + video
also good references from bdtalks
image source – 120 sided die -?inside science
CIO at Maxil Technology Solutions Inc
2 年Ajit, Agreed that you have a situation where Deep learning, needs to be trained with very large amount of?labelled data?– which is not often available. The need for data also creates a quest for hoarding data (Data is the new oil etc) and also to the need for more compute.?I think the most important factor is finding the solution and overcoming the challenges of deep learning.
The ability of humans to learn quickly from small amounts of data shows that today's Deep Learning is likely to be sidelined as new paradigms for neural networks emerge and take over. This is a huge opportunity for researchers. The emphasis on logic inspired AI will likewise be sidelined by approaches based upon plausible reasoning with imperfect knowledge, using ideas refined by a long line of philosophers including Carneades, Locke, Bentham, Wigmore, Keynes, Wittgenstein, Pollock and many others. Symbolic AI was seduced by the purity of mathematical logic, thanks to the influence of Russell, Whitehead and Quine. It is now time for fresh thinking that celebrates the rich ambiguities and uncertainties of everyday life. AI will never be the same again!
President at Artificial Labour Leasing, Inc
2 年Dear Ajit, future of artificial rational systems in the development of the artificial semi-autonomous subjective systems. Best wishes, Michael
Alexander von Humboldt Professor in AI; Co-founder and Chair of the Board of CLAIRE; Co-Chair of the Board of Directors of the AI Center at RWTH Aachen University
2 年Very interesting question. AI as a field has a history of extreme positions, and of people (within the field and beyond) being overly driven by bold extrapolation from a limited set of results. This led to the marginalisation of neural networks, which was a mistake, and more recently, to the belief that nothing but deep learning counts for the future of AI. There are many good reasons to believe that deep learning will play an important role going forward, and even more (including those you mention) to believe that this is only part of what will count. AI is more than learning; reasoning, for example, is the basis for any correctness proof, and hence for correctly functioning hard- and software. Learning from small amounts of data, and with limited computational resources, is becoming increasingly important. And there are serious doubts whether striving for AI systems that display broad-spectrum human-level intelligence is a good idea, economically and philosophically speaking. If I had to place a bet, it would be that most next-generation of AI systems will combine techniques from different areas within the field, likely across the (indeed misleading) symbolic/subsymbolic divide. Let's be open-minded and creative!