Can we generate intelligence about generative artificial intelligence?

Where’s my talking robot?

Robots and computers capable of holding conversations with human beings have been a staple of science fiction and visions of the future for many decades. Yet, until recently, they have seemed as elusive as flying cars.

And, while we’re asking, when’s the automated programmer arriving?

Since my first ever professionally programming job, the technology industry has threatened to do away with the job of programming - whether through 4GLs, low-code / no-code solutions, or other ways of avoiding the job of building code line by line. However, these approaches have seemed to do no more than push the need for programming somewhere else.

If I believe what I read in the press, though, both of these visions of the future may be about to come true. Generative AI models, such as ChatGPT, are being described as capable of writing text and code with the same facility as a human being (and if you can generate text, you can have a talking robot - text-to-speech is the easy bit these days).

And, just a few minutes with the public interface of ChatGPT gives results which seem impressive (and eerie). The same is true of the generative AI solutions such as DALL-E which produce images: the results seem so compelling that it is hard to believe that there is no human involved (of course, there are humans involved in the creation of the images that were used to train the model in the first place).

Despite all of the coverage of this new wave of AI, I must admit that I find it hard to understand how it works. I believe that I have a general and superficial understanding of how machine learning works, but I don’t know enough to figure this out without spending some focused effort on it. And this topic feels as if it might be kind of important - I just don’t know how yet.

So, I’m going to try an experiment similar to the one I tried with quantum computing a few weeks ago: I’m going to do some learning in public. In this article, I’ll briefly outline my current understanding of generative AI - which is more of a guess than anything else. I’ll also list some questions to which I don’t yet know the answers. In the next article, I’ll see what I can learn from some superficial scanning of popular articles on the topic. Then I’ll go a bit deeper into specialist literature. Finally, I’ll try to summarise what I’ve learnt. Given that my academic background is in philosophy, I may even attempt some rough and ready perspectives on the philosophical questions generative AI seems to raise.

Here’s my rough guess at how generative AI might work. I am sure that it will be wrong in many ways. If you want to correct me, please do so in the comments: that’s all part of learning. My guess is that there must be at least two models involved. One takes a prompt (the entry in the chat window in the case of ChatGPT, or the image prompt in the case of DALL-E) and generates something, then shows it to another model and asks whether it recognizes it as relevant to the prompt. And then they throw it backwards and forwards to each other until they get something that the second model recognizes. In both cases, neither model ‘knows’ anything about what they are producing; they are just finding something that humans would match to the prompt.

And here are my questions:

• Just how wrong is the guess above?
• What datasets are the models trained on? Where do the trainers get them from?
• How accurate are the results? How easy is it to fool the models?
• Why do the results (especially from ChatGPT) feel so bland and superficial? (I read an article recently which described them as ‘hollow’.)
• What are the operating characteristics of generative AI? Does it need masses of computing power, or could it run on my phone?
• How do we know how these models work? What level of transparency is it possible to achieve?
• What’s the future of this technology? Will the apparent speed of its development continue to rise, or will it meet limitations?
• Will we ever get talking robots and automated programmers?
• Will this technology really change the world in the way that some people are claiming? Does it genuinely have implications for the nature of human creativity?

As with quantum computing, I’m looking forward to learning more, and hope that learning in public will keep me motivated and focused. If you’ve got any good suggestions on books, articles, or other resources, please let me know in the comments.

(Views in this article are my own.)