A billion times more intelligent than Einstein

Axel Coustere recently posted a video (https://www.dhirubhai.net/posts/axelcoustere_ai-chatgpt-iq-activity-7212454420704313344-E61W/) quoting former Google(X) executive, Mo Gawdat saying that GPT 4 matches the IQ of Einstein, and we could be just a few months away (GPT-5) from a machine with 10 times the IQ of Einstein.? As Gawdat has said elsewhere, if someone with a much higher IQ tried to explain to you what he is thinking, you would not be able to comprehend even the topic. ?So, one possibility is that Gawdat is that much smarter than I.? Another possibility, which I think is more likely, is that I do understand what he is saying, and it is nonsense.

Gawdat is not clear on where he got his estimate of GPT-4’s IQ.? One source might be to give the model an IQ test and see how it scored. Eka Roivainen gave ChatGPT a human IQ test, the Wechsler adult intelligence scale (WAIS) (https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/) and claimed that its score qualified it for an IQ of 155.? Others have tried to estimate its intelligence from how the model performed on the SAT test, bar exams, or other tests.? These estimates are nonsense.

It’s not that I think that the estimates are wrong.? They make no sense.? When humans take an intelligence test, such as Wechsler’s WAIS, there is an assumption that the humans have not memorized the answers.? We cannot make the same assumption about a large language model.? Therefore, for a large language model to get an IQ question right, it could conceivably be because it is intelligent, or it could be because that question, or a close paraphrase, was in its training set.? Many example IQ questions are published on the World Wide Web and are included in the training.

Gawdat compares the scores achieved by an LLM with those achieved by Einstein. But Einstein is not considered a genius because of how he scored on IQ tests.? In fact, there is no evidence that he ever took an IQ test (https://www.askhandle.com/blog/everyone-says-einstein-had-an-iq-of-160-is-there-any-proof-for-this-was-he-ever-really-tested-and-if-not-why-is-everyone-so-sure-about-it).? Lewis Terman estimated Einstein’s IQ at 180, but this was probably estimated from the size of Einstein’s vocabulary, which, Terman argued, was correlated (in humans) with IQ (https://ia800708.us.archive.org/view_archive.php?archive=/28/items/crossref-pre-1923-scholarly-works/10.1037%252Fh0069483.zip&file=10.1037%252Fh0070343.pdf). But of course, that correlation would not hold for a language model, with an imposed vocabulary of about a million tokens.

Einstein is considered a genius because he invented new theories that could not be found in any existing writing at the time.? Before Einstein’s 1905 paper, it was known that certain materials would emit an electric current when exposed to light.? It was generally believed that the amount of current would depend primarily on the intensity of the light.? Einstein introduced the idea that light consists of particles, which came to be called photons, and it was the frequency of the light, not the intensity, that affected the current.? Reading all of the texts, papers, etc. before 1905 would have supported the intensity idea, but Einstein reconceptualized the problem as one involving particles of light interacting with electrons and his reconceptualization then turned out to be consistent with the experimental results, including subsequent results.

Science provides the best described examples of invention, but the process of invention also occurs in more ordinary circumstances.? Scientists are people.? They think essentially in the same way as everyone else, except they make their thought processes more explicit, and their interests tend to be specialized.??

When comparing the intelligence of LLMs versus Einstein or anyone else, the success on prescribed benchmark tests is simply not a good indicator of what is valuable in human intelligence.? The performance of a language model on an intelligence test is not meaningful because the way it answers the question is related to the material on which it was trained, not to its thinking ability. In fact, it has no thinking ability. It is designed to guess the next word (technically, the next token). Their performance on a test does not predict anything but their performance on that test.? IQ and aptitude tests are valuable (though biased) because they predict the performance of people beyond the test.? Medical school tests (MCAT) predict how well a student will do in the first year of med school (but not how well an ultimate physician will do). Language model performance on these tests predicts nothing.

Gawdat is concerned about a computer with 10 times or even a billion times the intelligence of Einstein, but being 10 times better at taking an IQ test is hardly a concern for the future of humanity.? Furthermore, it is difficult to see how any existing benchmark could even measure being 10 times better.? What would it mean to be 10 times better on the WAIS, which has a top score of 160?? Once you get all of the questions right, there is no further room for improvement.? There are people who are described as having even higher IQs, but it is not clear how those scores were determined, especially for people who died before IQ tests were invented.

In another video (https://www.youtube.com/watch?v=CFStx-FptMc), Gawdat refers to Moore’s law, which observes that the number of transistors in an integrated circuit doubles every 18-24 months.? From Moore’s law, he claims that we are doubling the compute capacity every year.? Therefore, “you know for a fact that the AI we have today will at least double every year.” There is no reason to conclude from Moore’s law that this pattern is inevitable, that it will double the compute capacity available for artificial intelligence, or that even doubling the compute capacity will lead to a doubling of intelligence.? More likely, exponential increases in compute capacity and data will be required to produce linear improvements in whatever capabilities these models are capable of demonstrating (https://arxiv.org/abs/2404.04125).

Claims that AI is soon going to take over the world, that it presents an existential threat to humanity, that we will be lucky to be kept as pets by computers thousands of times more intelligent than we are, are nonsense. There may be economic reasons for some people to make such claims, but they can also be dangerous.? They have no basis in fact, but a credulous public accepts these “mind blowing” claims as if they were the truth.? That, and not artificial intelligence, is the danger to society—researchers willing to make outrageous claims and a public willing to accept them uncritically.?