Large Language Model or Large Data Compression Technique? The Illusion of Intelligence.

The advancements in Artificial Intelligence, epitomized by large language models (LLMs) like Bard, ChatGPT and Grok have been both impressive and contentious. These models demonstrate exceptional text generation, language translation, and question-answering capabilities, igniting debates about the true nature of intelligence in AI.

A critical perspective suggests that LLMs may be better understood as sophisticated data compression algorithms. For instance, Bard's ability to compress its training data by a factor of approximately 27.3 million is noteworthy. If we assume that the uncompressed size of the data used to train Google Bard is 14 exabytes (EB) and that the size of the parameters in Bard's model is 128 petabytes (PB), considering that each parameter size is 4 bytes, then the compression rate for Bard is approximately 27.3 x 10^6. This means that Bard has managed to compress the data used to train it by a factor of 27.3 ?million.

This efficiency in data handling, however, raises questions about whether this constitutes 'intelligence' in the human sense.

Human Intelligence

True human intelligence involves not just pattern recognition, but also reasoning, inference, and making nuanced connections beyond the explicit data. LLMs, in contrast, appear to excel mainly in identifying patterns within their training data, which might not equate to genuine understanding.

Let's run a thought experiment as follow:

1. Isolation of a Large Language Model (LLM) and a Human: Both are isolated for 10 years, with no access to the internet or external world except for a limited amount of basic data (e.g., daily news), capped at 1GB per day.
2. Objective: After 10 years, both the LLM and the human are tested with questions that the LLM is initially expected to excel at.

Obvioulsy the outcome of this experiment would heavily depend on the nature of the questions asked. While the LLM might retain its strength in handling specific types of queries, its inability to evolve over the 10-year period could be a significant limitation. In contrast, the human's ability to adapt, learn, and think creatively could offer advantages, especially in more complex, nuanced, or creative queries.

Information Theory and AI

The limitations of Information Theory in fully capturing human intelligence are also notable. While it offers a framework for data storage and transmission, Information Theory may fall short in explaining the dynamic, adaptive process of human learning and reasoning.

The analogy with neutron stars is particularly apt in illustrating this point. Like neutron stars, which are incredibly dense yet possibly not fully understood, LLMs are dense repositories of information but may lack depth in understanding. This suggests a possible need to rethink Information Theory in the context of AI.

AI Singularity and AGI

The concept of an information singularity is an intriguing one, and it's interesting to speculate about the potential data size and processing power required to reach such a point.

The Tolman-Oppenheimer-Volkoff (TOV) limit is a physical concept that describes the maximum mass of a neutron star before it collapses into a black hole. The limit is determined by the equation of state of matter at extreme densities, and it's estimated to be around 2-3 solar masses.

Analogizing this concept to information theory, we could hypothesize that there might be a similar limit to the amount of information that can be processed or stored in a system before it reaches a critical point of complexity or intelligence. This hypothetical limit could represent an information singularity, where the system becomes so complex and self-referential that it is no longer understandable or controllable by humans.

The data size and processing power required to reach such a singularity would likely be astronomical. In terms of data size, it's estimated that the entire human knowledge base is currently around 10^14 bytes, which is equivalent to about 100 exabytes. However, this is just a fraction of what would be required to represent the full complexity of the universe and human understanding.

In terms of processing power, we're still in the early stages of developing computers that can handle truly massive amounts of data and perform complex computations efficiently. Even the most powerful supercomputers today are only capable of processing a few exaflops of data per second, which is far below what would be needed to reach an information singularity or AGI (Artificial General Intelligence).

It's possible that we'll eventually develop new computational architectures or materials that can achieve the necessary processing power, but it's also possible that the information singularity is simply beyond the reach of human technology. Only time will tell what the future holds in this area.

Conclusion

In light of these considerations, attributing true intelligence to LLMs warrants caution. Instead, the focus should be on enhancing their reasoning and genuine connectivity with the world. This might involve redefining our understanding of intelligence in the realm of AI.

The journey towards true AI intelligence is complex and may require a fundamental shift in our conceptualization of intelligence. By pushing the boundaries of AI research, we can unlock new insights into the nature of intelligence and its applications across various fields.