Everyone is Talking about Plato, but Kant is Lonely

Plato's Shadows: The Phenomenon of AI Model Convergence

Recently, the "Platonic Representation Hypothesis" has garnered widespread attention, partly due to the endorsement of former OpenAI Chief Scientist Ilya Sutskever after his departure. This hypothesis posits that different AI models tend to converge towards a common representation when faced with the same task. This phenomenon has sparked philosophical reflections on the nature of AI models and the development of machine learning.

To better understand this hypothesis, we need to revisit Plato's famous Allegory of the Cave. In "The Republic," Plato describes a group of prisoners who have been confined in a cave since birth, able only to see the shadows cast on the wall in front of them. These shadows are projected by a fire behind them, and since the prisoners cannot turn their heads, they believe these shadows constitute the entirety of reality (Read Philosophy Terms and StudioBinder for details).

One day, a prisoner is freed and steps out of the cave. At first, the bright light outside pains his eyes, and he refuses to believe what he sees. However, as time passes, he adapts to the light and realizes that the shadows he once saw were merely representations of objects, not the true essence of reality (See Allegory Explained).

In my previous article (Can AI Become Omniscient by Having All Data?), we discussed the limitations of models that learn solely through “hearsay.” Our current training methods for models are akin to the prisoners in the cave. These models lack autonomous exploration and continuous learning abilities. Their worldview is shaped entirely by the data fed to them by their trainers.

Immanuel Kant's Transcendental Idealism

Plato, from 2400 years ago, has recently gained much attention, sparking numerous discussions. However, the thoughts of Immanuel Kant from 300 years ago, despite being equally important, seem to have been overlooked. Kant's transcendental idealism is crucial for understanding the limitations and possibilities of AI models.

Overview of Kant's Ding an sich

In his seminal work, "Critique of Pure Reason", Kant introduced the concepts of Ding an sich (the thing-in-itself) and Erscheinung (phenomenon). The Ding an sich refers to the true nature of things, which exists independently of our perception and thus cannot be directly known. In contrast, what we can know are the phenomena—things as they appear to us through our senses and cognitive structures.

"Critique of Pure Reason" is regarded in the philosophical community as a groundbreaking work that launched modern philosophy. By systematically exploring the capabilities and limitations of human cognition, it established Kant's critical philosophy framework and profoundly influenced subsequent philosophical thought.

This distinction is highly relevant to understanding the limitations of AI models. The data processed by AI models are akin to what Kant described as phenomena—information processed through sensors (the AI’s “senses”) and algorithms (the AI’s “cognitive structures”). AI models cannot directly access the Ding an sich; they can only learn and infer based on the phenomena presented through data.

Feature Engineering and Modeling

In machine learning, feature engineering and modeling are two crucial stages. Feature engineering involves processing and transforming data to enhance understanding and representation, while modeling improves the ability to process and infer from these features. Feature engineering enhances the “recognition” capability, while modeling enhances the “knowledge” capability.

To better understand feature engineering, let's consider a concrete example. Suppose we want to use an AI model to recommend suitable movies to a user. To achieve this, we need to extract useful features from the user's behavior, which is analogous to feature engineering.

1. Data Collection: First, we collect the user's behavior data on the platform, such as which movies they have watched, the ratings they have given, the times they watched movies, and the frequency of their viewing.
2. Feature Extraction: Next, we extract certain features from these behavior data, such as:User's Viewing History: A list of movies the user has watched.User Ratings: The ratings the user has given to different movies.User's Preferred Genres: The types of movies the user prefers (e.g., sci-fi, comedy, romance).Viewing Time Preferences: The times when the user likes to watch movies (e.g., evenings, weekends).

These features are processed and transformed representations of the user's behavior (phenomena) and help the AI model better understand the user's preferences.

We extract various features to capture the essence of the data, similar to viewing the same object from different angles through different projections. Each feature represents a projection of the data's essence, but these projections can never fully reconstruct the complete reality of the object.

After feature engineering, these extracted features are fed into the AI model, which is then trained to improve its processing and understanding of these features. The architecture of the model determines how well it can "digest" the input information.

From this example, we see that AI models, through feature engineering and modeling, can extract and process useful information from the phenomena of user behavior. However, they are fundamentally still dealing with phenomena and not directly accessing the user's true inner preferences. If the extracted phenomena are incomplete or superficial, the model may be led astray. This illustrates the distinction between Kant's concepts of phenomenon and Ding an sich as applied to AI.

Inductive Bias in Deep Learning

Modern deep learning automates much of feature extraction. For instance, convolutional neural networks (CNNs) extract features from images, while recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) process textual data. Nowadays, Vision Transformers (ViTs) handle visual information, and various Transformer variants manage text. Transformers appear to be steering towards a "unified theory" of machine learning.

These models, though capable of extracting features without human intervention, inherently automate human inductive bias. Inductive bias refers to the assumptions or inclinations built into a model that aid in making reasonable predictions from limited data. For common network structures, inductive bias manifests as follows:

• Convolutional Neural Networks (CNNs): CNNs leverage the locality and translation invariance of image data. They assume that important features in an image are local and useful regardless of their position in the image. Thus, CNNs capture these local features through convolution operations.

• Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs): RNNs and LSTMs utilize the temporal dependencies of sequential data. They assume that earlier elements in a sequence are related to later ones, using their recurrent structure to remember and predict sequence elements.
• Transformers: Transformers, with their attention mechanism, assume that each position in a sequence can be related to every other position. This self-attention mechanism allows Transformers to capture relationships between any two points in a sequence.

These inductive biases help models perform well on specific types of data but also limit their generalizability. If the data characteristics don't align with the model's inductive biases, performance may suffer. While deep learning models are powerful in feature extraction, they remain constrained by their inherent assumptions, paralleling Kant's notion of cognitive limitations.

Humans also possess inductive biases, colloquially known as "filling in the blanks." For instance, when looking at an image of a Coca-Cola can, we recognize it immediately, even if the image lacks red color. Our mind fills in the missing information.

A Thousand Platforms, A Thousand Descriptions and Understandings of John Doe

The more complex an object, the greater the diversity in its descriptions, and the more significant the differences between limited descriptions.

By extracting user features (such as gender, age, address, etc.) for recommendations, we can't completely reconstruct the user. However, accumulating features improves recommendation accuracy.

Diversity of User Features

Let's illustrate how user features are extracted and utilized across different personalized products using "John Doe" a fictional user.

1. Video Recommendation Platform: John Doe, a 40-year-old male in New York, receives recommendations for sports, tech reviews, local news, and events. The system also suggests health, wellness, and finance content for his age group, as well as tech and food videos based on his viewing history.
2. E-commerce Platform: John Doe, a 40-year-old male in New York, gets recommendations for men's clothing, electronics, and local deals. The system suggests health supplements and fitness equipment for his age and new electronics and best-selling books based on his purchase history.
3. Social Media Platform: John Doe, a 40-year-old male in New York, is recommended popular male-focused communities, local events, and New York-related topics. The platform also suggests tech and food groups and posts based on his interests.

Accumulation of Features and Recommendation Accuracy

Although a single feature can't fully describe all of John Doe's needs and preferences, combining features from different platforms can gradually create a comprehensive profile. For instance, viewing history on a video platform, purchase history on an e-commerce platform, and interests on a social media platform complement each other, enabling each platform to make more accurate recommendations.

However, each platform's view of "John Doe" is just a part of him, presented through the platform's perspective and data characteristics—his "phenomena," not his "thing-in-itself." This aligns with Kant's philosophy: we can only understand things through their phenomena, not their essence.

Therefore, while AI models can achieve remarkable results through feature engineering and model improvement, they must continually refine feature extraction methods and model structures to better capture the diversity of user characteristics, enhancing recommendation accuracy and user satisfaction.

Through these examples, we see how the diversity in describing complex objects and the differences between limited descriptions manifest across different platforms. Even if we can't fully reconstruct the user, accumulating features across multiple platforms can significantly improve personalized recommendation success.

Differences in Scenarios: Cognitive Threshold Determines the Ease of Implementation

Different machine learning application scenarios have significantly varying cognitive requirements and levels of implementation difficulty, reflecting the minimal cognitive requirements for understanding an "object." Kant's distinction between the thing-in-itself (Ding an sich) and phenomena is directly applicable here: through AI models, we only grasp phenomena, not the thing-in-itself.

Tolerance in Ad Recommendation and Search

In applications like ad recommendations and search, some level of failure is acceptable. Machine learning models can iteratively improve accuracy through optimization. In these scenarios, AI models' approximate representations of phenomena are sufficient to meet the needs, making implementation relatively easy. For instance, an ad recommendation system can predict ads a user might be interested in based on their click history and browsing behavior. Even if the recommendations are not always accurate, it does not cause significant harm. This tolerance makes ad recommendation systems easier to implement in practice.

High Demands of Autonomous Driving and Surgical Robots

In contrast, scenarios like autonomous driving and surgical robots demand extremely high accuracy and reliability. Autonomous driving requires precise perception and understanding of a complex traffic environment, including vehicles, pedestrians, and traffic signals. Any minor error could lead to accidents, endangering lives. Similarly, surgical robots must perform precise operations where even the slightest mistake could jeopardize a patient's life. These scenarios have far higher cognitive requirements for AI models, demanding a closer understanding of the thing-in-itself rather than merely processing phenomena.

Thought Experiment

Imagine if we could observe our environment from an omniscient 360-degree perspective or had telepathy to predict other drivers' behaviors. We could better understand the limitations of AI models in handling complex real-world scenarios. An omniscient perspective would allow us to comprehensively and accurately acquire environmental information, and telepathy would let us accurately predict others' intentions and actions. However, real-world AI models can only rely on sensors and algorithms to gather limited information about phenomena, falling short of such comprehensive and accurate cognition. These thought experiments reveal the constraints of AI models in acquiring and processing information, explaining why implementing them in high-risk scenarios is more challenging.

Insights from Kant's Theory

Kant's theory reminds us that no matter how advanced technology becomes, AI models will always be limited by their interpretation of phenomena. While striving for higher accuracy, we must recognize the fundamental limitations of AI models in understanding and simulating the real world. Although we can continually improve sensors, algorithms, data processing, and feature extraction methods to enhance model performance, we can never fully transcend the boundary between phenomena and the thing-in-itself. This understanding helps us better appreciate the potential and limitations of AI technology and encourages caution, especially in scenarios demanding high accuracy and reliability.

Final Thoughts

Kant's philosophical insights provide profound guidance for understanding the limitations of AI models. Despite continuous advancements in AI technology, enhancing cognitive abilities and data processing capabilities, models remain confined to operating on the level of phenomena, unable to grasp the essence of data and reality.

Kant immersed himself in the sea of philosophy, engaging in dialogues with ancient philosophical sages. Though he remained unmarried throughout his life, he did not feel lonely. He lived a highly disciplined and regular life, akin to clockwork, until his final day.