Demystifying OpenAI's secretive "Project-Q*" - that's what it's really all about

The narrative surrounding OpenAI's Q*-Project unfolds against the backdrop of significant developments, commencing with the ousting of OpenAI CEO Sam Altman.

How Key Moments in the OpenAI Crisis focused Attention on "Project-Q*"

Ahead of OpenAI CEO Sam Altman’s four days in exile (how Reuters called it), several staff researchers wrote a letter to the board of directors warning of a powerful artificial intelligence discovery that they said could threaten humanity.

This revelation, still shrouded in secrecy, became a pivotal factor in the board's decision to remove Altman. At the heart of these concerns was the prospect of advances towards Artificial General Intelligence (AGI) without fully understanding the implications. The Board's fears also extended to the commercialization of cutting-edge AI tech and led to a broader debate within OpenAI about the ethical considerations and safety measures associated with these transformative breakthroughs, led by OpenAI Chief Scientist Ilya Sutskever.

What is Project Q*

Central to this discourse is the so called Project-Q*, with Q* (pronounced Q-Star) emerging as a potential breakthrough in OpenAI's relentless pursuit of AGI.

Q* is positioned as a powerful AI model demonstrating proficiency in solving mathematical problems, a domain traditionally considered a frontier in gen AI development.

Unlike current generative AI capabilities, which excel in language-related tasks, Q* showcased promising results in mathematical problem-solving, hinting at its potential to possess reasoning capabilities akin to human intelligence. This is a significant step towards Artificial General Intelligence (AGI), where systems can generalize, learn, and comprehend, rather than merely perform predefined tasks like a traditional calculator.

The Q*-Project also introduced a team of AI scientists tasked with optimizing existing AI models for improved reasoning and potential scientific applications. The unveiling of a demo within OpenAI caused internal tension, particularly among researchers focused on AI safety, highlighting the ongoing debate within the company regarding the ethical development and commercialization of advanced AI tech.

In an interview following his return as CEO of OpenAI, The Verge confronted Sam Altman with questions on the Q*-Project

No particular comment on that unfortunate leak. But what we have been saying [...] is that we expect progress in this technology to continue to be rapid and also that we expect to continue to work very hard to figure out how to make it safe and beneficial. [...] Without commenting on any specific thing or project or whatever, we believe that progress is research. You can always hit a wall, but we expect that progress will continue to be significant.

OpenAI CEO Sam Altman

[This kind of secrecy has proven to be a successful value driver in the past esp. for Apple. It will be fueling speculation and increasing the market value of OpenAI too.]

Q-transformers are not a miracle technology

In my decade spent on AI, I've never seen an algorithm that so many people fantasize about just from a name. No paper, no stats, no product. So let's reverse engineer the Q*-fantasy.

Dr Jim Fan, NVIDIA Senior AI Scientist

Integral to understanding Q* is the background of Q-Transformers, a pivotal element within Large Language Models (LLMs). Q-Transformers are specialized AI models designed for text data, particularly adept at understanding and answering questions, engaging in conversations, translating languages, and more. They are the focal point for developing the linguistic capabilities of large language models.

Complementing Q-Transformers is the concept of Q-learning, a model-free reinforcement learning (RL) algorithm with the unique ability to learn the value of actions in a given state without requiring a model of the environment. Q-learning, a key aspect of the Q*-Project, finds an optimal policy to maximize expected rewards over successive steps in a finite Markov decision process.

The synergy between Q-Transformers and Q-learning represents a promising frontier in AI research. Q-learning, traditionally employed in reinforcement learning, has evolved through recent advancements that combine it with Transformers and Large Language Models. The fusion of Q-learning principles with the capabilities of Q-Transformers has paved the way for Q*, positioning it as a potent force in the pursuit of AGI.

The specific advantages of Q-Transformers over other methods

The main difference between Q-Transformers and other transformer models lies in their architecture and design. Q-Transformers are based on the original transformer architecture.

Other transformer models may have different focuses or variations in their architecture. For example, BERT is a transformer model that is pre-trained for deep bidirectional representations of the input data (however, it is pretty complicated to prompt by users), while GPTs [Generative pre-trained Transformers] are transformer models that are designed for generative modeling tasks such as text generation (but also GPT models have numerous shortcomings compared to Q-Transformers esp. their dependence on training data).

The main difference between Q-Transformers and other transformer models lies in their specific design for efficient task learning, as well as potential variations in their architecture and pre-training objectives.

Q-Transformers might be superior to other models/techniques when it comes to

• learning new tasks faster and with fewer training examples, which can be a major benefit in certain applications
• projecting different potential paths for a task and select the most likely one as a starting point through the use of techniques such as the Monte Carlo system etc.
• operational efficiency

Which model a developer favors is primarily based on the goal being pursued - fast learning, simple prompting, source traceability, context understanding, etc. No model can handle all these and further tasks equally well.

Why OpenAI opted for Project-Q* and what it means for achieving AGI

OpenAI's decision in the much-discussed "Project-Q*" was most likely about the Q-Transformers' ability to continuously acquire new knowledge without regular, extensive pre-training (one of the core AGI goals) and to handle even larger model sizes with comparable compute usage through increased model-efficiency [absolutely subjective assessment]

The secrecy of Sam Altman in the interview on his return as CEO, as well as naming it an "unfortunate leak", might fuel the speculation rather than put it into perspective - that's clever marketing...

The project might have made a certain leap to narrow the gap towards AGI. Currently it looks like Q* is pretty good at math at high school level and can be considered harmless, at least for now.

Let's assume that GPT-5, which certainly will have Q*-capabilities implemented, might be much more powerful in logical thinking, problem solving and code generation and could raise more security and ethical questions - as will probably all models of the respective competitors coming onto the market in the future.

#OpenAI #QStar #AIRevolution #EthicalAI #GenerativeAI #Innovation #AGI

Sources

• The Verge, Alex Heath, 11/23, Sam Altman on being fired and rehired by OpenAI
• OpenAI Community, 11/23, "What is Q*? And when we will hear more?"
• Reuters, Anna Tong, Jeffrey Dastin and Krystal Hu, 11/23, OpenAI researchers warned board of AI breakthrough ahead of CEO ouster, sources say
• Reuters, Greg Bensinger, 11/23, ChatGPT loses its best fundraiser with Altman's departure
• The Information, Jon Victor, Amir Efrati, 11/23, OpenAI Made an AI Breakthrough Before Altman Firing, Stoking Excitement and Concern
• Twitter/X, Dr Jim Fan, 11/23, Let's reverse engineer the Q* fantasy
• Google DeepMind, Yevgen Chebotar, Quan Vuong, Alex Irpan, et al., 08/23, Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions