OpenAI's secret: Q*

This was originally published in the Sunday edition of my newsletter yesterday. You can now become a member for the next year with full access to research, essays, and conversations that will deepen your understanding of AI with a one-time discount for your first year. The offer ends in just a few hours. Upgrade to Exponential View Premium with 33% off

Just when you think the OpenAI drama is over, it bursts open again. Sam Altman is back as CEO, and now there’s talk that his firing was due to a secret project, Q*. According to Reuters:?

Several staff researchers wrote a letter to the board of directors warning of a powerful artificial intelligence discovery that they said could threaten humanity.

Reportedly, this secret AI can do grade-school maths on problems it hasn’t seen before. This has previously been a challenge for LLMs like GPT-4. While ‘grade-school maths’ doesn’t seem like much, with exponential progress of the type we’ve witnessed, grade-school maths could quickly become PhD maths. (The Information has more details on what this breakthrough could be. Others speculate it is about a different approach to reinforcement learning that can operate “model-free”, in complex or highly evolving environments. Yann LeCun suggests it could be about planning. But equally, this could be no more than breathless hype, diverting attention from OpenAI’s recent troubles.)

Let’s imagine that this “grade school maths” thing is real. So far, LLMs have built capabilities quickly. A recent study by Anthropic, Cohere, and NYU researchers pitted humans, GPT-4 and GPT-3.5 against a range of difficult graduate science problems. Highly skilled non-experts, pursuing PhDs in unrelated disciplines, were given access to Google to help them. They got 34% of questions right, better than GPT-3.5 but worse than GPT-4. PhD students in their own domains, naturally, did the best, scoring 65%. So, today’s state-of-the-art token predictor is nowhere near expert (PhD-level) performance, but it is better than your smarter-than-average-bear with a search engine–and will likely get better.

Perhaps, Altman had been giving hints. One day before his ouster, he said:

Four times now in the history of OpenAI, the most recent time was just in the last couple weeks, I’ve gotten to be in the room, when we sort of push the veil of ignorance back and the frontier of discovery forward, and getting to do that is the professional honour of a lifetime

This week won’t be the end of the drama. To help contextualise things and what might come next, I spoke with Karen Hao, a contributing writer at The Atlantic, who has been researching the company closely.