GenAI must ask questions, not just give answers

Some of Generative AI’s limitations, especially the models that are currently widely available, stem from a very simple thing: they’re calibrated for the “instant gratification” of their users. That’s a real problem when you're trying to solve complex challenges that have defined "right/wrong" answers, and that’s one of the reasons why sometimes language models provide well-structured, polished yet ultimately “middle-of-the-road” and unimaginative answers or even grossly inaccurate ones. There are ways to address this challenge, and they have to do with a rethink of what we mean by "generative".

The value of thinking slow

GenAI tools often remind us of over-eager interns who want to show off so they are perceived as smart. (To be sure, many other professionals fall into that trap: trying to impress colleagues by giving fast answers.) Our own, human, fast-answer mechanism, which Daniel Kahnemann dubbed “system 1” thinking, is often misguided.?We all - humans and machines - need “slow” thinking when looking for truly interesting answers. And that, I argue, starts with AI asking more questions to humans.?

Three examples illustrate the potential of this intuition.?

First, research conducted in mid-2023 by Harvard and BCG showed that large language models (LLMs) could already improve the output of junior consultants on tasks that don’t require much proprietary and private context, such as new product ideas for a business-to-consumer market segment. However, they struggle more with providing cogent company strategy, whose quality heavily depends on the understanding of the broader company context.?

A second set of examples comes from healthcare. While LLMs are already showing remarkable ability to provide answers when given thorough anamnesis, the reality is that most patients are unable to give a good enough set of symptoms. A large part of the role of doctors is (or should be) to also ask questions, and get more input for the diagnosis. An LLM could be instructed to do so, and even more so when using knowledge-graph databases to look into adjacent fields - which is something that even doctors struggle to do. Google AMIE does some of that, with solid results. Perplexity.ai and its CoPilot mode ask specific questions about the cause of health ailments before providing healthcare solutions.

A third example comes from education. Khanmigo, the tool built by Sal Khan (of Khan Academy) in collaboration with OpenAI, uses a sophisticated socratic flow, to ensure that children develop a full understanding of the knowledge they absorb by being asked to complete and build onto the progressively more complete inputs that the machines provide, instead of just passively memorizing answers.

To partially summarize these findings, let's focus on context, computational efficiency, and semantic vs symbolic reasoning.

• AI machines, and in particular GenAI, often lack context. While their impressive confidence may tempt us to think otherwise, they can’t read our minds, and many of us are tricked into thinking that they know more about our specific context than they really do.
• They also try to find answers in a computationally efficient way, which means that they won’t necessarily comb through all the components of a problem, and won't recombine possible parts of solutions if they have a shorter and cheaper yet likely path to the answer.
• Finally, while they do seem to have some representation of the world that transcends pure language structures, they reason mostly semantically, which means that they currently struggle more than humans with symbolic reasoning and respective abstractions and generalizations.

And yet, abstraction, context, and taking the less-traveled path are exactly what problem-solving often requires when facing complex problems.

Three vectors for questions

Understanding the problem well is a big part of any solution. Asking probing questions can increase the aperture, and enable better focus - it is the core of the first diamond of the design thinking’s double diamond, whose value is to identify the best angle for attacking a problem. But questions are extraordinarily useful in every other part of the ideation process too. ?

No question, whether human or digital, is a bad question, as long as it leads to at least one of three things.?

First, in these ideation processes, the interplay between diverse participants makes the difference between success and failure. Therein lies the opportunity: augmenting our collective intelligence by amplifying the diversity of views - through better questioning facilitated by smart machines. AI can ask many types of questions, using one of the many frameworks built by humans that embed logical structures - for instance, Socratic and other logical thinking methods (I discuss some here).?

Second, additional possibilities stem from using machines to radically open up the design space, by asking humans (or other machines) to look at the problem through the lens of distant analogies (e.g. “What communities, such as Wikipedia, look like good analogies to healthcare knowledge management?”); or using ideas from other spaces, such as what Markus Buehler at MIT recently did when using GenAI for investigating engineering materials properties (supersonic fractures) through the lens of biology. Even when machines may not have intelligent answers, they can tee up creative questions that humans - and other machines - can then try and address.?

Third, an emerging exciting avenue is through multi-agent generative AI: multiple agents that ask questions to each other, and humans. That process can be curated by humans who decide what types of agents are useful, or what threads of discussion are promising. Think for instance of a transformative idea for a large company, where much of the difficulty is in the change management and the acceptance by employees. What about a tool that uses multiple personas to critique the idea, acting as a synthetic, multithreaded town hall? For instance, one can gather the (simplified, possibly stereotyped, yet readily available) views of junior and senior employees, of people based in different regions, and of professionals in different departments. Humans could decide where to enter that town hall, decide which personas to amplify, or complement those voices, and draw conclusions and iterate on the transformative idea itself - as well as preparing a precise and thorough change management plan, including for instance detailed stakeholder engagement strategy. ?

These insights led to research and prototyping of alternative human-machine interactions for innovation. In our work at MIT's Center for Collective Intelligence (MIT Supermind Ideator), and subsequently in my collaboration with luca taroni on the “AI Collider” called Solver, (a simple example of which is the "Innovation Assistant S" OpenAI GPT) we built machines that ask humans to refine the questions before ideation. Solver also adds frameworks from insightful researchers, and theories - that put a lens on a problem, and through that prism it allows us to ask more and more pertinent questions.?These are just small, early prototypes hinting at the potential of a very large design space that will be no doubt explored further in the near term.

As a not-so-small aside these considerations should also remind us that is also dangerous to put humans into a position of dependency on the AI machine, as this might lead to atrophying core cognitive traits - such as symbolic and critical, logical thinking - that people have. Designing for active interaction between humans and machines is crucial to maintaining the vitality of human intelligence.

Building questions-asking machines today

As we stand, we already have the practical means to build interactions with machines that make human-machine collective intelligence more powerful. While there is much more that could be done, a few simple ideas are below:?

If you’re using this approach for yourself or your team:?

• Build a habit of never taking AI-generated answers at face value, and encourage the AI to ask you questions to help refine your thought process. For instance, systematically prompt the machine to ask you things like “What is missing?”, “What else should I be thinking about in framing this problem, or solution?” or "How might this fail". AI can complement your views, or vice versa, but you always want to be on the receiving end of such questions.
• Make yourself and your team acquainted with the best-known logical thinking frameworks (you can even ask ChatGPT for them), and embed those methods systematically into the flow of your interaction with GenerativeAI.?For instance, how would Christensen's Disruption Theory be applied to your problem? How would some Lean management principles (say, "The 5 Whys") shed light on it?
• Routinely submit your own work (e.g., drafts of presentations, solution ideas) to GenAI, asking AI to critique it by first asking questions to understand better what you're trying to accomplish so that the model can compare that with your artifact's content.

If you’re building enterprise or client-facing applications:

• Deliberately build workflows that include steps where the machine asks questions to gather human input - whether upfront or as a refinement of the ideas generated by the AI.?
• Consider adding multi-personas questions and critique (and even dialogue between the personas) into your workflows. This doesn’t need to be a fully-fledged “Mixture of Experts Models” dialectic, as some of it can be readily obtained by engineering prompts, possibly even as part of OpenAI's GPTs.?

Picasso said that computers are useless because they can only give answers. His point was also subtler: for true creativity and nontrivial problem-solving to occur, one needs to ask unusual, uncomfortable questions. Computers couldn’t do that then. And mostly, we don’t allow them to do that today either.

But with generative AI, they could. Generation is fueled by the right input and dialectic - it requires questions from us to the machine, but also from the machine to us. We just need to design the right interaction and workflows, and embed them into the flow of our work.?