How to talk to your robot: Why asking good questions is a critical skill in an AI-assisted economy

Markus Moos, with ChatGPT

Post-secondary institutions have been no exception to the rapid proliferation of AI, monitoring and responding to the growing use of generative AI tools like ChatGPT . Like many others, I have contemplated potential adjustments to my teaching methods in light of the benefits and challenges posed by AI. While responsible AI use undoubtedly offer promising avenues, there are genuine apprehensions regarding issues like academic misconduct, copyright infringement, surveillance, and the exacerbation of societal inequities

I've been contemplating how the students I instruct could leverage AI in their careers, strategically using it to steer clear of being replaced by robots. As an educator, my focus is on equipping them with the skills necessary to navigate a potentially AI-dominated future. One particular skill is emerging as crucial in this context: The ability to pose meaningful questions.

I decided to approach this experimentally by having a ‘talk’ with ChatGPT, delving into the significance of posing questions. My focus was on exploring AI's capability to respond to inquiries across various levels of Bloom's cognitive taxonomy , a framework familiar to many educators.

Interestingly, AI provided useful insights into the significance of mastering the art of questioning and the fundamental skills required to impart this ability to our students. But the conversation also pointed to pitfalls, and the need to remain well educated users.

Why we need to ask AI good questions

ChatGPT provided me with a rather lengthy response on why asking questions is a key skill to interacting with AI. These can be summarized into four overall points:?

1. Problem identification: Asking the right questions is usually key to solving a problem at hand. This requires contextual knowledge which is more difficult for AI to provide.?
2. Asking well-framed questions: This point is best summarized by the saying ‘garbage in - garbage out’. To receive meaningful and usable answers, the user needs to know how to ask well-structured questions.
3. Understanding AI capacity: Asking questions that leverage AI’s strength and minimize its limitations. This also includes the ability to ask about the quality of the information being provided and how to evaluate quality (i.e., what is garbage and how could we find out?)
4. Societal implications: Users need to ask about how AI works and how it can introduce and reinforce societal biases and inequities, and the ethical implications of automating certain decisions.

Is the list exhaustive? Likely not, but for the scope of this article, it arguably offers broad coverage. In essence, AI may not inherently possess the capacity to deliver an all-encompassing set of possibilities; rather, it tends to prioritize those most commonly discussed in the information it has been exposed to.

The implication is to use AI wisely in scenarios where having the 'complete set of information' is not imperative, and use it even more carefully when the absence of certain details could lead to harm. This is easier said than done: The information we receive can reinforce societal biases and dominant conversations and mute less prominent perspectives, as has already been pointed out by many critics .???

‘Testing’ ChatGPT’s knowledge?

I went on to ask ChatGPT about its familiarity with Bloom’s taxonomy. ChatGPT affirmed, explaining that Bloom's taxonomy aims to delineate the progression of thinking skills via various question types. Beginning with fundamental tasks such as memorization and recall, it advances towards higher-order thinking skills, encompassing application, analysis, synthesis, and evaluation.

I then asked ChatGPT to generate a table illustrating the comparative effectiveness of AI and Google search in responding to each question type in Bloom's taxonomy . I asked it to include additional columns outlining the potential workforce proficiency in posing each question type, along with the significance of this skill in fostering creativity and innovation (key attributes of a well functioning knowledge economy). I also asked ChatGPT to incorporate the underlying skills essential for formulating different question types and to propose teaching strategies aimed at imparting these skills to students.

The table below was made by ChatGPT, but it's crucial to note that this wasn't a seamless or single-step process. Multiple clarifications were necessary on my part to ensure the table included the desired information. In essence, I had to repeatedly pose increasingly detailed questions, many of which, I would argue, demanded prerequisite understanding of the subject matter.

Not unexpectedly, the overarching conclusion points to AI excelling more at lower levels of cognitive functions while demonstrating diminishing utility for higher-order tasks; and that the skills required by students to work at different levels of cognitive functions range from memorization and data retrieval all the way to debates, case analysis, and peer-reviews.

Table: AI Performance on Bloom’s Cognitive Levels and Implications for Teaching Strategies

What to teach students in an AI assisted world?

Through the lens of Bloom’s taxonomy, AI reasserts the importance of teaching at different cognitive levels for different purposes. Furthermore, it accentuates the crucial role of foundational knowledge as a prerequisite for executing more advanced functions. The findings in the table also imply that in an economy that prioritizes creativity and innovation, individuals possessing specialized expertise will be increasingly essential.

The good news is that most professors already understand the importance of starting with the ‘basics’ and working our way up. The concern is more that AI may become a distraction from that learning trajectory if we are not careful.?

It's akin to teaching children fundamental arithmetic skills before introducing calculators. While many of us rely on calculators, ranging from basic to advanced models, for everyday math tasks, it raises the question: would you, and should you, place complete trust in a calculator without a basic understanding of how numbers function in simple operations like addition, subtraction, and division?

Learning foundational knowledge ourselves, even though ChatGPT and Google already know the answers, instills a sense of accountability in our interaction with machines. This foundational knowledge not only aids in grasping the essence of the concepts at hand but also empowers us to critically assess the accuracy and reasonableness of the results generated by technological tools.?

Asking AI about AI

In the end, I also asked ChatGPT how I would know that it provided me with accurate information. It responded that the results would be ‘accurate within the realm of information that it was fed’. It told me to follow up with an Internet search and with detailed academic sources to confirm accuracy. In other words, it told me to go conduct the research myself. Thanks for nothing, ChatGPT!?

AI’s current utility in handling higher order cognitive questions, like evaluation, is limited in that it is not directly capable of understanding why it knows what it knows, other than to say ‘I was programmed to say this’. As experts we are able to articulate where our information comes from and why we think it to be credible. Others may disagree, especially in more normative realms, but we can evaluate each others’ arguments by the sources and assumptions they hinge upon.

We should be concerned that ChatGPT presents much of its information as ‘neutral’, without motivation behind it. When prompted about intention and biases, ChatGPT would try to reassure me that it was simply a tool that summarizes and synthesizes.?

ChatGPT may not have personal ‘motivations, intentions, or consciousness’, to use its own words, but the people creating it and feeding it information certainly do, embedding these into coding structures whether intentionally or not.?

The positive outlook for university education, particularly in the social sciences and humanities, is that an AI-driven economy appears to demand enhanced skills in analysis, synthesis, evaluation, and, notably, the ability to pose insightful questions. These are precisely the focal points that a university education has traditionally emphasized and can continue to cultivate.