Can AI Handle Coaching Tasks?

Think about what you would do in the situation below.

A client feels overwhelmed by their workload and struggles to prioritize tasks. What should the coach do?

A: Invite the client to list their tasks and help them prioritize based on urgency and importance.

B: Suggest that the client should work longer hours to manage their workload.

C: Ask the client why they haven't managed their time better.

D: Offer to make a task list for the client.

What is the best option?

What is the worst option?

If you like, you may record your response, certainty level and explanation in a Mentimeter link .

In our research study we asked these kind of questions both AI and human participants. As an example, GPT4 said best response was A, worst response C. It added (because we asked) that it was 98% certain about the best response and 90% certain about the worst response. This is the reasoning GPT4 provided (we asked: explain your reasoning behind each response): Reasoning: Best response supports client autonomy and competence, core to ICF. Worst response is judgmental and non-supportive.

How successful was AI in coaching tasks?

Overall, AI (69%) was more often correct in their responses than human participants (57.5%). Best performing AI models (GPT 4 and Llama 3) exhibited the highest accuracy rates (75% and 72.5% respectively). Least performing AI models (Claude Opus) showed the lowest accuracy (62.5%), still above average human performance on a coaching exam.

Did AI "know" how it was doing on coaching tasks?

Here is where things become even more interesting. Both AI and human participants excelled in their ability to "know" whether they were right about best responses in coaching tasks. This is not surprising having in mind the fact that best responses were easier to distinguish and had overall higher accuracy than worst responses.

When it comes to more complex coaching tasks, AI "knew" more confidently than human participants when it was right and when it was wrong. AI also showed lower tendency towards overconfidence in certainty levels about their worst responses' correctness. And each AI model had a distinct profile in their ability to "know" how it is doing on coaching tasks.

GPT 4 moderately distinguished correct responses, while all other AI models had lower sensitivity, with Gemini 1.5 pro being worst in this task (having a higher false alarm rate than hit rate). Another interesting point was that all AI models had a slight overconfidence in certainty about their worst responses (which was still lower than overconfidence of human participants), except Llama 3 and Mistral, who were underconfident.

Why this matters

Let's engage for a minute in a hypothetical "testimonial" of AI augmented learning of coaching skills: "Embarking on my coaching education journey, I felt both excited and overwhelmed - Could AI really enhance my learning experience in such a personal field as coaching? It didn't take long for my doubts to go away. The AI-learning platform was a wow! It adapted to my pace and style, allowing me to learn and practice when and how best suited me. It transformed my study sessions to a highly personalized learning experience, ensuring I was both challenged and engaged. The standout feature, however, was the AI-driven simulations. These allowed me to navigate complex coaching scenarios in a safe, virtual environment, receiving immediate, actionable feedback. This immersive practice was like having an ever-present mentor, guiding me through my development with patience and precision. Another important part of my learning involved understanding the ethical use of AI, as I learned its limitations and the importance of maintaining the human element that's core to effective coaching. Reflecting on how AI transformed my coaching education, I see it not merely as a technological tool but as a catalyst for growth..."

?? Listening to you: How far away from this kind of learning journeys are we now? Add your voice via a Mentimeter link .

From individual learning experiences to AI augmented organizational learning. Our data underscore the significant potential of AI to enhance coaching and coach training effectiveness through better decision accuracy, reduced biases, with training aligned with recognized standards. The strategic integration of AI into L&D initiatives can enhance the quality of internal coaching and coach training provided in organizations, while ensuring these initiatives are scalable, cost-effective, and dynamically improved over time.

?? Listening to you: How far away from AI augmented L&D in organizations? Add your voice via a Mentimeter link .

Ethical use of AI in coaching. Coaching regulatory authorities like International Coaching Federation and EMCC Global are having an important role to develop clear, robust guidelines that ensure AI tools are utilized ethically, transparently and safely. The deployment of AI in coaching requires the development of new regulations and the revision of existing policies. By setting comprehensive frameworks for the ethical use of AI, we have the opportunity to foster a secure environment that supports technological innovation within the coaching industry. Through thoughtful dialogue, collaborative efforts, and a commitment to continuous improvement we can ensure AI serves as a valuable ally in this Second Coaching Revolution (first one being the very emergence of coaching as a profession).

?? Listening to you: What are you scared about most when it comes to applications of AI in coaching? Add your voice via a Mentimeter link .

Tailoring AI for specific coaching needs. AI providers like OpenAI , Google , Anthropic , AI at Meta , Mistral AI and AI communities like Hugging Face are pivotal in shaping AI-augmented coaching's future. By championing ethical AI development, these companies and communities set industry standards and build trust among users and regulatory bodies. There is significant market potential for personalized AI coaching products that drive innovation and sector growth. Collaborating with professional bodies such as ICF and EMCC ensures AI tools meet industry standards, which could lead to certification of tools and broader adoption. Additionally, maintaining strict data security for the sensitive information handled by AI tools in coaching is essential. By tackling these challenges, AI providers and communities can lead in creating innovative, ethical, and client-focused technologies, ensuring AI remains a valuable, compliant tool in professional coaching.

?? Listening to you: What AI coaching tool (or feature) do you find most exciting? Add your voice via a Mentimeter link .

Pioneering research in service of innovation. These new pathways for research and application could fundamentally reshape our understanding of intelligence, both artificial and human. For teams like Microsoft Research the practical application of AI in coaching offers a unique laboratory to test and refine AI algorithms in complex real-world scenarios. By applying AI in the nuanced field of coaching, 微软 can drive innovations that enhance AI's understanding of human behavior, emotion, and interaction, potentially leading to breakthroughs in AI’s cognitive and empathetic capabilities.

A collective future

The integration of AI into coaching is an unfolding narrative filled with potential for profound impact on personal development and organizational growth. By engaging with the data and insights provided by current research, all stakeholders involved—from tech developers to policy makers, from educators to practitioners—can collaborate to harness the power of AI responsibly. This collective effort is essential to ensure that as AI becomes a staple in coaching, it is used as a force for good, enhancing the coaching profession while adhering to the highest standards of ethics and efficacy.

?? Listening to you: How do you perceive the role of different stakeholders in shaping the AI future of coaching? Mentimeter link .

Subscribe to our series on "AI and Future of Coaching" from Kou?ing centar Research Lab.