AI and Leadership Assessment – What the Science is Saying

For over 50 years, DDI has maintained a prominent leadership role in the science and practice of workplace assessment, having assessed millions of leaders in support of more science-based decisions in thousands of corporations across the world from the front line to the C-suite.

Now, as the world grapples with the high-velocity entrance of AI, DDI is once again helping to shape the science that ensures the effective and appropriate application of these electrifying and at times daunting new capabilities.

To understand more about the opportunities and risks that AI brings to leadership assessment and what we are learning from the science, I sat down with three of DDI’s leading researchers and product developers to gather their insights and learn about the future of AI-enabled assessment.

• Chris Coughlin, M.S. is the Senior Manager of Assessment Content Development and Design at Development Dimensions International (DDI). He leads a team of Industrial/Organizational (I-O) psychologists and data scientists that develop, test, validate, innovative assessments for leadership selection, learning and development, and executive solutions.

• Georgi Yankov, PhD, is Principal Research Scientist at DDI, a globally recognized author, and innovator in the realm of psychometric measurement. Georgi develops both traditional and AI-enabled leadership tests and simulations scored with machine learning algorithms. His expertise and publications are in computational psychometrics, machine learning and personality assessment.

• Brad Pitcher is a Research Scientist at DDI and a doctoral candidate in Industrial/Organizational (I-O) Psychology at Purdue University. At DDI, Brad contributes to the development of traditional and AI-based assessments of individual differences and behavior. Brad’s research interests are in emerging workplace technologies and their applications for assessment and development.?

There is a lot happening in the field of AI-enabled assessment and many HR leaders are not sure what to make of it. Is AI a good thing? A bad thing? Can it be used now? Should we be excited, or should we be afraid? Will it help our organizations, or put us at risk? How should I guide our senior leaders??Let’s see if we can sort some of these questions out.?

Let's dive in with the scientists.

Matt: Assessment is a broad category. It includes tests and inventories, simulations, interviews, 360 surveys, and more. Before we dive into the cutting-edge science, can you start by sharing how AI has been applied in assessment up to now? In other words, what is NOT new with AI in assessment?

·?????? Chris: That is a great place to start because many of the assessment techniques that fall under the umbrella of AI have been utilized by IO psychologists for decades. When we talk with new clients, it is not uncommon for them to be surprised by this. Take for example, Computer Adaptive Testing. CAT has been in use for well over a decade and uses numerous algorithmic methods that are forms of AI to adapt questions to a test-taker’s response pattern. This can result in a much shorter test that produces the same validity and fairness. Another example is Natural Language Processing (NLP) which has been in use for years to evaluate essays for style and grammar. These methods aren’t new in the realm of assessment, but their application is accelerating with AI.?

·?????? Brad:??Think of it this way: AI is powered by algorithms, which are simply sets of mathematical or logical rules for turning inputs (e.g., assessment responses) into outputs (e.g., competency scores).?All assessments are scored algorithmically at least in part.?The difference with emerging applications of AI is that the algorithms are becoming more sophisticated and being applied in broader ways in the assessment process.?

Matt: The most prominent assessment methods in use today are the product of research that spans most of the last century and has empirically determined the principles and practices that ensure reliability, validity, and fairness. How is AI changing the science of assessment in the workplace?

·?????? Georgi: This might surprise some people, but the scientific basics are not changing, because nothing is changing about the need to ensure that assessments are reliable, valid, and fair. Those principles remain foundational. That said, AI advancements are accelerating the intersection of computer science and psychological science. We are looking for the same proof points, but we are doing so with more advanced technology.

·?????? Chris: I agree with Georgi. The best practices and guidelines we adhere to in the assessment industry are fundamentally unchanged. And I would also add that AI is complementing our techniques and expanding some of the tools in our toolbox. For example, many AI techniques could help make classical procedures like job analysis and test item development more cost effective and efficient.

?

Matt: Given your research and the unique data sets that DDI draws on, what do you see as the most promising near-term applications in the assessment space? How will organizations benefit from these advances??

• Chris: Based on innovations already in place as well as many in development, organizations can anticipate advances in the precision of assessment insights, enhanced reliability and accuracy, cost savings, and immediate feedback for users to apply on the job.

• Brad: We are excited about near-term?applications in content development, evaluation, translation, and scoring. AI’s language, image, and video capabilities are showing great utility for enhancing our methodologies in areas like personality inventories, situational judgment tests, and immersive leadership simulations like business analysis exercises, strategic planning scenarios, and decision-making challenges. AI is helping us make all these methods faster, more cost effective, more scalable, more accurate, and more instantly applicable for leaders.?

Matt: Without giving away any secrets, how is DDI’s AI approach different from how other organizations are leveraging AI in assessment?

·?????? Georgi:? DDI’s data is unique because of its behavioral specificity. We don’t simply measure competencies; we measure the specific behavioral actions that make leaders effective in each one. So, we are putting AI to work in helping us do that better. For decades, we have been gathering behavioral observations of leaders as they work through interpersonal, leadership, and business judgment challenges (i.e., simulations). Large Language Models (LLMs) can now be trained to evaluate the leadership behaviors being targeted in those exercises.

Taking a simple example, DDI has gathered responses from thousands of people who have written on how to coach a direct report. With this large and contextualized vocabulary of coaching we can fine-tune the best LLMs to achieve similar or better reliability and validity than what humans demonstrate. We don’t see AI replacing human assessment completely, but for some aspects of leadership behavior, AI can augment human judgment and enhance reliability and accuracy. What this means in practice is that leaders and organizations will receive job-relevant insights from accurate, unbiased, predictive assessments, just as they always have, but they can now be scored in part by AI, resulting in higher quality in a more efficient manner.

?

Matt: We have heard of organizations that have completely banned the use of AI in leadership assessments. What is your reaction to this stance? Is it a prudent one, or shortsighted? Should organizations be afraid of AI in leadership assessment?

·?????? Chris: I understand that many organizations are concerned with the risks. We view this as healthy, if ALL the risks are considered.? A methodical approach may find (and often does) that there is greater risk in NOT leveraging AI than making appropriate use of it. For example, let’s say we find a new AI method that brings value in things like cost savings, real time feedback, more accuracy, and better ROI. Okay, then we find it to be equally or more interpretable, secure, unbiased, legally compliant, and predictive over traditional methods? How could you afford to not capture this competitive edge? This flips your original question about whether organizations should be afraid. I would say don’t be afraid of anything. Be diligent instead. Study AI-based products or services before accepting or rejecting them. The key is to know what AI means and to have a nuanced understanding of how it adds or subtracts from traditional approaches.

·?????? Georgi: That is exactly right. There is a lot of skepticism out there right now, and to Chris’s point, a lot of miscalculation of risk. Our task is to confirm and explain the benefits of using AI-enabled assessments, and to show that they can be built responsibly according to the highest professional and legal standards. After that, if organizations are still recalcitrant, they should understand that they may be at a competitive disadvantage.

Matt: Are you seeing trends that worry you? Are there risks, myths, or misperceptions that organizations should be wary of as they employ AI in assessment?

·?????? Brad: There is currently a trend toward thinking in false dichotomies about AI – either deifying it or catastrophizing it. AI, like all technology, is ultimately a tool that must be used responsibly. It’s not a panacea. It needs to be implemented thoughtfully and carefully.

·?????? Chris: I agree. This is not as simple as AI being good or bad. Like any complex technology, we must be thoughtful about its application. One trend that concerns me is the race to either deploy or reject AI without performing due diligence. While the field is moving rapidly, research and understanding are required before utilizing or rejecting a new method. We have done some work on this and have made recommendations that can get organizations started and moving in the right direction.

·?????? Georgi: As Plato said, the greatest evil in the world is ignorance. From our research and the rapidly evolving science, we can see clearly that the responsible use of AI brings tremendous possibility. We are standing at the doorway to the most powerful methods we have ever had for using assessment to help all people learn, grow, and perform to their fullest potential. At DDI we are choosing to be among those who use science to guide us toward the best technology and practices to see that future realized.

Matt: What guidance do you have for HR professionals as they consider assessment alternatives in the current marketplace? What questions should organizations be asking as they select assessment tools and partners?

·?????? Brad: This is a place where we can point to some of the work we have been doing. To call out few basics, users should conduct a review of technical documentation of any AI assessments with assessment and data science experts. As we have said, AI assessments should meet the same psychometrics standards that have long been used to ensure validity and fairness.?

It is also important to pay special attention to the training dataset of AI models used in assessment. Does the provider give sufficient information to assess the quality and appropriateness of the training data?

Additionally, if an organization’s proprietary data will be used in an AI model for assessment, are there safeguards to prevent model “jailbreaking” which could compromise data?

·?????? Georgi: I would like to reiterate something we mentioned earlier. Really, very little has changed with the process of carefully vetting assessment providers and their tools. AI continues to employ automated algorithms, and we will continue to evolve the sophistication of the technology that puts these algorithms to use, but the practices to ensure that a product or service is high quality are the same as they have been up to now.

Matt: Lastly, you have all been quite optimistic about AI when used responsibly. Taking that a step further, what inspires you most about the future of AI-enabled assessment for organizations seeking better leaders and a better future?

·?????? Georgi: The future is incredibly exciting. Imagine a personal assistant appearing as a hologram and role-playing with you to help you learn coaching or delegation, for example. Imagine, instead of taking a long test, you are interviewed by a similar assistant and your responses are recoded into the language of the assessed constructs and organized reliably to provide you with feedback and insight into competencies, personality, or motivation. Imagine, with a few clicks, keystrokes, or verbal requests, customizing a self-evaluation and practice protocol to help you cultivate your business decision making skills.

?

This is not science fiction. These and other exciting capabilities are already being worked on, and we are inspired to be part of the birth of this new wave of transformational technologies. The future of leadership selection and development is extremely bright.

·?????? Chris: Deployed and operated in a responsible manner, I am beyond excited of what the future holds for AI-enabled assessment. Specifically, I am enthusiastic about the potential to enhance our prediction accuracy (of leader performance) while also enabling more insights that are tailored specifically to guide a leader’s development and growth through real-time feedback. The tools that AI is unlocking for us will have a tremendous impact for identifying and developing the leaders of tomorrow while also enhancing the organizations they work for.

?

I’d like to thank Chris, Georgi and Brad for sharing their insights and helping us understand a field that is moving and changing quickly.

To learn more and follow DDI’s research in AI-enabled assessment and all things leadership, please join us here.

Matt Paese, Ph.D. is Senior Vice President of Leadership Insights for DDI. Matt is a globally recognized leadership coach, consultant, author, keynote speaker, and architect of proven approaches for helping leaders succeed. He is the lead author of?Leaders Ready Now: Accelerating Growth in a Faster World?(2016) and co-author of its award-winning predecessor,?Grow Your Own Leaders?(2002), which have shaped over two decades of impact in growing leaders into skilled, self-aware, compassionate enterprise executives.