Anything beyond productivity? A deeper look at the business impact of Generative AI

Tunnel vision is not a good basis for strategic decision-making We are looking at Generative AI predominantly through the lens of productivity. That's understandable because the impact is astonishing. But that is by no means all. GenAI is nearing its next infection point as companies move from experimentation to transformation at scale. Without keeping the entire picture in view, we run the risk of taking false turns in our Generative AI strategy.

There are two dimensions to GenAI beyond productivity: Looking deeper at the Future of Work and looking broader at Operations & Business Transformation.

| By now it is almost old news that GenAI is prevalent in nearly all companies (even if the companies don't know it)

BCG states that by now 60% of companies are implementing Generative AI [1] and McKinsey puts the number similarly at around 65% [2]. It is worth noting that in most cases employees are outpacing their organizations in adopting GenAI. Microsoft highlighted in its recent Work Trend Index that about 75% of global knowledge workers using AI & GenAI at work, however, 78% of them are not using tools provided by their employers and bring their own [3]. As impressive as initial adoption rates are, people are not willing to wait for their employers to provide them with the tools they need to work well.

Actually, it is important to remind ourselves the above adoption numbers do not mean that all these enterprises are already fully powered by GenAI. The majority of firms are still an initial adoption rather than scaling mode: 72% have GenAI deployed in one business function, 50% have done so in two, yet only 8% have done so in five or more, according to McKinsey [2]. In short, there is still substantial room to grow. In fact, Accenture recently put the share of work that can be augmented, automated or reinvented with GenAI at an average of 41% globally, with the highest impact among industries in Capital Markets (72%), Software & Platforms (68%) and Banking (67%) [4]. The journey has only just begun.

| Similarly well documented is the impact GenAI has on productivity

A survey of early adopters of Microsoft’s Copilot GenAI saw 70% of users reporting increased productivity and 68% furthermore saw the quality of their work improving [5]. Looking at data available on concrete GenAI cases, the average productivity gain ranges from 22% to 43% and taking an end-to-end view on efficiency, companies are looking at a total gain of around 20% on average.

These numbers are impressive and thus it is tempting to zero in on them. However, they represent only a small corner of the overall picture. Hence, this article explores the first of the two dimensions of GenAI impact: How it is reshaping the Future of Work at a fundamental level by giving people superpowers, super powering them (two different things) and how it levels the performance divide between top and bottom talents. Understanding these is essential for to make the right strategic decisions that maximise the upside from GenAI and mitigate its risks.

To capture the full value from Generative AI, organizations need to rethink how their people work, how their operations function and how they generate business value. To do they first need understand the full depth of how GenAI is reshaping the Future of Work.

Looking deeper: GenAI is reshaping the Future of Work at a fundamental level

Below we look at two studies that conducted experiments to better understand the full impact of Generative AI on employees beyond productivity alone. Although these experiments are focused on specific tasks, the outcomes can be generalised across knowledge workers are truly stunning with important strategic implications for companies.

| +40% performance: GenAI is super powering us by enhancing our existing capabilities

A joint study by Harvard, MIT, Wharton School, University of Warwick and the BCG Henderson Institute [6] took 750 consultants and gave them two typical consulting tasks to solve. One group did so the help of GPT-4, the other one without. The first task was focused on creative product innovation (think come up with a new offering and marketing plan for it), the second looked at business problem solving (what are the deeper issues in my company and which of its brand should I focus investment on).

The outcome was astonishing, not just for the obvious reason. Yes, the study did show a marked productivity improvement. Those assisted by GenAI completed on average 12.2% more tasks and 25.1% more quickly. More impressive was that in the creative product innovation task the performance or quality of work of the consultants increased by up to 40% (see Figure 1) compared to the control group, assessed by human reviewers based on four dimensional grading rubric. Overall 90% of the participants saw a marked performance improvement. In short, the GenAI enabled participants became even better at what they already knew how to do very well.

GenAI super powered these employees, enabling them to exceed in their field of expertise. Putting this in the context of companies’ talent strategies, ways of operating and goals for value output, this has tremendous implications. Capturing such boost is, alas, not as simple as just giving everyone a GenAI assistant. There are important caveats, which are covered in the risk section below.

| Instantly jumping to 83% expert level: GenAI adds superpowers, giving us new capabilities

In their latest experiment, the BCG Henderson Institute partnered with Boston University and Open AI’s Economic Impacts research team [7]. The focus was not on boosting worker’s performance at tasks they already were capable of doing. Instead, they tested if GenAI would be able to expand their capabilities to complete tasks that are beyond their current skillset. The study took 480 consultants with little or no coding and data science expertise and asked them to perform three typical tasks of data scientists: 1) write code for data cleaning, 2) build a predictive ML model and 3) run statistical tests.

Again the results were remarkable. The average performance across all three tasks compared to consultants without GenAI assistance was 61% higher, with the largest delta of 132% in the code generation for data cleaning task. Most astonishingly, the performance the GenAI using participants instantly reached an average of 83% of fully trained data scientists (see Figure 2).

The “AI exoskeleton”, as the study calls it, empowered workers to do things neither they nor the GenAI could do on their own. Consider the implications for companies in the context of onboarding junior employees or expanding output without an equivalent increase in cost of production or what this means for companies that cannot attract the experts they need in the war for talents.

| 84% narrower performance gap:?GenAI is levelling the playing field between talents

In the first study a baseline proficiency was established before the experiment. The ingoing performance gap saw the top half of participants perform 28.4% better than the bottom half. Here GenAI had an unexpected impact. When assisted by GenAI the gap between top and bottom performers shrank to 4.7%. That means GenAI helped narrow the gap by 84% - mind you, when comparing professional consultants (see Figure 3).

This has important implications: From now on being more capable without the aid of technology doesn’t give one much of an edge any longer; anyone can use GenAI to easily close that gap. From a company perspective this narrowing of the gap without any of the equivalent hiring or training costs is fantastic news. From an individual perspective this means we need to rethink where we add differentiating impact at work, not only compared to AI but also vis-à-vis our colleagues.

Generative AI fundamentally reshapes how individuals and companies work and how business value is generated. This means we need to fundamentally reconsider company's talent, operations and value generation strategies. To do so responsible though also requires us to take a deeper look at the risks to be mitigated.

You need to know the risks to manage them

To unlock these benefits responsibly and successfully, we cannot only look at the potential. We also need to closely consider the associated risks. There are several risks connected to GenAI that we well known. Like with any Artificial Intelligence, there are model, data and use-case related risks. Specifically to GenAI there are also risks such as hallucinations, jailbreaking and harmful content generation to name but a few. More broadly, there are also risks connected to underlying technological and operational maturity as well as execution risks.

Like before, here we consider some of the less obvious but just as crucial risks. Understanding these means the difference between unlocking the abovementioned upsides or incurring substantial downsides instead.

• Unthinking misuse: Always use the right tool for the right job. Inadvertently using GenAI for tasks that are outside its capability set is a major risk area. In the first experiment, for instance, the creative innovation task is something GPT-4 is well suited for. Complex business problem solving, the second task in the experiment, however not and was intentionally designed this way. Consultants who used it for this task actually saw a 23% decline (!) in performance. GenAI should be used as a copilot, not as an autopilot. That requires us, the pilots, to pause and think if what we are asking the technology is something it can reasonably be expected to do. We should not think of it as an ethereal, all-knowing AI but a newly hired intern. Just as with them we always need to think what tasks to give them. And, we always need to scrutinize outputs, rather than being asleep at the wheel (see below). If we do not discipline ourselves as users, teach this as employers and enshrine this in the operationalization of GenAI, the outcome may very well be minus 23% rather than the plus 40% in performance.
• Asleep at the wheel: Over trusting GenAI is another real danger. Humans need to be in the loop and take that responsibility as users seriously. Counterintuitively, this is not something we can solve simply by skilling people on GenAI prompting, it’s a question of mindset. Point in case, the above mentioned 23% performance decline were only the average. Looking deeper, the consultants who were previously trained on using GPT-4 actually saw a 29% decline in their performance, compared to "only" minus 16% for those who were not given GPT-4 training. The study observed those who received the training became overconfident in the GenAI and “kind of switch off their brains and follow what AI recommends”. Another study by Fabrizio Dell’Acqua from Harvard University [8] observes the same negative effect of overreliance on and overconfidence in the technology and is aptly called “Falling Asleep at the Wheel”. Mitigating this risk is not about learning how to prompt GenAI. Its about how we frame it in our minds: The new intern in need of supervision, not a superior brain beyond questioning.
• Skills erosion: We cannot supervise what we don’t understand. Yet paradoxically, increased use of GenAI might well erode the very skills we need to supervise it responsibly. The second BCG study highlighted that participants who had some coding experience, however little, outperformed the average. This was not because of their actual coding skills though. Rather, they had developed a certain “engineering mindset” enabling them to “break a problem down into subcomponents that can be effectively checked and corrected”. This meta skill or mindset was key to their outperformance. It enabled them to much better scrutinize and supervise the GenAI, resulting in their combined performance advantage. And that is where the crux lies. If, say, going forward we were to fully automate coding given the clear benefits of GenAI in this area, we risk that people will then never develop the very mindset they need to effectively supervise the GenAI they use.

There is no silver bullet, but one crucial skill we all need to hone stands out: Critical analytical judgement

The above example highlight that in abundance. And this is also the one skill highlighted by the majority leaders (30%) that will be essential for employees in an AI-powered future in last year’s Work Tend Index by Microsoft [9]. We need to reframe our mental models of GenAI. We have to stop treating it like a tool, as we did rightfully with previous generations of AIs. Instead, we need to frame it in our minds like a new junior coworker and engage with and supervise it accordingly.

Fortunately, nearly all employees (94%) are eager to learn new skills to work with GenAI according to a recent Harvard Business Review article [10]. Less heartening is that employers are not stepping up as the same article points out that only 5% of employers are actively providing training for their workforce. Given the upside to be captured and the downside to be avoided, this is a worrying gap that needs to be closed.

Sources for further reading:

[1] Boston Consulting Group (BCG) : "CEO’s Guide to Maximizing Value Potential from AI in 2024" by Nicolas de Bellefonds, Sylvain Duranton, Vladimir Lukic, Jessica Apotheker, Rich Lesser and Theo Breward

https://media-publications.bcg.com/BCG-Executive-Perspectives-CEOs-Guide-to-Maximizing-Value-from-AI-EP0-3July2024.pdf

[2] McKinsey & Company : "The state of AI in early 2024: Gen AI adoption spikes and starts to generate value" by Alex Singla, Alexander Sukharevsky, Lareina Yee, Michael Chui and Bryce Hall

https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

[3] 微软 & LinkedIn "2024 Work Trend Index Annual Report - AI at Work Is Here. Now Comes the Hard Part"

https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at-work-is-here-now-comes-the-hard-part

[4] 埃森哲 : "Work, workforce, workers Reinvented in the age of generative AI" by Ellyn Shook and Paul Daugherty

https://www.accenture.com/content/dam/accenture/final/accenture-com/document-2/Accenture-Work-Can-Become-Era-Generative-AI.pdf

[5] 微软 : "What Can Copilot’s Earliest Users Teach Us About Generative AI at Work?"

https://www.microsoft.com/en-us/worklab/work-trend-index/copilots-earliest-users-teach-us-about-generative-ai-at-work

[6] 美国哈佛商学院 , 美国宾夕法尼亚大学 - 沃顿商学院 , 美国麻省理工学院 - 斯隆管理学院 and 英国华威大学 with BCG Henderson Institute :

"Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality" (scholarly paper) by Fabrizio Dell'Acqua , Edward McFowland III , Ethan Mollick Hila Lifshitz Assaf , Kate Kellogg , Saran Rajendran , Lisa Krayer, PhD , Fran?ois Candelon and Karim Lakhani

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

"How People Can Create—and Destroy—Value with Generative AI" (BCG publication) by Fran?ois Candelon , Lisa Krayer, PhD , Saran Rajendran and David Zuluaga Martínez

https://www.bcg.com/publications/2023/how-people-create-and-destroy-value-with-gen-ai

[7] 美国波士顿大学 , OpenAI and BCG Henderson Institute :

"GenAI as an Exoskeleton: Experimental Evidence on Knowledge Workers Using GenAI on New Skills" (scholarly paper) by Emma Wiles , Lisa Krayer, PhD , Mohamed Abbadi , Urvi A. , Ryan Kennedy , Pamela Mishkin , Daniel Sack and Fran?ois Candelon

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4944588

"GenAI Doesn’t Just Increase Productivity. It Expands Capabilities." (BCG publication) by Daniel Sack , Lisa Krayer, PhD , Emma Wiles , Mohamed Abbadi , Urvi A. , Ryan Kennedy , Cristián Arnolds and Fran?ois Candelon

https://www.bcg.com/publications/2024/gen-ai-increases-productivity-and-expands-capabilities

[8] 美国哈佛大学 's Fabrizio Dell'Acqua "Falling asleep at the wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters"

https://www.almendron.com/tribuna/wp-content/uploads/2023/09/falling-asleep-at-the-whee.pdf

[9] 微软 "Will AI Fix Work?"

https://www.microsoft.com/en-us/worklab/work-trend-index/will-ai-fix-work

[10] Harvard Business Review : "Embracing Gen AI at Work" by H. James Wilson and Paul Daugherty

https://hbr.org/2024/09/embracing-gen-ai-at-work