Can GenAI Help Meet Changing Job Demands?

Can humans and GenAI really work together effectively? What are the risks involved in such a collaboration, and what does GenAI mean for the future of work? These are big questions—and to begin answering them, our think tank, the BCG Henderson Institute (BHI), completed an A/B test with volunteers from BCG’s own consulting and functional teams.???

This follows on BHI’s 2023 study, which assessed how GenAI could help workers with tasks they knew how to complete on their own. For the latest study, there was a key difference: we tested how workers could use GenAI to complete tasks that they were not formally trained to do. In other words, instead of learning about the potential to improve productivity, we wanted to find out about the effect of GenAI on skill aptitude.??

How the Study Was Structured—and What We Learned

The precise goal of the exercise was to understand three things:?

• Whether organizations can use GenAI to expand the capabilities of their people?

• How to manage the risks of human–GenAI collaboration?

• How to manage talent in this setting?

We compared two types of BCG employees: generalist consultants and advanced data scientists. Half of the generalist consultants used GenAI and half did not.?

The study assessed the ability of participants to complete three data-science tasks that could not be fully automated with GenAI:???

1. Coding to clean data?
2. Building a predictive sports-investing model?
3. Judging whether statistical analysis generated by ChatGPT were correct and meaningful?

They had 90 minutes to complete each task.?

Consultants who used GenAI to complete the coding task earned an average score of 86%, compared with the benchmark that was set by the data scientists. That was a surprising 49-point improvement over the participants not using GenAI (37%).??

GenAI-enhanced consultants did better than their peers on the other two tasks as well, though by smaller margins—13 points and 20 points, respectively.?

The coding task was well suited for GenAI-enhanced consultants. The technology is highly adept at coding, and consultants often perform data cleaning using no-code tools such as Alteryx, so they were more likely to recognize a correct output.??

But even in the other two tasks, where the margin of improvement was smaller, GenAI served as a valuable brainstorming and performance-improvement partner, assisting consultants in achieving scores of 76% of the benchmark on the second task and 88% on the third.?

While these high scores are impressive, we’ll still need data scientists or other experts to check and improve the work of generalists to fully close the gap. It will take teams of employees with different skills to harness GenAI’s strengths and maintain quality control.

What This Means for Organizations and Employees?

It made sense for us to study data science in this experiment, since we have several hundred expert data scientists on staff to serve as a benchmark. But the results could apply to other disciplines, with implications for all organizations:?

An Expanded Talent Pool. When GenAI augments individual performance, organizations have a larger and more fungible talent pool for skilled knowledge work. They can potentially minimize turnover costs and enable cross-pollination throughout teams and disciplines.??

A Continued Need for Experts. Even for tasks that GenAI-enhanced employees can handle well, subject-matter experts will still play a key role, performing more difficult tasks while also doing the vital work of overseeing and checking the quality of GenAI outputs.?

A New Engineering Mindset. GenAI-augmented participants with moderate coding experience performed 10 to 20 percentage points better on all three tasks than their peers, even when coding was not required. This implies that an engineering mindset—for example, the ability to break a problem down into smaller pieces that can be checked and corrected—is likely a valuable skill for organizations to develop in their broader employee base.?

Employee Engagement. Four of five consultants who regularly use GenAI at work said the technology made them feel confident in their roles and enhanced their problem-solving skills, according to the study. When properly managed, the technology enhances employee engagement.?

We still have more to learn about how people and GenAI can best work together. But it’s a great time for leaders to be thinking critically about how to plan for this future, including how to redefine expertise and what skills to retain in the long term.?

More of our latest insights on GenAI:?

• GenAI Doesn’t Just Increase Productivity. It Expands Capabilities. ?
• AI at Work 2024: Friend and Foe ?
• How People Can Create—and Destroy—Value with Generative AI ?
• AI Transformation for Future-Ready Functions: The C-Suite Playbooks ?
• Five Must Haves for Effective AI Upskilling ???