Generative AI productivity gains in coding and other professions

My friend’s job consists of talking to prospects and clients; all day, every day. It’s either in-person meetings or calls. She takes brief notes during the conversation, sometimes add a couple of bullet points immediately after the call has ended, then she copies and pastes it into a gen AI system asking it to rewrite it as full, professionally-worded sentences. Gen AI does its job well on this task. My rough estimate is that gen AI saves my friend some 30mins of work every day. Given that a standard working day in the UK is 7 hours, it’s a 7.14% improvement in efficiency; not bad at all.

However – even though sales is a very common job category – a call note summarizer doesn’t exactly sound like an economic game-changer on a planetary scale, which is what McKinsey has been promising us for the past 1.5 years with zeal that is second to none (and merits its own, standalone article, which I will write once I have some more time). From the Jun-2023 McKinsey “The economic potential of generative AI” report:

“Generative AI’s impact on productivity could add trillions of dollars in value to the global economy. Our latest research estimates that generative AI could add the equivalent of $2.6 trillion to $4.4 trillion annually across the 63 use cases we analyzed […] This estimate would roughly double if we include the impact of embedding generative AI into software that is currently used for other tasks beyond those use cases”.

(Also, a call note summarizer is a solution that can be deployed using non-generative AI)

?

By contrast, within jobs and professions I have worked at to date (compliance advisory, product management, management consulting) the scope for gen AI’s value-add is… limited to none. I could see how a gen AI-powered document search could help me find relevant documents on corporate intranets, or help browse regulations, but this kind of assistance would be seriously constrained: in professional services the tolerance for hallucinations is zero, so I would still need to cross-reference everything with the source texts. In other words, gen AI could save me some time and effort, but not a meaningful amount. This has been researched in the legal arena (an industry that is hotly tipped for being disrupted by gen AI) in great detail by Magesh, Surani et al. in their article “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools” , which, even as a pre-print, has been quite a sensation among legal scholars. I’ll give you a TL;DR: NOT hallucination-free.

?

One discipline we hear will be heavily improved / disrupted by gen AI is coding (btw whatever happened to “programming”? What exactly is so taboo about this word? It has all but disappeared from everyday use). I haven’t coded since I was a teenager, so I cannot claim much expertise here. Given the proliferation of coding jobs and their economic significance, it’s definitely worth researching how much impact gen AI may realistically have on the development profession. I will once again hand over to the experts who, in their recent paper “The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers” zeroed in on that (the depth of their research is nothing short of spectacular). They concluded that “usage of the coding assistant causes a 26.08% (SE: 10.3%) increase in the weekly number of completed tasks”. While that sounds most impressive, there is a postscript: “We find that Copilot significantly raises task completion for more recent hires and those in more junior positions but not for developers with longer tenure and in more senior positions”. There is also some unintended, pitch-black, real-world hilarity “We do not discuss at length in the main text another experiment which was run by Accenture in April 2023 and included a number of Accenture offices located in Southeast Asia. This experiment was abandoned by the company after Accenture laid off 19,000 employees that some month (cnn.com ), including 42% of the developers participating in this experiment”. That’s tech consulting in 2024 for you.

The comments about gen AI’s uneven impact by developers’ seniority have probably more significance than the researchers gave it credit for: senior developers became seniors developing their skills on the job, gradually over time. If junior developers skip that stage of development because AI will help them bridge their shortcomings, how will they attain higher levels of expertise? This is something my friend @Lev brought up only yesterday in his LinkedIn post concluding that “you'll end up calling your boomer senior engineer (whom you've fired some time ago to optimise costs) to come and fix things for you”.

?

Let’s round up this brief whistle-stop tour with creative work and industries. We hear those will be seriously disrupted by gen AI, and there might be some truth to that. Instead of hiring a freelancer on fiverr and paying them USD 20 – 40 for a simple logo design, we can ask ChatGPT or other image-generating systems like Ideogram to create one for us for free. The question is, once again, about the scale and limits of this impact. Will graphic design be impacted on an “existential” level? Will corporates switch from advertising agencies to gen AI? My own experience using ChatGPT for some graphic design assistance turned out to be somewhat… grotesque, which I described in great detail here “Race and dysmorphia: representations of masculinity in ChatGPT4o” . I’m not hating, I just can’t envision McDonald’s or Coca-Cola switching from human designers to gen AI ones – the cost saving, in the grand scheme of things, will be zero.

However, some academics disagree. In a recent Nature article (sic!) titled “The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks” a trio of researchers pitted random humans (who got paid USD 8.00 for their participation – yes, you got that right; EIGHT US DOLLARS) against ChatGPT on tasks such as “finding creative and original uses for items such as fork or rope”, “imagine people no longer need sleep – try and think of any and all consequences that might arise from this statement” or “Please enter 10 words that are as different from each other as possible, in all meanings and uses of the words”. Let’s just say that this paper failed to wow the academic community. Still, it got published in Nature. A viral post by Pau Aleikum Garcia concluded that “some academics are hyping GEN-AI irresponsibly” and I couldn’t agree more.

?

My personal opinion is that gen AI – as inherently “re-creative” rather than genuinely creative – is likely to lead to a huge increase in production of mediocre content across various media. While our ability to produce it is increasing, our ability to consume it is not – and I would argue that if there’s one thing we already have way too much of in this world, it’s mediocre content.

My second observation is that gen AI – through no-one’s ill-will or design in my view – seems to focus on a single individual and their optimized output, whilst reducing collective interactions / discussions. It doesn’t feel like a win to me.

?

Obviously, the above are just a handful of professions in an increasingly complex world; they’re either professions I worked in, know well enough to opine on, or know good research on. I can’t claim my reflections exhaust the topic – but they are a good start.?

Thoughts and comments welcome!

?

?

?

?

?