Is GPT-4 a good Data Analyst?

Here is a great video from Luke Barousse , giving an overview of a recent research paper. The paper compares the performance of GPT-4 (with python) against intern, junior and senior data analysts. The research paper in question is here, this is an excerpt:

"In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performances between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data analysts."

GPT-4 can replace data analysts? Not so fast… Maybe I'm being a curmudgeon, maybe I'm not.

Yes, based on the papers findings, it is blatantly obvious how much faster GPT-4 is at generating code and executing it against the underlying data set (which it also has to interpret itself) to answer a set of provided questions. This is nothing new - computers have been used to outpace humans at automatable tasks for a long time. The thing with this study though, is that the list of questions was pre-compiled, and constrained to the scope of the data set provided. Thereby removing any benefit that the human data analysts have from their domain knowledge.?It has been a long time since I was able to ablate answer a business insights question with a single data set, or only data that is private to the business unit.

Lets pull an example from table 12, comparing the Senior DA against GPT-4:

----

How to reduce human cost by shifting employees from different departments among these regions?

Senior DA

- Europe has only 2 employees, with 1 from Human Resources and 1 from Public Relationship?

- We think it may not be very efficient to set up an EU office with only two employees

GPT 4

- Europe has a very limited number of departments represented, with only Human Resources and Public Relations having one employee each. This suggests that there may be a need for additional staff in other departments in the Europe region, which could be addressed by transferring employees from the Americas region.

----

I’m actually going to fault BOTH the human and GPT-4 on this response. Do either of them realize the challenges of setting up an office in a new country, and the even?bigger challenge of “transferring employees from the americas to europe”?….. Let’s just say setting up a new international office and moving departments and people…. is extremely challenging. Brain drain is 100% likely to happen - and that might not be a price the business is willing to pay. Not to mention legal issues around passports and visas etc, and the massively impactful human factors....

Example: I moved UK to US when I was single and mid 20’s. I then declined a US to Germany relocation in my mid 40’s when I learned that my 16.5 year old son would have to file for his own visa at age 18?(without Brexit, it would have been fine). If he failed to get a visa, he would have had to go to the UK or back to the US alone, or we would have had to relocate ourselves internationally, at our own expense... Yes, a great job opportunity, but not feasible for where we as a family were at that time.

The human might, just might, have implied considerations like these with their “may not be very efficient” comment… but there is no evidence to support that. GPT-4 on the other hand, ignores those challenges entirely.

I believe that we humans still have the edge on unconstrained questions, but we are much slower than a computer. The question is though, how much longer will have the experiential advantage??GenAI will learn how to ask relevant questions outside of the initial scope, and how to get the additional data needed.

Are there lessons to be learned? Yes.

• Leverage GenAI now, for your core data analytical needs, especially for data that you already have. A starting point is Luke's "ChatGPT for data analytics: full course".
• As a human data analyst, always, always bring your domain knowledge into you analysis, especially if you have a different opinion from the GenAI findings.
• Channel your inner Trekkie - use GenAI for follow up questions as you dig into an insight, and be prepared to provide it with more data
• Do not ignore how GenAI continues to develop over the coming months (never mind the coming years)

This last one is perhaps the most important, because this object in your mirror, is definitely closer than it appears. I don't think we will have to wait too long for the appearance of Autonomous GenAI Agents that can do both 2nd and 3rd order data analysis for us. And when that happens, we will have eclipsed the Starship Enterprise's computer - The crew always had to ask follow up questions.