Looking for an AI to replace your market research supplier? Here it is! But you may (still) need a human validator!

With over 20 years of experience in market research spanning diverse geographies, sectors, clients, and methodologies, I've often found myself deeply engaged with data. As a researcher, some colleagues have noted my penchant for delving into data analysis, sometimes personally using tools like SPSS or Stata. This immersion in data has its pros and cons, which I won't delve into here.

In my current role at Wellspring Research , I see this data affinity as an asset. If time permits, I take the reins and run the data analysis myself. Otherwise, I delegate the task to a reliable freelancer.

Since the launch of ChatGPT last year, I've been eager to leverage it for data analysis. I applied for an OpenAI API license, enabling me to send requests via Python scripts. I took several courses from DeepLearning.AI , OpenAI and LangChain to grasp the nuances of prompting, chaining, and chatbotting. Though these courses were helpful, I yearned for a seamless way to load a survey database and begin querying the data immediately.

Imagine the dream scenario for any marketing or insight manager: instant answers, data analysis by simply asking an AI for frequencies, data filters, table or chart presentations – not in days, not in hours, perhaps not even in minutes. Instant results, ready to present to the big boss.

As such, I decided to test out the Code Interpreter from ChatGPT 4.0. It allows local file uploads and subsequent data interrogation. I used the Stack Overflow Annual Developer Survey 2023 , tasking the AI with deriving conclusions about AI tool usage among developers.

You can find my interaction with ChatGPT in the attached PDF, with critical steps highlighted in red. Here's a brief overview of what was done:

• Loaded and linked CSV data and metadata.
• Selected relevant data and inquired about variable details.
• Processed data: reorganized a multi-answer variable with data saved as multiple strings into a multi-answer vector.
• Ran frequency tables, a crosstabulation, and an association test.
• Generated bar charts for frequency tables and a stacked bar chart for the crosstabulation.
• Created main insights and used the conversation as content for a LinkedIn article (no, not this one!)

Now, let's look at some initial thoughts.

The Good:

1. ChatGPT mostly executed the tasks correctly. As Andrew Ng suggests, iterating and optimizing prompts gets you the desired output.
2. The AI formulates insights more effectively than many data scientists or market researchers.
3. It's fast, available 24/7, and handles changes without a fuss.

The Bad:

1. You need to know your data well. I asked for unique values for three variables, and the AI got two answers wrong. Later, it produced correct frequency tables and did not realise its answers were contradictory.
2. The same instructions can yield different results, some of which might be incorrect. This is a departure from traditional programming logic.
3. Some technical aspects could be improved, including data loading times, chart saving, system resets, and conversation exporting.

The Ugly:

If you can't verify the AI's answers, don't use it. The AI is swift and potent, but it doesn't think about its answers once it gives them. If it makes a mistake, it won't know unless it encounters a code error.

Conclusion:

AI for quantitative analysis shows promise. However, it's not ready to replace your market research function just yet. Perhaps market research will evolve to facilitate communication with AI. For now, we can view AI as an assistant, generating code faster than scripters in order to access insights quicklier. But remember, an assistant's work always needs a fail-safe. And that fail-safe can be reliably done only by market research consultants.

Here is the slideshow of the interaction with ChatGPT for quant data analysis.