Analyze Customer Product Reviews Using ChatGPT OpenAi API: A Step-by-Step Guide To Extracting Voice of the Customer VOC Business Insights Part 1

NOTE: I originally published this blog on startupstash.com, but that site is having trouble with its DNS, so I'm reposting it here with the full source code. This condensed version of the article uses the ChatGPT / GPT-3.5 TURBO API to conduct voice of the customer (VoC) and sentiment analysis on product reviews. The upcoming revised version of the article uses the new GPT-4 API and extends the use case. Stay tuned!

It seems everyone has gone crazy for AI chatbots recently, especially OpenAI’s ChatGPT. Although there are many interesting ways to use ChatGPT to brainstorm and explore ideas, how can you start using it today to generate real business value? One way is to use ChatGPT to analyze product reviews (either for your own product or competitors) to understand: customer sentiment about the product, perceived pros and cons and how the product could be improved to better suit customer needs. This used to be a tedious, machine learning natural language processing (NLP) task that required the collection of vast amounts of training data and careful pre-processing of the text. ChatGPT has changed everything, allowing you to generate insightful, actionable recommendations based on your product’s reviews without the need to hire a data scientist. Sound interesting? In this series of articles, I’ll provide a walkthrough of how you can achieve the same results using your own data with nothing more than a free OpenAI account and a free Google Colab account. The ChatGPT API allows you to summarize customer reviews using generative pre-trained transformer (GPT) technology, allowing you to extract insights from customer reviews quickly and accurately. Let’s look at how it’s done!

What you need to know to use ChatGPT

ChatGPT and now GPT-4 are a type of artificial intelligence neural network called a large language model (LLM) that excel at natural language processing and other advanced tasks with human-like ability. GPT stands for “generative pre-trained transformer” indicating that it can generate text responses based on input and keyword commands (called a prompt) from a user. It was developed by openai and available through a chatbot interface, or as we will see in this tutorial, as a far more powerful Application Programming Interface (API). Although that may sound technical, using an API is not difficult, you only need to obtain a free API key then provide the app with the required input data; all the AI magic will happen behind the scenes, outputting the results in a few seconds.

How does ChatGPT work?

ChatGPT is an AI-powered conversational technology that allows users to have conversations with an AI system. It works by using natural language processing (NLP) algorithms to understand the user’s input and then performs the corresponding tasks. In this tutorial, we’ll ask it to detect the sentiment of customer reviews and also summarize lengthy reviews. ChatGPT has been trained on vast amounts of knowledge and data, enabling it to respond to a wide range of queries and questions. The more interactions that take place, the more ChatGPT learns and improves its responses. This is known as machine learning. The system’s flexibility enables organizations and individuals to tailor its functionality to meet their specific needs. In summary, ChatGPT provides an easy way for non-programmers to access powerful AI natural language techniques through it’s chatbot or API interface, which fundamentally changes the types of data analysis that can be done.

How using ChatGPT to analyze customer feedback data can help your company

Using ChatGPT to analyze customer feedback and improve the customer experience can definitely benefit your company. By using AI models, ChatGPT can help your company create a better experience for your customers by finally making use of the massive amounts of data that your company has collected but never put to good use. By leveraging your existing company data and mining it for insights, you can increase customer loyalty, improve customer retention, and increase revenue. The ChatGPT API makes it possible for your company to analyze customer feedback with accuracy and speed. By improving the quality of your company’s data mining efforts, you can make faster and more effective data-driven decisions.

ChatGPT excels at language generation tasks using artificial intelligence

ChatGPT is a remarkable language generation tool that excels at generating human-like text, making it ideal for content creation tasks and other narrative-style reports that require logic and creativity. Its ability to generate text that is similar to that of a human makes it useful in generating engaging and diverse content for blogs, social media, and websites. ChatGPT’s text completion feature allows users to input just a few keywords and then have the AI automatically generate complete sentences or paragraphs based on those keywords. This makes the content creation process easier and less time-consuming, enabling your business to generate content quickly and efficiently. After first using the ChatGPT API to analyze our reviews in this tutorial, we can then leverage ChatGPT’s text generation capabilities to create a product improvement strategy that consists of a detailed list of product pros and cons and suggested product improvements ranked by importance and ease of implementation in the following tutorials.

AI-powered Analytics in Action: Extract Insights From Customer Product Reviews using sentiment analysis with the openAI ChatGPT API

Why sentiment analysis?

Sentiment analysis is becoming increasingly important in today’s world where people freely and easily express their opinions online. Sentiment analysis involves analyzing written text and then categorizing it as positive, negative or neutral in tone. This technique can provide a marketer with valuable insights into consumer behavior and preferences, which can then be used by businesses to improve products and services, improve the customer experience or provide better customer support. Previously, you would have had to conduct surveys or customer focus groups to try and learn what your customers were thinking, hoping that the process of collecting such data didn’t bias the opinions one way or another. Now, you can leverage what people have written online to quickly understand how your product is perceived. Sentiment analysis can also be used to monitor social media platforms for public opinion on certain topics, events, products or even competitors for business use.

Why use openai API instead of the ChatGPT chatbot interface?

By using the openAI API, you can automate the tedious task of cutting and pasting each review into ChatGPT. With a single command in Python, you can instruct ChatGPT to analyze customer reviews and determine the sentiment of each one. Once that is done, we can use chatgpt to generate the results on the screen and also save them to Excel and Word for safekeeping and further discussion with your team. The code will also give you a percentage breakdown of how many reviews were positive, negative or neutral. The API also allows us to take the output of one part of our analysis (i.e. Part 1 in this tutorial) and use it as input into the next step (i.e. Part 2, 3 and 4 in the next tutorials), eventually creating an insightful product improvement analysis in plain English that would be difficult, if not impossible, to do with only the chatbot interface.

Step-by-Step Machine Learning Sentiment Analysis

Assumptions

1. To use the openai API, you need to have an API access key. If you don’t have one already, follow these steps to create a free account for 3 months
2. You have a free Google Colab account

Step 1: Install the required Python libraries in Google Colab

# Used to access the openai API and send requests to it
!pip install pandas openai requests
# Used to create a progress tracker while the API calls are being made
!pip install tqdm
# Used to output the results to Word format
!pip install python-docx        

Step 2: Prepare the openAI API environment in Colab

1. Replace the section that says <REPLACE THIS TEXT WITH YOUR OPENAI API ACCESS KEY> with your private openAI API access key. Make sure to leave the “ ” surrounding your API access key.
2. We are going to use the chat/completions API endpoint rather than the older gpt-3 endpoints to ensure we are using the latest chatGPT version of the software, which is also cheaper to use than the older gpt-3 API.

import os
import pandas as pd
import openai
import requests
from tqdm import tqdm
import time
import docx

# Set up the OpenAI API
openai.api_key = "<REPLACE THIS TEXT WITH YOUR OPENAI API ACCESS KEY>"
GPT_API_URL = "https://api.openai.com/v1/chat/completions"        

Step 3: Load your review dataset

Here we are assuming that the reviews are in a CSV file named “reviews.csv”. The reviews are contained in a single column called “Product_Review” with one review per line.

In this example, we are using sample product reviews from a business, presumably our own. However, you could also use product reviews for competing products or services to understand how your competitor’s products are perceived by users.

We will print out the dataframe to make sure everything is loaded correctly.

input_file = "reviews.csv"
df = pd.read_csv(input_file)
df        

Step 4: Determine the sentiment of each product review using ChatGPT, output the results to Excel and Word

Note: If you are using a free trial account with openAI, they limit the number of times you can send data to the API per minute. To get around this limitation, we introduce a few second delay in the code between each request. If you are using a pay-as-you-go openAI account, you can remove this delay timer from the code below, time.sleep(4)

The openAI API sometimes encounters errors or becomes overwhelmed with requests from other users. To prevent the code from failing when this happens, we introduce a while clause in the code that will retry the API call 3 times; this is usually enough.

You can see we are using gpt-3.5-turbo, which is what openAI recommends at this time as the fastest, cheapest and most capable model for this type of analysis.

This is the prompt (aka the command) we are giving to chatGPT that will instruct it to act as a product sentiment analyzer and determine whether a review is positive, negative or neutral.

You are an AI language model trained to analyze and detect the sentiment of product reviews.
Analyze the following product review and determine if the sentiment is: positive, negative or neutral. Returnonly a single word, either POSITIVE, NEGATIVE or NEUTRAL        

def analyze_review(review):
    retries = 3
    sentiment = None

    while retries > 0:
        messages = [
            {"role": "system", "content": "You are an AI language model trained to analyze and detect the sentiment of product reviews."},
            {"role": "user", "content": f"Analyze the following product review and determine if the sentiment is: positive, negative or neutral. Return only a single word, either POSITIVE, NEGATIVE or NEUTRAL: {review}"}
        ]

        completion = openai.ChatCompletion.create(
            model="gpt-3.5-turbo",
            messages=messages,
            max_tokens=3,
            n=1,
            stop=None,
            temperature=0
        )

        response_text = completion.choices[0].message.content
        print(response_text)
        if response_text in ["POSITIVE", "NEGATIVE", "NEUTRAL"]:
            sentiment = response_text
            break
        else:
            retries -= 1
            time.sleep(1)
    else:
        sentiment = "neutral"

    retries = 3
   
    # add a delay of 4 seconds between requests to avoid hitting the openai free tier API call rate limit

    time.sleep(4)

    return sentiment

# Read the input Excel file containing user reviews
input_file = "reviews.csv"
df = pd.read_csv(input_file)

# Analyze the reviews and store the results
sentiments = []

for review in tqdm(df["Product_Review"], desc="Processing reviews"):
    sentiment = analyze_review(review)
    sentiments.append(sentiment)

df["sentiment"] = sentiments

# Save the results to a new Excel file
output_file = "reviews_analyzed_full_sentiment.xlsx"
df.to_excel(output_file, index=False)


# Save the results to a new Word file
output_file = "reviews_analyzed_full_sentiment.docx"
doc = docx.Document()

# Add table with headers
table = doc.add_table(rows=1, cols=2)
header_cells = table.rows[0].cells
header_cells[0].text = 'Product_Review'
header_cells[1].text = 'Sentiment'

# Add table content
for index, row in df.iterrows():
    row_cells = table.add_row().cells
    row_cells[0].text = str(row['Product_Review'])
    row_cells[1].text = row['sentiment']

doc.save(output_file)        

Step 5: Summarize each review using ChatGPT, output the results to Excel and Word

Note: Similar to the previous code, we introduce a 4-second delay between API calls to avoid exceeding the free trial account limitations for API calls. You can remove the time.sleep(4) line if you have a paid openAI account.

This is the prompt we are using to tell chatGPT to summarize the product reviews for us.

You are an AI language model trained to analyze and summarize product reviews.
Summarize the following product review, highlighting pros and cons        

def analyze_review(review):
    retries = 3
    summary = None

    while retries > 0:
        messages = [
            {"role": "system", "content": "You are an AI language model trained to analyze and summarize product reviews."},
            {"role": "user", "content": f"Summarize the following product review, highlighting pros and cons: {review}"}
        ]

        completion2 = openai.ChatCompletion.create(
            model="gpt-3.5-turbo",
            messages=messages,
            max_tokens=100,
            n=1,
            stop=None,
            temperature=0.8
        )

        response_text = completion2.choices[0].message.content
        print(response_text)
        if response_text:
            summary = response_text
            break
        else:
            retries -= 1
            time.sleep(1)
    else:
        summary = "Summary not available."

    # add a delay of 4 seconds between requests to avoid hitting the openai free tier API call rate limit

    time.sleep(4)

    return summary

# Read the input Excel file containing user reviews
input_file = "reviews.csv"
df = pd.read_csv(input_file)

# Analyze the reviews and store the results
summaries = []

for review in tqdm(df["Product_Review"], desc="Processing reviews"):
    summary = analyze_review(review)
    summaries.append(summary)

df["summary"] = summaries

# Save the results to a new Excel file
output_file = "reviews_analyzed_full_summaries.xlsx"
df.to_excel(output_file, index=False)


# Save the results to a new Word file

output_file = "reviews_analyzed_full_summaries.docx"
doc = docx.Document()

# Add table with headers
table = doc.add_table(rows=1, cols=2)
header_cells = table.rows[0].cells
header_cells[0].text = 'Product_Review'
header_cells[1].text = 'Summary'

# Add table content
for index, row in df.iterrows():
    row_cells = table.add_row().cells
    row_cells[0].text = str(row['Product_Review'])
    row_cells[1].text = row['summary']

doc.save(output_file)        

Step 6: Continue to Part 2 where we will automatically generate a list of product pros and cons from the user reviews

Here’s the link to Part 2 where we dive into creating the pros and cons list and derive a list of prioritized product improvement suggestions.

Conclusion

I hope you found this tutorial helpful and I’m happy to answer any questions.