Harnessing the Potential of Noteable and ChatGPT, PT3

Welcome back, everyone! Let's dive into part three of our exciting journey exploring ChatGPT and Noteable. We're rolling up our sleeves, ready to learn from our past articles and uncover insights that can help both the company and the parents whose kids use this app. The most interesting part? The main voices behind the feedback are the kids themselves!

Our journey starts by setting up our project on Noteable. This platform lets us make and run Python notebooks easily. With a little help from our friend, ChatGPT, we dig into our CSV file filled with Roblox reviews. This step helps us understand the structure of our data and gets us ready for the exciting analysis ahead.

We quickly notice that most comments are really positive, scoring a fantastic 5. We've gathered a total of 398 of such comments, all from June 2023. So, it's clear we can't just focus on the less positive scores since about half of our sample falls into this top score category.

Next, we want to understand how people feel about the game. With ChatGPT's help, we categorize each review as negative, neutral, or positive. This helps us see the overall mood of the reviews.

Even though we might have expected more negative comments, there aren't that many. Interesting, right? Let's find out what the kids have to say about this game and why they're so happy about it.

To dig deeper, we run a coherence test, which helps us identify the main topics in the reviews. We use these topics to build Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) models. By creating bar charts, we can visualize the distribution of topics in the reviews.

To further understand how the topics relate to each other, we create scatter plots for each model. These show us how the topics are grouped and how they relate to each other.

Even though the topics are pretty spread out, we turn our attention to the NMF model to dig even deeper.?

Surprisingly, the topics appear to be well spread out, with the major category being the Game Experience. But what does T-SNE reveal?

Both the UMAP and T-SNE models show us that the topics are well spread out. We know what the topics are, but the charts don't give us clear answers yet.

Lets the NMF with T-SNE how it looks if we add the sentiment color.

While most reviews are positive, there are still quite a few negative ones. To get a clearer picture, we decide to look at the data through stacked bar charts.

This helps us see that the main topic with negative sentiment is Game Experience in NMF which is also represented in the scatter plot, but so what? The quantity of comments still is not representative enough for a strong case so let's try something new. We then ask ChatGPT to select random comments to summarize each topic and this is the output.

Usually, that task can be done in the API, and depending on the model for reading actual text it will come with a cost, of course, I am not referring to the embeddings of ADA but parsing each comment to Davinci or chatGPT, so for testing I asked chatGPT, to select up to 10 random comments from each topic and to return a summary of feedback and actions that can help parents and Roblox, below the output from chatGPT.

From this analysis, we learned several important things:

1. Game Experience: Players have different feelings about the game. If we improve the game mechanics and fix technical issues, we could make the game even better.
2. Robux and In-Game Purchases: Some players don't like the cost of Robux or feel they need to buy too many things in the game. We could look at the pricing for Robux to address these concerns.
3. Technical Issues: Some players reported issues with lag, errors, and hackers. By improving the game's technical side and dealing with hackers, we could make the game more enjoyable.
4. Cyberbullying and Inappropriate Behavior: Some players reported bullying and inappropriate behavior. We could use stricter rules and provide more resources for reporting bullying to make the game a safer place.

After analysis, getting prompts, tables, text, chatGPT was able to come up with a summary of recommendations by asking to do a random sampling validation analysis.

In conclusion, our deep dive into Roblox reviews using ChatGPT and Noteable gave us important insights that can help improve the game, plugin usability and limitations. Both ChatGPT and Noteable were pretty easy to use. But, for someone who's new to topic?modeling, it might be a bit challenging, and they might even lose some of their progress.

I'd give the app a 5/10 for new users and 9/10 for users who are comfortable debugging code. In my future articles, I'll explain these scores. So, if you enjoyed this analysis and would like to learn more, stay tuned! We have more exciting deep dives planned, including a closer look at the plugins for ChatGPT.