How do I know if my AI information is correct?

Unless you're living under a rock, you've probably heard a thing or two about all the advances in AI over recent months. ChatGPT is dominating the conversation but they aren't the only game in town. For the purposes of this newsletter, thought, we'll pick on ChatGPT - just a little.

Reminder, tools like ChatGPT have a lot of great potential and we're already seeing the disruption of how we work. I'm a big fan of these tools and I think that they will help us refocus on really important work, while helping to automate more of the repetitive pieces. Keep in mind that this is not a post hating on ChatGPT. This is a remind to you, the end user, to use your noodle.

Since it has become available, the majority of users look at ChatGPT as a way to research and write content. Ever the skeptic, my first question is, "How do you know if the information is correct?"

The short answer is that you don't. Just like looking up something on the internet, taking the results at face value can be dangerous. Unless you're using GPT-4 through the Bing interface, it otherwise doesn't cite its source of information.

Chris told me that he was using ChatGPT to help him write code. I asked him how he knew if the code the system was writing was correct. He said that he had to go back through it and make sure it was what he intended. I could ask ChatGPT to write code too. But because I'm not a developer I would have no idea if it was correct and if I take it at face value it could be a risk to my project and my business.

When you start extending that scenario into how quickly ChatGPT can create content, the risks of misinformation start to get even higher. Some companies are using these systems to churn out thousands of pieces of content without doing a thorough fact check of each piece.

Here's a simple example. We asked ChatGPT: What do you know about Katie Robbert?

This is what it told us.

There is a person named Katie Robbert who is the CEO of Trust Insights, a marketing analytics and data consulting company. She is also a professor at Rutgers University and has been a keynote speaker at various conferences on topics related to marketing, analytics, and data science. Additionally, she co-hosts the podcast "Punch Out With Katie and Kerry" where she and her co-host discuss various topics related to marketing, business, and technology.

If you don't know me that well, you'd think this was a reasonable response. Well, friend, the devil is in the details. This is about 90% correct. However, I do not at this time, have any affiliation with Rutgers University. (I am however open to the possibility).

The point is, if you don't read carefully or aren't a subject matter expert you could miss the incorrect information.

When we recorded our podcast, In Ear Insights, the other day, we were walking through developing prompts. We decided to use "SEO in 2023" as the topic. We built the prompt, ran the query, and ChatGPT gave us a first draft of tactics that will be relevant for SEO in 2023. It said things like, video optimization and voice search.

To me, this was helpful advice. However, when Chris read it, he said that the advice was about five years out of date. The challenge is that I'm not a subject matter expert in SEO so I would have read this draft and thought it was good enough.

The examples I'm giving are pretty low risk. What happens when you start asking for content that involves medical advice or other high risk topics? There has to be a way to know if the information is correct, right?

Right.

Do your homework. Before you ask a tool like ChatGPT to write a post or develop code for you, make sure you know all the facts. In the example above where we ask the system to write a post on SEO in 2023, my first mistake was not being up to date on the latest SEO tactics. The way to get around the misinformation is to include the relevant facts in your prompt before asking the system to write for you. You can be as specific as you need to be. ChatGPT (on its own) isn't a search engine, you shouldn't be using it as one. Use it to rewrite your content, to summarize notes, to assist you, the expert.

This is the old, "Will AI take my job?" question. In this instance, if you're a subject matter expert, the answer is no. If you know how to research a topic, the answer is no. If you know how to fact check, the answer is no. Humans need to, and continue to be, in charge. Not the other way around.

Are you using ChatGPT? Join the conversation in our Free Slack Group, Analytics for Marketers