Slick, comprehensive and well structured - but can you trust it ? Experiments with ChatGPT

Like many people I’ve been intrigued and fascinated by the recent explosion in AI. Like many people I’ve been exploring ChatGPT which, according to Yahoo News “ fired the starting gun for a "Game of Thrones"-type war between some of the US's biggest listed companies”.

Like many people I’ve been curious to see what ChatGPT can do. So here follows three experiments which I have carried out with my students.?

Experiment 1: social media – Roger’s cat

?

Question Write a tweet about Roger’s cat

?Response

?"Just witnessed pure feline perfection in action! ?? My cat's grace and agility never cease to amaze me. Whether leaping across the room or curling up for a cozy nap, my furry friend brings endless joy to my life. #CatLove #FelinePerfection"

Analysis

At first glance this looks entirely plausible as a social media post. However when you analyse it further you notice that it is formulaic, predictable and bland. But then so is a lot of social media written by humans!

Key learnings

There’s a lot of discussion about AI replacing various jobs. So, if your job is to write formulaic, bland and predictable social media, then you should be worried.

In this case you should ask yourself what?the purpose of the content is, and make it relevant to your target audience. Most of all make it compelling. That's why I’ve always believed in Quality over Quantity when it comes to social media.?

Experiment 2: desk research – CRM Systems

?Question What are the most popular CRM systems worldwide

?Answer

1.?????Salesforce

2.??????Microsoft Dynamics

3.??????HubSpot

4.??????Zoho CRM

5.??????Oracle CX

6.??????SAP Sales Cloud

7.??????Pipedrive

8.??????SugarCRM

?

Analysis

When I did my own research I came up with this list, based on Worldwide sales from trusted research company, Statista.

1.??????Salesforce

2.??????SAP

3.??????Microsoft Dyanmics

4.??????Oracle Siebel

Source: Statista - https://www.statista.com/statistics/972598/crm-applications-vendors-market-share-worldwide/

The key differences were:

• ChatGPT didn’t provide references and did not explain the basis on which it used to decide which are the “leading systems”
• ?In fact, according to Statista, Salesforce, with a market share of 23.8 percent in 2021 worldwide, “leads the market with a big margin”.?And SAP, MS Dynamics and Siebel have only 5% of the market between them
• The way ChatGPT presents the information it implies that all of the various vendors are the same.?

Key learnings

Like any research, AI tools are only as good as the questions you ask. It all depends on what you mean by “most popular”. This could be highest sales (when), most number of installations (over what time period), or highest rated (by what method).

The scary thing is that, when presented with the ChatGPT answer, none of my students raised any of these questions or concerns – they just assumed it must be correct.

?

Experiment 3:?exam question – anti-virus software

I came up with this as a way to liven up revision for FE Digital Technology Computing exams – always a challenge !

First I got the students to work individually on an exam question. Then we asked ChatGPT the same question.

Question: ?Describe two ways that anti-virus software protects computer systems and their data.

Answer

ChatGPT came up with a really detailed answer, focused on:

1.?????Real-time Scanning and Detection

2.?????Regular Updates and Database Maintenance:

Analysis

The students looked at the Mark scheme for this question – which examiners use to decide on marks and critiqued ChatGPT’s response.

This was a great exercise because the comparison helped students develop:

• Critical thinking - again ChatGPT gave no references nor did it explain why it had chosen those two ways and not others. Are those the most important aspects of Anti-virus software?? If not, why not ? What are the other aspects of Ant-virus not covered ?
• Exam techniques – the ChatGPT answer was far too long for that specific question. How could you cut it down to fit the space ?

?

Key learnings

Exam questions normally relate to a specific scenario and are not generic so even if a student used ChatGPT to do some of the spade work, they would still have to ensure that their answer is relevant to the context.

This is actually much the same as using Google to do the same search – when researching information from the web you need to be careful about the source and the provenance of the information.

That's why I found that getting students to critique ChatGPT was a really activity.

Summary –?don’t be taken in by the grace and agility

I’ll let you in on a secret – I don’t actually have a cat.

And to me this illustrates another key thing about use of AI. ?Like my non-existent cat, ChatGPT’s grace and agility can never cease to amaze you. The answers are slick, comprehensive and well structured.

All in all, I think ChatGPT is a really useful and powerful tool. Just - like all information found on the Internet - make sure you assess the answers in a critical way.

I’m a trained Educator and Marketing Consultant.?Contact me?to find out how I can help train your team.?