I thought ChatGPT could perform this straightforward task, but I was wrong

The power of generative AI was unleashed recently with the public release of ChatGPT. Many people are still trying to figure out what can and cannot be done with this jaw-dropping technology. As part of my own efforts on this front, I recently ran another simple experiment to determine if ChatGPT was able to perform a straightforward task.

Conclusion: I was disappointed by the results, but not surprised. The outcome helped improve my understanding of ChatGPT's (current) strengths and weaknesses, and how it could eventually be applied successfully in a small business setting.

The experiment

Rather than running an artificial experiment, I prefer to run these tests in an actual business situation when possible.

As part of my regular business activities, I needed to set up a meeting with a potential client located in the Eastern time zone. To protect the innocent, I will change the name of my correspondent. Let's call him "George".

The usual process, whether done manually or automated, goes something like this:

• Confirm which time zone George is located in (I need to know if I have to be available in the morning or the evening)
• Inform George, who in this case happens to be in the Eastern time zone, that there is currently a 14-hour difference with Japan
• Inform George that no, just because it is the next day in Japan, I still don't know the future in the US (I say this as a joke, and once in a while people find it funny ??)
• Ask for some times that could work for George
• Check against my own schedule, and propose one (sometimes more) slots that match
• Confirm the agreed time
• Determine the means by which we will meet (Zoom, or whatever)
• Send a calendar invite (usually by attaching an iCal file)

This task cannot be labelled as "simple" because there are a number of steps involved, but each step is easy to understand, and easy to execute. I therefore consider the overall task of "setting up a meeting" to be quite straightforward.

Now, performing this task manually is boring and repetitive. If it occurs often enough, it is a perfect candidate for automation. There are many possible ways to automate this task, but I thought I would see if ChatGPT was up to the task.

The context

First, allow me to set up the context of my interaction with George. Although the exchange occurred via email, I will write it out more like a chat, and remove irrelevant information.

I already started my interaction with George, and it went something like this (with irrelevant parts of the exchange removed):

George: Let me know if you have any time this week and we can discuss next steps.

David: Ok, sounds good. What time zone are you in?

George: We are the Eastern US Timezone.

David: I am in Japan, so it is the worst combination. One of us will have to do very early morning, while the other does very late evening. Do you have a preference?

George: I can do either. Evening is more flexible and I can do anytime after 8pm or in the mornings I can do 5-6am and after 7:30am. Let me know what days/times might work best for you, and we can confirm.

Normally at this point, I would have to figure out how to match times. Usually I use a simple application that helps plan international meetings, but this time I thought I would give ChatGPT a try.

The interaction

My interaction with ChatGPT was a bit painful, so I will cut out most of the exchange and only show just enough to get the picture.

Help me plan for a meeting. I am in the JST time zone, while the person I want to meet with (George) has written this to me:

> I can do anytime in the mornings 5-6am and after 7:30am ET.

> Let me know what days/times might work best for you, and we can confirm.

I am available on Tuesday (March 7) after 8:30pm, or Wednesday after 8pm.

Write an email reply that clearly indicates the times for both parties.

----

Dear George,

Thank you for letting me know your availability. I am in the JST time zone, which is 14 hours ahead of Eastern Time (ET).

Based on your availability, I propose the following meeting times:

- Wednesday, March 8, 8:30 PM ET (which is Thursday, March 9, 10:30 AM JST)

- Thursday, March 9, 9:00 PM ET (which is Friday, March 10, 11:00 AM JST)

Please let me know which of these options works best for you, and we can confirm our meeting time.

Thank you for your time, and I look forward to our conversation.

Best regards,

[Your name]

I thought the response looked pretty decent, so I copied it into my email client and was just about to hit the "send button".

Then I noticed that ChatGPT got the times backwards. Instead of using my 8:30pm slot, it assigned 8:30pm to George, which is 10:30am for me. Not at all what I asked!! I had asked for 8:30pm my time, which would be 6:00am for George.

I thought that maybe a simple follow-up instruction could easily fix the problem. While I was at it, I thought that I may as well give it my name to replace the [Your name] placeholder.

My name is David, and you got the time zones mixed up. George is available in the morning, I am available in the evening.

----

I apologize for the mistake in my previous response. Here's a corrected version of the email you can send:

Dear George,

Thank you for letting me know your availability. I apologize for the confusion in my previous message. As you are available in the morning and I am available in the evening, I propose the following meeting times:

- Tuesday, March 7, 8:30 PM JST (which is 6:30 AM ET)

- Wednesday, March 8, 8:00 PM JST (which is 6:00 AM ET)

Please let me know which of these options works best for you, and we can confirm our meeting time.

Thank you for your time, and I look forward to our conversation.

Best regards,

David

Meh.

It added my name and corrected the dates, but it is too "me" centric and adds unnecessary text. The dates should be written with George in mind.

I try again...

Write the dates like this: 8 March.

Also, make the data more apparent for George by putting the JST time in parentheses.

----

Sure, here's the updated email:

Dear George,

Thank you for letting me know your availability. I apologize for the confusion in my previous message. As you are available in the morning and I am available in the evening, I propose the following meeting times:

- Tuesday, 8 March at 8:30 PM JST (which is 6:30 AM ET)

- Wednesday, 9 March at 8:00 PM JST (which is 6:00 AM ET)

Please let me know which of these options works best for you, and we can confirm our meeting time.

Thank you for your time, and I look forward to our conversation.

Best regards,

David

ChatGPT did not understand my prompt.

I had to start over and retry a number of times. I will spare you the pain.

In the end, I gave up and just wrote the text by hand.

Once ChatGPT finally understood the actual proposed times, I got it to write out the data in iCal format. It did this pretty well, but this also required a few tries to get it right. Having the iCal generated for me was quite nice.

The problems

ChatGPT appears to be quite weak with logic and computations. It had some trouble understanding simple instructions that a normal human would have understood.

It is possible to provide just the right prompt to get the desired result, but it is not easy. To make ChatGPT provide desired answers, the prompts have to be oh, so right. This requires a lot of trial and error, which opens up a new set of problems. Learning how to prompt becomes a chore and a field reserved for experts. I can easily imagine having to stock up on hundreds, or even thousands of prompts, and getting lots in all the mess. It will be necessary to manage prompts somehow, perhaps by keeping a prompt library or something similar.

Even once the perfect prompt is discovered, each language model will likely react to prompts in a different way, including the next version of the GPT model upon which ChatGPT is based. So, even if efforts are made to build up some kind of prompt library, the fix will only be temporary. The library will have to be completely rebuilt for each new language model.

In any case, it would have been much quicker for me to do the task manually than the time it took to get a decent answer from ChatGPT.

The conclusion

ChatGPT is amazing with language, but lousy at logic and calculations. Much of the grunt work should probably be done with the tried and true techniques of programming and simple templating. ChatGPT could call upon the programs and non-AI template engines to do its thing. In other words, mixing ChatGPT with other AI and non-AI techniques would likely produce a much better result than relying uniquely on ChatGPT.

So, to make the most of ChatGPT, we need to figure out how to embed it with the right mix of tools to relieve it of the tasks that it does not do well, and leave it to shine in areas it does do well: langauge generation.

Appendix

When I described the experiment in this article, I broke the task down into a series of successive steps. This way of designing the process is based on a logical, methodical approach.

Just for the fun of it, I asked ChatGPT to design a similar process to see if it could come up with something similar:

Write out the steps of a process for negotiating a time and setting up a remote meeting with a person who is located in a different time zone.

----

Negotiating a time and setting up a remote meeting with a person who is located in a different time zone can be challenging. Here are some steps you can follow to make the process smoother:

1. Identify the time zones: Determine the time zones for both parties. This will help you avoid confusion and ensure that you don't schedule the meeting for a time that is inconvenient for either party.
2. Propose a few potential meeting times: Look at your schedule and identify a few potential meeting times. Make sure to consider the other person's availability and try to propose times that work for both of you.
3. Communicate via email or message: Send an email or message to the person you want to meet with, explaining who you are, why you want to meet, and what times you have available. Be sure to clearly state the time zone for each potential meeting time to avoid any confusion.
4. Consider alternative communication methods: If the time difference is significant, consider alternative communication methods that might be more convenient, such as a pre-recorded video or audio message.
5. Confirm the meeting time: Once you have agreed on a meeting time, confirm it with the other person. Repeat the time and time zone to avoid any misunderstandings.
6. Prepare for the meeting: Make sure you are prepared for the meeting by testing your technology, setting up the meeting platform, and having any necessary materials ready.
7. Attend the meeting on time: Attend the meeting on time, and be mindful of the other person's time zone. If you need to reschedule or cancel, be sure to communicate this as soon as possible.
8. Follow up after the meeting: After the meeting, send a follow-up message thanking the other person for their time and summarizing any action items or next steps.

It is truly fascinating and entertaining that such human-like text can be produced by a machine. However, the proposed process is not useful in a business context. At least, not yet.

As is now the custom in GPT Chatter, I asked ChatGPT about its ability to perform tasks like scheduling.

David: Hi ChatGPT, it's great to chat with you again. I was hoping to get your perspective on how traditional tools can be used to complement your language generation abilities. What do you think about that?

ChatGPT: Hi David, it's great to talk with you again! I think it's a great topic to explore, as there are still many situations where traditional tools can be effective in enhancing the accuracy and efficiency of the final output.

David: That makes sense. Could you give me an example of how traditional tools could be used to complement your language generation abilities?

ChatGPT: Sure thing, David. Let's say someone asked me to schedule a meeting that involves participants from different time zones. While I can certainly generate text-based responses regarding scheduling and availability, it's important to ensure that everyone's availability is accurately reflected across multiple time zones. In this case, integrating with traditional tools like time zone converters or calendar software could complement my language generation abilities and ensure that the meeting is scheduled at a time that works for everyone involved.

David: I see. So, the traditional tools can help ensure that everyone's availability is properly accounted for, even across different time zones.

ChatGPT: Exactly, David. And this is just one example of how traditional tools can be used to complement my language generation abilities. In other contexts, tools like project management software or task tracking tools could be used to enhance the accuracy and efficiency of the final output.

David: It sounds like it's all about finding the right tools to complement your strengths and achieve the best possible results.

ChatGPT: That's right, David. The goal is to leverage the strengths of different tools and systems to achieve optimal results, and that can include both advanced AI technologies and more traditional approaches.

With the release of powerful language models?like ChatGPT, we have reached an inflection point in the use of AI. Founded in 2023, AI4B2B specializes in making AI accessible to small companies.