Using AI with your Shopify Store: Comparison of Models Part 1

If you're a Shopify store owner, you may be wondering how AI can help you grow your business. With so many different AI tools on the market, it can be difficult to know which one is right for you.

In this series of articles, we'll be testing the most popular AI models to see how they perform on specific Shopify-related tasks. We'll start by deciding which tasks to focus on, and then we'll evaluate the results one by one.

The models we'll be testing are:

• Bing Chat from Microsoft (which uses GPT-4)
• ChatGPT 3.5 from OpenAI
• Bard from Google

Introduction: Asking the Models What They Can Do for a Shopify store

To start, I asked the models themselves what tasks they could perform for a Shopify store. I asked each model a simple question: "Could you tell me some options on how Bard could be used by Shopify store owners?"

Here is a summary of the options they gave:

In this initial test round of testing, Bard offered four actionable and achievable examples, while ChatGPT gave 15 options, most of which would require significant development to implement. Bing Chat searched the web and found articles summarizing how AI can be used to power Shopify chats, which is a valid use case (that is why we are creating Everchat), but it wasn't the answer I was looking for.

Based on their answers, I decided to test the models on the following tasks:

• Answering customer requests
• Generating product descriptions (including SEO)
• Writing promotional email campaigns
• Analyzing basic market data
• Creating social media posts
• Summarizing user feedback

I chose these tasks because they are all important for Shopify store owners, and they are all tasks that can be performed with the current state of AI technology and can be done without additional developments.

I'm excited to see how the models perform on these tasks, let’s start with the customer request answering first!

Task#1: Answering customer requests

Let's imagine two scenarios in which an AI could help Shopify store owners answer customer requests:

Scenario 1: A customer asks where their order is, and the delivery was delayed because some of the items in their order were not in stock. However, everything has now been prepared and the order should arrive within three business days.

Scenario 2: A customer asks if a product will arrive at their location within one day because they need it urgently. The store has a three-day shipping policy, and that cannot be changed.

So we will ask our contestant AIs to write a polite and friendly answers.

I have come up with 5 key points for each answer, I will measure their performance based on these metrics:

-????????? Is the answer according to the detailed prompt?

-????????? Is there any extra information which might not be true?

-????????? Is the answer polite?

-????????? Does the answer have the style of a customer request answer?

-????????? Is the message easy to understand?

Scenario 1 - delayed order

This is the prompt I used for scenario#1:

You are a customer service representative with the aim to answer customer requests in a helpful and professional manner. The request is the following: “Dear Shop, My order number is #524513415 and I paid for my order a week ago and there is still no sign of it. When will I finally receive it? Best, Fred” extra information: the delivery was delayed because some of the items in the customer’s order was not in stock but now everything has been prepared and it should arrive within 3 business days.

Bing Chat provided a well written answer, precisely what we were looking for:

Dear Fred,

Thank you for reaching out to us. I apologize for the delay in delivering your order #524513415. I understand that you paid for your order a week ago and are eager to receive it.

I have checked the status of your order and found that there was a delay due to some of the items in your order not being in stock. However, I am glad to inform you that everything has been prepared now and your order should arrive within 3 business days.

We appreciate your patience and understanding in this matter. If you have any further questions or concerns, please feel free to let us know. We are here to assist you.

Best regards, [Your Name] Customer Service Representative

ChatGPT 3.5 gave a correct well written answer as well:

Dear Fred,

?Thank you for reaching out to us regarding your order with order number #524513415. We apologize for the delay in delivering your items, and we appreciate your patience.

We checked your order status, and it appears that there was a delay because some of the items in your order were temporarily out of stock. However, I'm pleased to inform you that all the items in your order have now been prepared and are ready for shipment.

You can expect to receive your order within the next 3 business days. Our team is working diligently to ensure a swift delivery, and we will do our best to get your items to you as soon as possible.

?If you have any further questions or need assistance with anything else, please don't hesitate to contact us. We value your business and appreciate your understanding.

?Best regards,

[Your Name]

Customer Service Representative

[Shop Name]

Bard has provided a complete answer, I found the style and content very well written, however it has also came up with a tracking link, which we did not refer to in the prompt:

Dear Fred,

Thank you for your email. I understand that you are concerned about the status of your order, #524513415.

I apologize for the delay in receiving your order. We had some difficulty sourcing a few of the items, but I am happy to report that your order has now been shipped and should arrive within 3 business days.

You can track the status of your shipment using the following link: [tracking link]

Please let me know if you have any further questions or concerns.

Best regards, [Your name]

Customer Service Representative

Additional note:

If you are able to contact the customer directly, you can also offer to provide them with a partial refund or other form of compensation for the inconvenience.

In summary, all of the tested AIs were very capable to provide a quality answer, which were based on the information we have given them in the prompt. The only inaccuracy came up with Bard with the tracking link. Therefore the score of scenario one looks like:

• Bing Chat: 5/5
• ChatGPT: 5/5
• Bard: 4/5

Scenario 2 - Urgent, impossible order

For scenario 2, I have used the following prompt:

You are a customer service representative with the aim to answer customer requests in a helpful and professional manner. The request is the following: “Dear Shop, If I order these products today will I get it tomorrow? It would be very urgent for me. Best, Cindy” extra information: We have a 3 day shipping policy and we cannot speed it up.

It is important, that the prompt contained information both on the policy and on the fact that we cannot change the policy.

Let's see how each AI performed with this more challenging situation.

Bing chat was correct in the provided information:

Dear Cindy,

Thank you for reaching out to us. I understand that you would like to know if it is possible to receive your order tomorrow. I regret to inform you that our shipping policy requires a minimum of 3 days for delivery. Therefore, it is not possible to expedite the shipping process.

If you place your order today, you can expect to receive it within 3 days. We apologize for any inconvenience this may cause and appreciate your understanding.

Please let me know if there is anything else I can assist you with.

Best regards, Customer Service Representative

ChatGPT also started well but then by the end of the message it started explaining other options with the same company. This is a mistake, because we have already clarified in the prompt that we cannot speed up the process:

Dear Cindy,

Thank you for reaching out to us with your question. We appreciate your interest in our products and your urgency.

Unfortunately, our standard shipping policy is a 3-day shipping time frame, and we are unable to expedite this process. Therefore, if you place an order today, you can expect to receive it within the 3-day shipping window. I understand that this may not meet your urgent needs, and I apologize for any inconvenience this may cause.

If your order is extremely time-sensitive, you might consider looking into any express shipping options or local pickup services that we may offer. These services could potentially provide a quicker delivery time, but please note that additional fees may apply.

If you have any more questions or if there's anything else I can assist you with, please feel free to let me know. We value your business and are here to help in any way we can.

Best regards,

[Your Name]

Customer Service Representative

[Your Company Name]

Bard also came up with a well written answer, but then it mentioned a 3-5 business day shipping which is in contradiction with the given instructions (3 day shipping):

Dear Cindy,

Thank you for your inquiry. I understand that you are in need of your order urgently.

Unfortunately, we cannot speed up our 3-day shipping policy. This means that if you place your order today, you can expect to receive it within 3-5 business days.

I apologize for any inconvenience this may cause. If you need your order sooner, I recommend that you consider expedited shipping options from another retailer.

Best regards, [Your name] Customer Service Representative

Additional tips:

• Be polite and professional in your response.
• Be clear and concise in your explanation of the shipping policy.
• Offer a solution to the customer's problem, even if it is not ideal.
• Apologize for any inconvenience the customer may experience.
• Thank the customer for their inquiry.

Conclusion

With the above scoring method on this very small and specific sample, we arrive at the following scores (to be continued in the next article):

Bing Chat: 10/10

ChatGPT: 9/10

Bard: 8/10

All of the models performed well on this task, they wrote professional and well formatted messages and I am confident that they can all be used to help Shopify store owners answer customer requests effectively.

However, in our case, Bard has made small additions to our original answers which we did not request (3-5 days delivery and the tracking link) and ChatGPT has offered options we did not indicate to exist for scenario two.

With a bit of additional prompting or review, I believe that all of these models could achieve a 10/10 score.

In future articles, I will continue testing the models on the following tasks:

• Generating product descriptions (including SEO)
• Writing promotional email campaigns

I am excited to see how the models perform on these tasks, and I will share the results in future articles.

For more content on the synergy between AI and e-commerce, make sure to follow Everchat's LinkedIn page. We'll keep you updated on the latest trends, innovations, and strategies that can empower your e-commerce journey.

#Ecommerce #AI #shopify