An Experiment with ChatGPT

As I get older, I have become more aware of my limitations with technology. I have seen how my younger colleagues are often better able to use new technologies to their advantage. So when it became clear that ChatGPT is likely to be the “next big thing,” I wanted to make sure that I was on top of this new tool, rather than lagging behind.

I decided to do a little experiment to learn about ChatGPT and better understand the question: How can I use ChatGPT to improve my work?

First, a little disclaimer: I am not an expert in ChatGPT. I only explored the free version of ChatGPT without any additional programming or software integration. Things change rapidly with technology, and I imagine this whole article will be outdated in a matter of months if not days. Additionally, there are ethical concerns with using AI software, which I don't address in this article.

But now, on to my experiment…

First, I needed something concrete that I could use to compare my own work vs ChatGPT’s work. As part of a recent project, I developed a Diversity, Equity, and Inclusion (DEI) audit checklist – a tool that could assist me in evaluating organizational policies from the lens of DEI. Could I use ChatGPT to re-create this checklist for me, in a more efficient way and to the same level of quality as my original checklist, or even better?

Let's find out! I signed up for ChatGPT, and entered my first prompt. The answers I got at first were... underwhelming. I thought, 'Ok, maybe I should enter a slightly different prompt.' So I tweaked my question, and the response was closer to what I wanted, but still not great. So I entered another prompt, and another, and another... I kept playing with it and fine-tuning my question. When I saw a response was lacking something specific, I asked ChatGPT again, but made it clear in the question that I wanted ChatGPT to address the point that was lacking. With every question I learned more and got closer to what I wanted.

2+ hours later, I can say that I went on a journey from excitement and curiosity, to confusion and frustration, and ended with (some level of) understanding.

Below is a side-by-side comparison of a ChatGPT-created checklist compared to my human-made checklist. This is focused specifically on DEI in compensation and benefits policies. This was the closest I came to being able to re-create part of my original work through ChatGPT. Note that the ChatGPT list is the complete list it provided after 2 hours of fine-tuning, whereas my list is only a small portion of my full checklist on this topic.

So, what are my main conclusions after this experiment?

1.??????ChatGPT can be a great resource to increase work efficiency, but not necessarily quality (at least not yet). While I couldn’t get ChatGPT to create a DEI checklist to the extent that I’d be satisfied with it, it created something that I could have used as a decent starting point. It could have potentially helped me to brainstorm a first draft quicker than I could on my own. In the future, though, ChatGPT may be able to produce higher quality work, as it learns every day and can be integrated into other programs.

2.??????Using ChatGPT effectively requires learning what works and what doesn’t. Fine-tuning my prompts to ChatGPT and asking follow-up questions gave me much better results than my first question alone. I’ve already learned a bit about how to get the answers I’m seeking from ChatGPT, but I’m sure that the more I use it the better I’ll become.

3.??????ChatGPT is limited in its ability to be forward-thinking. ChatGPT is trained on available data, which means it is limited by what is already out there on the internet. This was clear to me, as most of the information ChatGPT produced was information that I could have easily found online with a Google search. In a field like DEI that is changing quickly, this means that the information ChatGPT provided was already outdated in some ways.

I started this experiment thinking about ChatGPT as a robot that could replace human intellect. But I think a better comparison is to a Google search. Like Google, ChatGPT can be an incredibly useful tool. Being able to use it may become an essential work skill. (Can you imagine working without Google today?) And like Google, using ChatGPT requires some experience to use it most effectively. (Have you ever seen someone type a search into Google that you just knew wasn’t going to give the best results?) The question should not be, "What can ChatGPT create?" but "What can I create with ChatGPT?"

So, will I use ChatGPT in my work? Absolutely. But when I use ChatGPT in the future, I won’t see it as a way to produce an end result, but as a tool to help with the starting point. It can get me maybe 10% of the way, but adding my (human) interpretation, expertise, and knowledge is 90% of the work. At least for now.

If you want to read more about ChatGPT and how you might be able to use it, here are a few articles I recommend:

• Chat GPT 101 for HR Pros (SHRM)
• Four Ways ChatGPT Could Help Level the Humanitarian Playing Field (The New Humanitarian)