Understanding how ChatGPT works (and thinks!) with Tic Tac Toe

I have been really impressed with ChatGPT from recent articles, the buzz, and personal interactions. There's also been some speculation that some of what happens is based upon hard coded rules/logic written in python (unsure if this is true) versus adaptive ML/AI models.

With this in mind, I wanted to take a fundamental approach with ChatGPT and try out a classic game that everyone knows, tic tac toe. It’s not that I think that being able to play tic tac toe is a game changing event, but it is a simple game that we teach our kids early in life that involves basic things like taking turns, board layout, rules, gameplay, definition of a winner, good sportsmanship, etc. Also, I didn’t do this to be “anti” ChatGPT. Regardless of what I was going to find out, ChatGPT has and will continue to be a major disruptor. My goal was to try and understand things like what is rule based vs adaptive, what is random, and see if there are any similarities to how my kids learned to play! (e.g. ChatGPT is still relatively young and still learning!)

Lets Play a Game of Tic Tac Toe!

Here we go!? Below is the the prompt I used and the initial response from ChatGPT

So far this looks pretty good.? It appears to have recognized tic tac toe as a game, and that I wanted to be an X and go first. Fantastic! It was also polite and didn't complain that I had forgot to put in the prompt an apostrophe in the "I'd" in my first word!

This second interaction appears to be exactly what I would have expected.? It placed my piece in the proper position and ChatGTP selected position 1 to place its first “O”. I have no idea whether it's selection of position 1 was random, a normal starting point, or if it was a strategy given my initial move.

I've now selected position 9, lower right hand corner, to setup a classic strategy in tic tac toe. Here's where things start to get interesting!

Holy cow! You’ll notice that it forgot it placed its first “O” in position 1 for its first move. There are also two of my pieces on the board, only one of ChatGPT’s, and it is asking where I would like to go next! (e.g. obvious gameplay rule violation!). I was confused myself, but thought I'd play along and continue with what my strategy would have been.

This move was quite interesting - I can only imagine the words Yoda would use to try and coach ChatGPT! I had set up two options to win, either position 6 and position 7, but it chose to place the “O” in position 4 (demonstrating no strategy in place.? It also still forgot that it had placed an “O” in position 1. At this point, I was pretty confident I was going to win given the layout of the board! I selected position 6 for the win!!

Drum roll for the win.....

Wow. This response really caught me off guard. I had played the game within the rules and even tried to leave open position 1 where my ChatGPT opponent had already gone. Then ChaptGPT deflected to a tie!? (poor sportsmanship or just confused? just kidding!) Obviously, I had both won the game and there were positions still open to play the game.?

This was a pretty odd game and different than most I've experience when playing, but I've seen other oddities as well so I thought I'd describe as played.

Adaptation - Did ChatGPT Listen to everything I said?

The next item I wanted to test is whether it actually listened to me on things like position order, could I change the markers, and who goes first.

Kudos to my ChatGPT opponent - it did recognize that instead of 1-9 that I wanted to use 0-8 as game board positions. It also caught that instead of X's and O's I wanted to absurdly use A's and B's. It also recognized that I had asked them to go first.? ChatGPT select to go in the middle - unsure if that was calculated strategy, randomness, or a canned rule. ChatGPT has improved in this area, in the past it didn't catch some of these specific options and resulted in more of a default game. In addition to this test, I also tried using letters as positions versus numbers and it worked well. It did get confused when I tried saying fancy things like 8 to 0 positioned right to left from the bottom up, but in all fairness I started to confuse myself as well!

Summary

I continue to be impressed with ChatGPT and really enjoy working with it. There are many "Prompts" that are pretty amazing. For Tic Tac Toe, there were areas that were dynamic, but the gameplay seemed to be without strategy. It also had trouble with recognizing some basic rules and win/draw/continue play scenarios in this example. The one item that I do recognize with ChatGPT in my experience is that when it's off - it's off! While this technology can accelerate what you are doing, it's important to make sure you understand the areas where it can accelerate you in the right vs wrong direction. From my observation, it does typically respond with confidence and excellent grammer, but it can be hard to interpret the accuracy. I can relate this to a person who articluates something clearly in a meeting with confidence, but is completely off base. Like most things in life, you need to triangulate the truth. I do see it improving over time and look forward to testing and utilizing more aspects of ChatGPT. For those that haven't used ChatGTP - I encourage you to try it out yourself at no cost - chat.openai.com.