Supersize Me: Extreme Programming with ChatGPT!

Everybody has seen the movie Supersize Me, about a very healthy man who decides eat only at McDonalds for a month. It didn't work out well.

As an IT guy I decided to do something very similar: extreme programming with ChatGPT. The goal was to create a fairly complex client/server web application (CRUD), using the technologies Python/FastAPI, Vue.js and the PostgreSQL database. But here's the rub: I’ve decided to let ChatGPT do 100% of the coding, so my role is restricted 100% to prompts, putting the code into Visual Studio Code, and testing. Even in cases where I could easily do something by hand, I asked ChatGPT to do it.?

ChatGPT and I carried out this job over the last four days, probably spending around 5 hours per day.

Now that I'm pretty much done, I wanted to share my thoughts.

What’s worked well

1. ChatGPT exceeds where I want and need the help. I’m not a full-time application developer, so I have no desire to fill my brain with things I will never remember, like HTML and CSS and stuff like that. ChatGPT has done a great job: “Can you please make the box a bit longer and a slightly brighter shade of green?”
2. ChatGPT doesn’t just give me the code; it explains to me what’s happening and highlights important topics. This is important for me, since me acting as “dumb scribe” together with ChatGPT does not work; see the next section for bad examples! This means I need to have as deep a knowledge of the code as possible.
3. ChatGPT helps me learn. Before I started this project I had no experience with Vue.js - and I neither had nor wanted skills with JavaScript. Thanks to ChatGPT’s constant inputs and help, after just 2 days I have a much better knowledge of this product.
4. ChatGPT is an accelerator. Even if I can do something, ChatGPT can do it faster and often better. For example, whenever I can, I like to package everything into Docker containers, since it simplifies deployment in the cloud. ChatGPT always kept the Dockerization process up-to-date, pro-actively informing me when changes to e.g. requirements.txt were needed.

What’s worked badly

1. For whatever reason, ChatGPT seems to have good days and bad days. I have no logical explanation for it. I can only guess it has to do with my prompt history (e.g. “context”) – but that’s just a guess. One example: yesterday I had to stop our session many times: I’d give it a source code file, instruct it clearly to change only the business logic and not the look-and-feel, and ChatGPT would change both. But today, working with that exact same file, ChatGPT only makes the changes I request. Another example: yesterday, no matter what I tried, ChatGPT would also regurgitate the complete source file; but today, it (thankfully) only gives me the bits-and-pieces that have changed. Degree of frustration: *****
2. Infinite debugging loops. Yesterday was a particularly bad day for this, but it happens frequently. ChatGPT gives me code that has an obscure bug. I feed the error message into ChatGPT, and it spits out essentially the same code as what created the bug. I call this an “infinite debugging loop” – and I’ve needed to “break out of this loop” many times – but sometimes telling ChatGPT to take a different approach (and have it really take a different approach) is easier typed than done! Frustration factor: ***
3. ChatGPT givews me more than I need. Yesterday was also a particularly bad day for this. I needed a simple update to a single file (e.g. change the button text from “Save” to “SAVE”). ChatGPT would then not just give me the needed files, but also 4-5 other files (usually with the text “Please ensure file XZY looks like this:”.? Frustration factor: *
4. Unable to debug some problems. There have been a number of bugs during development that ChatGPT was incapable of solving. After quite some frustrating time and back-and-forth, I carried out my own debugging. I found the need for this could be reduced by ensuring ChatGPT always took “baby-steps” that I could continually test. But I did want to point out: total elimination of the human factor in debugging is not yet possible. Frustration factor: **
5. ChatGPT often goes into “guess mode.” I encountered this situation this morning where a Vue component with a dynamic table suddenly started adding duplicate entries into the table. ChatGPT: “Let’s see if this is race conditions…” It was not. ChatGPT: “OK, let’s see if they are added to the local array and not the server…” They were not. Guess after guess. ChatGPT was literally fishing. I gave up. You get the idea. Frustration factor: ***

Other considerations

1. Pair programming with ChatGPT is lousy. I’ve done “extreme programming” or “pair programming” many times, and the good social interaction between programming partners can really keep the momentum going. Even with ChatGPT prompted to treat me nicely and call me Dr. Ken, I can only stomach about 90 minutes of work with ChatGPT without having to take a “frustration break.”
2. Don't be afraid to keep one the hand on the brake. At first it was often difficult for me to stop ChatGPT in flight. But as our time together progressed, I had no qualms about pulling the emergency break when I got the information I need. Of course, I am curious about how many "hidden calories" it burned, e.g. the environmental impact of the information I never saw.
3. Resetting the context is almost always necessary. What do I mean by this? The pair programming with ChatGPT always started well, but as it progresses usually more and more bugs would crop up. This is surprising, since I would hope that by spending more time with the code, ChatGPT would be more effective. In each of my sessions, it was necessary for me to finally end the ChatGPT session so it would forget our "context." Losing context prematurely can sometimes be painful - but in these cases I could club it together with a "frustration break."

What surprised me

1. ChatGPT can learn the context easier than you think - or maybe, "a word to the wise." I realized I wasted time at each session telling ChatGPT I was working with Vue 3, FastAPI, etc. I learned in most cases I can give it code to fix and I don't need to supply much more than that. Interestingly, I think this is a case where I "anthropomorphized" ChatGPT - in other words, I thought, which information would I need to solve the problem? ChatGPT probably needs information, too, but its requirements are much different than mine!

Advice for students and others

Naturally your mileage may vary, but I’ve found that programming with ChatGPT requires not only constant attention on my part and a deep knowledge of the code, but also the ability to carry out my own debugging. I can easily see that without some prior programming experience — and especially debugging experience — working in this way might be very slow and tedious.? When you do enough debugging, you get a “gut instinct” of where the problem could lie – and that was essential for me to climb out of the holes ChatGPT would occasionally dig for me.

I'd love to have another go at this with a different ChatBot - or try again next year. I have a feeling these are just "teething" problems, and that we are on the cusp of letting ChatBots and LLMs do our development for us.