An AI Coding experiment and the 10 things I learned

With the enormous popularity of AI, there’s no denying it is everywhere—whether I’m looking, reading, watching, or listening. I decided to run an experiment to test AI’s ability to write code. The article that follows is divided into three parts and a conclusion.

Part One: I’ll explain my experiment, including what my application is and which AI engines I used to help create the code.

Part Two: I’ll provide some basic information about the AI engines I worked with.

Part Three: I’ll share the ten key things I learned.

Conclusion: Finally, in the conclusion, I’ll share my overall thoughts.

PART ONE: My Application

My test application addresses a specific need I have. I read a lot of business books, mostly as eBooks, and I frequently highlight passages that I want to remember. However, simply highlighting text doesn’t commit it to memory, and I rarely go back to review my highlights—so that doesn’t help either. I came across a very useful service that aggregates your highlights and emails them to you, one per day, as a memory aid. It’s an elegant and well-designed app—better than what I’ve created. However, with two kids still to put through college, I couldn’t justify even its minimal cost.

If you know me, you know I love Python. I also love Django, a Python-based framework for creating web applications (Instagram is perhaps the most famous Django app). My app serves a dual purpose: providing some simple static web pages and performing two complex functions.

The first function is processing files uploaded by users. The paid app connects to your Kindle account through an API and automatically retrieves your highlights. My app, on the other hand, requires users to manually download their highlights from their Kindle reader as an HTML file, and then upload the file through a web page. My code parses the HTML file to extract the highlight data, which is then stored in the app’s SQL database.

The second function is a daily email system. Each day, the app iterates through its users and emails each one a random highlight from their stored data.

PART TWO - About the AI Engines I Tested

My AI Test Subjects Included: ChatGPT, Gemini, CoPilot with GITLens, and Claude.? Here is a high-level comparison of the AI engines I tested.

Foundational models and training are the backbone of AI engines. Understanding their architecture, training, and design philosophy is key to evaluating their strengths, weaknesses, and suitability for various applications. When comparing AI systems, the foundational model provides a lens to understand how they work, what they excel at, and where they might fall short.

PART THREE: The 11 Things I Learned from Using AI to Help Write Code for My Application

1. The way you phrase your questions matters. I struggled with an error for two days and had exhausted all the suggestions from AI. The error persisted until I rephrased my explanation to include the word "intermittent." That single word helped ChatGPT lead me to the solution.?
2. Apple’s AI was disappointing. Once, I asked why my new MacBook didn’t come with Pages pre-installed. Apple’s AI responded, “I don’t understand the question.” ChatGPT, on the other hand, provided all the information I needed.?
3. The code editor/IDE makes all the difference. An editor/IDE with integrated AI is the best way to go.? When I started coding, I used a very simple and lightweight text editor called Sublime Text 4. With Python, I didn’t initially see the need for a more complex IDE. Sublime even supports add-on packages for additional functionality, such as code completion—a very helpful feature that predicts and suggests the rest of the statement you’re typing. As much as I like Sublime Text, there are two reasons I decided to learn a more complex code editor like Visual Studio Code: First, Visual Studio Code’s code completion is incredibly impressive. Instead of suggesting the remainder of the current line, it can suggest the next 5–10 lines. Second, using an AI tool through its web interface often requires providing lots of context. Sometimes I needed to share code from five files, totaling nearly 1,000 lines. With Visual Studio Code’s integrated Co-Pilot, I could chat with it directly within the editor. I could also split windows, have Co-Pilot open in one, and drag relevant files into the window as context for my questions.
4. Visual Studio Code’s code completion is phenomenal. I can’t count how many times I’ve been impressed by VS Code’s code completion. It often seemed like it was reading my mind. While Sublime Text might achieve similar results with the right plugins, my current setup only completes words or maybe a single line, whereas VS Code suggests multiple lines.
5. Coding with AI reminds me of driving with a navigation system. With a navigation system, I can easily navigate complex routes, but if I had to make the same trip without it, I might struggle. Similarly, with AI, I can quickly create working code that I don’t fully understand. If I had to reproduce that code in the future, I might not be able to. Some might argue that AI is making us less intelligent, but this is no different from how calculators replaced the need to do complex math in our heads.? Also, unlike a calculator, I can ask AI to explain the code I do not understand.
6. Don’t “ask to ask” another question. My daughter jokingly says she’s polite to AI so that if it ever takes over, it might spare her. While I don’t necessarily believe AI will take over, I’ve also found myself being polite and gracious. One time, after AI solved a frustrating problem, I said, “Thank you! May I ask one more question?” It replied, “Yes, of course, whatever you need.” I eagerly asked my next question, only to receive the message, “Sorry, you have exceeded the number of questions allowed for your free/trial account.”
7. Break prompts into manageable tasks. Initially, I tried to use AI for large, complex tasks, like generating a full Python function to display users’ highlights in a table with editable links. After two hours of frustration, I changed my approach. I asked it to first generate code to display the highlights, then modify the code to add a column with edit links. This took just 10 minutes. Breaking down prompts made a huge difference.
8. If one AI doesn’t help, try another. Over time, I learned to trust my instincts. If an AI’s initial response was way off, I’d switch to another. Each AI has its strengths and weaknesses, and experimenting helped me identify what each does well.
9. Say goodbye to Stack Overflow. As a hobbyist coder, this was huge. For years, Stack Overflow has been the go-to resource for coding questions, but it can be intimidating. Responses can be snarky if your question seems basic or repetitive. With AI, I can ask any question—even “dumb” ones—and get immediate, tailored responses. I can follow up without re-explaining everything. Since I started using AI, I haven’t returned to Stack Overflow.
10. Recognize when you’re in a loop. Sometimes, when troubleshooting with AI, I noticed it would start repeating solutions. For example, I had an error where the code worked fine in my Mac’s terminal but failed in VS Code’s terminal. After several rounds of feedback, the AI reverted to its initial suggestion. When this happens, it’s time to switch to another AI.

My Conclusions

I was frankly amazed by the complexity of the problems AI could solve and the speed at which it did so. One notable example was when I had an issue with my hosted server. While using ChatGPT, it identified that I was using DigitalOcean simply from the features I described in my prompt.

That said, there were times when I knew enough about Django to recognize that some of the information AI provided was incorrect. These instances reminded me that AI still has a way to go before it can fully replace programmers.

While I didn’t make a conscious decision to favor one AI tool over another, I noticed that by the end of my project, I was almost exclusively using ChatGPT and CoPilot integrated with Visual Studio Code. CoPilot was my go-to when I needed to provide extensive context for a prompt. Django, by design, organizes code across multiple files, and sharing this data was much easier with CoPilot’s seamless integration in VS Code.

I also found that different AI tools excelled at different tasks. For instance:

• CoPilot was exceptional for specific coding questions, especially when I needed to include a lot of code for context. However, it struggled with configuration issues, like Python interpreter problems, which ChatGPT resolved quickly and effortlessly. One exception was troubleshooting VS Code itself, where CoPilot proved highly effective.
• Claude wasn’t bad. It performed better than CoPilot for certain tasks and was certainly better than Apple’s AI. But over time, I found myself naturally gravitating to ChatGPT, Gemini, and CoPilot instead. Frankly, I often forgot about Claude entirely.
• Gemini stood out in Chrome’s address bar searches, providing useful results directly within the browser. However, I noticed it sometimes repeated previous suggestions when running out of ideas.

Of all the tools, ChatGPT was the clear standout. Its interface felt warm and user-friendly, and its answers were consistently accurate and insightful. It even excelled in creative tasks. For example, when I asked it to make my base.html template more elegant and inviting, it exceeded my expectations. My main challenge with ChatGPT was providing sufficient context for complex prompts. In some cases, I found myself pasting nearly 1,000 lines of code to ensure it understood my needs. My app can be found here: https://134.209.220.170/home/

Final Thoughts

My experiment showed that AI can drastically increase the speed at which a hobbyist like me can produce code. However, this often comes at the cost of fully understanding the code being generated. While AI is a powerful assistant, it’s not yet ready to replace programmers. There were times when even I, an intermediate-level programmer, could see that the solutions AI suggested wouldn’t work.

That said, the future potential is undeniable. AI has already transformed how I approach coding projects, and I can only imagine how much more capable it will become. Someday, it might be able to fully replace programmers—but for now, it remains an incredible tool for accelerating development and enhancing productivity.