Can ChatGPT Teach Secure Coding (and Make Learning Fun)?

Executive Summary

• ChatGPT can host interactive games to help software professionals practice identifying common security vulnerabilities from vulnerable code snippets.
• However, the limited variety of vulnerable code patterns generated can hinder its effectiveness for repeated use.
• While gamification helps to reinforce learning from security trainings, it cannot substitute the need for such trainings.

When I was in 5th grade, my lifelong love for computers was sparked when my father brought home an IBM PC/XT. My first personal computer boasted an Intel 8088 CPU running at 4.77 MHz, 128 KB RAM, a 5.25" floppy drive, and a monochrome monitor. This was where I discovered the fun of BASIC programming and text-based interactive adventure games. One of my favorite game series was King's Quest by Sierra On-Line, which not only provided hours of entertainment but also helped improve my English vocabulary (at least that was how I convinced my parents to let me play).

Recently, I was reminded of this fond childhood memory and turned to my friend ChatGPT for help. While many people are familiar with ChatGPT as an AI assistant for tasks and questions, not everyone realizes that ChatGPT can host a text-based adventure game, reminiscent of the old days.

The concept of text-based adventure games is similar to "Choose Your Own Adventure" books, but you are given even more choices. To play, simply give commands like "Look at map," "Ask king about the Golden Crown," or "Leave castle." ChatGPT will then reveal the next part of the story based on your decisions and actions at each step of the way. Each story is unique, offering endless possibilities for exploration and adventure.

In my own adventure, I successfully retrieved the Golden Crown, rescued the princess, and vanquished the evil dragon. King Edward then challenged me to defeat the formidable Black Knight before assuming his throne. All in all, it was an enjoyable 20-minute adventure!

Inspired by my childhood experience using adventure games to learn English, an idea game to mind: How can I leverage ChatGPT's game hosting ability for both fun and education in my field? In fact, I am no stranger to the concept of gamification. Six years ago, I co-founded HACK@EVENT to inspire participants to gain more hands-on experience in hardware security through captivating capture-the-flag competitions. With this in mind, I turned to my trusty friend ChatGPT for assistance once again:

ChatGPT invited me to play "Secure or Insecure," a name perhaps inspired by the TV show "Deal or No Deal." Being the first and the easiest challenge of the three, I quickly spotted the use of an unsafe function gets() that could lead to a classic stack-based buffer overflow attack.

ChatGPT acknowledged my correct answer and took the opportunity to explain in detail the vulnerability and the solution, which involved replacing gets() with its safe equivalent, fgets(). I earned 10 points in the first round and was eager to see what the next challenge entailed.

The code snippet for Round 2 was a bit longer. Here, I spotted the use of strcmp() for password comparison in line 11. Given how strcmp() is implemented, it opens up the possibility of a timing attack, making password brute-forcing much easier than any constant-time string comparison implementations. I provided my answer to ChatGPT with confidence.

It turned out that ChatGPT had a different answer in mind. He was correct that hard-coding secrets in plaintext in the source code is a bad idea. Attackers can easily dump printable characters inside a binary file by running UNIX commands like strings. Yet, I believed my answer was also correct, and so I protested.

ChatGPT was diplomatic in calling the timing attack that I had reported (and he had missed) as a subtle issue. He went on to explain how strcmp() was vulnerable to timing attacks. However, I was surprised he recommended a vulnerable solution. Replace strcmp() with memcmp() was not helpful because memcmp() was not a constant-time implementation neither. ChatGPT might have been confused with the guidance used to handle buffer overflows, where limiting the comparison within the buffer length would avoid read access violations beyond the end of the buffer. A proper fix would require the use of a constant-time string or memory compare implementation such as CRYPTO_memcmp() from OpenSSL. ChatGPT also got my score wrong, but it was not the time to challenge the game host again.

As the game approached its climax, I expected ChatGPT to present code that might be more complex or convoluted. It did not take long to spot an "off-by-one" error in the destination buffer size in Line 6, as strlen() did not include the null terminator when calculating the string length.

Perhaps our game host was getting tired. First, ChatGPT made a perplexing statement about how strlen() calculates the length of a string by including the terminating null character. Second, he proposed to rectify the code by hard coding the array size to 14 bytes. However, what if I later changed Line 5 to a string with a different length and forgot to update Line 6 accordingly? Third, he got my final score wrong. I could not let this slide.

ChatGPT was quick in admitting fault and provided an update on my scoring. Overall, ChatGPT managed to offer a fun and educational experience to help learners exercise their secure coding skills. However, there are a few key takeaways to keep in mind:

1. ChatGPT can host a fun, interactive game show that is highly adaptable to your needs. If the challenges are too easy, you can ask ChatGPT to level up the difficulty or switch to a different type of vulnerability. Conversely, if you get stuck, you can request hints to help you along. If you prefer a more lively game host, you can ask for a more dramatic tone. However, there were also times ChatGPT misunderstood and simply dumped his entire script to me when provided the same instructions. I had to remind him to wait for my answers so as to keep the game interactive.
2. ChatGPT offers a limited variety of secure coding challenges. We have seen buffer overflow, secret handling, and off-by-one vulnerabilities in the examples above. Other runs have also presented format string, error handling, and input validation on user-specified file/path as security concerns. Nevertheless, most of the challenges focused on the use of unsafe functions that lead to buffer overflows, and the code constructs look quite alike. It may not provide enough diversity for repeated learners.
3. ChatGPT does make mistakes on security best practices. ChatGPT is only as good as the data he was trained on. As software security is a continuously evolving field, what was once considered safe code may no longer be sufficient as new attacks emerge. Therefore, learners are encouraged to exercise judgment and seek advice from a human security expert if they have any doubts about ChatGPT's recommendations. ChatGPT may be better suited to hosting trivia that involves facts that are more black-and-white, such as Hacker Jeopardy.

While gamification can be a valuable tool to reinforce learning, it is important to note the day when ChatGPT is ready to replace comprehensive secure coding training or proper code reviews is still ahead of us. As software professionals, it is crucial to continue deepen your secure code development skills while staying up-to-date on the latest security best practices.