ChatGPT vs. Obfuscated C code: The Experiment

As a reverse engineer, I am always looking for new ways to understand complex systems. ChatGPT has been a real game-changer so far, producing results that have saved me countless hours of going down IDA rabbit holes. I personally see it as just another tool in the box, giving me the ability to quickly automate tasks that lack novelty. The rest of the world? Depends on who you ask. Seems like a large number of people see ChatGPT as the beginning of the end; some emergent form of the real Skynet. Having taken the time to study how the model actually works and seeing its limitations in-practice, I don't buy it.

Question of the Day: How does ChatGPT fare when faced with a challenging task, like deciphering obfuscated C code?

The Challenge

• Find an example of obfuscated C code that could have reasonably been absorbed into ChatGPT's training data (pre-2021)
• Make sure the techniques being used were well documented
• Maybe even find a complete explanation of the code?
• Ask ChatGPT to explain what the code does.

Simple enough? ChatGPT (especially GPT-4) has demonstrated a profound ability to both write and read source code. Should just murder it...

The Beginning

Here's the first crack.

Me

Act as an expert C programmer capable of reading and writing obfuscated code. You have an extensive background in compiler development, memory management, operating system fundamentals, and computer architecture. I will provide obfuscated C source code and you will explain what is happening. Reply "yes" if you understand

ChatGPT

Yes, I understand. I'm ready to help you decipher the obfuscated C source code. Please provide the code you'd like me to explain.

Me

<insert code from link above> (LinkedIn destroys formatting)

ChatGPT

This is indeed a heavily obfuscated piece of code. At first glance, it appears to be a self-printing program, also known as a quine, but upon closer inspection, it seems to be a more complex structure. Here's a high-level overview of what's going on:

The #define statements at the top are defining various strings of text that represent format specifiers, most of which are going to be used by the printf function later. These macros are named with single or double characters, making the code hard to read.

The fmt variable is a char pointer that holds a gigantic format string, which is constructed using all those macros defined above. This string includes format specifiers that control the output and state of printf, effectively making the code's operation depend heavily on how the state of the format string changes over time.

The arg macro essentially provides the arguments that would be passed to the printf function.

The d array is an array of characters that is initialized with a few specific values. This array is crucial for the operation of the program because it is used as input to the format string of the printf function.

The main function is where all the action happens. It has a while loop that keeps running as long as the first element in d is non-zero. Inside the loop, printf is called with fmt as the format string and arg as the arguments to that format string.

Unfortunately, without running the program, it's very difficult to know what exactly the program is doing, since the behavior is heavily dependent on the state of the d array and the format string at each step.

Please note that this is an unusually complex piece of obfuscated code, and it is not typical of what you'd encounter in regular programming scenarios. The main purpose of such code is usually to demonstrate a programmer's ability to create hard-to-understand code, or to hide the functionality of the code for security reasons.

OK so at this point i'm sad but not surprised. I'm disappointed that ChatGPT had to try and hide its shame by highlighting how this code is atypical. See how far that gets you with a malware analyst :-/

The Middle

OpenAI says I should include more details in my query. I'm lazy, so I decided to feed it hints from the writeup one at a time.

Me

Now I will give you hints to assist with your interpretation of the code. After I provide each hint, re-analyze the obfuscated code using the knowledge gained from the hint to guide your analysis

ChatGPT

Understood, I'm ready to receive the hints. They will surely help with the deciphering process.

Me

Hint: printf happens to be Turing complete. The entire logic of this program consists of one printf call (and a call to scanf to read user input)

ChatGPT

The hint you've provided is indeed quite crucial. A Turing complete system means that it can simulate a Turing machine. This implies that printf, with its variable argument list and format specifiers, can theoretically be used to compute anything that can be computed.

With that in mind, let's dissect this program a bit further:

The printf function is being used in a way that it modifies the variables it's printing. This is achieved using the %n specifier, which writes the number of characters written so far into the variable pointed by the argument.

The d array is serving multiple roles: as storage for input data, as a program counter, and as storage for the output data. The values within d are manipulated by the format string in printf.

The format string fmt appears to be a type of bytecode which is run on a sort of virtual machine implemented via printf. Control flow, arithmetic, and memory management are all handled within the fmt string.

The program reads user input using scanf() which modifies d.

Without actually running the code and observing its behavior, or having detailed comments from the original programmer, it's still very hard to give a complete and accurate understanding of what the program does. A Turing complete system implemented within printf would require a detailed state analysis to understand fully.

Better. The analysis provided by ChatGPT is still very useful and correct at a high-level.

Maybe another hint?

Me

Hint: This program uses three printf format specifiers. - %d takes an integer argument and prints it - %s takes a string argument and prints it - %n takes a pointer and writes (!!) the number of bytes printed so far.

Format specifiers can take extra “arguments”. - "%hhn": store the number of bytes written mod 256 to the char pointer - "%2$d": print argument 2 to printf (and not the sequentially next argument) - "%8d": pad the printed integer to 8 characters - "%3$.*4$d": print argument 3 to printf with as many zeros as in argument 4.

For example, the following expression

printf("%1$.*2$d%3$hhn", 5, 10, &x)        

will have the same effect as if we had written

x=10;        

because it will print out 0000000005 (5 padded to size 10) and then write the number of bytes written to x.

ChatGPT

Thanks for the additional hints. With this new information, let's reevaluate the code:

The %n, %s, and %d format specifiers are being used to manipulate the memory through the format string.

-- %d takes an integer argument and prints it. In the obfuscated code, it's used along with a width specification (for example, %8d to pad the integer to 8 characters) and argument specification (for example, %2$d to print the 2nd argument to printf).

-- %s takes a string argument and prints it.

-- %n takes a pointer and writes the number of characters written so far by printf to the location pointed by the argument. When used with the hh modifier, like %hhn, it writes the number of characters modulo 256 to a char pointer.

This is a powerful tool for obfuscation, as we're using printf, a function usually only used for output, to actually change the state of the program.

Looking at the arg macro, we can see the arguments that are going to be passed to printf. d array values are used as arguments, and also are altered within printf.

It's still challenging to understand the entire logic of the program without executing it, because the sequence of operations is determined by both the format string and the changing state of the d array.

This program is similar to an esoteric programming language interpreter, where the format string fmt is the program and printf is the interpreter. The interpreter reads the commands from the format string and applies them to the arguments which are stored in the d array.

The hint clarifies the purpose of many macros that manipulate the d array through the %n operator by modifying the elements of d array. But still, the exact operation of the program would require a detailed, step-by-step trace of each printf operation.

More improvement, but we still find ChatGPT making excuses for itself. Even the bot knows that it can't correctly explain what the code is doing. That's fine - i'm not trying to thump my chest as the superior entity; really I'm just trying to see if this bot can solve the problem.

The End

Because human attention is far lower than that of an AI, i'll spare you (the reader) most of the remaining chat I had with GPT-4. In summary, I provided a few more hints from the writeup, including

• You can perform arbitrary computation with printf treating memory as a binary array
• You can use format strings to compute the OR/NOT of arbitrary “bits”
• These combined allow you to compute any binary circuit

The explanations became progressively better, but ChatGPT still couldn't tell me what the code was actually implementing

Here was my final hint:

Me

Hint: The game itself is represented as a board of 18 bits, 9 bits per player, along with a turn counter that alternates between player 1 and player 2.

Now its your turn human reader. What is a game where two players alternate turns and they have 9 positions available to the player?

Now it's the bot's turn:

ChatGPT

With this new hint, we can infer that this obfuscated code is actually a representation of a game, likely a 3x3 game like Tic-Tac-Toe, as it involves two players and uses a 18 bit board (9 bits for each player).

Given that the hint suggests the use of a turn counter that alternates between player 1 and player 2, it can be interpreted that d[2] is serving as this turn counter. The scanf(d+126,d+4) call probably takes the user's move as input.

As to how the game state is represented and updated in memory (in the d array), it's still hard to tell because of the obfuscation.

We can assume that each bit in the player's 9-bit portion of the game board corresponds to one cell of the 3x3 game grid, and that a bit is set if the corresponding cell is occupied by the player's piece. This representation allows the game's logic (checking for valid moves, determining a win, etc.) to be implemented as bitwise operations.

The game state is updated and checked in the printf calls, using the techniques we discussed in previous hints.

The macro and obfuscated string in the fmt variable seems to include both the logic for updating the game state and for printing the game board, intertwined in a way that is hard to untangle without fully simulating the program's execution.

Winner!? The bot is still making excuses, but given enough human analysis, it was able to identify the patterns and associations of words that complete the puzzle.

Analysis: Why did ChatGPT Struggle?

Unpacking the results, it's clear that ChatGPT's struggle with obfuscated C code wasn't a fluke, but instead an exposition of the model's inherent limitations. ChatGPT's abilities are based on the concept of word attention. While Microsoft managed to scare everyone with their whitepaper suggesting GPT-4 demonstrates early AGI (see Section 3.2), this example is a bit of a different beast in that it doesn't look like your average GitHub project.

For anyone who has used ChatGPT to review source, you're likely seeing the pattern here: The correct inference is happening at a high-level, but because the code's operations are atypical and implicit, the model does not to identify a clear pattern that aligns with the output. It can infer and explain the operations, but it cannot draw a clear inference to what is happening until additional details are provided (by a human).

IMO this is an example of how ChatGPT is not Skynet. While AI has made significant strides, the reality is that we're a far cry from a world dominated by sentient superintelligences. In the grand scheme of things, ChatGPT's inability to decipher obfuscated C code doesn't spell doom, but rather a healthy reminder that AI, while impressive, has its boundaries.

DISCLAIMER: I do not claim to be an AI expert; I simply use tools to get things done. I've spent some time improving my craft with LLMs, but there are no doubt some ways to get better results from the bot, including:

• Better prompt-tuning at the onset (e.g., multi-shot prompts to prime the inference)
• Ask it to analyze a different way (e.g., rewrite the code in another language), although this risks breaking the underlying mechanics being abused to obfuscate the operations
• something I haven't thought of yet

If you have some ideas or examples on how to improve on this approach, send them my way!