From Debuggers to Docs: My Reverse Engineering Toolkit (And a Laugh or Two)

In this article, I want to share some of my experience in setting up an environment for reverse engineering various software (e.g., malware or game engines) and the tools I use most frequently. Please note that I primarily reverse-engineer Windows binaries such as x86, x64, and .NET applications. Additionally, as the majority of my reversing work is done under Windows, the tools mentioned here are selected with that use case in mind.

I could easily write a separate article for most of the topics covered here, so I've focused on providing a concise overview of the most important and valuable information. If you'd like to learn more about specific details or have any questions, feel free to contact me!

Also, please note that I'm not claiming that my approach is the best. I'm always open-minded to new ideas and methods. :)

0x1 Introduction

I began my journey into reverse engineering nearly eight years ago. Over this period, I spent countless days facing various problems and even more time finding solutions to them.

I started with reversing old games and their engines. I was fascinated by how brilliant some game ports were back then, particularly on the Game Boy console. From leveraging every bit of hardware (e.g., bank switching for more available registers) to highly optimized software (e.g., pre-calculated lookup tables), it was a marvel. I highly recommend checking out Randal Linden 's Quake port for the Game Boy Advance. He accomplished some amazing feats, which continue to fascinate me to this day. If you're interested read this great article about Randal's port: https://www.xda-developers.com/how-quake-ported-game-boy-advance/ (Thanks Randal for sharing the link!)

This was when I truly began to appreciate the elegance of well-written assembly code. The most important lesson I learned was the necessity of thoroughly understanding the basics. You need to know your hardware, the OS you’re developing on, the programming languages you're using, and many other factors. This is quite challenging in today's environment, where the complexity of these topics is incredibly high.

Today, most applications I reverse-engineer are obfuscated and/or virtualized, making the task even more difficult and time-consuming. This also highlights the importance of being able to develop custom tools tailored to your needs. These might be custom plugins for your favorite disassembler or debugger, or entirely custom software that handles specific tasks for which no existing tools are available. For me, this meant that when I started reverse engineering eight years ago, I also began learning C++, Assembly (MASM x86 and x64), and Python.

0x2 Host system & Windows VM

Since most of the applications I reverse-engineer are malicious, I need a highly isolated and clean environment. This involves more than just a VM on my personal computer; it includes an isolated network (separate VLANs), a clean host system to virtualize the Windows VM, and more. If you're interested in the specifics of this setup, feel free to reach out to me here on LinkedIn.

Starting with the host system: I use a fresh Ubuntu installation (Arch also works well) on lab-specific hardware (a separate computer used exclusively for these tasks). I prefer this setup because even if malware escapes the VM, it won’t affect the host system, which is different from the VM’s environment. Additionally, internet access is restricted, and specific firewall rules are applied when needed.

For virtualization, I prefer VirtualBox, though KVM and Virtual Machine Manager are also solid options. If you need to pass through your GPU to the VM, I recommend the latter.

The Windows image I use comes directly from Microsoft with no modifications, aside from installing the necessary tools and software (which I’ll cover in more detail later). The exact version of the Windows image depends on the application I’m reversing. For example, kernel drivers sometimes require specific Windows versions because they rely on undocumented structures or functions within Windows internals.

0x3 Tooling

Now, let's dive into what might be the most anticipated part of the article: what tools do I use? As I mentioned earlier, I’m a big fan of the basics.

If I had to choose just two tools for reverse engineering, they would be a disassembler and a debugger. These are my go-to tools for the applications I work on. They become even more powerful when enhanced with plugins, such as anti-anti-debug plugins for the debugger or byte signature scanners for the disassembler.

I also highly recommend building your own basic framework for reverse engineering. I started developing my own Windows memory manipulation framework two years ago and customized it to suit my needs. This offers the significant advantage of speeding up repetitive tasks in a way that’s tailored to you. For instance, I often needed to support multiple versions of an application/game in my tools, so implementing a pattern scanner that scans the memory for specific byte sequences and retrieves offsets was incredibly helpful. Being able to execute shellcode in processes, dump them from memory and fix their Import Address Table (IAT), or suspend and resume them has also been invaluable. Sometimes, user-mode applications include detection mechanisms that require direct syscalls to the Windows API, which my framework handles seamlessly within my daily workflow.

So, in short, as you gain experience, you’ll identify tasks for which no generic or existing solution exists - this is when you should start writing your own tools and plugins.

0x31 Static Reverse Engineering

Over the years, I’ve experimented with many disassemblers and similar software. Here are my thoughts on the ones I’ve used the most. Ultimately, it comes down to how well you can work with a particular program, but each tool has its own pros and cons that you’ll need to evaluate for yourself.

• Interactive Disassembler (IDA)

I think anyone who has delved into reverse engineering has come across these three letters: IDA from Hex-Rays . Most of my peers prefer IDA as their disassembler of choice. One reason is that it supports a wide range of processor architectures and comes with decompilers for them. It also has an excellent plugin system that allows for significant customization, and its macro-like language (IDAPython/IDC) is great for non-standard use cases. Their debugger also works well [1]. I really doubt it but if you didn't stumble upon them until now make sure to inform yourself about them.

• Binary Ninja

Binary Ninja from Vector 35 . is another great disassembler. It offers a solid range of features, including a disassembler, plugin system, custom API (Python and C++), and the ability to select the abstraction level of disassembly, from low-level binary code to high-level pseudocode [2]. It’s also more affordable than IDA, which is why I frequently used it during my time at university. Students can even get a discount. Though I found its decompiler to be a bit less accurate than IDA’s at the time, the smooth workflow was a huge plus.

• ImHex

This is by far the most supreme HexEditor I used in my life. Period.

It has so many different features which synergize - it's really amazing. I used it for Reverse Engineering different Savefile formats from multiple games, fast patching of binaries and also just browsing and analyzing binary data. I recently noticed that it could also be used to attach to processes, view their memory and also analyze them, which is just another great feature [3] [4]. And on top of all that it is open-source and available under: https://github.com/WerWolv/ImHex.

• dnSpy

Did you ever have to deal with an .NET application, which you needed to reverse or heavily debug? This is your tool of choice.

I can't count how many applications I reversed and also modified with it, doesn't matter if they were obfuscated or not. As I'm really not into .NET and always hated to reverse this kind of software, I really liked the features and workflow of this tool [5]. Most of it is self-explanatory but there are also a lot of good tutorials out there. It's also open-source and available under: https://github.com/dnSpy/dnSpy

0x31 Dynamic Reverse Engineering

As I said I only need a good debugger besides my Disassembler but if I could choose more tools, here are the ones I can recommend. Also consider my slightly bias from reversing game engines. ;)

• x64dbg

x64dbg is just as well-known as IDA, and if you haven’t encountered it yet, it’s worth checking out. It’s open-source, has a great plugin system, combines disassembly with built-in assembly, and features an excellent breakpoint system. You can also automate quite a bit with it's scripting language [6]. There are also plugins available, which are helping you with memory dumping and also fixing the relevant IAT like e.g. Scylla. Also there is ScyllaHide, which is great for evading a lot of usermode anti-debugger techniques [7]. They are also open-source and available on github.

• Cheat Engine

I think almost everybody used this tool when they were young and tried to cheat their money, health points or similiar things. In the good and older times of hypercam, dreamscape 009 and notepad cheat engine tutorials on youtube most of my friends and I knew this tool already. If I could only choose one tool for reversing this would be my choice. It has everything you need to get started: A disassembler, hex editor, lua scripting, memory patching, byte pattern scanning, mid function hooks, a kernel driver and debugger, different usermode debugging methods (soft- and hardware breakpoints) and so much more [8]. This is a great allrounder and I'm using it still today for reversing. It is also open-source and freely available on github: https://github.com/cheat-engine/cheat-engine

• ReClassNET

The combination of a Disassembler and this tool is great. Analyze the disassembly, grab your information like e.g. needed offsets and just put them into ReClass to see the live memory from the area and reverse the structures in the memory [9]. I can recommend reading the article of Adam Henault : Cube World Reversing - Cheat UI & LocalPlayer. He shows the workflow pretty good in my opinion and also how you combine e.g. Cheat Engine with ReClass to gain valuable information.

I would really like to go deeper with all of these tools and also add more but I think these are the essential ones for me. You might wonder that a lot of these tools were intended for reverse engineering of games but I use them also for malware reversing. The workflows aren't the same but for me it is all about information gathering and understanding functionalities when reversing. Some honorable mentions I still have in mind for this section of the article are: API Monitor, frida, CrySearch and radare2. If you have questions regarding this section, make sure to contact me!

0x4 Documentation (official, unofficial)

When reversing and analyzing a software you are happy about every bit of information you find about it. Most often you stumble upon documentation. When working with the Windows API or windows internal structures and functions, documentation plays a key role to get your things together.

0x41 MSDN

The classic. When starting with working with the Windows API this is your choice. Function prototypes, arguments, structures or behaviour - almost everything is there. Overall it has good documentation but it also really lacks in some places. You will find it yourself, there is no chance you'll miss it.

https://learn.microsoft.com/en-us/windows/win32/apiindex/windows-api-list

0x42 PDBs

The times the official documentation fails you and you have to power on the disassembler to have a look at yourself - you want to have a PDB from Microsoft. I recommend downloading them through SymChk, which is a official tool from Microsoft.

0x43 Vergilius

If you ever have to work with undocumented structures in Windows I can recommend the site: Vergilius Project. There you have the possibility to choose between different windows versions and explore a lot of their internal components. It's really helpful and also completes the "stack" of official und inofficial documentation.

Thank you for reading this article! If you're interested in more details about the topics I covered here or have any questions - don't hesitate to contact me.

0x5 Sources

1 https://hex-rays.com/ida-pro/

2 https://binary.ninja/

3 https://github.com/WerWolv/ImHex

4 https://imhex.werwolv.net/

5 https://github.com/dnSpy/dnSpy

6 https://x64dbg.com/

7 https://github.com/x64dbg/ScyllaHide

8 https://github.com/cheat-engine/cheat-engine

9 https://github.com/ReClassNET/ReClass.NET