Reverse-Engineering and Fuzzing Custom Network Protocol

Introduction

The communication protocol de?nes the format and semantics of message exchange between applications. In modern times there are a myriad of proprietary application protocols like Skype Protocol, Dropbox Protocol, etc which applications use to achieve various goals like bandwidth e?ciency, custom encryption/compression, etc. These protocols could have security vulnerabilities. Protocol Reverse Engineering (PRE) is not only useful for o?ensive purposes but also used by modern Intrusion Detection Systems(IDS), they use the knowledge of protocol speci?cation to do Deep Packet Inspection(DPI) which can enhance its capabilities, where it earlier relied just based on pattern matching which may produce lots of false positives. Custom protocols are not only used by legitimate applications but also by malware and botnets like Zeus, Emotet, etc. By reversing malware protocol you can connect to malware servers and track their campaigns.

Protocol Reverse Engineering(PRE) is an art and science of recovering the protocol speci?cation of the obscure/proprietary protocol whose documentation is unavailable or poorly documented. There are e?orts to develop automated PRE tools but they are largely academic and are not mature enough to be usable, and can't give the accuracy a human analyst can o?er. Automated tools face the challenges of heterogeneous protocol data which is often a mixture of text and binary, and it has di?erent data types and variable-length ?elds and this is the reason I have created this training, it is to help you understand these challenges and learn to recover protocol speci?cation.

This training is divided into two parts, in the ?rst part we will learn about Protocol Reverse Engineering principles. We will look at some of the common data formats and other protocol structures and with that understanding we will write a protocol dissector using Scapy framework for a target Desktop game Minetest (open source implementation of Minecraft). Minetest is online multiplayer game in which di?erent players can connect to the server and play with other players, there are also many public servers which you can connect and play. Once we have written the decoder we will sni? the connection and look at the communication ?ow between the client and the server which we will capture and re-analyze the tra?c to improve the dissector further, using this newly improved dissector we will implement a custom game client/bot which will connect to the server and play as a Bot player.

In the second part, with a decent understanding of the Minetest Protocol we will move on to the o?ensive side of the training and try to fuzz the game server to ?nd some security vulnerabilities, we will start with basic Fuzzer and try to do incremental improvement such that we have good code coverage. Leveraging their reverse-engineered understanding of the protocol, participants will employ Generational Fuzzing by defining the protocol specification in the Boofuzz fuzzing framework and subsequently fuzzing the application. The training will also explore Mutation Fuzzing as an alternative approach to identify potential crashes or vulnerabilities.

This hands-on game hacking training is a takes project-based learning approach, ensuring a comprehensive and practical understanding of Protocol Reverse Engineering. In summary, this training aims to equip you with the knowledge and skills to reverse engineer and understand obscure protocols, enhance IDS capabilities, and explore o?ensive techniques such as protocol fuzzing to uncover potential security weaknesses. The ultimate goal is to empower participants to create their own tools in the realm of protocol security.

Why did I design this course?

1. Understand the structure of the requests and response especially useful for a malware analyst.

2. Construction of protocol decoders useful for writing gaming clients or to add support for a third-party proprietary product.

3. Reverse engineer communication of online games like MMORPG which can help you do security testing of multi-player online games.

4. Create Network signatures for malware communication that can be integrated with IDS and IPS, understanding the protocol specification can help you to do deep packet inspection.

5. Write a protocol fuzzer to feed the remote server with crafted randomness in the data to crash the data processing part of the application with the intent of finding security vulnerabilities.

6. Identify security vulnerabilities in protocol implementation like authentication bypass, replay attack, information disclosure, DOS, RCE etc. PRE can also help you to do deeper black-box testing of the application.

7. Build a protocol specification for a vaguely/undocumented protocol.

8. Audit the privacy and security of an application running on your phone/computer by looking at what data is it exporting.

Who should take this course?

1. DFIR Practitioner - to investigation malicious activity in the network

2. Reverse Engineer - write a custom client that fully replicates the existing client software/game.

3. Bug Hunter - Write protocol fuzzer for Black Box testing for application processing remote data, for example, lots of IoT Devices use custom protocol for efficient communication.

4. Malware Analyst - To decode C&C server commands and the data which is exfiltrated

5. Threat Hunting - write network signatures for new emerging APT threats or it could be an intruder in your network, this course will help you decode network and analyze network traffic.

6. Developers - who don't have access to source code or protocol documentation, it usually happens when you are dealing with a legacy system which is too old and the company cannot find any documentation and you intend to migrate the system to new technology.

7. While debugging software over the network, writing a protocol dissector can help you to get a deeper understanding of network communication done by your software.

8. Helps you to do network debugging/diagnostics of application layer data.

9. Privacy - It helps you understand what is really transmitted over the network.

10. RED Team - take advantage of what the Security Operation Center (SOC) doesn't know. Look for data leaks, do attacks like inject, replay and spoofing.

11. Vulnerability Researcher/Exploit Developer - this will also help exploit developer and vulnerability research to reproduce remote vulnerability and find zero-day bugs.

Course Outline

1. Networking Basics

2. Capturing Network Traffic

1. Passive analysis

1. Network Sniffing

2. Syscall hooking (strace)

2. Active analysis

1. Network Proxies

3. Protocol Reversing

1. Protocol Structure

1. Common data format

2. Data Encoding

3. Binary Protocol Structure

4. Text Protocol Structure

2. Protocol Flow

4. Protocol Encode/Decoder (targeting Minetest game)

1. Scapy 101

2. Implementing Protocol Encode/Decoder in scapy for Minetest game.

1. Protocol decoding TLV format

2. Packet decompression

3. Packet Reassembly

5. Custom Client (Bot Player for Minetest Game)

1. Brief Understanding of Application

2. Authenticate the client

3. Establish a valid session

4. Some game hacks like making the player fly

5. Create A Bot Army (if time permits)

6. Protocol Fuzzing (targeting Minetest game)

1. What is fuzzing?

2. Implement Mutation Fuzzer

3. Implement Dumb Fuzzer

4. Implement Generation Fuzzing (Protocol Aware Fuzzing)

5. Vulnerability Triaging (Harness)

Tools of the Trade

Below are some of the tools that you will learn in this training that will make you Protocol Reversing experience more fun.

1. Protocol Reversing tools

1. Wireshark

1. Scapy

2. strace

3. scapy

2. Protocol Fuzzing Tool

1. Boofuzz (Sulley) fuzzing framework

References

1. Reverse-Engineering and Fuzzing Custom Network Protocol - Nullcon Goa 2023
2. Reverse-Engineering and Fuzzing Custom Network Protocol - Nullcon Goa 2020