BUFFER OVERFLOW AND REVERSE ENGINEERING: UNDERSTANDING THE RISKS AND SOLUTIONS [PART 1]

Computer programs comprise multiple functions, each with its own stack frame allocated to store local variables and arguments passed to the function. The stack is a data structure that makes it easy to transfer control and data between functions, allowing for the insertion and deletion of data only at the top of the stack.?

However, what happens when we overflow the stack with data and make the return address point to a malicious code? This is known as buffer overflow, a dangerous vulnerability attackers can exploit to execute arbitrary code and gain unauthorized access to a system.

The article explores buffer overflow and reverse engineering, the risks, and solutions associated with these vulnerabilities, and how developers can protect against such attacks.?

For your understanding, we have divided the article into two parts. In the first part, we will discuss buffer overflow along with its types, examples, and defense techniques. And the second part will explore reverse engineering and how it can be used to detect buffer overflows.?

Understanding Buffer Overflow?

As mentioned, computer programs allocate variables using memory blocks that have a fixed size. After allocating memory, data can be stored and retrieved from memory blocks. Buffer overflows occur when data written to a memory block exceeds its allocated size, resulting in overwriting memory allocated for other purposes, potentially leading to various consequences in the program.

Buffer Overflow Attack

A buffer overflow is a type of cyberattack that exploits a vulnerability where data controlled by the user is written to the memory. Attackers leverage this vulnerability by intentionally writing more data than the allocated memory's block's size, resulting in the overwriting of adjacent memory locations.?

The consequences of buffer overflow attacks can be severe, ranging from program crashes to the execution of malicious code. Attackers can use buffer overflow attacks to change critical values, modify the program's execution flow, or inject malicious code.?

Attackers can use buffer overflow vulnerabilities for various purposes, as given below.

• Code Execution: Attackers use buffer overflow attacks to execute their code in the vulnerable application. This enables the attackers to run code on the affected system, having the same permissions and access as the exploited app.?
• Denial of Service Attacks: An application's memory space contains code, pointers, and other data critical for execution. When this data is overwritten, it can make the program crash, and as a result, a DoS attack is executed.?
• Access Control Bypasses: When an attacker takes advantage of buffer overflows to execute code, it can result in granting them increased privileges to a targeted system. This newfound access can subsequently be utilized to launch additional attacks.

Types of Buffer Overflows Attacks

There are different types of buffer overflow attacks. Some of them are given below.

• Heap-Based: One of the primary functions of the program heap is to allocate memory dynamically to variables whose size is not predetermined during program compilation. However, this dynamic allocation also creates a potential vulnerability for attackers to exploit. An attacker can flood the system heap through a buffer overflow attack and overwrite essential application data, thus compromising the system's integrity.
• Stack Based: Within an application's architecture, the program stack holds crucial control flow information, for instance, function return pointers. This data is often a prime focus for attackers looking to exploit buffer overflows. By manipulating the function return pointer through overwriting, an attacker can redirect the program's flow to data under their control and execute it as code, allowing them to gain elevated access permissions to run code within the application.
• Format String Attacks: C/C++ programming languages offer a printf family of functions that utilize format strings. These format strings enable the memory’s reading and writing. However, if user-supplied data is erroneously interpreted as a format string, it may result in the leakage or modification of confidential information. This can compromise sensitive values and potentially lead to a security breach.
• Unicode Overflow Attacks: Such attacks exploit the memory requirement discrepancy between Unicode format and American Standard Code for Information Interchange (ASCII) characters when storing strings. This makes programs that exclusively accept ASCII character inputs susceptible to such attacks.
• Integer Overflow Attacks: Integer overflow can occur when the result of an arithmetic operation exceeds the maximum value that can be stored in the integer type. This can potentially lead to buffer overflow vulnerabilities.

Examples of Buffer Overflow Attacks

Here, we will explore two examples of Buffer overflow attacks.?

C/C++ programming languages are prone to buffer overflow vulnerabilities, typically when programs reserve a predefined memory size and transfer data without adequate security measures. Below is an example code sample demonstrating a buffer overflow vulnerability:

char buf[BUFSIZE]

gets(buf);        

In the provided code snippet, there is a variable named "buf" with a static size. The function "gets"? is employed to receive input data, which is then stored in "buf" until a null terminator is encountered. This can lead to a potential security vulnerability, as an attacker may exploit this situation to overwrite data over the specified limit of "buf."

The Format string attack represents a specific category of buffer overflow attacks. Below is a sample code for this attack.

#include <stdio.h>

void main(int argc, char **argv)
{
 ? ? printf(argv[1]);
}        

If the user's input contains a format string, attackers can potentially access or modify the memory of the system in the above example, where a user-provided input is accepted by the program from argv[1], and it outputs it to the terminal.

Why is Buffer Overflow so Dangerous?

Buffer overflow attacks are highly dangerous due to their prevalence and the diverse range of ways in which they can be executed. Developers can make these errors in numerous ways, including heap and stack-based buffer overflows and other specific vulnerabilities such as Access of Memory Location After End of Buffer (CWE-788) and Out-of-bounds Write (CWE-787).?

Given the immense amount of code that exists in modern software applications, it is highly probable that buffer overflow vulnerabilities exist in many programs. In fact, The MITRE Corporation rated buffer overflow attacks as the most dangerous software weakness of 2021 . This indicates that if left unchecked, buffer overflow vulnerabilities have the potential to result in significant damage, potentially leading to data breaches, system crashes, or even complete system takeover by attackers.

Tools for Detecting Buffer Overflow

Several tools are available to detect buffer overflow vulnerabilities. Here is a list of some common tools developers can use.?

AddressSanitizer (ASan)

Address Sanitizer is a runtime tool developed by Google for detecting and debugging an array of memory errors, such as accessing heap, stack, and global buffer overflows. One of its key features is the ability to provide a detailed stack trace of any invalid memory access, along with a memory map that can help identify the root cause of the error. By leveraging ASan, developers can detect and resolve memory-related issues more easily, resulting in a more secure and robust codebase.?

Valgrind

Valgrind is a programming tool for memory debugging and profiling. It can be used to detect buffer overflows, memory leaks, uninitialized memory, and other memory-related errors. Valgrind works by instrumenting the program's executable and intercepting calls to memory allocation and deallocation functions.

American Fuzzy Lop (AFL)

Fuzzing is a technique for detecting buffer overflow vulnerabilities by sending large amounts of random or semi-random data to a program and monitoring its behavior. There are three types of fuzzers known as evolutionary, generation, and mutation.?

American Fuzzy Lop is a mutation fuzzer used to fuzz programs taking input from a file or STDIN. AFL uses a genetic algorithm to generate input data designed to trigger a buffer overflow, format string, and other vulnerabilities in target software. The tool takes a sample input file, mutates it, and then runs it through the target software to see if it triggers any crashes or other errors.

Klocwork

Klocwork is a static code analyzer for developer productivity, DevOps/DevSecOps, and SAST. It helps identify software security, reliability issues, and quality and detect buffer overflow vulnerabilities. Also, it detects software defects, security vulnerabilities, and compliance issues in code.

GNU Debugger (GDB)

GDB is a powerful debugger inbuilt into every Linux system that can be used to detect buffer overflow vulnerabilities. It allows developers to examine the state of a running program and to inspect and modify the memory contents. By setting breakpoints and examining the program's execution, developers can identify buffer overflows and other memory-related issues.

How to Choose the Right Tool to Detect Buffer Overflow in Your Project?

Due to the availability of several tools, choosing the right tool to detect buffer overflow vulnerabilities in your project can be daunting. Here is a list of factors to consider to choose the right tool for your project.

• Type of application: The first thing to consider is the type of application of your project since different types may have different requirements for security and performance. For instance, an embedded system can have different constraints than a web app.?
• Programming language: Different programming languages have different memory management models and may be more or less susceptible to a buffer overflow. For instance, C and C++ are more prone to buffer overflow than Java and Python.
• Tool features: Evaluate each tool's features and capabilities to ensure it meets your project requirements.
• Tool compatibility: Ensure your chosen tool is compatible with your development environment and workflow. For instance, if you use continuous integration tools like Jenkins or Travis CI, you'll want to choose a tool that can integrate with those tools.
• Cost: Some tools may be free or open source, while others require a paid subscription or license. Consider the cost-benefit ratio of each tool to ensure that it provides value for the investment.

Protecting Against Buffer Overflow

While tools to detect buffer overflow vulnerabilities can be useful in identifying and addressing security issues in software applications, developers should not entirely depend on them. They must take a proactive approach to security by following best practices and secure coding techniques to prevent buffer overflow attacks.?

Let's explore some best practices to prevent buffer overflow attacks and guidelines for writing secure code.

Best Practices to Prevent Buffer Overflow Attacks

To prevent a buffer overflow attack, consider the following best practices:

• Perform regular code auditing using automated or manual methods to identify potential vulnerabilities.
• Provide training to developers on secure coding practices, including bounds checking, the use of unsafe functions, and adherence to group standards.
• Utilize compiler tools like StackShield, StackGuard, and Libsafe to add an additional layer of protection against buffer overflow attacks.
• Use safe functions like strncat and strncpy instead of more vulnerable functions like strcat and strcpy.
• Keep web and application servers up-to-date with regular patches, and stay aware of bug reports related to any dependent applications.
• Periodically scan your application with available buffer overflow scanners to identify any potential vulnerabilities in your server products or custom web applications.

Guidelines for Writing Secure Code

Here is a list of guidelines for developers to write secure code while developing applications.?

• Never trust user input: Input validation is crucial for writing secure code, as attackers can easily manipulate user input. To ensure safety, always check data length, character set, and format and restrict user input.?
• Send inputs to other systems after sanitizing data: To send inputs to other systems safely, it's essential to sanitize data first. This can be done using a whitelist or blacklist to control what data is allowed or blocked. Additionally, escaping inputs can also help ensure safety.
• User authentication & passwords: To ensure user authentication and password security, use transport layer security (TLS) client authentication, implement authentication error messages, store and manage passwords safely, and transmit passwords securely.
• Use the principle of least privilege: To improve security, follow the principle of least privilege. This involves validating permissions on every request and creating tests to validate permissions before release. Additionally, reviewing permissions to ensure they remain appropriate periodically is important.
• Deny access by default: To write secure code, ensure that access is denied by default. This means that unauthenticated users should not have access to sensitive resources. Apply this policy to new user accounts and new features to reduce the attack surface of your application.
• Make your architecture unique & Secure: Ensure unique and secure architecture by using subsystems, following OWASP's Secure Architecture (SA) practice, and loading only secure plugins and libraries.
• Check your code quality and follow coding standards: Ensure code quality and follow coding standards for improved security. Review code, use effective quality assurance mechanisms, and follow coding standards developed by international bodies. Additionally, use threat modeling to prevent attacks.

Buffer Overflow Mitigations?

Let's explore some techniques to mitigate buffer overflow exploitation.

Stack Protections

A buffer overflow can lead to the attacker executing malicious code by manipulating the program's control flow. A technique attackers use is Return-Oriented Programming (ROP), which allows them to chain together small snippets of code (gadgets) to perform arbitrary actions.?

You can implement various countermeasures such as stack canaries, address space layout randomization (ASLR), and executable space protection (NX) to prevent ROP attacks.?

Stack Canaries

To enable Return-Oriented Programming (ROP) on the stack, an attacker must have the capability to alter a function's return address and direct it toward a memory region they control. To mitigate this, the use of stack canary was devised as a safeguard.

A stack canary is a recognized value inserted on the stack prior to the return address. The program confirms the canary's value before returning from a function and throws an error if it's incorrect. This technique aids in identifying and preventing buffer overflow attacks.

Data Execution Prevention

ROP exploits the fact that user input meant to be interpreted as data can be interpreted as code because control information and data are often mixed without clear boundaries on the stack. As a result, attackers can leverage this to manipulate the program's flow.

To prevent these types of attacks, Data Execution Prevention (DEP) was developed. DEP designates specific memory regions which are non-executable and can help prevent buffer overflow exploits.?

When a memory region is marked as non-executable, any attempt by an attacker to run a shellcode by modifying the return address will be prevented. However, this protection can be bypassed through a return-to-libc attack. To counter this type of attack, address space layout randomization (ASLR) can be implemented to randomize the locations of key program components in memory.

Input Validation

Buffer overflow attacks occur when an attacker writes more data to a memory block than what the application allocated for that data. This vulnerability arises for various reasons, but the most common cause is unbounded reads that continue to read until a null terminator is discovered in the input.

One possible solution to prevent buffer overflow attacks is to use fixed-length reads that fit within the allocated buffer space, making the application immune to such attacks. By setting a maximum length on the reads, the program can ensure that no excess data is written to the memory block, preventing buffer overflow attacks.

Address Space Layout Randomization

Object-oriented design is commonly used in software applications, with shared libraries being a critical component imported into the program's memory space. However, these shared libraries are not only beneficial to legitimate code but can also be exploited by ROP attacks.

Address Space Layout Randomization (ASLR) is a technique to prevent buffer overflow attacks by randomly assigning memory locations to executable files in memory. By randomizing memory addresses, attackers are less likely to know where specific code (such as ROP functions or libraries) is located, making it more difficult to exploit vulnerabilities. Rather than eliminating vulnerabilities entirely, ASLR aims to make it more challenging for attackers to exploit them.

Standard Libraries

Although C++ allows manual memory allocation for user input, it may not always be a safe option. The C++ Standard Template Library (STL) provides functions, such as strings, which correctly manage memory without the need for manual allocation. Switching from C-strings to STL strings is a simple solution to reduce the risk of buffer overflow vulnerabilities. Using strings, the program can ensure that the memory allocated for user input is managed securely behind the scenes without leaving room for vulnerabilities.

Integer Overflow Checking

Integer overflow vulnerabilities can also lead to buffer overflow attacks. This happens when a variable stores a value too big to fit in its capacity, causing it to drop some of the most significant digits. This can lead to a large input being interpreted as having a shorter length than it actually does, which can cause an undersized buffer allocation.

To prevent buffer overflow attacks, checking for integer overflows in input lengths is crucial. By verifying that the input length does not exceed the variable's capacity, the program can ensure that the allocated buffer is of the appropriate size, reducing the risk of buffer overflow vulnerabilities.

Now that we have a clear understanding of buffer overflow attacks, their types, potential risks, and solutions, let’s explore what reverse engineering is, how attackers can use it to exploit a buffer overflow, and how reverse engineering can be used to detect buffer overflow.?

What is Reverse Engineering?

Reverse engineering is the practice of deconstructing an object or system to understand its functionality and design. Its primary purpose is to gain insight into the inner workings of a product, but it can also be used to create copies or improve upon the original.?

This technique can be applied to a wide range of objects, from software applications to complex machines, military hardware, and even biological processes such as gene expression. By breaking down and examining the components of a system, researchers can uncover new knowledge that may lead to advancements in technology or other fields.

Takeaway?

We have given a brief introduction to reverse engineering. While it is not an exploit itself, the knowledge gained from reverse engineering can be used by attackers to identify vulnerabilities in the system and create exploits to take advantage of those vulnerabilities, such as buffer overflow. Head over to part 2 of this article to learn more about reverse engineering.?