What are the Encryption Algorithms Used in Modern Malware? - A Comprehensive Guide

Malware authors increasingly rely on encryption to obfuscate their code and evade detection by security tools such as YARA, Suricata, and other static file analysis solutions.

For Security researchers, Analyze malware in an Interactive sandbox such as?ANY.RUN?is a more convenient and secure way, with a variety of monitoring features.

Encryption is a cornerstone of modern malware. It is used to encrypt network traffic, obfuscate command and control (C2) strings, and protect payloads. This guide provides an in-depth overview of the most commonly used encryption methods in malware.

Types of Encryption Algorithms

Stream Ciphers

Stream ciphers encrypt data in a continuous stream, one bit or byte at a time, akin to water flowing from a tap. This method is fast and efficient but generally results in a weaker cipher compared to block ciphers.

Stream ciphers are often used in malware for various reasons, primarily due to their simplicity, speed, and low computational overhead. In the context of malware analysis in real time, understanding how stream ciphers are used can help analysts detect, decrypt, and mitigate the threats posed by malicious software.

Examples of Stream Ciphers:

• RC4: One of the most well-known stream ciphers, though its use has declined due to vulnerabilities discovered over time.
• Salsa20: A modern stream cipher known for its speed and security.
• ChaCha20: A variant of Salsa20, widely used in various protocols, including TLS (Transport Layer Security).

Stream ciphers offer an efficient and straightforward approach to encryption, making them suitable for specific applications where high speed and low complexity are essential. However, choosing a secure and modern stream cipher is vital to ensure the confidentiality and integrity of the encrypted data.

XOR Cipher:

XOR ciphers are often used in malware to obfuscate code, encrypt stolen data, and protect configuration information. Their simplicity and low computational overhead make them effective for these purposes.

By extracting the XOR key from the code or memory, analysts can decrypt the data to understand the malware's behavior. Despite their weak security, XOR ciphers are useful for evading basic detection mechanisms in malware analysis.

The XOR (exclusive OR) operation is a simple binary operation used in many stream ciphers. It takes two bits and returns 1 if exactly one of the bits is 1, and 0 otherwise.

The XOR cipher is widely used in modern malware due to its simplicity. Each bit or byte of plaintext is XORed with a corresponding bit or byte from the key, making it reversible.

Despite its simplicity, the XOR cipher can be surprisingly effective when used with a sufficiently complex key.

The XOR cipher is a simple encryption algorithm based on the XOR (exclusive OR) logical operation. It is a type of symmetric key algorithm, meaning the same key is used for both encryption and decryption. The simplicity of the XOR cipher makes it easy to understand and implement, but it is not secure for most practical purposes unless the key is as long as the message and truly random, as in the one-time pad.

Analyze malware in?ANY.RUN Sandbox?with no limits?- Create a Free Account

How the XOR Cipher Works

XOR Operation

The XOR operation takes two binary inputs and returns true (1) if the inputs are different, and false (0) if they are the same. Here is the truth table for XOR:

Encryption and Decryption Process

The XOR cipher encrypts and decrypts data by applying the XOR operation between the plaintext and the key. Because XOR is a symmetric operation (i.e., A XOR B XOR B = A), the same operation is used for both encryption and decryption.

1. Encryption: Ciphertext = Plaintext XOR Key
2. Decryption: Plaintext = Ciphertext XOR Key

Block Ciphers

Block ciphers are used in malware to encrypt payloads, exfiltrate data, protect configuration files, and secure communication with C2 servers, often using AES due to its strong security.

Ransomware like WannaCry and Petya utilize these ciphers to encrypt victims' files, making them inaccessible until a ransom is paid.

Analysts use tools like ANY RUN, an interactive sandbox for static and dynamic analysis, to identify encryption routines and extract keys, helping to decrypt and understand the malware's operations.

Notable examples include WannaCry's use of AES-128 and Petya's use of Salsa20 for encryption.

Block ciphers encrypt data in fixed-size blocks, usually 64, 128, 192, or 256 bits at a time. This method requires the key to match the block size exactly during decryption, or errors will occur.

• Modes of Operation: Block ciphers can operate in different modes (e.g., CBC, ECB, CTR), which determine how plaintext blocks are processed and combined. Selecting the correct mode is crucial for successful decryption. For instance, CBC (Cipher Block Chaining) mode uses an initialization vector (IV) to ensure that identical plaintext blocks produce different ciphertext blocks.
• Initialization Vector (IV): An IV is a random value used to initialize the encryption, ensuring that even if the same plaintext is encrypted multiple times with the same key, the result is different each time. The IV adds an additional layer of security by preventing pattern analysis.

To decrypt a block cipher in malware, you need to extract the key, the mode, and the IV, along with knowing the encryption algorithm used.

Characteristics:

• Fixed Block Size: Operate on blocks of data, e.g., 128 bits for AES.
• Symmetric Key: Same key for encryption and decryption.
• Modes of Operation: Handle data larger than block size using modes like ECB, CBC, and CTR.
• Security Structure: Use substitution and permutation operations for secure transformation.

Common Block Ciphers in Malware

AES (Advanced Encryption Standard)

AES is a symmetric block cipher and the de facto standard for encrypting sensitive data. It operates on fixed-size blocks of 128 bits and supports key sizes of 128, 192, or 256 bits. AES involves several steps:

• SubBytes: Each byte in the block is replaced according to a substitution table (S-box), which provides non-linearity in the cipher.
• ShiftRows: Bytes in each row of the block are shifted to the left by a certain number of positions, depending on the row index.
• MixColumns: A linear transformation is applied to each column, combining the bytes to provide diffusion in the cipher.
• AddRoundKey: A modified key (round key) is XORed with the block. Each round key is derived from the original key using a key schedule algorithm.

The number of iterations (rounds) depends on the key size: 10 rounds for 128-bit keys, 12 rounds for 192-bit keys, and 14 rounds for 256-bit keys. AES is highly secure and efficient, making it a popular choice among malware authors for encrypting payloads and C2 communications.

DES (Data Encryption Standard)

DES is a symmetric-key block cipher that was once the primary encryption standard in the United States. Although largely replaced by AES, DES is still found in some older or less sophisticated malware samples.

It uses a 56-bit key and operates on 64-bit blocks, going through 16 rounds of transposition and substitution. The small key size of DES is now considered insecure, making it vulnerable to brute-force attacks. However, it may still be encountered in legacy systems or less advanced malware.

RSA (Rivest-Shamir-Adleman)

RSA is an asymmetric encryption algorithm that uses a pair of keys: a public key for encryption and a private key for decryption. This method is known as public key cryptography.

RSA relies on the difficulty of factoring large prime numbers for security, making it robust but slow. Malware authors typically use RSA to encrypt small chunks of high-value data, such as C2 addresses or keys, due to its computational complexity.

RSA is often combined with symmetric encryption algorithms, where RSA encrypts the symmetric key used for the actual data encryption.

Are You From SOC/DFIR Teams? Meet ANY.RUN

If you're interested in malware analysis and go deep into the malware files and their operations, then ANY RUN is a place where you can find your time.

ANY.RUN is an interactive sandbox that helps over 400,000 cybersecurity professionals worldwide analyze malware threats targeting both Windows and Linux systems. The platform offers threat intelligence products like TI Lookup, Yara Search, and Feeds to help identify Indicators of Compromise (IOCs) and respond to incidents faster.

Advantages of ANY.RUN

• Rapid Detection: Detect malware in under 40 seconds, allowing for quick identification and response.
• Interactive Analysis: Interact with samples in real-time, providing a hands-on approach to malware analysis.
• Cost Efficiency: Save time and money on sandbox setup and maintenance, as ANY.RUN provides a ready-to-use environment.
• Comprehensive Recording: Record and study all aspects of malware behavior, from network activity to file system changes.
• Team Collaboration: Collaborate with your team seamlessly, sharing insights and findings in real-time.
• Scalability: Scale as needed, accommodating growing analysis demands without compromising performance.

ANY.RUN offers a robust platform for cybersecurity professionals to understand and mitigate malware threats effectively. Try the full power of ANY.RUN for free and enhance your malware analysis capabilities.

By understanding these encryption algorithms and leveraging tools like ANY.RUN, cybersecurity professionals can better protect systems and data from malicious attacks.

Interact with malware in the ANY.RUN sandbox like you would on your workstation. - Book Free Trial