Navigating Chaos: Lessons Learned in Incident Response - Web App Compromise
In cyber security, the common saying is, "it's not if - it's when"
So what do we do when the when happens? That's what incident response is for. Incident response as it relates to cyber security is the preparation for an incident, the identification of a security incident, the containment to limit damage to systems and data, the eradication of the root cause, the recovery of affected systems, and documenting lessons learned from the process. NIST defines incident response into four phases: preparation, detection & analysis, containment, eradication, and recovery, and post-incident activity. No matter which framework you follow, there are some key similarities and steps that are required in order to efficiently and effectively handle a security incident.
Those that know me understand that my interests lie with penetration testing more than incident response, however as a well-rounded cyber security practitioner, sometimes I have to put my background and knowledge to use in other domains outside of penetration testing. That's what this article is about - getting more tangible and real-world experience within cyber security by leveraging a broad skillset.
What Happened?
It's 11:30pm in the middle of the week, and I have to wake up early for work. Probably a bad time to get a phone call stating:
Hey Ryan, some of my websites were hacked.
Well, damn. Guess I won't be going to bed somewhat on time tonight.
The person calling me gives me some details - the websites are redirecting to other websites that are less than pleasant in nature, one of them is kind of like a captcha website asking to click the "I'm not a robot" checkbox and input some credentials, and another one is a website to buy stuff from China.
At this point, I'm thinking to myself that I'll have to jump in and help here, because the person calling wasn't at a computer and wasn't able to do any analysis or shut anything down.
So now we're in the detection & analysis phase of NIST's incident response plan, and the next steps are to preserve data artifacts for later analysis, contain the attack to prevent further malicious activity, eradicate the root cause, patch the exploited vulnerability, restore the websites so they're operational, and talk at a later point about 20-20 vision and hindsight.
The Detection & Analysis
The attacks against the websites were detected by my friend who called me. He checked a few websites and saw the same behavior, and he called me right away. Alright, so how do I as an adversarial thinker start to tackle the incident response and begin the analysis into what happened? I put on my hacker hat and spun up my Kali VM, opened up Burp Suite and ZAP, and started looking at requests and responses.
My initial hypothesis was one or more of the following: server-side request forgery (SSRF), vulnerable WordPress plugins, or a compromise of the hosting provider. Burp Suite and ZAP identified that there were many GET requests being sent to a domain that had nothing to do with any piece of the websites. Some of the requests were being sent to a "trace" subdomain, some being sent to a "away" subdomain, and each request had a weird cookie attached. Here is just a snippet from ZAP from the sites that were being accessed.
I also noticed the Referer HTTP header was not coming from the same domain as the legitimate website:
I gathered that information in my notes and continued with analysis. My friend provided me with login credentials to the hosting provider website so that I could take administrative action on his behalf. I started with looking at the file structures for each website. I sorted by files last modified most recently, and this is when I noticed a few things:
These files stood out as potentially suspicious due to the naming and recent modifications. I made copies of these files for later analysis. After about 45 minutes of looking through these file structures, the decision was made to restore the websites from a known-good backup.
The Containment, Eradication, & Recovery
I'll be the first to admit that because of how late it was, how sudden this was, and interruptions due to work, this phase of the incident response plan could have been improved upon. But we'll talk about that in the next section.
Because my analysis and knowledge of how the websites were developed was limited, we determined it would better to revert to backups rather than trying to only purge malicious files. That's not to say that we didn't try though just to get some more information. I started with removing some of the recently modified files and renaming the current .htaccess file to deactivate the website for a brief period and reset configurations. When I opened a new browser tab, it seemed like the website that I accessed was loading normally and was "back to normal". But better safe than sorry. What I did was snag the before and after .htaccess file for analysis after the incident was resolved.
Going through each website one by one and grabbing the file structure both in the as-is state and after the restore, I was able to retain as many artifacts as possible without compromising the integrity of most of the files. I say most because two of the websites we did make some changes to, including deleting and modifying files instead of grabbing them right away in the as-is state.
After about 2 total hours of working on this incident, each website was restored back to a known-good state, and I had analysis to perform the next day. I spent about 30 minutes typing up a small report and sending files to my friend to also look at in the morning.
Post-Incident Activity
As mentioned in the last section, the integrity of the files in the as-is state were not fully maintained. Looking back at how this was handled, I realized that I made a mistake in making changes without having the full file structure saved first. The additional thing to note during this is that I did not have contact information for the hosting provider and I was limited in my knowledge of web hosting and the interface, so certain changes to the websites were not able to be made by me. This also included mitigation steps to prevent this from reoccurring, so the vulnerabilities still existed. This was noted in the report and via text and phone call as well, and was something that my friend would be looking into in the morning.
The incident was not yet over, and there was still further information to be gathered. This includes finding where the attack vector was, applying mitigations, doing vulnerability scanning, and analyzing the files that I had. Because the incident response lifecycle is a cyclical process, I had to jump back to the analysis phase to further understand what happened.
The first thing I started with was analyzing some PHP files. I started with the "wp-track" file, and it was just a long, one-liner of obfuscated PHP code. Here's what it looked like cleaned up:
领英推荐
Pretty suspicious right? Leveraging ChatGPT allowed me to clean up the code for this screenshot, and find out the following:
The code exhibits behaviors that are often associated with malicious intent. It's attempting to execute dynamic code based on data from superglobal arrays and might involve downloading or executing external PHP files. The exact malicious intent can only be fully understood with a detailed review of the decoded content and the context in which this script is deployed. If this code is not something you intentionally put in your application, it's recommended to remove it immediately.
A few things stood out in particular about this - there's some base64 decoding, an eval function, and an unlink function. Also remember the weird cookie that I referenced earlier? The requests in Burp Suite were showing a cookie named "_ge" which is referenced in this PHP file. Additionally, the JavaScript file mentioned also was analyzed, and I saw this:
/*:wp-toucher:8839e51e262d47d534007a0d7b1a18c5:wp-toucher:wp-working-1693300362.php:wp-toucher:*/
Notice how this JavaScript file is also referencing the "wp-working" PHP file? Great, I have some data artifacts now and some pieces to the puzzle, but still no definitive answers as to what exactly happened, how it happened, or why it happened. That's a bit beyond my personal scope, but I can theorycraft and at least provide some initial technical analysis. The important part is that we try to prevent this from happening again.
The Mitigations
I pivoted into my hacker hat again and did some vulnerability scanning with wpscan to check for WordPress vulnerabilities. Although the vulnerability scanning is great to identify potentially vulnerable plugins and themes within WordPress, the real benefit is using an API key to get deeper with vulnerabilities, which will actually provide CVEs and more detailed information on them. Here are a few outputs from wpscan that make me more confident that the root cause was because of WordPress:
My best mitigation advice - nuke WordPress.
Following Up
We now have a plan in place so that I can provide some further ongoing scanning to detect and correct outdated plugins and software before this happens again. Thankfully none of these websites were super critical in nature, and it only goes to serve as a learning experience for why security professionals should be an integral part of any IT team.