DoS attack and two hours

Notification: It was usual morning when we started work in office. An email flashed on my computer screen. Email was from one of our clients and seems to be normal email about having some issue in accessing server.

Action: Our team started looking into the issue promptly. Initially, we thought it as the issue with database connection. People who are using MySQL on EC2 can understand this, when there is lack of memory on EC2, sometimes it stops some of your services (including MySQL). As it happened in past for couple of times, we already suggested to make use of separate AWS RDS instance for database services.

Analysis: But the issue was not as simple as we thought. Database service was running fine. However, the portal was not responding as usual and it was extremely slow. It leads us to check for the AWS resource monitoring service. It was found that there was unusual number of connections were made with the database and surprisingly all were made from our own EC2 instance. It was DoS (Denial of Service) attack. Resource (CPU and Memory) utilization on server was on peak and all the CPU credits got exhausted.

Looking into server logs gave us clue about the type of requests, which were flooding the server. This was XML-RPC requests, which is used when we need to access WordPress admin from mobile device.

Solution: As there was not any such requirement in the system, we configured server to deny all such requests. 

Result: Prompt response and right approach for problem solving from the team helped us to get the system to function normal in less than 2 hours.