DevSecOps+ML - Prevention Against DoS or DDoS Attack

What is DoS & DDoS?

A Denial-of-Service (DoS) attack is an attack meant to shut down a machine or network, making it inaccessible to its intended users. DoS attacks accomplish this by flooding the target with traffic, or sending it information that triggers a crash. In both instances, the DoS attack deprives legitimate users (i.e. employees, members, or account holders) of the service or resource they expected.

A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt normal traffic of a targeted server, service or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic. DDoS attacks achieve effectiveness by utilizing multiple compromised computer systems as sources of attack traffic. Exploited machines can include computers and other networked resources such as IoT devices. From a high level, a DDoS attack is like a traffic jam clogging up with highway, preventing regular traffic from arriving at its desired destination.

The Top DDoS Statistics and Trends You Need to Know

So, what are some of the biggest DDoS attacks and trends we’ve seen so far in 2019? We’ve pulled together a list of DDoS statistics concerning the year’s biggest DDoS attacks, trends, and future industry projections for 2020 and beyond:

1. Global estimates of the total number of DDoS attacks are anticipated to double to 14.5 million by 2022, according to 2017 data from the Cisco Visual Networking Index (VNI). 

2. Also according to Cisco, DDoS attacks, both in size and number, have been on an overall downward trend since the FBI shut down 15 of the largest DDoS-for-hire websites in December 2018.

3. DDoS attacks are a dominant threat to the vast majority of service providers — and their impact is widespread. These attacks can represent up to 25 percent of a country’s total Internet traffic while they are occurring.

4. Research from Kaspersky’s SecureList shows that in Q2 2019, there were more high-profile DDoS attacks than in the previous quarter. China and the U.S. ranked as the top two targets for DDoS attacks in Q2 2019, with 63.8 percent and 17.5 percent of the attacks, respectively. However, Q1 2019 itself was nothing to sneer at — the quarter also saw a significant number of DDoS attacks, which increased by 84 percent.

5. Some of the largest DDoS attacks in history were recorded this year by Imperva. The cybersecurity solutions provider reported on its blog that the largest application layer DDoS attack in history occurred in the spring over a 13-day stretch. The attack, which targeted a streaming service client, and peaked at 292,000 requests per second (RPS).

6. The beginning of the year was truly a busy time for Imperva’s clients. The cybersecurity company also reported that one of their clients was able to thwart the largest packet-based attack on record, which peaked at 580 million packets per second, in April. This followed an attack against another of Imperva’s clients in January, which topped out at 500 million packets per second.

7. One of the trends Neustar discovered in its DDoS attack research is the increasing trend of what their researchers refer to as strategic, “low-intensity incursions” that degrade the performance of servers over time. Using these lowball attacks enables hackers to carry out longer attacks that fall below the level of intensity that would trigger DDoS defenses.

8. According to Neustar, more than 82 percent of all attacks mitigated by Neustar in Q2 2019 used two or more vectors. Their report also indicates that they “observed a significant number of attacks that featured over four vectors.”

9. The number of Internet of Things (IoT) devices that are estimated to exist by 2020 is 20.4 billion, according to a press release from Gartner, Inc. Because IoT devices (connected devices such as smart thermostats, refrigerators, and even baby monitors) are notorious for lacking any real IT security or cybersecurity measures, they’re vulnerable to hacking, eavesdropping attacks, and DDoS attacks.

10. Ever wonder what the average cost of one of these attacks equates to monetarily speaking? Bulletproof’s 2019 Annual Cyber Security Report indicates that a DoS or DDoS attack could cost up to $120,000 for a small company or more than $2 million for an enterprise organization. That equates to a lot of apologies to customers and shareholders.

11. A report from A10 Networks indicates that the company tracked more than 20.3 million DDoS weapons — infected computers, IoT devices, and servers — in Q2 2019 that were available to launch DDoS attacks. In Q1 2019, the company tracked nearly 23 million such weapons.

12. According to Kaspersky’s Q2 2019 report, although DDoS attacks were down overall, the second quarter showed a clear increase in politically-motivated DDoS attacks.

13. DDoS attacks are affecting some industries more than others this year. Research from Akamai indicates that financial service organizations, in particular, are seeing a rise in reflection DDoS attacks. Between Dec. 2, 2018 and May 4, 2019, Akamai observed more than 800 DDoS attacks against the financial industry — a number that represents more than 40 percent of all unique DDoS targets they observed during that time.

14. Another popular set of industries DDoS attacks target are the media/information services and insurance industries. Research from IBM X-Force indicates that more than 80 percent of all observed activity from Mirai botnet variants so far in 2019 targeted these two industries. Mirai (and Mirai botnet variants) also have been the most common malware to target enterprise IoT devices so far this year.

15. With a current value of $2.4 billion in 2019, one research firm estimates the DDoS protection and mitigation market will nearly double to $4.7 billion by 2024. That’s a compound annual growth rate (CAGR) of 14 percent.

Article link is here to refer.

In this article, I will show you how to integrate Cyber Security with Machine Learning and then automate this thing using DevOps concepts which will make this a MLDevSecOps kind of setup. So let's see what all things we need to do this and how we can integrate this all to detect the DoS or DDoS attack and block the IP Address.

First of all we will list out somethings that we will need.

1. Docker
2. Jenkins
3. GitHub

Now first we will create a web-server using docker before going further, I have created one script which will do some services management for us so that jenkins and docker can work efficiently without any firewall issue. We need to make this setup once and it will run forever until it's shutdown. So we don't have to worry about the DoS or DDoS attack once we setup this structure of integrating multiple technology.

Now this will first stop firewall so that we have connectivity of jenkins from our host system that is windows in my case as I am running Rhelv8 as VM on top of Windows and inside we have Jenkins and docker running. Then we are restarting docker service otherwise it will fail due to iptables issue. Then we are disabling SELinux. We are then making it executable and executing it.

Now we need to create a workspace where we will manage our whole process. So, I made a /loganalysis/ directory and inside it, I have made another httpdlog directory. We will download all the code and necessary script from GitHub and save it in this /loganalysis folder and the httpdlog will be mounted to the web-server to collect log. On this log we will do our KMeans Cluster and detect the Blacklist cluster and block the IP in the blacklist cluster.

Now, we will create Jenkins job to automate our task.

JOB 1 To download from GitHub

This is the configuration of the job to download the github codes. We have basically provided the github link to the project where our python script and some other files and script is there which is developed my the developer.

JOB 2 To execute the whole process automatically

Here we have made the job scheduled to run every two minutes using build periodically. In here we are first downloading 3 files and copying it in /loganalysis folder in job1.

These three files are:

1. apache_log_to_csv.sh : This will convert the log file into csv file using basic linux command, here we have done some data cleaning. Even you can think it as a ELK stack but not completely we have just made one mini version of Logstash using basic linux commands.
2. DoS_Attack_KMeans_Clustering.py : This the main file where we are creating cluster, saving it as a .html file for the client to see and detecting the cluster which is doing DoS or DDoS and finally blocking it.
3. DoS_Blacklist.csv : This file will save the blacklisted IP.

#!/bin/bash
	

tac /loganalysis/httpdlog/access_log | awk -F' ' '{print $1","$2","$3","$4","$5","$6","$7","$8","$9","$10}' > /root/HTTPD_log.csv
	

sed -i 's/-/0/g' /root/HTTPD_log.csv
sed -i 's|[[]||g' /root/HTTPD_log.csv
sed -i 's|[]]||g' /root/HTTPD_log.csv
	
sed -i 's|["]||g' /root/HTTPD_log.csv

So this the code of apache_log_to_csv.sh here the first command is taking first 10 fields which are separated by space as a delimiter and saving it in the root directory with HTTPS_log.csv name. In the following four command we are finding and replacing things. sed is a basic linux command to edit files and many more things using command.

sed -i <to update the file> 's/<thing to be replaced>/<replace with this>/g' <filepath>

So, if you analyse the log there are - , [ ] and " in the appearance so we need to clear it out, it's called data cleaning. This we have done using simple linux shell command.

Next we check whether the web-server with name webos is running or not. If it's running we will for ensuring it will run start command which will do nothing if it's already running. Next what we will do is run the python file to do our clustering and blocking part automatically, else it will just start the web-server and after every two minutes it will verify the the log files. So the main thing here is this auto checking of log files every two minutes so the security team and operation team don't have to worry about the attack because if it happens this program will block them.

import os
	import numpy as np
	import pandas as pd
	import plotly.graph_objs as go
	import plotly.offline as pyof
	import csv
	import plotly.express as pxp
	from sklearn.preprocessing import StandardScaler
	from sklearn.cluster import KMeans
#importing dataset from file
	dataset= pd.read_csv('/root/HTTPD_log.csv', names= ['IP', 'identd', 'user_id', 'time', 'time_ext','req_method', 'req_dir', 'req_http_header', 'status_code', 'bytes_trans'])
#we only need the IP Address & Status Code
	dataset= dataset[['IP','status_code']]
#modifying dataset by aggregating count of status code against IP Address
	dataset= dataset.groupby(['IP','status_code']).status_code.agg('count').to_frame('Total').reset_index()
#We are inserting the Index No as it needs it, otherwise it will give Shape of passed values is (13, 2), indices imply (13, 3) error
	dataset.insert(0, 'IndexNo', range(len(dataset)))
#we are droping IP Column as instead of this we will take the Index No as reference of IP and scale it
	train_data= dataset.drop(['IP'], axis= 1)
	

	sc= StandardScaler()
	scaled_data= sc.fit_transform(train_data)
#We have used here 3 as a cluster because it's a good practice to give odd number due to the calculation of points which are crucial between two cluster
	#This solely depends and varry from data to data
	model= KMeans(n_clusters= 3)
	pred= model.fit_predict(scaled_data)
	#here IP_Scaled is actually IndexNo because IP Address is treated as string
	pred_ds= pd.DataFrame(scaled_data, columns= ['IP_Scaled', 'status_code_Scaled','Total_Scaled'])
	pred_ds['Cluster']= pred
	ds= pd.concat([dataset, pred_ds], axis= 1, sort= False)
#Here we are creating graph of Request per IP vs Count
	Graph= pxp.scatter(ds, 'Total', 'IP', 'Cluster', hover_data= ['status_code'], color_continuous_scale= 'Jet') 
	layout= go.Layout(title= 'Request/IP', hovermode= 'closest') 
	figure= go.Figure(data= Graph, layout= layout)
	graph= pyof.plot(figure, filename= 'Cluster_graph.html', auto_open= False)
#Here we will see which cluster is violating the number of request against a threshold of 500 
	black_cluster= []
	for index, row in ds.iterrows():
	    if ds['Total'].loc[index] > 500:
	          black_cluster.append(ds['Cluster'].loc[index])
	black_cluster= max(set(black_cluster), key= black_cluster.count)
#Here we have created a CSV file which will keep record for all the IP which     are under blacklist cluster
	filename= "DoS_Blacklist.csv"
	with open(filename, '+w') as csvfile:
	    csvwriter = csv.writer(csvfile)
	    csvwriter.writerow(['IP_Blacklist']) 
	    for index_in_data, row_in_data in ds.iterrows():
	        if ds['Cluster'].loc[index_in_data] == black_cluster:
	            #Check whether we have the IP already in the file
	            if ds['IP'].loc[index_in_data] not in np.array(csvfile): 
	                csvwriter.writerows([[ds['IP'].loc[index_in_data]]])
	                print("Blocking IP {0}".format(ds['IP'].loc[index_in_data]))
	                #Blocking IP Address by writing a rule in iptables
	                os.system("systemctl start firewalld")
	                
                    os.system("iptables -A INPUT -s {0} -j   DROP".format(ds['IP'].loc[index_in_data]))

This the python file, here we are using the concept of KMeans and iptables to determine the attack find IP and block it respectively. Here we have kept the threshold of request as 500. Which solely depends on the company. We have even kept the cluster as 3, it's always a good practice to keep it odd and the number totally depends upon the dataset. It varies from data to data. In my experience if you have around 5000 observations you can have 3 cluster and if it's around 8000 to 10000 you can use 5 or more accordingly.

Now we will host a web-server and think it like some company's website is hosted upon it. We will do it using docker.

So as we can see there is no container running with name webos and we have our web-server image named webserver.

Now we will run the container with attaching the /loganalysis/httpdlog to the folder where the httpd server saves the log /var/log/httpd and will expose it on port 7575. After doing so you can see the access_log and error_log files come up in the folder. Initially when we see the content of the access_log it's empty because the web-server is freshly started and no clients have hit the webserver. Now we will create the attack or say we will flood the webserver with the request and eventually it will cause some downtown or destroying the server.

For creating the DoS attack we will create one another docker container using my previous image of kali linux which has shellinabox enabled. On top of it we will install python3 and slowloris.py this is the tool to create DoS attack on the webserver. We will even use Kali Linux VM to do the same attack and local command line trick to increase the load on the server. So, finally we create a mini DDoS attack using three ways at the same time. So let's create the container first. You can also download it from docker hub. I have created the final image and uploaded it. Even, I will show you how to make a image out of the container and upload it to the docker hub.

You can see here after making a request to the website it generated log. here tail -f <filename> is a linux command to see live update in a file.

Here we have created the kali linux container. Now we will install slowloris here from github.

Copy the URL and download it using git clone as shown below.

Now to run slowloris we need python, hence we will download python v3

Now you can see that slowloris is working fine. Now we will write the command to do DoS.

python3 slowloris.py -s 1000 --sleeptime 2 -v -p 7575 192.168.99.102

This is the command to create DoS of request 1000 every 2 second. Here -s 1000 is called socket, it means it will throw request to this many ports on the webserver. --sleeptime 2 means it will wait for 2 seconds between each attack. -v means verbose i.e. to see the background process. -p for port number where the website is working and finally the target host or domain name.

Same attack we will do from Kali linux VM, for this you need Kali Linux setup as VM.

Sorry for the clarity of the image but the command is same.

Now we will create a simple linux command to just increase the load on the webserver.

This is just a for loop which runs 200 times and do curl operation on the website.

Now, we will execute this three things and we will see the real time update in the log file with 100's of request flooding.

You can see the log file flooding.

Similarly after starting it from kali container it will increase.

Then if we see the output of our for loop you will face some down time getting the output. This is because of DoS.

Finally within some time we terminated the attack and we can see we created a flood of around 6000 request or flood we have made. Now we need to run the jenkins job. But before that we will again start the attack. Why? Because when we will start the jenkins job and within a span of 1 minute the entire process will execute automatically and these IP will be blocked and there won't be any further log updated and the slowloris script wont find the host running for them, but for other clients it will still work.

Here for once we have to press the build button, even we can add trigger through the git hooks so as soon as developer commits it will start. For this you can refer my previous articles. This will download and copy the files. This will automatically trigger the next job.

Within some seconds you will see that they have successfully find the DDoS IP and blocked them. To confirm we can go to the attacker machine and see the slowloris attack is not not happening as it's not able to find host. Even the log file is now not flooding.

Kali Container

Kali Vm

Log file status: not flooding

The other users or client can still access

??

Now if you are not able to find the DoS tools I am making the image of the container and uploading it in the docker hub. You can download and perform the task.

So using docker commit, I have made image out of container.

And then uploaded to the docker hub. You can download and practice this or setup for your personal use.

Now, you can see every two minutes its running the job and doing the necessary things for us. So now the security team and devops team don't have to waste time or focus on DoS or DDoS.

??GitHub https://tinyurl.com/y7zf9555

Thank you very much for reading this article patiently?? Feel free to connect and share your views!