Securing your web server in 10 easy steps. A practical guide.

Hi! I'm Ihor Chyshkala and as a CTO in IC Studio I responsible for infrastructure solutions for clients and the company as well.

Some IT-savvy people believe that clouds solutions can provides you not only with easy-to-use resources like VPS instances or lambda function or some RTDB (real time database) as a service BUT also done all security tasks at once. That's not true. Today we're gonna make your web server more secure.

I will try to write this article in a light manner with elements of humour in order to brighten up the abundance of technical terms.

All of the following tips we apply in our work and they really help us to cope with the influx of various attacks to which publicly available services are exposed.

We will touch on aspects that sometimes go beyond basic security settings, but nevertheless reduce the number of potential vulnerabilities by 80%.

This article will be useful for devops or full stack software engineers whose responsibilities include deploying, maintaining and monitoring Laravel and/or Next.js based web applications.

I also hope you are using nginx web server. If for some reason you are using Apache... well, God is your judge.

Let's get to the tips.

1. Monitoring and backups

The first and most important aspect. No matter how well and deeply you have designed the security of your infrastructure, there is always the possibility of a 0-day vulnerability or human error that will lead to a breach.

Like a patient resuscitation or a fire service, you should have a clearly worked out plan to respond to such situations.

Which managers should be notified, even if it's 2 a.m., which services should be ‘silenced’ to prevent further damage spreading. After all, the hack of one node of the infostructure can lead to further hacks, if the attackers acquire new access, for example, private keys ssh, API keys, passwords to the database and so on.

2. Block bad IPs in bulk

Let's start with the fact that will not let even close to the perimeter of publicly known IPs with a bad (what is there, terrible) reputation. Thanks to such a phenomenon as Bulletproof hosting (BPH), hackers can rent attack servers on such sites without fear and non-stop, 24 hours a day, 7 days a week and 365 days a year to look for vulnerabilities simply by the list. From open ports to publicly accessible .git folders and .env files

Hopefully you already have iptables installed on your server. It is usually included in all Linux distributions that I know of. For convenience, I'm going to assume that you have Ubuntu 22.04 installed

Login as root and create this wonderful bash script ipset-rules.sh

touch ipset-rules.sh        

chmod +x ipset-rules.sh        

The chmod +x command allows this file to be executed. Now let's fill it with its contents. Keep in mind that it is good practice on Linux systems to add a blank line at the end of the file. Sometimes it doesn't matter, and sometimes without that space nothing may work.

#!/bin/bash

# Function to show progress
show_progress() {
    local current=$1
    local total=$2
    local progress=$((current * 100 / total))
    printf "\rProgress: [%-50s] %d%%" $(printf "#%.0s" $(seq 1 $((progress/2)))) $progress
}

# Install ipset if not already installed
sudo apt-get update
sudo apt-get install -y ipset

echo "Downloading and processing IP list..."
# Download the ipsum list and clean it
wget -qO- https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt | cut -f 1 -d"#" | sed '/^$/d' | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" > /tmp/bad_ips.txt

# Count total IPs
total_ips=$(wc -l < /tmp/bad_ips.txt)
echo "Total IPs to process: $total_ips"

# Calculate the number of ipsets needed (assuming max 1 million IPs per set)
num_sets=$(( (total_ips + 999999) / 1000000 ))

echo "Creating $num_sets ipset(s)..."

# Create ipset(s) with larger capacity
for ((i=1; i<=num_sets; i++)); do
    sudo ipset create bad_ips_$i hash:ip maxelem 1000000
done

echo "Adding IPs to ipset(s)..."
# Add IPs to the ipset(s) with progress
current_ip=0
current_set=1
while IFS= read -r ip; do
    if ! sudo ipset add bad_ips_$current_set $ip 2>/dev/null; then
        current_set=$((current_set + 1))
        sudo ipset add bad_ips_$current_set $ip
    fi
    current_ip=$((current_ip + 1))
    show_progress $current_ip $total_ips
done < /tmp/bad_ips.txt

echo -e "\nCreating iptables rules..."
# Create iptables rules to block the ipset(s)
for ((i=1; i<=num_sets; i++)); do
    sudo iptables -I INPUT -m set --match-set bad_ips_$i src -j DROP
done

echo "Saving iptables rules..."
# Save iptables rules
sudo iptables-save | sudo tee /etc/iptables/rules.v4

echo "Ensuring iptables rules are loaded on boot..."
# Ensure iptables rules are loaded on boot
sudo apt-get install -y iptables-persistent

# Clean up
rm /tmp/bad_ips.txt

echo "Firewall updated with ipsum list using ipset(s)"        

Here we use the utility ipset, which is more convenient to create and add whole lists of ‘bad’ IPs for complete blocking. As the list itself we use the well-known public and constantly updated list https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt.

There are about 190k IP-addresses in this list that have been seen in malicious activity. I think it would take a month to add that many addresses manually. But even a script can take a considerable amount of time to run for the first time, so be patient. On average on a new server it took us from 30 minutes to 4 hours, depending on the resources.

3. PhpMyAdmin, phpPgAdmin, Adminer...

Database managers are quite popular even on production servers. Although we strongly recommend not to use them at all, there is sometimes a need to use them. Sometimes even periodically.

1. Make sure that remote access is not allowed to the database server

2. Please do not make the address for phpMyAdmin a default address like /phpMyAdmin

3. Be sure to use TLS certificate and HTTPS protocol to work with tools like phpMyAdmin. Because if you want to download a database dump for yourself and do it in a public place, then over the protocol http all your data will be available for man-in-the-middle attacks.

4. Add http authorisation to use phpMyAdmin. Here is a sample for nginx conf:

# phpMyAdmin Nginx configuration with HTTP auth and random path
location /db_admin_x7k9v2 {    # Random string instead of 'phpmyadmin'
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.htpasswd;
    
    root /usr/share/phpmyadmin;
    index index.php;

    location ~ ^/db_admin_x7k9v2/(.+\.php)$ {
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php/php-fpm.sock;  # Adjust this path based on your PHP-FPM setup
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }

    location ~* ^/db_admin_x7k9v2/(.+\.(jpg|jpeg|gif|css|png|js|ico|html|xml|txt))$ {
        root /usr/share/;
    }
}        

4. Limit the number of requests per second from one IP

In nginx it's easy to limit the number of requests from one IP and it's cool!Let's assume that 15 requests per second is more than enough for your client to comfortably use your public service.Then add this line at the top of your nginx config:

limit_req_zone $binary_remote_addr zone=your-domain.tld:960k rate=15r/s;        

Don't forget to replace your-domain.tld with your one.

5. Block access to sensitive files and directories

Laravel or Next.js can be as secure and clever frameworks as you want, but they have no power over web server settings. So don't give attackers a chance. As a rule of thumb, all files that start with . (dot) in Linux are service files and contain sensitive information. So there is no need to access them from a browser.

location ~ ^/(\.git|\.env|\.htaccess|\.htpasswd|\.user\.ini|php\.ini) {
        deny all;
        return 404;
    }
location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }        

6. Security for your HTTP headers

add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=()" always;
add_header Strict-Transport-Security "max-age=63072000" always;        

7. If you don't use PHP, disable it!

Surprisingly, hackers love PHP too. It is not uncommon to see ‘planted’ php scripts in other projects, if the server had support for this server language.

But let's surprise hackers and forbid execution of any php scripts. This will be relevant for node projects. In this case, even if an attacker can somehow upload his malicious script to you, he will not be able to execute it.

# Block all PHP execution
    location ~ \.php$ {
        deny all;
        return 404;
    }        

8. Block access to sensitive files and directories for node (deno) app

As with Laravel, node (deno) applications have their own service files and folders that should not be accessed from the browser. So make sure to explicitly close access to them.

All files from the root of your project (webpack-mix.js, webpack-config.js version.md...) can be added here, because the node web server runs inside your application and is accessible by port (even in docker mode), this is a fundamental difference from php applications.

location ~ ^/(\.git|\.env|\.next|node_modules|package\.json|package-lock\.json) {
        deny all;
        return 404;
}        

9. ClamAV

ClamAV is a popular open-source antivirus engine for detecting trojans, viruses, malware, and other malicious threats.

# On Debian/Ubuntu
sudo apt install clamav clamav-daemon

# On RHEL/CentOS
sudo dnf install clamav clamd

# Update virus definitions manually
sudo freshclam

# Enable automatic updates
sudo systemctl enable clamav-freshclam
sudo systemctl start clamav-freshclam        

Basic Scanning Commands:

# Scan a single file
clamscan file.txt

# Scan a directory recursively
clamscan -r /path/to/directory

# Scan and show only infected files
clamscan -i /path/to/scan

# Scan and remove infected files (be careful!)
clamscan --remove /path/to/scan

# Scan with detailed output
clamscan -v /path/to/scan

# Scan and log results to a file
clamscan -r /path/to/scan --log=scan_results.txt        

Using the Daemon (clamd):

# Start the daemon
sudo systemctl start clamav-daemon

# Scan using the daemon (faster for multiple scans)
clamdscan /path/to/scan        

10. Fail2ban

Fail2ban is a tool used to enhance the security of a web server by preventing brute-force attacks. It works by monitoring log files for suspicious activity and then creating temporary rules to ban IP addresses that exhibit malicious behavior.

sudo apt install fail2ban        

Configure Fail2ban: The default configuration file is located at /etc/fail2ban/jail.conf, but it's recommended to create a local.

Copy to avoid overwritten settings during updates:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local        

Customize Settings: Edit the jail.local file to configure general settings, such as ban time, find time, and max retry attempts.

Manage Fail2ban: Start the Fail2ban service and enable it to run at boot:

sudo systemctl start fail2ban
sudo systemctl enable fail2ban        

Monitor Fail2ban: Use the Fail2ban client to check the status and logs for any banned IPs:

sudo fail2ban-client status
sudo fail2ban-client status <jail_name>        

root@server2:~# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	16258
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	1
   |- Total banned:	989
   `- Banned IP list:	128.1.141.22        

I realised as I was writing this article that I still have a lot of material and would be happy to share it as I have free time. Unfortunately, CTOs don't have much of it. Next time we will talk about TCP ports, honey traps (my favourite!) and more about fail2ban.

Kind regards,

Ihor Chyshkala

Full stack software engineer

CTO at IC Studio