Analyzing SSH Attempts on a Linux Server

(7 min read)

The past couple of weeks were a sweet reminder that the internet is indeed a very noisy place, when I started analyzing some of the SSH attempts on a virtual server that I had recently set up. The first attempt took place by the eighth minute after setting up, leading the way to an increasing number of consecutive logs. These attempts led me on to further research this trend for patterns and gain more insight about the subject. Needless to say, any attempt to log into a private server is illegal. Yet, as the ex LulzSec hacker Mustafa Al-Bassam pointed out, cyber-crime is constantly becoming more industrialized.

As a server I decided to use an Amazon EC2 instance with a default Ubuntu 20.04.1 LTS operating system on it. One of the easiest ways of connecting to it is via the ubiquitous SSH protocol, also referred to as Secure Shell. SSH is generally used for managing servers and has been designed as a replacement for insecure protocols such as Telnet and rlogin. It can also be used for file transfers, which could be dangerous if used with the wrong intentions. The most common connectivity tool is OpenSSH and in this case the default OpenSSH_8.2p1 was installed. This can be used on Unix-like operating systems and now also on Windows. In fact, Windows 10 uses OpenSSH as its default SSH client and SSH server. A public IP has been assigned to the server and incoming traffic from the internet is only allowed on port 22, the standard TCP port for SSH. Public-key authentication has been set up and password authentication has been disabled with no further hardening being carried out. 

As Outlined in “Cryptography and Network Security: Principles and Practice”, by William Stallings:

“Public-key cryptography, or asymmetric cryptography, is a cryptographic system that uses pairs of keys: public keys, which may be disseminated widely, and private keys, which are known only to the owner. Effective security only requires keeping the private key private; the public key can be openly distributed without compromising security.”

If such keys are not automatically generated, they can be generated using the ssh-keygen OpenSSH authentication key utility.

For this project, I have decided to use Linux commands/utilities (i.e. less, grep, sort, uniq, awk and sed) and Bash scripting for data cleaning and Python (i.e. Matplotlib and Plotly) for data visualization. The aim was to practice the above-mentioned tools, however there are other resources for ingesting logs and visualizing them, such as Splunk, Elasticsearch and Grafana with Loki.

In Debian systems, the SSH logs would be found in the authorization log file named auth.log, which is located in /var/log. For Redhat/CentOS, it would be /var/log/secure. Such logs are captured and processed by default using the rsyslog log processing system. You can configure logrotate to periodically rotate and compress log files, discard old log files after a certain period of time and also add the compression date to the filenames using dateext. The logs can also be emailed. The SSH logs would have a sshd tag assigned to them and would end with [preauth] if they fail before the authentication occurs.

Upon the first perusal, four kinds of logs could be identified depending on the different scenarios outlined below:

1) Attempting to log in without any username or if the server and the remote system don't share at least one cipher. Without any agreed cipher no encrypted channel is possible.

 2) Inputting a non-existent or invalid username.

3) A valid username attempting to use a password or cancelling the connection without trying a password after key attempt failure.

4) The attacker might probably be using some custom code to brute-force, which results in sending malformed authentication requests. Thus, the connection is killed by the server.

Further details regarding OpenSSH Logging and Troubleshooting can be found here.

After two weeks of running the server I started off by sorting out the relevant logs by date, so as to get an idea of the number of attempts per day.

1.A bar graph showing the daily SSH attempts over a span of two weeks.

As one can observe, the number of attempts increased considerably by the first week and eventually decreased whilst remaining significant after two weeks. The decrease might indicate that upon repeated failures, certain brute-force scripts could permanently stop attempting access and move on to other IP addresses.

The attempted usernames have been extracted and a word cloud with the top 70 can be seen below.

2. A word cloud showing the top 70 attempted usernames.

The percentage of valid username attempts (i.e. root & ubuntu) is around 65%. These are usually created by default on Ubuntu servers. This highlights the importance of disabling SSH for root, by setting the PermitRootLogin to “no” in the sshd_config file.

Next up is the top 10 invalid username attempts. Most of them are the default usernames on a variety of devices.

*Information Source: WePrivacy, which is a site dedicated to providing online privacy and security tips.

*Nagios is a free and open-source computer-software application that monitors systems, networks and infrastructure whilst *OSMC is a free and open-source media centre.

The table shows that scripts are trying to brute-force a variety of devices using default usernames and passwords. Dictionaries of such credentials can be found online and compromised devices might then be recruited in DDOS attacks. Therefore, any default credentials should always be changed.

An analysis of the IP location of around 7800 attempts which took place over a span of 2 weeks was translated into choropleth map. 

3. A choropleth map showing SSH attempts by country.

The majority of the attempts came from China, Hong Kong, USA, Taiwan and Russia. Whilst on the map China seems to be the top attempter, it is Hong Kong who wins the category but is not clearly shown due to its geographic dimensions. Whois has been used to get location details, however there are companies having accurate databases, such as MaxMind and IPinfo, which support APIs and provide a number of free requests per month. Shodan can also be used for further analysis, especially when it comes to compromised IOT devices. However, the above is merely just an exercise in mapping since any attacker can hide behind a proxy server which changes the source IP or use a Virtual Private Server that can be hosted anywhere.

4.The Top 10 Attackers

One can point out the ISPs are all hosting companies, indicating the possibility that they are either proxy servers or poorly secured SSH servers which have been compromised by DDOS botnets and are in turn trying to propagate. All of the above IPs have been reported as malicious on Virus Total. They have consistently been reported on AbuseIPDB by global distinct system administrators, with some being reported more than 10,000 times.

I then used Fail2ban which operates by monitoring log files and updating the Firewall rules to reject the IP addresses which fail repeatedly. The aim was that of decreasing log noise. This has been set up to block IPs that failed 3 times. Till now it has banned a total of 215 IPs based on 9950 failures, however, the number of logged daily attempts did not show any significant decrease.

Setting up a more aggressive approach might not be a better solution since the IPs change very frequently and can also be spoofed. Banning entire countries that are well known for brute-force attacks or regularly banning the top attackers might also not be a solution since as indicated in the top 10, attackers can use cloud-hosting providers. The best solution would be that of setting up Static IP addresses on the clients who require to SSH the server and only allow their IP addresses on the Firewall. Some administrators even change the default SSH port which can easily be done. This is shown to drastically reduce the number of attempts; however, it is considered as security through obscurity and should never be the only security mechanism in place.

Just to summarize all of the above, next up is a list of key take aways and further notes if you’d like to give it a go yourself:

• Use SSH version 2 as opposed to version 1 since the latter has inherent design flaws which make it vulnerable.
• Audit your OpenSSH configuration. ssh-audit is an advanced SSH hardening Python tool for auditing client and server configuration.
• If you really have to use Password Authentication, change the default credentials, ensure a strong password and use a Password Manager.
• If not required, always disable root login and disable Password Authentication.
• Public-key authentication is a must since it provides cryptographic strength.
• If you don’t require SSH for automation, the Private Key can also be secured with a passphrase.
• Banning entire subnets of the usual suspects or regularly banning the top attackers is impractical.
• If possible, assign Static IPs to clients and block access to the rest.
• The default port of SSH can be changed but it is also considered as security through obscurity.
• Logs can be shipped to a separate segregated server which ingests them to a SIEM tool for a better log indexing, alerting and dash-boarding.

Just keep in mind that exposing any management ports to the internet should be avoided since they could be an entry point for malicious players. Make sure that any commissioned service is protected and hardened as much as possible.