What is an open source IDS and IPS?
Recently I have been reviewing the courses Pluralsight has on open source IDS's and IPS's. This has resulted in me creating our skills path for our learners entitled "Enterprise Security Monitoring with Open Source Network IDS & IPS". I thought it may be of value to others to see what I discovered during that research.
Wikipedia says that an intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any intrusion activity or violation is typically reported either to an administrator or collected centrally using a security information and event management (SIEM) system. A SIEM system combines outputs from multiple sources and uses alarm filtering techniques to distinguish malicious activity from false alarms.
Wikipedia goes on to say that IDS types range in scope from single computers to large networks. The most common classifications are network intrusion detection systems (NIDS) and host-based intrusion detection systems (HIDS). A system that monitors important operating system files is an example of an HIDS, while a system that analyses incoming network traffic is an example of an NIDS. It is also possible to classify IDS by detection approach. The most well-known variants are signature-based detection (recognising bad patterns, such as malware) and anomaly-based detection (detecting deviations from a model of "good" traffic, which often relies on machine learning). Another common variant is reputation-based detection (recognising the potential threat according to the reputation scores). Some IDS products have the ability to respond to detected intrusions. Systems with response capabilities are typically referred to as an intrusion prevention system (IPS). Intrusion detection systems can also serve specific purposes by augmenting them with custom tools, such as using a honeypot to attract and characterise malicious traffic.
Veronis created this helpful diagram to show the difference between an IDS and an IPS:
Open Source IDS and IPS Tools
When searching online for the top 3 open source and free tools, there are 3 that stand out; Snort, Suricata and Zeek (formally known as Bro).
Snort
Snort acts as both an IDS and IPS, has a high level of customization that makes it the choice for a lot of different organizations. This Linux utility is easy to deploy and can be configured to monitor your network traffic for intrusion attempts, log them, and take a specified action when an intrusion attempt is detected. It has the ability to perform real-time traffic analysis and packet logging on Internet Protocol (IP) networks. It uses a series of rules that help define malicious network activity and uses those rules to find packets that match against them and generates alerts for users.
It performs protocol analysis, content searching and matching. The program can also be used to detect probes or attacks, including, but not limited to, operating system fingerprinting attempts, semantic URL attacks, buffer overflows, server message block probes, and stealth port scans. Snort can be configured in three main modes: 1. sniffer, 2. packet logger, and 3. network intrusion detection.
There’s no GUI here and it lacks an administrative console, although lots of other open source tools have been created to help out, such as Snorby, BASE and Sguil. These tools provide a web front end to query and analyze alerts coming from Snort IDS.
Major commit version changes:
- 2021 Snort 3.0 stable release (beta 2018)
- 2003 Snort 2.0
- 1998 Snort 1.0
Suricata
Suricata’s architecture is different from Snort, however it behaves the same way as Snort and can use the same signatures. The Suricata engine is capable of real time intrusion detection (IDS), inline intrusion prevention (IPS), network security monitoring (NSM) and offline pcap processing. There are third-party open source tools available for a web front end to query and analyze alerts coming from Suricata IDS. It has standard input and output formats like YAML and JSON, meaning it integrates with tools like Splunk, Logstash/Elasticsearch, Kibana, and other databases.
A single Suricata instance is capable of inspecting multi-gigabit traffic. The engine is built around a multi threaded, modern, clean and highly scalable code base. There is native support for hardware acceleration from several vendors and through PF_RING and AF_PACKET. Suricata will automatically detect protocols such as HTTP on any port and apply the proper detection and logging logic. This helps with finding malware and C&C or C2 channels.
Through the Unified2 output format and the Barnyard2 tool, Suricata can be used with BASE, Snorby, Sguil, SQueRT and other tools. For all the features see the following web page.
Major commit version changes (https://suricata-ids.org/category/release/):
- Oct 2020 6.00
- Oct 2019 5.0.0
- Jul 2017 4.0
- Jan 2016 3.0
- Mar 2014 2.0
- Jul 2012 1.3.1
Zeek
With Zeek you can detect suspicious signatures and anomalies, track DNS, HTTP, and FTP activity. You can also monitor SNMP traffic, which enables you to check SNMP trap conditions and any changes to device configurations. Event data mining is performed by policy scripts. Each policy is essentially a collection of rules, and you can have as many active policies or protocol stack layers as you want. This tool is capable of automatically downloading suspicious files it spots on the network, sending them for analysis, notifying relevant parties if anything is uncovered, blacklisting the source and shutting down the device that downloaded it.
Zeek functions as a network traffic analyser and an intrusion prevention system, with alert conditions provoking predefined actions. Policy scripts are customisable, but they’re limited by the standard framework, which involves anomaly detection, analysis of connections, and signature matching. There’s no native GUI but there are third-party open source tools available for a web front end to query and analyse alerts coming from the IDS, you could consider using ELK stack. Zeek also allows the import of Snort signatures.
Major commit version changes:
- Dec 2020 v3.2.3
- Sept 2019 v3.0
- 2011 v2.0
- 2005 v1.0
- 1997 v0.2
Extensions and Integrations
What is Arkime (formerly Moloch)?
Arkime (formerly Moloch) is a large scale, open source, indexed packet capture and search tool. It is used to augment your current security infrastructure and provides more visibility to store and index network traffic in standard PCAP format. The interface offers a web application for PCAP browsing, searching, analysis, and PCAP carving for exporting.
What is the Elasticsearch with Logstash and Kibana (ELK)?
Elasticsearch is developed alongside a data collection and log-parsing engine called Logstash, an analytics and visualisation platform called Kibana, and Beats, a collection of lightweight data shippers. The four products are designed for use as an integrated solution, referred to as the "Elastic Stack" (formerly the "ELK stack"). ELK can help to manage, aggregate and visualize logs generated by Snort.
What is BASE? (Snort/Suricata)
BASE is the Basic Analysis and Security Engine. It is written in php and based on the code from the Analysis Console for Intrusion Databases (ACID) project. This application provides a web front-end to query and analyze the alerts coming from a Snort IDS system. The interface allows the classification of alerts in groups, the display of diagrams and the search for alerts according to different criteria.
What is Pulled Pork? (Snort/Suricata)
PulledPork is an open source perl script that can automatically update your rules. Using a regular crontab you can keep your Snort or Suricata rules up to date automatically. It provides Checksum verification for all major rule downloads.
What is Barnyard2? (Snort)
Barnyard2 is an open source interpreter for Snort unified2 binary output files. Its primary use is allowing Snort to write to disk in an efficient manner and leaving the task of parsing binary data into various formats to a separate process that will not cause Snort to miss network traffic.
What is Snorby? (Snort/Suricata)
Snorby is a front end web application (scripted in Ruby on Rails) for any application that logs events in the unified2 binary output format. It is important to understand that Snorby is a front end for other applications, and that the administration of your Intrusion Detection System (IDS) (ie.: Snort, Saga, Suricata), will not always be done through the Snorby interface. It is also important that you become familiar with the underlying IDS for proper tuning and updating.
Snorby supports OpenFPC, the capability to have a full transcript of the network traffic. This enables an analyst to be able to see the entire conversation surrounding an attack.
What is Sguil ? (Snort/Suricata)
Sguil (pronounced sgweel) is built for network security analysts. Sguil's main component is an intuitive GUI that provides access to real time events, session data, and raw packet captures. Sguil facilitates the practice of Network Security Monitoring and event driven analysis. The Sguil client is written in tcl/tk and can be run on any operating system that supports tcl/tk (including Linux, *BSD, Solaris, MacOS, and Win32).
What is Sagan? (Snort)
The Snorby E Book explains that Sagan is an open source (GNU/GPLv2) high performance, real-time log analysis and correlation engine that run under *nix operating systems (Linux/FreeBSD/ OpenBSD/etc). It is written in C and uses a multi-threaded architecture to deliver high performance log & event analysis.
Sagan's structure and rules work similarly to the Sourcefire "Snort" IDS/IPS engine. This was intentionally done to maintain compatibility with rule management software (oinkmaster / pulledpork /etc) and allows Sagan to correlate log events with your Snort IDS/IPS system.
Sagan can also write to Snort IDS/IPS databases via Unified2/Barnyard2. Sagan is compatible with all Snort "consoles". This compatibility makes it possible to use Sagan with Snorby.
Sagan supports many different output formats, log normalization (via liblognorm), script execution on event detection, automatic firewall support via "Snortsam", GeoIP detection / alerting, multi-line log support, time sensitive alerting.
What is SQueRT? (Suricata)
Squert is a web application that is used to query and view event data stored in a Sguil database (typically IDS alert data). Squert is a visual tool that attempts to provide additional context to events through the use of metadata, time series representations and weighted and logically grouped result sets. The hope is that these views will prompt questions that otherwise may not have been asked.
What is EveBox? (Suricata)
EveBox is a web based alert and event management tool for events generated by the Suricata network threat detection engine. EveBox consists of a JavaScript frontend, and a very minimal backend written in Go. It is also included in SELKS which provides Suricata and an ELK stack configured and ready to go. You can also install EveBox with Docker.
What is Scirius? (Suricata)
Scirius Community Edition is a web interface dedicated to Suricata ruleset management. It handles the rules file and update associated files.
Pluralsight current courses on Snort, Suricata and Zeek
My research has resulted in us looking to commission a further six courses in this series.