Suricata Log Parsing using jq [DL Series-7]

In this seventh part of the ‘Detection Lab’ series, we focus on the powerful JSON parsing tool jq. Logs are the backbone of any security monitoring system, and efficiently parsing them is essential for actionable insights. In this article, you will learn the basics of jq, understand its role in log parsing for security engineers, and apply it to Suricata logs. By the end, you’ll have a firm grasp of using jq to extract meaningful data from JSON logs to enhance your security operations workflow.

What is jq?

jq is a lightweight and flexible command-line tool for processing JSON data. It allows you to parse, filter, transform, and extract specific information from complex JSON structures. For security engineers, working with tools like Wazuh, Suricata, or Sysmon, which produce JSON-formatted logs, jq becomes indispensable.

How Can jq Help a Security Engineer?

• Efficient Analysis: Extract meaningful information quickly from large datasets, such as identifying the source IP from a bulk log file.
• Custom Queries: Focus on fields relevant to specific incidents or investigations.
• Integration: Seamlessly integrate with scripts or monitoring pipelines. For example, jq can be used in active response scripts to dynamically process JSON data, such as extracting file paths from VirusTotal alerts for automated remediation.
• Log Verification: Check if logs are properly collected and contain the expected fields. For instance, use jq 'has("alert")' to verify the presence of an alert field in the log data.

To install jq:

sudo apt-get install jq        

Basics of jq

jq works by applying filters to JSON data. Here’s a simple breakdown of how it works:

jq [options] 'filter' <input_file>        

? [options]: Optional flags that modify how jq works (e.g., -c for compact output).

? ‘filter’: The jq filter used to select or modify data.

? <input_file>: The file or data being processed.

Using jq for Parsing Suricata Log Sample

Selecting Events Containing Alerts

jq -c 'select(.alert)' /var/log/suricata/eve.json        

Command Breakdown

? jq -c: Enables compact output, showing each JSON object on a single line.

? select(.alert): Filters events where the .alert field exists.

? /var/log/suricata/eve.json: The Suricata log file being parsed.

This extracts all events that include alert-related information, such as signatures and timestamps.

Filtering TLS Events with Matching Subject and Issuer DN

jq 'select(.event_type=="tls" and .tls.subject==.tls.issuerdn)' /var/log/suricata/eve.json        

Command Breakdown

? select(.event_type==”tls”): Filters events where event_type is tls.

? .tls.subject==.tls.issuerdn: Ensures that the certificate’s subject matches the issuerdn, indicating a self-signed certificate.

This identifies potential anomalies where certificates are self-issued.

Finding Established SSH Flows

jq 'select(.event_type=="flow" and .flow.state=="established" and .flow.age<1800 and .app_proto=="ssh")' /var/log/suricata/eve.json        

Command Breakdown

• select(.event_type==”flow”): Filters for flow-related events.
• .flow.state==”established”: Matches established connections.
• .flow.age<1800: Filters flows active for less than 30 minutes.
• .app_proto==”ssh”: Identifies flows using the SSH protocol.

This identifies active and recent SSH connections, detecting ongoing or unauthorized access.

Detecting Dshield Alerts

jq -c 'select(.event_type=="alert")' /var/log/suricata/eve.json | jq -c 'select(.alert.signature | contains("Dshield"))'        

Command Breakdown

? select(.alert.signature | contains(“Dshield”)): Matches alerts where the signature contains the word “Dshield.”

This identifies alerts related to Dshield, a threat intelligence source, enabling detection of potential malicious activity.

Counting Alerts by Source IP

jq -c 'select(.event_type=="alert" and .signature=="ET DROP Dshield Block Listed Source group 1")' /var/log/suricata/eve.json | jq .src_ip | sort | uniq -c        

Command Breakdown

? .signature==”ET DROP Dshield Block Listed Source group 1": Matches specific alerts.

? jq .src_ip: Extracts the src_ip field from the matched events.

? sort | uniq -c: Counts occurrences of each unique source IP.

This identifies the frequency of specific alert activity by source IP, highlighting patterns of malicious sources.

Avoiding Null or Empty Alert Signatures

jq '.alert.signature | select(.!=null)' /var/log/suricata/eve.json | uniq -c        

Command Breakdown

? select(.!=null): Ensures only non-null values are included.

This identifies meaningful alert signatures, filtering out irrelevant or incomplete data.

Investigating Specific Alert Details

jq -r 'select(.alert.signature == "ET CINS Active Threat Intelligence Poor Reputation IP group 28") |
       .timestamp + " " + .src_ip + " " + (.src_port | tostring) + " " + .dest_ip + " " + (.dest_port | tostring) + " " + .alert.signature' /var/log/suricata/eve.json | while read line; do
    read timestamp sip dip sp dp signature <<<$(echo $line);     
    echo "timestamp = $timestamp --- srcip = $sip --- dstip = $dip --- srcport = $sp --- dstport = $dp --- signature = $signature"; 
done        

Command Breakdown

Part 1: Using jq to Extract Data

This function filters the events to only include those where the alert signature matches “ET CINS Active Threat Intelligence Poor Reputation IP group 28.”

The part .timestamp + “ “ + .src_ip + “ “ + (.src_port | tostring) + “ “ + .dest_ip + “ “ + (.dest_port | tostring) + “ “ + .alert.signature formats the output by combining key fields for investigation. The src_port and dest_port are converted into strings using the | tostring pipe to ensure the port numbers are treated as text.

Part 2: Piping the Output to while read Loop

| while read line; do

The output from the previous jq command is passed into this while loop. The loop processes each line of the input one by one.

? read line: Reads a single line of input into the variable line.

read timestamp sip dip sp dp signature <<<$(echo $line)

This line splits the contents of $line into multiple variables.

? read timestamp sip dip sp dp signature: Reads and assigns the space-separated values from $line to the variables timestamp, sip(source IP), dip (destination IP), sp (source port), dp (destination port), and signature.

? <<<$(echo $line): This syntax sends the contents of $line to the read command. The <<< operator feeds the string into read, effectively splitting it based on spaces.

echo “timestamp = $timestamp — — srcip = $sip — — dstip = $dip — — srcport = $sp — — dstport = $dp — — signature = $signature”;

This prints out the values of the variables in a readable format, making the output easy to interpret.

? It uses the echo command to format and print the values of the variables with labels (e.g., timestamp, srcip, dstip, etc.), showing the data for each alert.

done

This marks the end of the while loop. Once all lines have been processed, the loop ends, and the script finishes executing.

Investigating Specific Alert Details — Download in .csv

jq -r 'select(.alert.signature == "ET CINS Active Threat Intelligence Poor Reputation IP group 28") |
       .timestamp + " " + .src_ip + " " + (.src_port | tostring) + " " + .dest_ip + " " + (.dest_port | tostring) + " " + .alert.signature' /var/log/suricata/eve.json >> suricata_alerts.csv | while read line; do
    read timestamp sip dip sp dp signature <<<$(echo $line);     echo "timestamp = $timestamp --- srcip = $sip --- dstip = $dip --- srcport = $sp --- dstport = $dp --- signature = $signature"; done        

This will download the result in suricata_alerts.csv.

Conclusion

jq is an invaluable tool for security engineers dealing with JSON logs. By mastering its syntax and capabilities, you can streamline log parsing, reduce noise, and focus on actionable data.

Upcoming

In the next article of the Detection Lab series, we’ll dive into creating a customized Wazuh dashboard that transforms logs into meaningful insights. Whether you’re new to Wazuh or looking to enhance your existing setup, this article will guide you through configuring dashboards to visualize security events, streamline monitoring, and drive more effective incident response. Stay tuned as we unlock the full potential of Wazuh’s visualization tools!

Check out the next article here: The post will be available on 26/01/2025. Stay tuned!

Feel free to ask questions or share your feedback in the comments section — I’d love to hear from you! You can also connect with me on Gibin John to to clarify any doubts or continue the conversation.

Follow my Medium profile to stay updated on the full series: Gibin John.

#wazuh #WazuhSecurity #CyberDetectionLab #WazuhSIEM #ThreatHunting #CybersecurityLab #Wazuh #SecurityMonitoring #SIEM #IncidentResponse #ThreatDetection #SecurityOps #WazuhDetection #SOCLab #CyberThreatIntel #WazuhAlerting #LogAnalysis #CyberDefense #MalwareDetection #Sysmon #SOCAnalysis