SIEM Big Data Visualization [05] : P2PComm_GeoTopology_Map_Plugin_App

This article will introduce the Peer to Peer Communication Geolocation Topology Map Plugin App developed for SIEM big data analytics. This is part of the SIEM big data visualization project, if you are interested about other dashboard plug in, you can check below links:

• SIEM Big Data Visualization [01] : Dashboard for Monitoring Scam Events in Critical Infrastructure
• SIEM Big Data Visualization [02] : Dashboard for Summarizing SG National Cyber Threats in Critical Infrastructure
• SIEM Big Data Visualization [03] : Graph-Based SIEM Log Analysis Dashboard
• SIEM Big Data Visualization [04] : Data Transmission Latency SIEM Log Analysis Dashboard

Program Design Purpose:

The goal of this project is to develop a Flask-based web App plugin that can be integrated into other Security Information and Event Management (SIEM) system for devices' communication data and state visualization. This plugin is designed to visualize real-time peer-to-peer (P2P) communication status within a distributed system with key metrics such as data flow throughput between encrypted devices and data transmission speed between servers and clients will be displayed alongside the network topology on an interactive geolocation map. This visualization Web-App will enable security administrators to effectively monitor, analyze, and manage real-time P2P communication states, helping to identify potential performance bottlenecks, latency issues, and abnormal communication patterns across the distributed network.

For the program demo, please refer to this video:

# Version:     v0.2.1
# Created:     2020/05/22
# Copyright:   Copyright (c) 2024 LiuYuancheng
# License:     MIT License         

Introduction

The P2PComm_GeoTopology_Map_Plugin is a Flask-based web application designed to visualize real-time peer-to-peer (P2P) communication diagrams and network topology over an interactive geolocation map. The plugin app use Google Maps API and full customizable configuration settings to provide an all-in-one HTML webpage which can be easily integrated in other into other Security Information and Event Management (SIEM) systems to make it a valuable addition for monitoring and managing distributed network communications. The Web plug UI is shown below:

This tool can help the security administrators with enhanced capabilities to monitor and analyze distributed system communications. The four features provided by the program are:

• P2P Device Data Flow Throughput: Visualizes the transmission rate and data volume exchanged between encryption devices.
• Data Transmission Speed: Tracks connection state, latency between servers, clients, and other network nodes.
• Network Topology: Dynamically maps peer-to-peer relationships and data exchange paths.
• Geolocation Mapping: Displays the physical locations of nodes and their communication links over an interactive map interface.

System Main Components Introduction

The project is structured into three main components:

1. GeoTopology_Map Front-End Application: A Flask-based front-end module that integrates with the Google Maps API to render an interactive map for real-time visualization.
2. Backend Data Database: A lightweight SQLite3 database backend to store and manage device communication data, making it accessible to the web application.
3. Data Collection Module: An interface program that gathers communication data from devices and updates the database in real time.

System Design

The system workflow diagram is shown below:

The system includes four function modules:

Module[1] : Flask Web Host Application

• Function: Hosts the web interface and serves data to the user.
• Key Features: Interactive map visualization using Google Maps API, User-configurable settings for customizing the displayed topology and Real-time data updates based on user setting.

Module[2] : Data Management Module

• Purpose: Coordinates the overall workflow and integrates different modules.
• Key Features: Loads initial configurations (e.g., database connections, API keys), Manages data threads for processing and communication, Queries the database for the latest node and link data and Creates JSON objects for nodes and communication links.

Module[3] : Database

• Function : Store the Node metadata (e.g., device IDs, geolocations) and Node metadata (e.g., device IDs, geolocations). Two table are included with below config:

CREATE TABLE IF NOT EXISTS gatewayInfo(id integer PRIMARY KEY, name text NOT NULL,ipAddr text NOT NULL,lat float NOT NULL,lng float NOT NULL, actF integer NOT NULL, rptTo integer NOT NULL,type text NOT NULL)        

CREATE TABLE IF NOT EXISTS gatewayState(time float PRIMARY KEY,id text NOT NULL, updateInfo text NOT NULL)        

Module[4] : Data Collector

• Function : Gathers data from monitored devices and updates the database.

User Interface Design

In the demo web app, we added a sidebar attached beside the map which allow users to change various map settings. The user can choose the data update rate of the flask webserver calling GET request through a dropdown menu. A filter function is also added to adjust the display certain types of communication links (active, gateway, control hub).

The design of the control side bar is shown below:

The design of symbols, icons and links on the map panel is shown below:

Program Setup

Development Environment : Python3.7.4, HTML+flask, socketIO+eventlet, SQLite3

Additional Lib/Software Need :

Hardware Needed : None

Program Files List:

Program Usage

Follow the below steps to execute the program

Step1 : Setup the configuration file

Rename the src/config_template.txt to config.txt , then fill in the your google map API Key and set the parameters as shown below:

# This is the config file template for P2P communication display map
# emulator program <p2pCommMapApp.py>
# Setup the parameter with below format (every line follows <key>:<value> format, the
# key can not be changed):
#-----------------------------------------------------------------------------
# Database file name
DB_NAME:node_database.db
# Node information record
NODES_INFO:nodes_record.json
# Google Map API key
MAP_API_KEY:AIzaSyBoHBPqxFw40DFvCbXrj1IWNcvkzb6WkkI
# Set the Hub GPS location
HUB_LAT:1.2945315
HUB_LONG:103.7746052
#-----------------------------------------------------------------------------
# Init the PLC local web Flask app parameters
FLASK_SER_PORT:5000
FLASK_DEBUG_MD:False
FLASK_MULTI_TH:True        

Rename the src/nodes_record_template.json to src/nodes_record.txt and add your node in the json file with below info:

"<Node_ID>": {
    "no": 1,
    "name": "NUS",
    "ipAddr": "10.0.0.1",
    "lat": 1.2964053,
    "lng": 103.7690442,
    "type": "GW",
    "rptTo": 0,
    "actF": 0
}        

To find the Geolocation GPS of a IP you can use the online tool or this app: https://github.com/LiuYuancheng/IPGeoPosLocater

Step 2 : Run the test data creator and map P2PComm_GeoTopology_Map App

Start up the data insert simulation program to add new gateway

python3 databaseCreater.py        

If you want to link to your system call the module function updateStateTable(self, gatewayID, infoStr) to insert the new gateway state in to database.

Run the flask webserver:

python3 p2pCommMapApp.py        

Step3 : View the P2PComm_GeoTopology_Map

After running step2, wait 10 sec make sure the database thread fully started then do step 3, open web browser and enter URL: https://127.0.0.1:5000, then check the web plug in as shown below:

You can use iframe to integrate the map in your SIEM dashboard, for example in Grafana dashboard you can set the AJAX config as shown below:

Then you can see the map will show on the Grafana dashboard page:

Project GitHub Repo Link:

https://github.com/LiuYuancheng/P2PComm_GeoTopology_Map_Plugin

Thanks for spending time to check the article detail, if you have any question and suggestion or find any program bug, please feel free to message me. Many thanks if you can give some comments and share any of the improvement advice so we can make our work better ~