Network Statistics Analyzer: Simple Programming to Solve Simple Problems...

Contents:

1. What problem was I trying to solve?
2. How did I implement the solution?
3. Discussing the workflow, process, and troubleshooting / debugging.

What problem was I trying to solve?

Since I recently acquired Starlink ISP service, I have been curious to know what its general performance is. I previously had Bloosurf which, while being serviceable enough to get the job done, was horribly unreliable, and lacking in speed / bandwidth. Traditionally, one can simply run an internet speedtest, such as Google speed test, Ookla, or SpeakEasy speed test. They have a fatal flaw, though. A manual speed test only tests real bandwidth at a given fixed point in time. This is not accurate for assessing what the long-term real bandwidth is on average, as delivered to my end-user devices. Thus, the problem-to-be-solved was made clear.

I needed a program that would query my real-time bandwidth numbers throughout the day, during peak AND non-peak hours, log that data, aggregate and average that data, and make it plainly available to me anywhere. That way, I could monitor the performance of my WAN uplink, as measured from within my network, with a far more wide-angle lens.

Implementing the solution!

I initially waffled between a pre-packaged solution, and creating my own. Truthfully, through, I could not find a pre-packaged solution that was simple, lightweight, and that suited my needs. As a result, I decided that I would create my own solution. What follows is what I came up with...

Part 1: BASH and Speed Logging...

As outlined in my introductory statement, the crux of the problem here dealt with getting data from a bandwidth tester and automatically storing it on a host inside my network, multiple times a day. The first chunk of that was made simple by my discovery of an apt package that comes with binaries to query the Ookla speed test API from the command line. Results are then output to the console. This is "speedtest-cli."

netuser@NetworkAnalyzer:~$ speedtest-cli

Retrieving speedtest.net configuration...
Testing from Starlink (129.222.220.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Connecticut Education Network (Hartford, CT) [156.81 km]: 79.996 ms
Testing download speed................................................................................
Download: 138.57 Mbit/s
Testing upload speed......................................................................................................
Upload: 12.08 Mbit/s        

Since this package could grab the data from Ookla and output it to the console, all I had to do was get BASH to store and parse this data from the output. Between a few clever included command switches, and my trusty 'grep,' I was able to cobble together a functional script to organize and store this network data.

#!/bin/bash
source /home/netuser/.bashrc
#run the speed test
result1=$(speedtest-cli --simple --secure | grep -E 'Ping|Download|Upload')

#Wait 45 Seconds
sleep 45

#Gets the current time in 12Hour format
Time=$(TZ='America/New_York' date +"%m-%d-%y %I:%M %p")

#append the result to a log file
echo "$Time" >> /home/netuser/SpeedTestLog
echo "$result1" >> /home/netuser/SpeedTestLog
echo "--------------------------------------------------------------------------------------------" >> /home/netuser/SpeedTestLog        

In this simple script, I call the speedtest-cli command with the --simple and --secure command switches. The simple switch reduces the amount of extra text that wraps the output, thus making parsing easier and more efficient. The --secure switch forces the use of HTTPS when making the request which, while more secure, also improved reliability during debugging and testing. Once the API returns the data, the result is immediately piped into grep, which searches the output for lines containing strings matching 'Ping, Download, and Upload.' The parsed result is stored in the variable 'result1'. After that, I wait 45 seconds. This is because rapid testing reduced the reliability of the program. It seemed that the API did not respond well to queries placed less than 45 seconds apart from one another. Furthermore, I initialize a variable called Time and set it equal to my local time zone. This is to ensure that log entries come paired with a timestamp for readability purposes. Finally, I echo the result variable, Time variable, and a buffer of hyphens to a LogFile called SpeedTestLog. That end result is represented by the following log snippet:

--------------------------------------------------------------------------------------------
10-30-23 12:08 PM
Ping: 70.259 ms
Download: 117.67 Mbit/s
Upload: 24.23 Mbit/s
--------------------------------------------------------------------------------------------
10-30-23 01:08 PM
Ping: 71.498 ms
Download: 100.32 Mbit/s
Upload: 8.09 Mbit/s
--------------------------------------------------------------------------------------------
10-30-23 02:08 PM
Ping: 77.039 ms
Download: 94.28 Mbit/s
Upload: 20.06 Mbit/s
--------------------------------------------------------------------------------------------
10-30-23 03:08 PM
Ping: 68.378 ms
Download: 98.56 Mbit/s
Upload: 10.47 Mbit/s
--------------------------------------------------------------------------------------------
10-30-23 04:08 PM
Ping: 74.045 ms
Download: 144.86 Mbit/s
Upload: 26.80 Mbit/s
--------------------------------------------------------------------------------------------
        

Now that the task was functional, and outputting readable, stamped data, it was time to automate it. In Linux, CRON is the baked-in daemon for automating scripts and tasks within the operating system. I set my CRON file in the following manner:

7 11-16 * * * /home/netuser/speedtest1.sh

7 23,0-4 * * * /home/netuser/speedtest1.sh
        

A CRON entry like these performs the script's function on the 7th minute of the hours 11AM through 4PM (5 repetitions), and then again from the hours 11PM through 4AM (5 repetitions). For me, and many people, peak hours of use occur during the day, and into the mid-afternoon. This is represented by the first CRON job. The second CRON job pulls data during low-use hours, while I, and others, are asleep.

Part 2: Google Cloud Platform's Service Accounts & APIs...

Given that I now needed a free-tier platform to utilize for data visualization and access, I naturally gravitated towards what I know and use daily. Google Drive. In order to make my data get up to Google Drive, and into a file, automatically, I would need to sign up with their Google Cloud Platform (GCP) and build a project to interface with that data. So here is what I did...

After signing up with a free account in GCP, I created a new project called "Network Statistics Script." In this project, within the IAM & Admin section, I generated a new service account with the purpose of logging into my Google Drive to move the data between my Linux Container and a file in the Google Cloud. For this service account, I generated a Private Key, stored in a JSON file, that would be used to authenticate my client container to the Google API server as my service account. After securely transferring this key-file to a directory on my Linux host, I was nearly finished within the cloud console. Lastly, I simply enabled the Google Sheets and Google Drive APIs within the APIs section of the project. I also enabled the Health Check API to monitor the heartbeat of my Service Account's connection to the GCP. With these APIs enabled, my service account would be able to establish a verifiably healthy and secure connection to the necessary APIs to perform the function of moving the data!

Part 3: Python, Google APIs, and more...

Now that I had the data AND a service account, I had to figure out how to push it to a Google platform that was available everywhere. The goal, after all, was to make the data available to me anywhere, for monitoring away from home. Furthermore, I wanted to make the data visual, a simple graph would suffice. As a result, I came up with the idea of using a GoogleSheet with a dynamically populated line graph. Prior to this project, I had never written my own program to interface with Google APIs before, so I did some web research and found that Python had some fantastic pre-written libraries for this very purpose! What follows is the final scripted product, with some security-minded redactions, followed by an explanation.

#!/usr/bin/env python3

# Importing the necessary libraries
from datetime import datetime
import pytz
import time
import re
import gspread
from oauth2client.service_account import ServiceAccountCredentials

#Define the scope of G-Sheets API
scope = ['https://spreadsheets.google.com/feeds', 'https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/spreadsheets']

#Specify credentials for service account in KeyFile
creds = ServiceAccountCredentials.from_json_keyfile_name('/home/netuser/NAME_OF_JSON_KEY_FILE', scope)

#authorize the task
client = gspread.authorize(creds)


# Initialize variables to store the total and count of each metric
total_ping = 0
total_download = 0
total_upload = 0
count = 0

# Open the log file
with open('SpeedTestLog', 'r') as file:
    for line in file:
        # Use regular expressions to find the metrics in each line
        ping = re.search('Ping: (.*) ms', line)
        download = re.search('Download: (.*) Mbit/s', line)
        upload = re.search('Upload: (.*) Mbit/s', line)

        # If the line contains a metric, add it to the total and increment the count
        if ping:
            total_ping += float(ping.group(1))
            count += 1
        if download:
            total_download += float(download.group(1))
        if upload:
            total_upload += float(upload.group(1))

# Calculate the averages
average_ping = total_ping / count
average_download = total_download / count
average_upload = total_upload / count

# Print the averages
print(f'Average Ping: {average_ping} ms')
print(f'Average Download: {average_download} Mbit/s')
print(f'Average Upload: {average_upload} Mbit/s')

#Get the current date and time
DateTime = datetime.now(pytz.timezone('US/Eastern'))
DateTimeStr = DateTime.strftime('%m-%d-%Y %I:%M %p')

#Send averages to Google Sheet
sheet = client.open('Starlink Performance').sheet1
data = [average_ping, average_download, average_upload, DateTimeStr]
time.sleep(5)

try:
    sheet.append_row(data)
except Exception as e:
    print("An error happened:", e)
        

While far from perfect, this script properly performs the task at hand, and I can explain how!

First, I initialize a script in Python with the Shabang, and import the necessary libraries for the program. I will need datetime for establishing the date, pytz for establishing EST as the timezone, Time to count seconds out for sleep pauses, re stands for "regular expressions" and will help me search through the log file using RegEx, gspread is the library that imports functions for interacting with google spreadsheets, and finally oauth2client provides me with the libraries necessary to render commands to a service account in the Google Cloud Platform.

To start, I define the scope of my API calls to Google. This is important so that my service account is allowed to perform ONLY actions that I authorize and pre-allow in the program. After that, I authenticate to the Google API servers with my service account. This is done with a private JSON key file that is stored securely within my Linux Container's password-protected file system. This process concludes with authorizing the connection with said credentials.

Next, I initialize some variables to store the data that I will pull from the log file. Subsequently, I open the log file and loop through it line by line, searching with RegEx queries to find lines with Ping, Download, Upload, and ms, Mbit/s, Mbit/s respectively. Then, if a given line in a given iteration contains that string, the numerical value is converted to a floating point numeral, and appended to a running total.

Finally, I initialize a variable to house the current date and time EST. After that, all that is left to do, is to invoke an object for the GoogleSheet, name it in accordance with the real-world GoogleSheet, and create an array for the data itself. After sleeping for 5 seconds, to allow for some breathing room, the program tries to append the array of data to the earliest available row in the spreadsheet. If it fails, for any reason, it prints an error statement to the console for me to be alerted.

With the main program finished, I had to automate its execution. I did so in the same way that I automated the BASH script earlier in this article, CRON! What follows is my CRON file, and a brief explanation of what it is doing...

1 6,12,18 * * * /home/netuser/myenv/bin/python3 /home/netuser/averageNetStats.py        

In this CRON file, the Python program is executed on the first minute on the hours of 6AM, 12PM, and 6PM. When it executes, it does so within a virtual Python environment. This is best practice for security and isolation of the libraries that are used. There is no reason to do a system-wide installation of these python libraries when only the "netuser" user will be calling them.

Part 4: Data Visualization and Accessibility...

In keeping with the theme of simplicity, I used a simple Google Spreadsheet to make the data accessible to me anywhere, anytime, so long as I have internet access. As the program runs throughout the day, it passes sets of rolling averages to the spreadsheet which get progressively more accurate over time, as the log file they are pulled from grows by the day. From this I am able to see how the total average of the speed is pulled up and down over time in a graph that is dynamically updated by a formula in the spreadsheet...

As you can see, I am able to gain some important insight into the average performance of my Starlink over time. If the line for download bandwidth, for example, pulls down, I can tell that the overall log file is aggregating lower than previously. This must mean that a significant enough change has occurred over time that the trend is now settling lower...thus I am able to monitor what I receive in service versus how much I pay for that service.

Debugging, Troubleshooting, and Closing Rambles...

While not the most flashy, or overly complex, of programs, this program holds a special place in my heart. It solves a practical problem with a practical solution. Using openly available APIs, Libraries, and Packages it actually solved a real issue and use case for me. I have only ever written programs, commands, or scripts either for work, or for a school project. This is the first time I wrote one to solve a problem of mine.

That being said, it was not easy at first. I wanted to take some time to discuss one of the main issues that ended up being a great lesson in Linux / Python best practice!

When trying to import the necessary libraries for the Python side of the program, I was consistently hit with errors because of my attempts to perform system-wide installation of packages. Evidently, the default configuration of the pip3 package manager, and of Ubuntu, seems to be to NOT allow such a thing without changes in security configuration. As such, I was directed to the correct way of doing things. By running the program in a Python Virtual Enviroment (pve), I was able to execute it securely in the userspace of the account that I set up to run the program. In doing so, all other accounts and Python instances were left unaltered. This is better for two main reasons. First, it is more secure and keeps the files and utilities of a given user in THAT user's home only. Second, it prevents unnecessary, and potentially conflicting, python versions and libraries from being installed across the entire system.

Another thing that I wanted to discuss was my use of GPT-4 to help me learn Python and troubleshoot my code.

As a native student of C++ and Java, and somebody that is, albeit at a preliminary level, educated in Computer Science, I am aware of the fundamentals, and even advanced concepts of general programming. That being said, I had never written in Python prior to this project. The wonderful thing about having the knowledge, but not the syntax, is that I can ask generative AI to help me do "X, Y, Z" tasks in a language, but show me how to write it in Python! This helps me not only learn the general syntax of Python, but it helps me to see how it compares to other languages with which I am more familiar. Generative AI has truly been revolutionary in helping me familiarize myself with, and learn, a vast variety of technologies, languages, and concepts by providing me with real-world examples and demonstrations. It has totally changed the way I learn, for the better, it seems.

Lastly, I wanted to talk about my choice of the LXC container for this project, as opposed to a traditional Virtual Machine.

Within Proxmox, my hypervisor of choice, I am given the options of virtualization within a traditional VM or an LXC (Linux) container. There are benefits and drawbacks for either fork of the concept, but I chose the latter over the former. Here is why...For a small and lightweight function/program such as this, the name of the game is CPU instructional efficiency, and minimal resources used. I wanted to keep the potential for expansion into other functions, while minimizing the footprint of the machine on my physical resources. LXC containers share their kernel and low-level instructional binaries with their host-system's operating system. That is to say, my container shares a kernel with the Proxmox host. As such, the NetworkAnalyzer container is able to run all day, performing its tasks, without using even ONE PERCENT of the resources assigned to it. Combining this with dynamic resource allocation with Proxmox, I can effectively run the programs of interest for...free.

Containerization, in many ways, and for many applications, is the future of computing. It is fast, lightweight, and highly efficient, especially for tasks like this.

All in all, this project was a fantastic way to experience the joys of simple programming for simple solutions to practical needs. It taught me much about Linux containerization, Python, BASH, GCP, APIs, CRON, and more! This has really reinforced the idea of mine that nothing, not a textbook, course, or video, can compare, in terms of learning-value, to raw lab experience.

Actually performing the task is the best way to learn anything in technology...

Thank you for reading, if, in fact, you did...

Tyler Sell