New CLI and Log Management System Proposal for OWASP SecureCodeBox (Project Proposal)
Bulitha Kawushika
Cryptography Enthusiast | Cybersecurity Engineer (UG Trainee) | Computer Science (Hons) (Cyber Security Specialization) | Cyber Security blogger
Project Proposal Overview —
I propose enhancing OWASP’s SecureCodeBox tool with a user-friendly Command Line Interface (CLI) for effortless scan creation and execution, while also introducing functionalities for logging scan details and accessing/downloading scan history. I have identified existing difficulties and issues within the current system and suggested solutions for them. Additionally, this enhancement aims to improve the user experience and introduce non-repudiation qualities to the improved system.
Project Title: OWASP secureCodeBox
Project Proposal
Introduction
SecureCodeBox is a Kubernetes-based toolchain for continuous security scans of software projects, integrating security testing tools into CI/CD pipelines and aiding penetration testers in recon and discovery phases. The primary interface in the current system to interact with secureCodeBox is through its custom resources (CRs) in the Kubernetes API.
This project aims to create a custom (but optional) secureCodeBox CLI that automatically connects to the Kubernetes API to make these interactions easier and more user-friendly. The main task of the project is to create a new command-line client that connects to the Kubernetes API and interacts with the CRs of secureCodeBox.
As an additional feature, every search executed through the SecureCodeBox CLI will log into the S3 cloud storage (through the Amazon API). If there are any errors occurring in the processes, they will also log in to S3. It will be helpful for secureCodeBox developers to gather details about issues and solve them.
In this secure CodeBox CLI, users can check their search history and download log f iles in Syslog (.log) format. It enables users to track their scan history and easily monitor actions using log analyzing tools and third party softwares.
After creating SecureCodeBox CLI, writing tests and integration tests should be implemented to identify and solve potential issues. This is a complete end-to-end test (e2e test) to verify the functionality of CLI.
GitHub Actions are used in this project for automating the Continuous Integration (CI) pipeline, which includes running tests (including integration and end-to-end tests) whenever there are changes pushed to the repository. Lastly, deploy the new functionality into the current system.
Project Background and Functionalities
1. Main functionality of secureCodeBox CLI -
In most cases, users want to write resources (e.g., scans) when calling the scan method. But writing the resource is generally not hard but can be cumbersome as it requires the creation of a new file or multi-line string in the command line.
Because of this, when users use cmd, powershell, or bash shell to execute commands, it is very hard to find previous search histories. Also, when we create several resources, it will be hard to manage these folders and move between them during the penetration test.
Through this new secure CodeBox CLI, users can do their basic and medium scans very easily. It will save time for the penetration tester as well. It is added as an optional feature for the existing system. Through the secureCodeBox CLI, users can input basic information for scanning and let the scan be executed very easily. SecureCodeBox CLI will automatically create resources and initiate the scanning.
In the scan process, the user should enter a command and provide the necessary details for scanning.
As an example, if the user wants to run nmap scan to scanme.nmap.org, they should enter the,
“scbctl create scan nmap scanme.nmap.org” command.
“scbctl create scan” is used to create a scan (resource file) for the scanning process.
Then the “nmap” part of the above command is used to identify the scan function that the user wants to do. SecureCodeBox has several scanning facilities. The user should mention the scanning type in there.
“scanme.nmap.org” users can replace the URL or IP address that they want to scan in there.
When the user enters “scbctl create scan nmap scanme.nmap.org”, the command component of CLI will create a scan for scanning automatically. Also, CLI will prompt the message that “scan is created. waiting for a run.” It will give the user confirmation that the scan was successfully created.
Then the user can enter the “ scbctl run ” command to run this scan. When the user enters this command, scanning will initiate and run. When the user enters this “scbctl run” command, the CLI will automatically grab the scan created in the previous step and start to perform scanning. After that, the results of the scanning will be displayed in the CLI.
2. Log scan history -
When the user executes a scan, SecureCodeBox can store the log file (.log file) in the S3 cloud. It will log details about this scan. So users cannot directly access these log files. If some errors occur during the scanning, it will also log this log file.
Importance of this feature —
● User scan easily find and gather details about their previous scans.
● It enables nonrepudiation quality to secureCodeBox —
Nonrepudiation provides proof of the origin, authenticity and integrity of data. It provides assurance to the user that its scan was executed, as well as proof of the user’s identity to the secureCodeBox team. This way, neither party can deny that a scan request was sent, received, and processed.
● In the organization environment this log files can use for monitoring the user actions —
The log file cannot be altered by the user. This information can therefore be utilized within the organization to look into user behavior and spot illegal or unethical behaviors.
● This log file is saved in the Syslog (.log) file type, making it easier to analyze by using log analysis tools like ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog etc.
3. View scan history and download log file -
The first option users can download log files in Syslog (.log) format. Then it can be easily used for future analysis. By using the second option, the user allows the user to view the top 20 scans performed recently.
When users enter “scbctl download log”, they can easily download the user’s log file of the current month. It will download to the user’s computer’s folder. CLI will create a folder in the local storage (user’s computer) to save this downloaded file. Once CLI creates a folder to keep downloaded log files, it will save the next download files in there as well.
As a further enhancement, We can enhance this download mechanism, for example, we can create a command to download the user’s complete log folder, or they can download a selected log file. A log folder will be created once a month. (The log file management mechanism is explained in the design chapter.)
Users can analyze these log files very easily using log analysis tools such as ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog, etc.
The user can view the top 20 recent scans using “scbctl history”. CLI will locally store the top 20 scan details as a small separate log file. This is accessible for the user because it is on their computer.
Instead of that, we can store data about recent 20 scans in memory (RAM), because it takes up very little space in memory. But RAM is a volatile memory. Volatile memory does not retain stored information after power is removed. So I do not encourage you to do it for this project.
Functional Requirements
● CLI should work properly with existing services and APIs in the current system. It is essential to be compatible with the Kubernetes API.
● If CLI creates resource files (scans) such as .yaml files in the local storage during the scanning, they should be managed properly.
● Log files should be created, altered, retrieved, copied, and maintained properly.
● Propertesting (end-to-end tests)
● CIpipeline (Github action)
Non Functional Requirements
● The CLI should be user-friendly, so a new user can start their work without any hassle. Commands should be easy to memorize, and their corresponding action words should be self-explanatory. (Proper documentation containing commands andtheir functionalities will also be provided.) For the actions that are not clear, a help tooltip can be provided. Also, the error messages and import details can be made simpler, so non-technical people can understand them easily.
● Also, proper documentation regarding the whole project will be provided for further understanding by users. It describes the functionality of the SecureCodebox CLI.
● Complete documentation should be published to inform users about SecureCodeBox’s commands and how they are used during different scan operations.
● The scope of the project should be covered in a short period (3 months).
Design
Overview diagram
The following figure is an overview diagram designed to provide a high level of abstraction for an interaction model of a system. It also describes the main component of the overall system.
This CLI will add a separate component to the CLI. Users can optionally install that feature. By creating it as a separate part, the developer can push an update solely for this CLI component. It reduces the complexity of architecture, and bug fixing is also madeeasier with it.
This diagram provides a high-level idea of where the secureCodeBox CLI will be placed in the system and how it will interact with other existing components in the system.
Note that UML component symbols in all internal components (green boxes) in the secureCodeBox container are removed because it is easier for visualization.
This CLI is completely running on the user’s computer or laptop.
Sequence diagram (Runtime View)
This section describes the runtime view of secureCodeBox CLI with other components. In contrast to the static building block view, this view shows the interactions of participants (components or actors) over time. This is the sequence diagram of the system.
To maintain the simplicity of the sequence diagram, some internal and external components that do not directly interact with CLI are as compact as all other components and all other external systems, respectively.
Runtime Scenario 1: Basic Scan
This scenario describes a simple ZAP scan using the secureCodeBox CLI, which is initialized by a developer actor and persists the findings in Elastic and DefectDojo (mentioned as other external systems).
CLI directly interacts with Users, Engine (through kubernetes API), and the S3 cloud storage bucket ( through S3 Amazon API).
The first user can provide inputs to the CLI. If necessary inputs are provided, CLI initiates Scan (kubectl) through the Kubernetes API. The engine component will get this initiated the scan, and other functionalities will run as usual in the existing system.
In other words, the user should input scan details through CLI, and then CLI initiates scans in the engine through the Kubernetes API. Then the rest of the scan operations will run as usual on the current system.
If the necessary inputs are not provided, then CLI will raise a warning message. (It will not draw in a sequence diagram because otherwise it can be hard to view.)
If the scanning process runs smoothly without issues, the CLI receives a confirmation message from the engine through the Kubernetes API. The CLI then initiates the write log command and sends it to the cloud S3 bucket. (Detailed discussion on logging follows in the subsequent part.)
The CLI also retrieves the results of the scanning process through the Kubernetes API and then presents them to the user for viewing.
How does that logging process work?
When the scan successfully begins, the CLI will receive the confirmation message, as explained earlier. It is crucial to create and send log files to S3 storage via the Amazon S3 API. The significance of this confirmation message lies in its exclusion of logging any user mistakes, such as network errors; instead, it solely logs scan details and system-side errors.
I suggest creating a separate folder in the S3 storage bucket to save log files. (name this folder as “Logging Record Folder”)
Then I suggest building a pipeline connection between CLI and the S3 bucket. We have facilities to do it in the Amazon S3 API. Creating a pipeline using GitHub Actions to send files from the command-line interface (CLI) to an S3 bucket can be a convenient and automated way to manage your deployments or uploads. We can also update existing log files using a github action (basically using the aws s3 cp command, or alternatively, similar commands are available in the aws documentation with extra features). The main advantage of using Github actions to pipeline is that it can automate log writing functionality.
In this logging process, CLI will maintain a separate log folder for each user. This folder will be in the “Logging Record Folder.” While the user installs this secureCodeBox CLI, it will automatically create a new log folder for the user in “Logging Record Folder” (which contains the S3 bucket).
CLI will create a new log file (a .log file) every month. When the user opens CLI at the very first moment of the new month, it will create a new log file (.log file) in the operating user’s log folder.
File Structure in here,
A new Syslog (.log) file is only created when the user starts CLI at the very first moment in the month. If the user never opens it for the whole month, it will not create a log file for this month.
When the user executes the scanning process in CLI, it will write a log-line in the log folder for the current month.
Runtime Scenario 2: Obtain history and download log file
In this scenario, users obtain a copy of the log file, which is in S3 cloud storage bucket.
In here also, CLI interacts with the S3 storage bucket through the pipeline that we created in the github action (the Amazon S3 API is also applicable here). The same pipeline is used to download a log file. The user can download a log file for the current month using the CLI.
This is the sequence diagram for downloading the log file. The first user initiates a request to download a log file using the CLI. then it will pass to the Amazon S3 API through the pipeline. The Amazon API will obtain a log file from the S3 storage bucket and send it to the CLI. CLI has the facility to save it on the user’s computer or laptop.
领英推荐
Further enhancement of this feature -
We can further enhance this part, as the user can download the whole user log folder or select a log folder later. In this project scope, it is not part of the scope to develop that feature. The Amazon API has features to facilitate selecting and obtaining certain log files according to the search.
Implementation
This section explains how the implementation was done according to the previously gathered requirements and system design. An explanation of the technology and programming languages utilized in this project, along with the reasons why they were chosen. Also, it has a code snippet and screen shots of the small demonstration.
CLI Implementation
Since this project mainly aims to create a CLI, the design and features of the CLI are discussed in the above sections.Here I suggest creating a Unix-style Linux CLI. So I hope to use bash scripting to write this CLI. I created a small bash script to demonstrate how to create a scan for nmap scanning. This is the code that I wrote. I uploaded it to my Github account.
Github repository -
#!/bin/bash
# Welcome Banner
echo "________WELCOME TO SECURECODEBOX________"
# Function to create a new scan YAML file
create_scan() {
local scan_type=$1
local url=$2
# Ensure scans folder exists. This is the folder which contain all scan files.
mkdir -p scans
# Generate YAML content
local yaml_content="apiVersion: \"execution.securecodebox.io/v1\"
kind: Scan
metadata:
name: \"$scan_type-$url\"
spec:
scanType: \"$scan_type\"
parameters:
- $url"
# Write YAML content to file
echo "$yaml_content" > "scans/$scan_type-$url.yaml"
echo "Scan '$scan_type-$url' created in 'scans' folder. Waiting for run....."
}
# Main function
main() {
while true; do
read -p "secureCodeBox\CLI: " command
case $command in
"scbctl create scan "*)
input="${command#*scbctl create scan }"
read -r scan_type url <<< "$input"
create_scan "$scan_type" "$url"
;;
"exit")
echo "Exiting..."
exit 0
;;
*)
echo "Invalid command. Usage: scbctl create scan <scan_type> <URL>"
;;
esac
done
}
# Run the main function
main
When execute it, it will work as following,
First, it prompts a welcome banner. When the user enters the “scbctl create scan nmap scanme.nmap.org” command, it creates a new yaml file in the scan folder and prompts that,
“Scan ‘nmap-scanme.nmap.org’ created in ‘scan’ folder. Waiting for a run…”. Then the user can verify that their command was executed successfully.
If the command is incorrect, it will prompt a message and inform user of the correct command syntax and command format.
When navigating the scan folder, we can see the yaml file was created for scanning.
The content of the YAML file is written automatically according to secureCodeBox documentation.
Then the user can use the scbctl run command to run scanning. (But this part is not created in this demonstration.)
We can add commands to all basic and medium functionalities of the secureCodeBox tool.
Through this CLI, users can perform the basic functions of
● Repeating scans on a schedule
● Scanning Networks
● Scanning Web Applications
● Port-processing with hooks
● Storing Findings with Persistence Hooks
● Automatically Scan Your Cluster with Auto-Discovery
● Enforcing Engagement Scope
This will reduce the complexity of creating scans and running the scanning process.
CLI has the feature to maintain a log file for each user in S3. Its functionality was discussed previously under the Design section. CLI is connected to the Amazon S3 API using a pipeline. We created this pipeline using a Github action, and a Github action has features to automate this log-writing process. One log folder will be created for each user who installs CLI on their computers. Every month, a new log f ile will be created in that user’s log folder to log their scans. This file creation and writing are automated by github action.
CLI also has features to view and download log files.
‘The ‘scbctl history’ command can be used to view the top 20 recently scanned details. CLI maintains a small log file to track the most recent scans. When the user executes it, the CLI will grab data from this log file and view that data.
The ‘scbctl download log’ command is used to download the log file. CLI is connected to the Amazon S3 pipeline, and the Amazon S3 API has facilities to send f iles from the S3 bucket to components of CLI. The component in the CLI will take this file and save it to the user’s computer.
When this CLI is embedded in a Docker container, it is cross-platform. So I can run any OS, like Windows, Linux, or Mac OS. Instead of that, we can use another two solutions.
First one is, we can distribute them through package managers like apt, yum, Homebrew, or Chocolatey for easier installation and updates on Linux, macOS, and Windows, respectively.
The second is that since this CLI tool is relatively simple and doesn’t have complex dependencies, we might consider compiling it into a cross-platform executable using a tool like PyInstaller, Py2exe, or Nuitka for Python scripts, or using a similar tool for other languages.
Log Files
Log files are created in the Syslog (.log) filetype, which is a famous file type for writing logs. It can be easily analyzed by using log-analysis tools like ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog, etc. The size of log files is very small, so it is easy to store and manage.
Testing
● Writing test and Integration tests -
This includes unit tests, which check individual components of CLI in isolation. Here, verify the functionality of separate subcomponents and mechanisms in CLI, such as file handling and creating resources.
Integration tests, which verify interactions between different components or services. In other words, integration tests verify that different parts of the system work together correctly. These tests often involve multiple components, such as APIs, file handling, or external services. They ensure that the integrated system behaves as expected.
Integration with the Kubernetes API and the S3 Amazon API should be especially checked.
● End to end test (e2e test) -
End-to-end tests (e2e tests) simulate user interactions with the application from start to finish. They validate the entire user journey, including CLI interactions, API requests, and backend processing. E2E tests help ensure that the application functions correctly from the user’s perspective. This e2e test will help to identify and solve potential errors in the CLI.
CI pipeline with Github Actions
● Setting up a CI Pipeline with GitHub Actions -
GitHub Actions allow automating the CI/CD process directly within the GitHub repository. Define workflows in YAML files to specify the steps needed to build, test, and deploy a CLI application.
● Incorporating Tests into the CI Pipeline -
Within the CI workflow defined in the GitHub Actions configuration, include steps to run tests. This typically involves executing unit tests, integration tests, and E2E tests against the application codebase.
● Reporting Test Results -
After running tests in the CI pipeline, GitHub Actions provides feedback on the test results. The secureCodeBox development team can view the test output directly within GitHub and receive notifications about test successes or failures.
● Github Action has the facility to automate the log-writing process. When the user successfully executes the scan, it will automatically write a log line in the user’s log file.This pipeline is also used to download log files from the S3 storage bucket. The Amazon S3 API has facilities to find and send files; these log files will be obtained by CLI through this pipeline.
Timeline
Further Enhancements
As a further enhancement, we can develop a user-friendly CLI (optional) for secureCodeBox. It also has features to log, view scan history and download a log file. It will make it easy for very beginners to interact with SecureCodeBox.
secureCodeBox CLI -
This is the outline of the CLI.
First, the user can choose whether to initiate scanning or view the scan history. By default, initiate scanning was already selected, so users can quickly skip this function by clicking enter.
When the user selects the initiate scanning option, they should select the scan type available in the secureCodeBox. Then the user should input essential information for a basic scan. Additionally, users can choose advanced scan options. By default, it is disabled.
When users select advanced options, they can add some additional scan methods, like flags, for this scanning process. secureCodeBox CLI has several most usable f lags.
As an example, in the nmap scan, the user can add this flag to their resource file (.yaml file) scan code. Also, they can add the number of threads and specific port numbers here.
The user is then prompted to confirm the search, after which the scan will begin. The results will also be displayed in this CLI.
View scan history and download log file
The user can choose the scanning history option to view history.
The first option allows the user to view the top 20 scans performed recently. By using the second option, users can download log files in Syslog (.log) format. Then it can be easily used for future analysis.
It is suggested to use the Go language (golang) for that user interactive CLI.
Advantages of using the Go language -
● Most of the components in the existing secureCode Box system were written in the Go language. Because of that, CLI can easily integrate with current system components.
● Go has built-in support for concurrency via goroutines and channels. This makes it easier to write concurrent programs, which can be particularly useful for CLI tools that need to perform multiple tasks simultaneously, such as handling user input and output while executing commands or managing background tasks.
● Gois compiled into machine code, which generally results in faster execution compared to interpreted languages like Python or Ruby. So performance is high.
● Go comes with a rich standard library that includes packages for handling tasks such as input/output, networking, encryption, and more. It reduces development time and effort.
● Goprograms can be compiled to run on various operating systems, including Linux, macOS, and Windows, without modification. So secureCodeBox CLI is platform-independent.
Gathering information from users and creating resources (scans) through the CLI is one of the main subtasks in this project.
This is my own code written to gather information from the user and create a resource file.
Github Repository -
This code is written to gather basic information for an Nmap scan, and it creates a YAML resource file automatically. The user should only enter a URL, and then this CLI will create a new YAML file in a separate folder in the project. I enable the central management of resource files.
The Yaml file was created, and its name format is output_. Therefore, users can easily find and manage local resource files.
Yaml file content is created according to secureCodeBox documentation.
Summary
As described in the project proposal, first the project background was identified and the requirements were gathered and analyzed. After that, the system design of the new CLI was created and the Implementation was planned according to that. The project will be done following the above timeline, so the working product will be available at the end of the coding period of Google Summer of Code 2024.
References