Black-box cyber security testing of products and integrated systems

Black-box security testing refers to a method of software security testing in which the security controls, defenses and design of an application or system are tested from the outside-in, with little or no prior knowledge of the application’s internal workings. Essentially, black-box testing takes an approach similar to that of a real attacker.

Since black-box security testing does not assume or have knowledge of the target being tested, it is a technology independent method of testing. This makes it ideal for a variety of situations, particularly, when testing for vulnerabilities that arise from deployment issues and systems server miss-configurations.

While the term black-box is well defined, there is no common accepted standard on how and what should be covered under black-box tests. This is subject to specific use case and customer requirements.

1. Black-box Methods

Black box methods typically include vulnerability scanning, penetration testing and systems security posture assessment. During these tests, no information about the applications or infrastructure is supplied to the testing expert – traditionally targeted at seeing the target application from a ”hacker’s perspective”.

1.1. Vulnerability Scanning - Vulnerability scanning tools - either manual or automated are designed to find implementation and software applications/packages vulnerabilities. Application scanning tool examples include: HP Web Inspect, NTO Spider, IBM Appscan, Acunetix, etc. Automated scanning tools for operating system and service vulnerabilities include: NeXpose, Qualys, OpenVAS, NMAP etc.

The tools are very effective at identifying many implementation and operational vulnerabilities. However, there are a high number of false positives, the tools cannot identify effectively design vulnerabilities, they can impact network resources and can potentially disrupt ICT systems.

In order to avoid disruption of operation, it is recommended that vulnerabilities scans are conducted during off-peak hours or maintenance windows

1.2. Penetration Testing - With penetration testing, an application is considered a black box in which data goes in and results are delivered as output. A list of ‘test cases’ is exercised in the application’s native environment. An example test case might include a check for an SQL injection, testing using a browser plus man-in-the-middle (MITM) proxy, communications protocols security testing etc.

The benefit of penetration testing is that it tests the actual implementation and can quickly exploit issues with the application. It can find Implementation, Design and Operational vulnerabilities and can ultimately have very little impact on organization resources. However, it can be very time consuming. For example, a system GUI with 50 pages may require 50 test cases per page equating to 2500 test cases. It can impact production systems (if no test environment is available) and is dependent on the availability of the applications. While it exposes what a hacker would see, it may not find all implementation vulnerabilities.

1.3. System Security Posture Assessment - Based on the definition by the National Institute of Standards and Technology (NIST), the term security posture refers to the security status of an enterprise’s hardware and software its capability to manage its defenses and its ability to react as the situation changes.

There are several available open-source and commercial tools that can provide security posture analysis for operating systems and end user devices. The tested systems are assessed for implementation and configuration errors/mistakes that could result in security exposure.

For purposely built applications or closed OS – for example vendors’ switches, routers etc., Linux or Windows security posture analysis tools will not be effective. The common practice is for ICT vendors to release security configuration hardening guides for customers to follow to improve the security posture of the systems in question.

2. Evaluated Tiers in Black-box Testing

Compliance and regulatory requirements, the rising threats to the information infrastructure and the increasing cost of successful cybersecurity attack are resulting in more and more businesses and organizations conducting solution vulnerability assessment and penetration testing to establish the state of enterprise security, the resilience of the solution and how effective the current cyber defenses are.

Black-box testing methods can be applied to many parts of an integrated the system or individual product. Below are outlined the recommended tiers for black-box testing of integrated solutions and individual systems:

2.1. Tier 1: Products system software security test - The purpose of this test it to ensure that the overall system software is free of security vulnerabilities and undocumented features. This type of test is a low-level software and machine code test using specialized tools and test-bed configured specifically for the purpose. The following is tested although it is possible for organisations to expand the test cases:

- Product binaries test

- Attack simulations

- Product functions security test

- Communication interfaces security test

2.2. Tier 2: Web/Application layer and system interface security test - The purpose of this test is to ensure that the product user interaction and management interface is free of security vulnerabilities and cannot be compromised and taken over by malicious actor. This is application level test using specialized tools and test-bed configured specifically for the purpose. The following is tested although it is possible for organisations to expand the test cases:

- Attack simulations and penetration testing

- Application logic test

- OWASP Top 10

- Communication protocols security test

- DOS/DDoS attack test

- User interface functions security test

- Common Vulnerabilities Exposure (CVE) test

2.3. Tier 3: OSS Layer security test - The purpose of this test is to ensure that the product operation system management layer and interface are free is free of security vulnerabilities and cannot be compromised and taken over by malicious actor. This is application level test using specialized tools and test-bed configured specifically for the purpose. The following is tested although it is possible for organisations to expand the test cases:

- Attack simulation and penetration testing

- Management channel security

- Vulnerability scanning

- DoS/DDoS attack test

- OS and applications abnormal processes scans

- Network communications and protocols security test

Tier 4: Configuration and security posture test - The purpose of this test is to ensure that the products and the systems are configured in a secure manner following industry best practices and Huawei recommendations. This is a high level system and system configuration testing, using a test lab specifically configured for the purpose. The following is tested although it is possible for organisations to expand the test cases:

- Attack simulation and penetration testing

- Vulnerability scanning

- Base line security scan

- Virus, malware, root-kits, malicious software scan

- Targeted implants, stealth implants

- Configuration hardening

- Network communication security test

Tier 5: Solution commissioning and System Security Acceptance Test (SSAT) - The purpose of this test is to test the overall system for security vulnerabilities prior to commissioning and entering into operational service. This is an overall system test in an environment similar to the day-to-day operational conditions. The scope of this test is similar to Tier 4 test, however rather than individual products, the overall system is tested, simulating internal and external attacks:

- System attack simulation and penetration testing

- Vulnerabilities scan

- Baseline security scan

- Configuration hardening

- DOS/DDoS attack

- Systems and applications network communications security test

3. Conclusion

When designed and conducted correctly, black-box type cyber security penetration and resilience testing of products and integrated system/solutions is crucial in determining the efficiency and effectiveness of the cyber security controls and measures employed by the business and organizations, to withstand modern day cybersecurity attacks. Therefore they need to be part of every company cyber security operation, plans and strategy.

However, it also must be pointed out that an effective cyber security operation is a combination of technology, people and processes. While penetration testing can help to verify the technical part, organizations and business need to consider all three parts!

There is no silver bullet and focusing only on the technical, while overlooking the other two aspects (people and processes), is a recipe for cybersecurity disaster! 

Note from the Author

In this article I have attempted to explain and put some order a rather complex and extensive subject. Please also note that each of the tests mentioned in the different tiers can be quite complex and require specialized equipment, software and skills.  

Agree, disagree, have comments suggestions recommendations? Please let me know - I will be glad to hear your opinion!

Vladimir M. Yordanov