Testing Outside the Box

I don't think outside the box. I think of what I can do with the box.

Henri Matisse

One of the ongoing battles in the penetration testing community is what kind of testing to perform: white box, black box, or something in between.

The debate rages because these different styles of testing also serve different functions and goals.

White Box Testing. White box testing provides full access to testers to any/all information available. Testers are, essentially, treated as local administrators, with privileged access to network diagrams, DNS records, log sources, source code, and even administrator accounts. As a result, testing is extremely thorough and comprehensive, but may also lead to large numbers of findings and issues that are unlikely to be discoverable or exploitable by a remote hacker.

Black Box Testing. The opposite is black box testing, where the client doesn't provide any information at all to testers, excluding the company name and potentially the main domain name. This lack of information forces a penetration tester to emulate a remote attacker most closely and discover any relevant information related to the target on their own. While fewer findings are usually reported in a black box test, every single one of them could be discovered and exploited by any anonymous attacker on the Internet.

Gray box testing combines the pros and cons of both approaches and is most often used in web application penetration tests where the pentesters are provided credentials to the application, but not back-end access to logs or source code.

So, what is the big controversy? After all, each type of testing process has its own purpose.

The problem arises with limited budgets and time.

I have yet to meet a client with an unlimited budget for security testing and more than 24 hours in a single day. As a result, clients usually have to pick one type of test or the other - but not both.

So, if we only perform one (or maybe two) penetration tests per year, what do we prioritize? White box style? Or Black box?

Do we prioritize more findings? Or more realistic hacker scenarios?

And that is where the controversy (or at least the difference) comes in, because each type of test is valuable and each client may be different in their respective goals for testing.

Nevertheless, the overall trend in the industry is towards more white box testing for the majority of projects.

In fact, the PCI standards state (emphasis is mine): “PCI DSS penetration tests are typically performed as either white-box or grey-box assessments. These types of assessments yield more accurate results and provide a more comprehensive test of the security posture of the environment than a pure black-box assessment. Performing a black-box assessment, when the entity provides no details of the target systems prior to the start of the test, may require more time, money, and resources for the deliverables to meet the requirements of PCI DSS." [https://listings.pcisecuritystandards.org/documents/Penetration-Testing-Guidance-v1_1.pdf]

In addition, NIST states “White box techniques still tend to be more efficient and cost-effective for finding security defects in custom applications than black box techniques” [https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-115.pdf]

For us and our clients, we always encourage white box testing these days as the standard test if any of the following are true:

• Testing is performed a maximum of twice per year
• Testing has not been performed before, or it has been a while since the last test
• There is not a dedicated security team monitoring alerts or logs from the application

Because white box testing is more comprehensive, it is particularly useful for organizations that are not constantly testing and identifying issues already. The additional issues that white box testing identifies should be appropriately assessed and handled according to the organization's risk tolerances (i.e. fix them, mitigate them, ignore them, or transfer the risk to users/vendors).

Similarly, we recommend black box testing in situations where testing is regularly performed and a dedicated security team (often called a blue team) is present to monitor and potentially respond to testing activity (depending on the nature and complexity of testing, this type of approach is also sometimes called Red Team testing). In this scenario, the reconnaissance and information gathering phases of the test represent important early warning signs of attack that security teams can review and, hopefully, correlate to later attacks.

If possible, of course, the best solution is simply to do both - white box and black testing. But, if your team is like most and you don't have infinite budget for testing, then start with white box testing until you feel confident that additional testing won't continue to reveal significant issues - then black box testing may be the most relevant to focus on testing alerting as opposed to simply finding vulnerabilities.

In either event, don't over-focus on the general industry trends as much as your own goals and needs.

In the end of the day, it is still your box.

Security News

• A vulnerability found in the Really Simple Security plug-in allows an attacker to remotely gain access to any account on an affected website, including the administrator, when 2FA is enabled.
• ChatGPT exposes significant data pertaining to its instructions, history, and the files it runs on, placing public GPTs at risk of sensitive data exposure, and raising questions about OpenAI's security on the whole.
• Palo Alto Networks (PAN) put out an advisory on Friday, Nov. 15, warning its customers that a critical, unauthenticated remote code execution (RCE) bug is under exploit by cybercriminals in its Expedition firewall interface — making this the tool's fourth vulnerability under active attack identified in just the past week.
• Experimental counter-offensive system, dubbed ‘Mantis’, responds to malicious AI probes with their own surreptitious prompt-injection commands, turning attackers into prey.
• Multiple threat actors have been found taking advantage of an attack technique called Sitting Ducks to hijack nearly 800,000 legitimate domains for using them in phishing attacks and investment fraud schemes for years.
• Google appears to be readying a new feature called Shielded Email that allows users to create email aliases when signing up for online services and better combat spam.
• Cybersecurity researchers have disclosed a high-severity security flaw, CVE-2024-10979, in the PostgreSQL open-source database system that could allow unprivileged users to alter environment variables, and potentially lead to code execution or information disclosure.
• A newly patched security flaw impacting Windows NT LAN Manager (NTLM) was exploited as a zero-day by a suspected Russia-linked actor as part of cyber attacks targeting Ukraine.
• The U.S. federal government's repository for security vulnerabilities is struggling to clear a backlog of tens of thousands of unanalyzed flaws after failing to meet a self-imposed deadline for making the database up to date.
• A threat actor with suspected ties to Russian nation-state hackers has listed thousands of vulnerable IoT devices as proxy networks within minutes of their initial compromise. A campaign that began in 2020 has so far infected 20,000 IoT devices, according to security firm Trend Micro.

Until next time!

The Craft Compliance team