Is Penetration Testing broken?

I recently came across the following question on twitter: "Why are Management teams afraid of having a PenTest or Risk Assessment?". As this is such an important question, i thought it deserved a slightly longer and more thought out response than could be achieved in a 140 character tweet.

First, some background on me. I'm a penetration tester that moved into security after spending time as a software engineer. I've seen it from both sides, so to speak. I know the pressure there can be to meet a deadline (and yes, i hate to admit it but I have shipped vulnerable code) but I also know how important it is to get security right. 

So, why are management teams afraid of getting a pen test?

I've performed more pen tests and security reviews than I can count. I've performed testing for small startups and global organisations and whilst there does seem to be a trend in avoiding pen test I don't think its fear. I think it's more accurate to say it's about priority. Due to the nature of pen testing, we require the testing environment to be as similar to production as possible (if not production itself). On a standard pen test, we have a testing window (usually between one and two weeks) and anything that will delay the test, such as unstable environment, software changes or connectivity issues will eat into that time.

Because of this requirement, pen testing is requested at the end of development. Best case, nothing is found and the application can ship (was testing therefore a waste of money?). Worst case though....a complete lack of security. What's the right thing to do? Unfortunately, I don't think there is often a choice. Many organisations will need to release an insecure product with the "best intentions"(tm) of addressing the security issues in the next release.

As someone that cares about security, I tell myself there must be a better approach and actually yes, I think there is. The solution is to move away from this model of security testing. Security has had trouble keeping up with modern software development practices. Pen testing doesn't always work well in agile environments (again, pen testing works best when we're looking at production ready software). When the code is constantly changing, any security testing is invalidated when the code changes. This would definitely be the wrong time to bring in expensive external consultants. 

I put some blame on the InfoSec community for this. We are often too far removed from development teams. It's easy to sit on an ivory tower and say "this software is broken, fix it" but we shouldn't be surprised when it isnt when the software works as far as the requirements dictate. The priority is getting the software out the door and making money. A PDF of vulnerabilities found during a short testing window is of limited use, especially when the report is littered with far too many low/informational risk issues (the truth is, we don't put these in to pad the report. It's because they do represent risk, and it would be negligent not to report them).

The InfoSec community should be working better at integrating with development teams so that security is embedded into the whole process. Rather than making tools for other hackers, we should be making tools for developers. This doesn't mean tools with nice GUIs, but tools that will integrate into the development pipeline. Tools that can be extended with plugins or that will work with knowledge of the code base. Security training should also take another approach too. Much of the security training i've seen teaches people to hack which, although usually high quality training, the training doesn't target the people that really need it; developers, QA Testers and Project Managers.

So, to start to address the issue of security here is what I propose:

• Removing the ego in InfoSec - yes, we're hackers but that doesn't mean we know everything. Sometimes there is a reason a vulnerability exists whether that's due to budget, deadlines or usability. Ignoring business requitements pushes us further out of the teams that need our knowledge. We're there to help, not humiliate.
• Rather than selling penetration tests, security consultancies should and integrate security into the teams. Work with Project Managers to create Abuse Stories that describe security risks. The architects can make design choices that can be tested against the Abuse Stories. Work with developers to write tools that are useful to them. 
• Teach QA testers to perform basic security testing. Despite how we make it sound, much of what we do in a pen test is fairly trivial and some basic training would go a long way.
• Companies should start being proactive about security. Engage with security professionals early. Talk to them before and as code is being written. Make sure the design is solid and the developers understand how to write defensive code.
• Get rid of the PDF report. Instead, consultants should deliver usable test cases to development teams.
• Hire developers that do care and understand security and have them use techniques such as mob programming to share knowledge.
• Consider a DevSecOps approach to working.

Once security stops being something that is bought in and becomes another measure of code quality, I think we'll see a drastic change in how security is seen within organisations. It'll become easier and cheaper to do right and therefore will be a higher priority. At the moment, the cost of bringing security in can seem daunting so it is often ignored. The InfoSec community should be creating the tools, techniques and methodologies that development teams can use to make security easy.  

There will always be a need for a penetration tester but it shouldn't be to report on issues that could be found and fixed automatically during product development.

The link to the original tweet(s) can be found: