Crowdsourced Pen Testing: Does it work?!

I know what you’re thinking. I’m guessing it’s something along the lines of “Amy, who the actual f*ck crowdsources their security testing?”. If you’re a CISO, you’re probably also thinking “how the hell can I get in on this action?”. But please, read on to hear some thoughts I had over a glass of wine with Shaun before calling in the masses…?

Ok so you might not have actually been thinking either of the above things. You were likely wondering what I mean by ‘crowdsourcing’ security. So let’s start there. By explaining what I really want to talk about.?

Bug bounty programmes.?

Over recent years, we’ve seen a big rise in bug bounties. Programmes that define parameters for hackers or security researchers to test for vulnerabilities in specific products, applications, environments within an organisation and report them back to the company in exchange for cold hard cash and a bit of kudos.

Tesla famously offered $1 million if you could hack into one of their cars. Microsoft run a really successful programme with thousands paid out weekly for new exploits found in their products. Hundreds of companies have their names on bug bounty platforms like HackerOne, Synack Red Team, Bug Crowd, etc.?

So what’s the deal??

Well I guess the thing I want us to consider here is whether they're a good idea for businesses and what the issues that might arise as part of a bug bounty programme.

Continuous pen testing.?

I’ve seen a few studies when I was googling ‘bug bounties’ that described running this type of programme as ‘continuous pen testing’. I disagree. Continuous pen testing for me is either a team of offensive security professionals looking at an environment on an ongoing basis. Most banks run this kind of operation. Or… it’s a vulnerability scanner (maybe with a fancy dashboard).?

Bug bounty programmes are neither of these things. There’s no guarantee that anyone is checking the entirety of your environment. Usually because you’ve scoped only a small part of it to be included in the bounty. Mostly because you cannot ensure you have a constant stream of people interested in enough in YOU.?

I know you all think you're mega interesting and important. I think the same about myself too. But let's face it, it's just not true.

Unless you're a huge corporations bringing out new products and services, where innovation is regular and there are new things to test every couple of weeks or months. The Teslas and Microsofts. Even the Ubers and Rockstars. The kudos someone would get in the community for popping a big hitter would be great. The money you’d get would be great too. And herein lies another issue.?

It’s expensive to crowdsource.

The bounties I’ve seen are all in the tens of thousands. Apple pays six figures for vulnerabilities. Tesla was paying $1 million. The severity of the vulnerability often determines the price tag. So it’s no wonder that some people entering the industry want to focus on bug bounty rewards - it’s an attractive proposition.?

But can most companies afford the bounty to make their programme attractive? Do YOU have tens or hundreds of thousands to spend on the report of a single vulnerability? Based on how hard it is to pry cash out of your hands for a test where we tell you ALL the vulnerabilities, I’m gonna guess your answer is “No.” And from a security researcher perspective, why would they focus on finding a vulnerability for ￡5k when a different organisation is paying ￡50k??

Sure, some researchers find an issue and go round attempting to replicate in all environments they know have a programme, so you might get swept up in the ‘gotta catch em all’ mentality of some bug bounty experts… but then you’re still not getting a holistic view of your security posture.?

You’ve been given ONE vulnerability for a hefty price. Surely that money would be better spent on penetration testing or red teaming with some experienced and reputable security consultants?!

Consultants with integrity.

Now I’m not casting shade on security researchers and those who have picked the bug bounty life. It's a tough gig and the wins are hit-and-miss for them. But where are the guarantees that they’re handing over ALL the information. How are you reliably monitoring the integrity of the data you’re given??

With between 35-55% of all bug bounty reports being invalid, the integrity of the vulnerability information provided as part of these programmes is clearly up for question.?

Using a bug bounty programme as part of your security strategy is supposed to be a help to you, not a hindrance. So imagine being the poor security team that has to deal with the influx of reports on the reg and has to validate them all.?

I guess you could restrict the programme to only allow specific people to participate, but then you run the risk of deterring researchers. So then you’re stuck with a toss up between a million reports you need to look into, or no one bothering to do any research on your environment at all.?

So, it’s fair to say that I’m not enamoured with the bug bounty thing, particularly for smaller organisations. I'm obviously going to be biased towards using penetration testing for a holistic view of your infrastructure and applications though, given what I do for a living. So with that in mind, I’d be super interested to hear YOUR thoughts on it and whether you think they’re worth your time as a researcher or from an end client perspective.