Why Managed Triage Is A Poor Economic Proposition: A DarkHorse Security Research Paper

Managed triage is a poor economic proposition.

In this writeup I’ll explain why I believe managed triage (as it's being sold and delivered in the market today) is a poor economic proposition and choice. Doing so will also inherently explain why DarkHorse has chosen not to offer it.?

Given my background, saying “managed triage isn’t worth it” is not something you’d expect to hear from me. I started my career in crowdsourced security as a triager, and spent the last decade at Bugcrowd promoting its benefits. I even oversaw the triage group for a number of those years. If anyone could argue for the benefits for managed triage, it’d be me.

And yet as I’ve gained perspective after leaving Bugcrowd, I’ve come more and more to the conclusion that the way that bug bounty / VDP providers sell managed triage today is fundamentally disadvantageous to the buyers.

I’ll explain.

First, let’s set some base assumptions:

• Vendr.com lists the average deal size for Bugcrowd and HackerOne to be ~$40,000. Some higher, some lower. It also asserts this includes an average savings of 12.7%, which puts the average deal size closer to $45k.
• As per data from HackerOne’s public programs, the average number of reports over the last 90 days for a public program (at the time of this writing) is ~71; this number fluctuates up and down at any given time - it was 65 earlier in the year, etc. Note that this number is for public programs, which is significantly higher than what one might expect from a private program. To get to the number of reports annually, we’ll be generous and extrapolate that 90 day number across a full year to be 284 (71*4) reports. We’ll also be generous and say that’s roughly true for all other platforms, which we’re confident isn’t the case - but given that nobody else publishes any such public data for us to use, this is the best we can do for now.
• HackerOne has indicated in some of their online collateral that public programs get 3.5x more active participants. We’ll take this to mean that the number of reports increases by approximately at the same rate.?
• For the purposes of providing as much benefit-of-the-doubt to these providers, we’ll also continue to be generous and operate with the assumption that there are equal numbers of public and private programs (there are most certainly more private than public programs - likely anywhere from 2-5x). Additionally, based on the above information on 3.5x the activation for public programs, we’ll say that private programs get one third the average of a public program. We’ll call that 95 reports per private program over the course of a year.
• With our assumption that there are equal numbers of public and private programs, and that private programs get 95 reports per year, and publics get 284, we get an average blended number of 190 reports per organization (where each contract costs ~$40k).
• Again, all of these assumptions are extremely favorable towards the providing organizations here (HackerOne, Bugcrowd, Intigriti, et al). The true numbers are likely to be even less flattering than what we'll get into in a bit.

As a quick aside, for those unaware of what I’m talking about when I say “managed triage”, I’ll explain quickly. It works like this:

• Bug bounties and responsible vulnerability disclosure programs can get a lot of reports / submissions.
• As per HackerOne’s data, only ~20% of those reports are valid - meaning that approximately 80% of all reports are “noise”.
• So, the platforms that provide bug bounties and VDPs bundle in a service called “managed triage”, where their in-house teams do first-level triage on all the incoming reports, weeding out the noise, and passing along the signal.

On the surface, this sounds like a pretty great and straightforward thing; however, as given away by the title of this report, I’ll dig into why this isn’t actually the case.

Again, using these extremely favorable numbers, we have an average of 190 reports across an average contract size of $40k. This gives us an average cost per report of $210. And that’s not including rewards (if one is running a bounty).

Before we go any further, it goes without saying that these assumptions are imperfect. For instance, a single organization may run multiple programs under a single contract, etc. However, this is why I’ve been extremely generous in my assumptions. Some organizations may run multiple programs under a single contract, BUT there are also many contracts that go for well under $40k (such as for pentesting or attack surface management) that likely drag down the average contract value across all contracts. AND there are likely significantly more private programs than there are public ones (which would drag down the average number of reports per program, while simultaneously raising the cost per report average). All things considered, I’d wager a decent amount on the belief that the average $40k contract gets significantly less than 190 reports per year. If there’s data to support this being incorrect, I’ll happily update my position - but it’s unlikely anyone is going to get transparent here. Said differently: the real average contract size for VDP/bounty is likely higher than $40k, and gets fewer reports than 190; which is to say that the real average cost per report is also significantly higher than $210.?

Since the existing providers bundle everything together, in order for us to get into triage costs (so as to establish whether or not it makes sense financially), we first have to make a quick n’ dirty assumption around how much we think the platform should cost.

Fundamentally, the platform (paid for by whatever costs we don’t associate with triage) allows for one to receive and manage vulnerability reports, along with a few other things such as integrations, crowd management, and so on. Going on gut, how much is this functionality worth per report? $50 per report feels like a good number, considering that 80% of all reports are “noise”, and you still have to pay this amount per report, noise or not, it seems reasonable that when accounting for noise, you’re actually paying approximately $250 per valid report. That seems palatable as a starting point.

Given our starting cost of $210, this leaves $160 left to cover triage costs per report.?

NOTE: you can also do this math for your own program / contract - just divide your contract cost by the number of reports you got for the year. I think you’ll be surprised to see how much you’re paying per report when you start looking at things objectively.

So, is $160 for triage per report a deal… or no? Let’s see.

Across 100 reports, let’s make these generalizations:

• 20% of all reports are valid, unique issues (HackerOne reports this number being 19%, but we’ll round up for easy math)
• 30% of all reports are duplicates
• 30% of all reports are not applicable / junk / noise
• 10% of reports are out of scope.
• The remaining 10% of reports are not reproducible.

Loosely speaking, I think these numbers are directionally accurate. As always, I’m happy to update assumptions based on any official data, should anyone be willing to provide it.

Now, let’s give some weighting to these reports in terms of effort. One effort point is how much work it takes to triage a single valid, unique vulnerability report.

• Valid, unique: 100% of one effort point
• Duplicates: 33% (e.g. one can process three duplicates in the time it’d take to process one completely new report. This is again, a fairly generous estimation - I’d assume in actuality it’s closer to 4:1 or 5:1)
• Not applicable: 25% of one effort point (4:1 seems reasonable here)?
• Out of scope: 15% of one effort point (out of scope is extremely easy to process, as it takes only seconds to look at the target to determine whether something is in scope or not. Exceptions do apply, but this is a loose average)
• Not reproducible: 80% (NR reports tend to take a lot of effort, as there can be a lot of back and forth; this number could be as high as 100%)

So, as an average across 100 reports, we get the following total effort scores:

• 20 valid, unique reports: 20 effort points
• 30 duplicate reports: 10 effort points
• 30 not applicable reports: 7.5 effort points
• 10 out of scope reports: 1.5 effort points
• 10 not reproducible reports: 8 effort points

47 effort points in total.

In terms of what gets triaged, you (as an organization) should just see the 20 valid, unique reports, and then we’ll say another 10% of total reports where you need to weigh in on NA/NR/OOS and so on. In my experience, most program owners will agree that this 10% number is probably too low - it’s fairly common for triage to ask the client for input on a large number of reports since they don’t have business context - e.g. asking things like “is this a valid attack in your threat model” or such. We’ll go low and call this an extra 2 effort points in total.

Now, with everything that makes it through triage (valid issues, false positives, etc), you still need to review, reproduce, process, and give input on those reports… and while it’s possible / probable that some of that effort may be reduced as a function of good triage notes, we also need to take into account the presence of false positives and false negatives - which we’ll say is ~5%. On average, we’ll say that any speed gained from good triage notes is offset by the presence of false positives / negatives.

In all, after triage has done their job you’ve got 22 effort points to cover on your own.

Out of an initial total of 47.

Said differently: even after paying 100% of the cost for triage to look at your findings, you still end up having to perform 47% of the work or more (22 is 47% of 47… just in case all the 47s start to get confusing).

I’ll say it again for emphasis: you pay 100% the cost of triage, while having to re-perform nearly 50% of the effort. And again, all of this is based on fairly generous and favorable assumptions. It's highly possible that in many cases the level of effort exceeds 50%.

I’m unsure where you draw the line in regards to where something becomes inefficient, but if you’ve only got 50% output for 100% input, that’s inefficient no matter how you cut it. If I got out 50cents for every dollar I put into something, I’d stop doing whatever that was real quick.?

But inefficient can sometimes be ok - if you can pay ten people that do the work of three for the price of two, you’re still coming out ahead, regardless of how inefficient it is in the aggregate. Is this true for triage?

Earlier we said the average dollars per report that (ostensibly) go toward triage is $160. Maybe that’s still a good deal? Let’s see how much it’d cost you to triage…

The average triager can process 30-40 reports per day. But one of the big arguments made in favor of managed triage is that the managed team is significantly faster to triage. So we’ll again be generous and say that you can probably only process half as many as a trained triager: 15-20 per day. So, if you had someone on your team do the same amount of work, assuming you’re paying a security engineer $50/hr (which is ~$100k/yr), your average cost per report with someone processing 20 per day would be: $400 (daily cost) /20 (number of reports processed in that day)…

Wait.?

That’s $20 per report. That can’t be right.

What’d we miss?

Well, for starters, you have to pay benefits to your employee and there are other costs (assuming you don’t just 1099 offshore contractors, which would cut costs by well over half…). ChatGPT tells me that employees actually cost 1.25- 1.4x their salary. Let’s go with the high end of 1.4x. Additionally, let’s increase that salary to $120k. Now where do things sit??

As a note, considering that performing triage is generally considered an entry-level position, paying $120k in salary for someone who performs triage is pretty darn expensive in 90% of the United States, and 99% of the world. Anyways, let’s proceed with this number. Said differently: on average, this is a generous salary for triage.

120 base * 1.4 = $168k/yr OR $84/hr

Surely managed triage will be a veritable steal now that our costs have nearly doubled. Let’s see!

Across 20 reports processed per day, that’s $33.6 per report.

Huh.

And again, all of this assumes you’re paying a pretty decent, high-cost-of-living-area wage to the person doing the triage (especially considering that triage is often an entry-level role). If this role were offshored (which is what you’re typically getting when paying for managed triage), your costs would be halved (or less).

That can’t be right. You can’t be paying effectively $160 per report (not including the effort that you have to re-do!), when the real cost is $34 to do it on your own. What if we cut it to 15 reports per day? Does that fix it?

Not really... now it's just $51 per report.

For reference, when I was a triager there was no platform tooling to support, and when I finally worked my way up to making $3 per fully processed report, I was making anywhere from $12-15 per hour. Said differently: I was a part-time triager who certainly wasn’t the most brilliant or prolific triager in history, and I did 4-5 reports per hour, on average. If I could work through 30+ reports in 8 hours on a 10” tablet with 2gb of ram (where most of my time was spent waiting for my VM to respond), I’m absolutely confident someone on your team (or even an entry level intern) can process 15 (let alone 10) in a day at a whole lot less than $120k in salary (again, a mere ten years ago, I and a bunch of other people were doing this for effectively $15 an hour with no benefits).

But wait, the math gets even worse.

Since you have to expend ~50% of the effort, no matter what (even if you're paying for managed triage), that's going to cost you ~$1700 per 100 reports (50% of $34 per report).

So, your base cost, again, no matter what, is ~$1700 per 100 reports.

To do the other 50% of the work across those 100 reports, it's another $1700.

How much is it for managed triage?

At $160 per report, to do the other 50% of the work, that'd cost $16,000.

Said differently: managed triage is nearly TEN TIMES the cost.

Ten. Times.

Shoot.

It doesn’t take an MBA to see the better option between spending an extra $1700 to triage in-house, or paying $16,000 for managed triage (though it does take an MBA to spin the narrative that managed triage is somehow a good deal in light of all this).

Managed triage isn’t just 3-5x more expensive (as it would appear on the surface), it’s TEN TIMES more expensive.??

There’s certainly something to be said for the ease of not having to manage other people and what not… it most certainly is easier just to let someone else handle it all. BUT remember, you still have to do approximately 50% of the work, no matter what. And instead of handling the other 50% yourself, paying someone else to do it will cost you ten times what it could cost you to pay someone on your own team to finish the job. When you start framing things in context, it starts to make sense to just do the other 50% in-house.

So, what’s happening here? And why??

Fundamentally, these businesses market themselves as SaaS businesses, but they carry a huge amount of overhead in their triage teams, which are effectively services groups. However, valuations for services companies aren’t all that great (much smaller multiples), so they need to make SaaS margins (~80%) across the business, even on services (which more typically have a much smaller gross margin of 20-30%).

We can see this by doing the math… Assuming the cost of triage is approximately the same for the organizations selling it, if we take our $34 report cost (20 reports/day) and apply 80% margins to it, we get $170. Not all that far off from our initial estimate of $160.

On principle, I don’t love inefficiency - paying 100% of the cost for 50% of the value is already a tough proposition to swallow. Add on spending $16,000 on something I could do myself for roughly 1/10th of that cost, and this just seems like bad economics.

It’s up to you what you do with this information, but I know this isn’t something I’d pay for with my budget.

Maybe it’s a deal for you, in which case, I applaud your deep budget and how much you must pay your employees for this to make economical sense.

It’s also possible to argue that one is paying more for the platform than we estimated earlier ($50). But that also becomes a tough pill to swallow once you start paying $100 per report to the platform… just for existing. Sure, there’s R&D and features n’ such, but is it worth that much per report? AND that’s paying $100 for every single report, just for the luxury of it existing. Keeping in mind that only 20% of the reports are valid - so you’re paying $100 per duplicate, invalid, not applicable, and so on. Does that feel like a good deal? Only you can make that call.

And even if the platform gets $100 of the $210 per report, now you’re still paying over $110 for triage per report. And, as we’ve established previously, across 100 reports that’d effectively be $11,000 in cost per 100 reports vs. doing it in house for $1700. Still a whopping 650% increase, AND you still have to find a way to justify paying $100 to the platform for simply existing.

All things considered, this just doesn’t sit right with me at this point in time.

In fact, I feel so strongly about the massive pricing disparity here that I built a platform where the per-report cost is literally $7 (which, you’ll note is a lot-lot-lot less than $50 or $100 per report). I won’t try to sell you on managed triage either. Just a low cost platform that puts you back in control with 80-90% of the features, for a fraction of the cost. You can learn more at https://darkhorse.sh, or shoot me a message and I’d love to help you out.

Of course, this review wouldn’t be complete without going over some of the common reasons given for why managed triage is better and/or required. Let’s go through a few of them…

Isn’t this why people pay for managed services in the first place? Nearly anything is cheaper if you do it yourself… people pay premiums all the time to be able to have a qualified expert do things for you. Why is this any different?

• At a surface level, this makes sense. One often generally pays more for managed services - that’s just how it is. However, there is one key point of difference here: no matter how much noise is eliminated by managed triage, as we established earlier, you still have to do ~50% of the same work. What this means is that you still have to be capable of doing triage work via your team in the first place. It’s not like you’re hiring someone to do something you’re incapable of doing, or that ONLY they can do, or that they do significantly better (e.g. a plumber, roofer, or electrician) - in fact, it’s the opposite. You’re hiring someone to do something you’re likely over-qualified for. In the vast majority of cases it’s not that you can’t triage bugs - if you’re paying for managed triage, it’s because you have bigger things to do. Massively overpaying for a less skilled role is very, very different from paying a premium for a more highly skilled position.
• Paying 10x the cost of doing it yourself is also highly unusual. Again, this type of premium is reserved for if you’re bringing in someone to do something wildly complex. Paying this much for someone to do something less skilled is unheard of. Except here.

Managed triage is faster than doing it yourself.

• I’m not completely convinced this is inherently true. It’s one of those things that feels true, but I think the differences are likely far less significant than one would be led to believe. It’s covered in other places, but it’s not all that complex to perform bug triage - I’d argue that after a few months of exposure, the differences in performance are likely to be negligible.
• Furthermore, as we showed in the math, even if you run at 1/4th the speed of an average triager, the cost savings are still significant. Managed triage would need to be 20x faster than you doing it for it to start being a remotely competitive conversation.

Managed triage is more competent than doing it yourself. They see more bugs and do a better job as a function of it.

• Same as the above, on the surface, this seems like a valid argument until you (1) account for it not being a particularly complex role; and (2) consider the incentives. Managed triage providers are incentivized to do as many reports as quickly as possible, NOT to do the best job possible. In fact, I’d argue that many reports are poorly processed as a function of this. Triagers, knowing they’re measured on throughput, work to do as many as they can as quickly as they can, and there's often fallout as a result of this. Again, to be abundantly clear I’m not saying this is negligent on their behalf or that triagers are bad. That's 100% not the case. Most triagers that I know and have worked with are fantastically hard workers and genuinely good people; any rapid processing errors here more a function of the system they’re operating within. Unfortunately, as a result there’s no shortage of fairly public complaints by hackers around triagers racing to hasty judgement, which is again a function of the incentives not being properly aligned. All things considered, even if you're marginally slower at the job or even have to ask more questions to get to the end result, I'd argue that most organizations would benefit more from doing the triage themselves, as opposed to utilizing a third party.

Managed triage ensures a better experience for hackers by enforcing tight SLAs.

• It’s true that many clients are slower than they should be when it comes to managing their program and responding to reports. However, managed triage doesn’t inherently solve for this. All it means is that reports get a first touch more quickly, but it absolutely doesn’t mean they get resolved or rewarded any faster. A triaged report still has to sit and wait for the client to reward it (the same goes for cases where a triager is awaiting client input on a report) - managed triage cannot and does not make that happen any faster. It can help process out all the noise faster - so that a duplicate is called a duplicate sooner, but they cannot lead to positive final outcomes (e.g. getting paid) any quicker than if a client did their own triage. Paradoxically, it’s possible that by adding the triage layer on the front end that it creates the effect of removing the client’s focus from their program - where, if they managed it end-to-end, they’d be more responsive and engaged overall. And who knows, with more budget unlocked by not paying for managed triage, clients may even have faster SLAs as a function of being empowered and having a greater sense of ownership. There are a number of great studies where outcomes improve across the board when people are given a sense of ownership and agency. Of course, with adequate support, which we’ll cover next…?

All the above being said, my (and the DarkHorse view) on all of this is:

You / program owners are more capable than they’re given credit for.

• If organizations can make things seem big and scary, they know you’re more likely to want and use their solution. While triage may appear scary, it’s actually not all that complicated. I don’t mean to insult the people doing triage, but I think they’d agree when I say it doesn’t take a particularly high level of skill to do triage. Beyond some generally basic application security knowledge, you simply follow the provided reproduction steps, and if they don’t work, you go back to the reporter and ask for clarification. On top of this, by being inside the business, program owners have important context around the scope and how to contextually look at vulnerabilities that a 3rd party triager simply isn’t able to provide. The only other requirements for triage are (1) attention to detail; and (2) being an understanding human. I’d go so far as to say that 90% of what makes a good triager is communication. Long story short: organizations are absolutely capable triage, and I’d argue that they may even make better triagers, when done right. Technical competence as it relates to triage is overrated; while treating hackers with respect and communicating clearly are highly underrated in terms of what makes a quality hacker experience. I don't think many would disagree that a well-intentioned, but technically novice triager is preferable to a highly technical, but abrasive triager. The willing novice can learn; the rude genius is unlikely to adapt. Most hackers are extremely helpful by default - if you earnestly ask a question or for help, the vast-vast majority of them are more than willing to assist. Which leads us to…

You / program owners benefit enormously from directly interacting with the crowd.

• Sure, it can be tempting to offload all the relational work to a 3rd party that’s located in a low-cost offshore location (as is the case with the current platforms), but the value of a quality relationship cannot be understated when it comes to bug bounties and vulnerability disclosure programs. When you triage personally, you get to build and nurture those relationships directly - instead of the hackers dealing with the same triagers as always, they get to see you. Which gives you a chance to differentiate your program through your words and actions. As you build trust with hackers, they also build trust and relational capital with you and your organization, leading to improved engagement over time. You may even hire some of them in the long run!

The platform should support and guide you / program owners.

• This is a key DarkHorse belief that’s gone into everything we’ve built. If we’re going to empower you to set up, manage, and triage your program, you’re probably going to need some support along the way, and that’s exactly what we’ve built the platform for. To guide, remind, and prompt you along the way to do the things that will help make your program successful - all while having to deal with fewer people (who for some reason are always asking you for more money), and putting the controls back in your hands where they belong. Because, again, you’re far more capable than what others may want you to think you’re capable of… because they want to make money off you. So, in summary...

Managed triage is expensive.

• With one of the major crowdsourced security providers (HackerOne, Bugcrowd), you’re likely paying $210 or more per report (you can do the math for your own program: contract value / # of reports).

Managed triage is inefficient.

• Approximately 50% of the work performed by managed triage needs to be duplicated by you.

Doing triage yourself can save you a lot of money.

• If you have an employee that can triage 20 bugs per day and gets paid $120k/yr, your cost per triaged report is only $34.
• Across 100 reports, since you have to re-do 50% of the work, regardless, your base cost even with managed triage is ~$1700.
• For another $1700 you can do 100% of the work.
• Or you can spend TEN TIMES as much ($16,000) to pay for managed triage.
• Said differently: across the average number or reports (190) and a full year of not using managed triage, you’re likely to save nearly $30,000!
• $30k that can be used to hire, buy tooling, give raises, and other essential items.

Despite the stories, managed triage isn’t significantly better than doing it yourself.

• You’re just as capable, and with enough reps, I’d argue you’re likely to (1) be better; and (2) see significant benefits to in-house triage over time (relationships, etc).

DarkHorse is an affordable alternative.

• Less than 1/10th the platform cost for 80-90% of the same features as the bigger platforms.
• We won’t sell you managed triage, because we don’t think it’s a good use of your money.
• To extend that... we won’t sell you anything we don’t think is a good use of your money. That’s just not what we do. We’re here to make the world a better place.

Hopefully this guide is helpful. What you do with it is up to you; as they say, knowing is half the battle.

As always, if you're looking to save money while being more secure, send me a message and I’d love to help you out!