Lack of Talent is Not the Problem in "Cyber"!

Today I saw the following headline in the Washington Post:

Following the OPM data breach, Uncle Sam needs to step up recruitment of cyber talent

No. That isn't possible, wouldn't help, and might make things worse. Let's let Frederick Brooks (The Mythical Man Month, 1975) explain:

adding manpower to a late software project makes it later

The "cyber" security problems that the US Government, and every other government, and every large and medium enterprise are all coping with today do not stem from lack of "cyber" talent. Rather, the way we build, and buy, and manage our technology dooms us. If we change those things, we can stop or at least slow the losses we're all experiencing. If we don't, then no amount of new "cyber" talent is going to make any difference, other than by adding to the general confusion, getting in the way, and wasting money, brains, and time.

Build?

The hardware and software being built to automate and enhance much of the human experience and the world economy has a relevance problem -- makers cherish the relevance of their work, often measuring that relevance in "total revenue". Time to market and new whiz-bang features are the solution to that problem -- get new stuff out into the market faster than your competitors! It's extremely rare that anybody measures or manages for "total quality level" or "attack resilience" unless we're building military or space systems.

Buy?

The hardware and software being bought to automate and enhance much of the human experience and the world economy has its own relevance problem -- buyers have a list of features they need and problems they have to solve, but they generally do not have a corresponding list of problems they have to avoid. Most technology goes straight into production when delivered, and most of what sees the inside of a test lab on its way into production is never competitively "red teamed" -- there's rarely incentive compensation for "good hackers" who can break into this new technology and take it over.

Manage?

The hardware and software being deployed to automate and enhance much of the human experience and the world economy has a TCO (total cost of ownership) problem. A lot of software is apparently free these days, and a lot of business plans now depend on this apparent freeness. But nothing is free in the long run, if you count the cost of monitoring it, fixing it when it breaks, upgrading it, and especially patching it. As a result, we tend to avoid monitoring it, make fixing it somebody else's problem, never upgrade it, and mostly can't patch it.

We're on the Supply Chain to Hell

Today one of my personal VPSs (virtual private systems) told me that one of the software packages I've installed (called "gnutls-3.3.16") has a reported vulnerability. Another system reports similar problems in two other packages ("bind910-9.10.2P2_5" and "pcre-8.37_2"). This means I have to go through my entire inventory of systems both virtual and physical and install updates until I can get a clean audit. This isn't an emergency -- I have to do it every week, and I have some automation to help me with it. But to decide what this means for the rest of human society and the world economy, consider the following:

1. My systems are all within a week or so of being up to date, at all times, and so I don't anticipate a lot of conflicts or downtime.
2. I was once a powerful "cyber" wizard and I recall enough of that craft to be able to take care of my systems in this way, and to patch things by hand if there is a conflict.
3. These are my personal systems for the most part, and downtime isn't going to cost me any revenue, in fact is likely to not be visible to anybody but me.
4. I have only a few dozen systems in my inventory.
5. I don't have to argue with anybody about how to manage them.

Those conditions are unusual. Large enterprises like Google and Facebook have invested hundreds of millions of dollars in being able to keep their systems safe in the face of an ever-evolving supply chain. But when your average webmaster presses the "install now" button on a new VPS, they're going to get the latest versions of the operating system, web server software, and layered applications like Wordpress or Drupal, and then they're going to carve content into all those components, and then for the most part they're not going to touch any of it again unless there's a problem. 90% of the Internet's hardware and software infrastructure is managed that way, 9% is managed the Google/Facebook way, and if we're lucky then as much as 1% is managed the way I do it at home.

And even doing it my way leaves systems vulnerable for a few days until I get around to my once-a-week patch-a-thon. Maybe two weeks, if I take a holiday. And of course there are the bugs that haven't been reported yet, either because no-one has found them yet, or because only malicious people have found them and they're still conducting the auction to see which national government is willing to pay the most money for this so-called "0-day".

Most damningly, the most common use case for a new vulnerability in some part of the supply chain is not to break into it and steal data. I say "damningly" because if that were the common use case, then everybody would care more about this, since there would be a self-defense aspect to it -- more people would care! But the common use case is to break in and then use these new resources as launch points for other attacks, either to amplify those attacks, or merely to reflect them and thus hide the real launch points. This is a far-away problem experienced by other people, and is hardly ever cause for local alarm.

What, then?

If the US Government, and other governments, and large to medium enterprise everywhere, wants to make their world safer, they can only do so by making the whole world safer. Just as they are "too big to fail", so too they share other fate. To make any difference in their own measured outcomes, they will simply have to invest in things which will, as a side effect, make a huge difference to all of us. They will have to change the way technology is built, bought, and managed. To do this they will have to change they way they buy it and manage it. Examples:

1. Do not buy network connectivity from any IP transit provider who does not practice Source Address Validation on or before July 1 2016 on all of its customer connections except where a specific exemption is warranted by customer need and configuration. Also require that they disconnect any and all IP peering connections, no later than July 1 2017, where the "peer" network does not also practice Source Address Validation on all of its own customer connections.
2. Do not buy equipment or services that have not been professionally "red teamed" by the maker of said services. Asking your supplier for their privacy and security policy is fine, but it's time to ask them for the certificate they got from some public accounting firm who hired the best "good hackers" in the business and were unable to break into the product or service (or back office for such products or services) you're thinking of buying.
3. Do not install hardware or software that lacks either a monitoring and patching plan, or a specific expiration date. You'll also have to ask for a complete list of internal components, and an extended warranty period during which the supplier is required to notify you of vulnerabilities and provide you with a patch. If the warranty expires or the company goes out of business, you have to immediately remove and replace the product or service that just became your worst nightmare: "zombie technology".

There's more. Much more. Security isn't about the strength of the walls and locks, but rather their fitness for the attacks you should know to expect, and their usability and auditability and maintainability by your non-wizard staff. Sadly, there isn't a lot of money to be made telling people what to avoid, and it's hard to build a resume by listing all the things you refused to do because they would not have been prudent. In other words, real security isn't sexy, so we're by and large going to do something else. (Now you know what my customers mean when they ask me, "Paul I want to buy a hole, why are you trying to sell me a shovel?")

Conclusion

Katherine Archuleta should not have had to resign, because she should not have had to be an expert on "cyber" security, and also because she had a reasonable expectation that somebody, somewhere, knew how completely and ruinously bad all of the IT (Information Technology) in the world was, and would have told her that there was no safety anywhere except on paper, in filing cabinets, guarded by the U.S. Military.

Nothing that happened at OPM, or failed to happen at OPM, was the fault of its leadership team. Rather, there are systemic defects in the way we build, and buy, and manage, "cyber" technology. More talent would change nothing. We're in a hole, here, folks. The first thing we should do is: stop digging.