Responding to Aaron B. Comments on "Making Security in a Software Factory"

Thanks to Aaron B. for a lengthy analysis of my article on "Making Security in a Software Factory." I really appreciate the thoughtful comments. We seem to have failed to connect on a number of ideas, and I take the blame for not having explained them better.

There are as many ways to do security as there are organizations. Perhaps, in your organization, it works best to keep developers and security in separate silos. And perhaps they are achieving the outcomes that you want. But my experience is that in almost every organization, there is a massive gulf between what security is expected by customers, employees, board of directors, and the public and what is actually being produced. It's often difficult even to *find* what security is actually being produced. And that's the problem I'm focused on.

-------------------------------------------------------------------------

These comments probably won't make sense unless you read Aaron's comments. I listed the page number so you can find the comments I'm responding to.

P1. I'm not from the development community. So I'm not sure how to respond to your assertion that I'm representing the "developer view." My bias against how security is done today comes after spending 30 years in the field.

P2. It's not whether but *how* the software factory must produce security. Putting aside the current results, there simply aren't anywhere near enough security specialists to handle all the work.

P3. I don't think it's arrogance to see the stunning similarity of the problems that software used to have and the problems in cybersecurity. Following the path DevOps blazed is obvious to me.

P4. Yes, I leveled some harsh criticism towards my own industry. I think it's important to take a hard look at what's really working if you want to improve. FTR, I have *massive* respect for the *people* working in cybersecurity and development. Happy to provide additional backup for any of my critiques.

P5. I think perhaps you missed the idea that teams should tackle a single risk at a time across the entire SDLC. See DevOps concept of "flow." I'm talking about *repackaging* the work and turning the output into something useful. But it doesn't mean there's no relationship between what I'm suggesting and traditional activities. Mostly it's massive streamlining and repackaging.

P5. I have no idea why you think a security story is "limited." Anything you consider a threat should be in your story and addressed with a defense strategy, defenses, and evidence. In fact, I talk about previously unknown threats in C8. It might be worth mentioning that taking steps to prepare for unknown threats -- even if you might miss -- is something you should consider as part of your defense strategy in C5.

P5. Is your objection to “outcome based” related to outcome based contracts (OBC) for security? That is not at all what I mean. I’ll have to make clear that I only care about the *actual* security outcomes (what you might call “good sausage”) – have I identified the right threats, put in the right defenses, and proven that they’re effective. I don’t care how you get *those* outcomes. 

P6. I fundamentally disagree with the idea that you should try to work on *all* possible threats simultaneously. I will make sure the front and back doors are locked before I seal the chimney.

P6. I’m not suggesting that all security activities be done by development. Throughout the document there are discussions of where security expertise is recommended. But I'm quite sure that we don't have enough security experts to do all the security work. So I highly recommend using the "big machinery" of software development to "build security." Most of the work can be packaged for the factory just fine.

P7. (C1) Your complaint about C1 seems to be that it doesn’t ensure that defenses are used properly. That is what C2 is all about.

P8. (C3) Traditional IV&V for security is a joke. C3 is about verifying against the security story which clearly identifies the threat, defense strategy, specific defenses, and how they are to be used. The outcome is strong evidence (hopefully automated) that the threat has been properly controlled, linked into the security story forever.

P8. (C4) Outsourcing supply chain security to a third party? I think I'm going to need a little more explanation of how this helps. I'm not particularly a fan of optimizing for "independence" or "objectivity." My experience is that when security is done in the sunshine, where you have to show all your work, that the kind of cheating you're suggesting doesn't happen.

P8. (C5) Continuing to reference “TRUE Cybersecurity” while talking about a bunch of requirements documents that have had spectacularly little influence on anything is making my point.

P9. (C6) I need to be clearer in this section. We need to bridge the gap between broad policies at the enterprise level down to specific “as code” enforcement that translates those policies down into reality.

P9. You can’t improve what you can’t measure. Your chart is literally a metric.

P9. I’m not sure we do need to focus on the unknowns. The overwhelming majority of breaches are due to known vulnerabilities. It's really all about basic blocking and tackling. Should I stand up an advanced research center with world leading security experts like Google? Well, you could. But most companies would be compromising the easy stuff to focus on the amazingly unlikely.

P9. The solution for development and operations was not to shore up the silos and work on how they “behave, communicate, and support each other.” Read the Phoenix Project and Unicorn Project and then let’s talk again.

P10. Every company, including the ones you think are following all those regulations, are wide open to a SolarWinds attack on the software factory. If they had added “software factory compromise” to their security story, they might have a defense strategy, etc…

P11. I’m disappointed at this point that you are quoting statutes at me and suggesting that the solution is for development teams to know what they’re about. I'm not sure what I can do to be clearer that this has not, does not, and will never work.

P11. Thank you for the reference to Infection Monkey. I need to add chaos engineering to C3. Take a look at Chaos Monkey and what you can learn by really breaking things.

P12. Another list of traditional security practices. Sorry, but I’m not changing my opinion. It’s easy to show these things just don’t work. I'd be more convinced if you pointed out something of value that those activities produce that's *not* produced better and faster by the approach I've described.

P12. We had a disconnect here. Security stories don’t replace assessments. Assessments feed evidence into security stories so that you have a complete (and continuous) line of sight from threats to assurance.

P13. I couldn’t disagree more about automation. I know most security folks are scared of speed. But your instincts are leading you the wrong direction here. DevOps has produced amazing increases in both velocity *and* quality. The same quality benefits yielded by DevOps can be shared by security. In fact, security can take advantage of much of the DevOps infrastructure instead of standing up their own. Check out the “three ways” of DevOps and the benefits of tight feedback loops.

P13. The vast majority of cybersecurity documentation is useless. If you get rid of the 99% that doesn’t actually matter, cut up the rest, and link all the pieces together into a compelling argument, guess what you have…. A complete security story.

P14. I like the idea of automating security documentation. But I suspect you mean managing a bunch of documents. I mean literally turning it into code that runs continuously. This is called “security as code” in the DevSecOps community.

P14. I think we’ll just have to agree to disagree about whether developers and security should stay a siloed teams. You say combining them is an excuse for not solving organizational problems. I say keeping them separate *is* the organizational problem.