The DevSecOps Paradox

Hi all,

Today I am bringing up a discussion that I have already taken many times, and it is time for a dedicated article.

DevSecOps is a totally overused buzzword nowadays, defined as the solution for everything while enabling agile product teams to drive innovation and on the other hand, it includes security in the software lifecycle and, therefore, mitigates all security concerns that arrive through agility.

But at the same time, driven by the "former" DevOps shift, the following paradigm became popular and is used in product-oriented companies:

You build it, you run it.

And here is where these two typically collide because some people understand now that the product teams will take care of everything and are also responsible for everything.

Spoiler: You can expect that from your teams, but this will definitely not going happen.

The Scaling Issue

Let us start simple and ask yourself the following question:

Okay, now I have dozens or even hundreds of product teams - will all be able to provide the same maturity level of security?

I mean - just think about it from a skill perspective. You want to have lean and agile product teams, consisting of five to nine people, and rarely more; And now you expect to have the same but high security maturity in all of them while everyone owns their whole environment?

What to Secure

Let us step one step back and have a simplified view of what needs to be secured. Let us say your product teams are running on Cloud and Kubernetes. Then you will need to secure/harden your Infrastructure and Platform properly. This results in a proper setup of your project/subscription and a proper configuration of your identities, network, and the Cloud resources, e.g., the Kubernetes cluster. In the visualization below, these will be the underlining areas of the Cloud Solution Provider (CSP) and the Cloud Resources.

A rule of thumb here - the more you move towards the direction of FaaS, the less you need to care, but you will ALWAYS need to care.

Next up is the Pipeline itself. You definitely want to avoid access to it or the used credentials from any "unexpected persons."

This is directly followed by the Outcome of the Pipeline till its Deployment. The outcome can be a binary, a web page, an API, or something else. But no matter what it is, you want to see secure code, quality gates, and respected best practices in between. And then, you want to securely deploy it somewhere and protect it, e.g., against DDoS attacks.

The last phase is when you want to ensure that nothing "strange" happens while your production is up and running. Therefore, you want to have Security Monitoring and Incident Response established and properly defined.

Okay - after knowing all this, do you still think that your product teams will be able to accomplish all of this properly? All of them?

// And this was - only - the Technical Security part.

Shift left

Typically, the answer to this is

But what does it mean? It is extremely easy to understand when you visualize it.

You want to shift all possible automated security and configuration tests to the left / as early as possible in your pipeline to include them directly from the beginning before deployment.

Also know, the later in the pipeline you identify security issues (the more on the right) the more concerning they will be for you, being the worst scenario receiving a finding from the Incident Response team after they identified your product had been the entry door for a Ransomware attack you were just hit by.

SSDLC

In the following picture, you can have a look at the most important key pieces of a so-called Secure Software Development Life Cycle.

Tools - the solution for everything?

There is no doubt that integrated security scans / linters / config scans in the SSDLC are helpful. Still, in many conversations, I got many times that these would be the solution for establishing a full-proof security no-brainer.

And here, I need to be very honest - the tools are helpful to mitigate the most typical issues, which are done repeatedly and are then codified as policies for being captured in the SDLC. They are just as good as the policies of the tool, but they rarely defend you against stupidity or general issues on your platform and misconfiguration of the Cloud Resources.

In short, they tend to raise a wrong feeling of established security, though leaving back a lot of blind spots.

If it were just that easy to secure things, we would not have so many data breaches nowadays, impacting almost every company on this planet. Don′t be so naive to think that using now some security scans will secure your whole environment.

I mean - you just scaled your agility and increased the code's change exponentially and recently grew your overall attack surface by adding one or even more CSPs using many different Cloud resources, which all needs to be properly taken care of.

Also, I see many companies focusing primarily on Static Application Security Testing (SAST), e.g., Sonarqube Code Testing, and ignoring all the other different areas where to possibly identify weaknesses like misconfiguration or security issues, e.g., with the platform or the deployment itself, or lacking further insights from the Interactive Application Security Testing (IAST) and Run-time Application Security Protection (RASP). (context here)

It is like saying, I will just have a continuous look at the paint of my car but I will never review the engine or the tires.

But - does this mean DevSecOps is wrong?

No - it is definitely the right direction, but you will not be able to create a fully secure environment by just using some security tests in your pipeline. You need to do much more.

In my last article, I pitched the Governance lifecycle and "shift your Governance left" to catch Governance issues (including technical security) more proactively than reactively.

But even in a high-maturity environment, you will never catch all the things.

Therefore, you need to have centralized Continuous Monitoring well-implemented to identify them as quickly as possible, and you need to transition the more reactive actions into proactive or implemented ones over time.

You want to invest your time in the challenging and special things and not in those that occur over and over again.

You should try increasing the maturity of your whole company by transition finding captures to the area where things have not yet been deployed, e.g., by having policies as code implemented to prevent misconfiguration globally proactively before creation or by having basic security configs implemented in templates that are used by all of your product teams.

Economy of Scale

And here is where the economies of scale help. There will be some basics or requirements that many or even all product teams need to take care of. You need to define the things which don′t make sense to give to the product teams because these things require special knowledge and/or can be set up centrally. You don′t need to let your product teams secure all their Kubernetes clusters when you can do this internally for all teams.

Another way is to enable all your teams in parallel with training sessions or provide guides for every team.

Keep also in mind that you need to take security as a risk-based approach. Every product team should normally do a threat modeling exercise to understand which pieces in the product should be properly hardened and monitored, as these are known to be most sensitive or attacked most of the time.

Responsibilities

The tough part will be the responsibilities cut. Typically, it is great to have a dedicated platform team to take over common functionality and catch the most severe config issues globally, e.g., using policy as code.

On the other hand, you want to provide your product teams as much autonomy and agility as possible. Here is where you should think about different team types, e.g., "platform teams" to provide the platform and "enablement teams" that positively impact many product teams, e.g., with training and guides.

In the picture below you can see different operating models of companies. On the left (positioning doesn't matter) you can see the typical central-based approach primarily focusing on platform teams and on the right more agile and decentral team setup. The more you do centrally the less agility and freedom are available for the agile product teams. The more freedom and responsibility you shift to your agile product teams, the less control you have initially. This can be seen on the right side, demonstrated as decentrally.

Typical agile product team patterns are DevOps and DevSecOps, as stated here in the document, but also similar methodologies like SRE which primarily focuses on availability and new feature creation.

In the end, it is a question of what and how much you want to do centrally and how much you want to allow to be done decentrally, while still keeping your environment under control.

Typically the best answer to this is a mixture of both, as you cannot just say that the product teams will solve everything.

This allows me to transition to a great model, which I would call the next maturity level for a product-based company.

The next maturity of a product-based Company

You should quickly understand that there are different types of teams, and you should define the best model for your company. Depending on the vertical your company is in and on how your people's culture looks like, the perfect model for your company can be totally different compared to other companies.

In the following model called "Team Topologies," the stream-aligned teams can be translated into the product teams creating a product with business-relevant features.

The platform team set up the platform and integrate commonly used functionality and requirements in it - e.g., this can be the team maintaining a Private Cloud or a Public Cloud - for example, a CSP like the Google Cloud Platform.

The enabling teams can be teams that try to identify patterns and scale the impact. This can be to provide dedicated enablement from this team directly, e.g., the Cybersecurity team, or gather highly impactful ideas from stream-aligned teams and scale them to many other stream-aligned teams, e.g., identify new solution architectures and sharing them widely in the whole company.

And the complicated subsystem teams provide something as a service, e.g., compliance checks at some point in time for the stream-aligned teams.

Furthermore, you want to have relevant data and insights on how all your teams perform. So, it would be a fantastic idea to have dedicated KPIs and insights to validate if the enablement teams have the right impact and which topics you need to address centrally because these are the topics the majority of your product teams are struggling with.

Sounds trivial? In a large company, it is definitely not. ;-)

____________________________________

Okay - I will just stop here. The team setup is easily a single article in itself.

Let us do a recap to understand the quintessence of the DevSecOps Paradoxon.

Recap:

• DevSecOps is great, but just the start.
• "You run it, and you own it" does not transition the complete responsibility to the product teams.
• Shifting left only will not solve all of your security issues.

As a result, you will need to do much more like:

• Working with data and insights.
• Changing the team setup by creating highly specialized enablement and platform teams.
• Making use of Economy of Scale.
• Shifting your complete Governance lifecycle model left.

I hope you liked the article and stay tuned for future ones. Don′t forget to check the follow-up materials!

Best,

David

________________________

Follow-up materials:

• Team Topologies described including Security integration
• Fantastic session, using KPIs to demonstrate enablement value of Security trainings.