Find me if you can ... Security in DevOps

I'm writing this post following several discussions, readings, trainings and certifications in the field of DevSecOps.

I confess that a few years ago, I was essentially considering static (SAST) and dynamic (DAST) Application Security Test (code analysis) behind the term DevSecOps .

Unfortunately, today it's impossible to stay with that idea, given that the DevOps approach is at the heart of the digital transformation, and with it the creation of new businesses and the ongoing releases of new features in a complex and dynamic technical environment ( including cloud).

Let's start with a few definitions to set the scene :

What is DevOps ?

DevOps is a software development approach that aims to improve collaboration between development (Dev) and operations (Ops) teams within an organization. The main goal of DevOps is to accelerate software delivery by automating development, testing, and deployment processes while fostering effective communication and collaboration between development and operations teams.

Key principles of DevOps include:

• Automation: Automating manual tasks such as compilation, testing, and deployment reduces errors, speeds up processes, and ensures better consistency.
• Collaboration: DevOps promotes communication and collaboration between development and operations teams, as well as other stakeholders, to more effectively solve problems and meet user needs.
• Continuous Integration (CI): CI involves frequently integrating source code into a shared repository, followed by automated tests to quickly detect errors.
• Continuous Delivery (CD): CD extends CI by also automating the delivery of software versions, enabling the rapid availability of stable versions for users through manual or automated production deployment (CD also stands for Continuous Deployment).
• Monitoring and Feedback: Continuous monitoring of software performance in production helps quickly identify issues and gather data for product improvement.
• Infrastructure as Code (IaC): IaC involves managing IT infrastructure as code, facilitating automated deployment and management of resources.

By adopting DevOps, organizations aim to enhance the flexibility, responsiveness, and quality of their software development processes, enabling them to deliver products more quickly while reducing risks and costs.

What are the main softwares for DevOps ?

Everyone has seen this image of the Infinity Loop ... the first time I think I saw it was in Ulysse 31 ....

This is a continuous evolution cycle. The need is specified, then it is developed, tested, and validated in a version before being deployed on the information system, all while being monitored to prepare for the application's evolution.

There are numerous tools and platforms available for implementing DevOps practices, and it can be challenging to distinguish major vendors because most DevOps solutions result from open-source efforts or contributions from various companies.

In a simplification attempt, which DevOps purists may contest, I would distinguish six major categories of vendors and some of the leaders :

1. Source Code Repository / Version Control System (VCS) - The three leaders are GitHub, GitLab and Bitbucket derived from the open-source Git. They manage source code . Most companies turn to cloud-based solutions when working with significant remote or distributed teams (for more details)
2. CI/CD Integration - There are several opensource solutions , such as Jenkins and Gitlab CI/CD that aims to streamline and automate the entire process of building, testing, and deploying software. 6 Open Source CI/CD Tools in 2024 | Estuary
3. Container - There is The Open Container Initiative, an open governance structure for the express purpose of creating open industry standards around container formats and runtimes. Docker, mixing open source and enterprise components is leading this space. It provides thousands of pre-built images to easily deploy the produced code in a microservices and elasticity logic. (Open Container Initiative - Open Container Initiative (opencontainers.org). Kubernetes is the most famous open source container orchestration engine for automating deployment, scaling, and management of containerized applications (also known as K8s) (Kubernetes Documentation | Kubernetes) .
4. IaC - Infrastructure as Code (IaC) is an approach to manage IT infrastructure using code, usually in text files, to automate deployment, configuration, and management of IT resources. The solution interprets these needs, sequences the necessary steps for the deployed infrastructure to be operational, and for the code to work. There are several Open-source solutions such Ansible, Chef and Terraform. HashiCorp is the editor behind Terraform, which is a widely used open-source solution for IaC. There are two methods, Declarative and Imperative, used by IaC editors, each with its advantages and disadvantages (for more details).

These editors have tended to expand the features they provide to cover other services within the DevOps practice and even add security services.

Microsoft Azure, Amazon AWS, and Google GCP cloud providers also offer their solutions, which are particularly tailored to each of their environments.

It is important to note that the choice of DevOps tools depends on the specific needs of each organization, its technological infrastructure, and its development pipeline. Many companies opt for combinations of tools that best meet their requirements.

?What are the main Cybersecurity Vulnerabilities in DevOps ?

Unfortunately, they will appear throughout the DevOps process.

In other word, vulnerabilities can be in the code being developed, in the application dependencies or operating system (OS) packages, through credentials in plaintext , and in all the misconfigurations that may occur throughout the entire process, especially when the application is running.

Common vulnerabilities include missing data encryption, buffer overflows, missing authentication for critical functions and insecure interactions between software components.

Exposed secret is a risk at the different stages of the Software Development Lifecycle (SDLC) in some many components.

App code development relies heavily on open source software which is frequently built by combining multiple packages. Note on the graphic below the high percentage of codebases that contain a vulnerability.

Just as a reminder, three well-known exploits that were caused by vulnerable open source software and impacted significantly many organizations.

• Mirai Botnet - Mirai (malware) - Wikipedia
• Log4J - Apache Log4j Vulnerability Guidance | CISA
• HeartBleed - Heartbleed Bug

The survey below illustrates the challenges organization have to deal with and the risks they want to adress through DevSecOps .

As CI/CD environments, processes, and systems are the beating heart of any modern software organization, OWASP, well-known for the “Top 10 Web Application Security Risks” framework, released in november 2022, the “Top 10 CI/CD Security Risks”framework :

• CICD-SEC-1: Insufficient Flow Control Mechanisms
• CICD-SEC-2: Inadequate Identity and Access Management
• CICD-SEC-3: Dependency Chain Abuse
• CICD-SEC-4: Poisoned Pipeline Execution (PPE)
• CICD-SEC-5: Insufficient PBAC (Pipeline-Based Access Controls)
• CICD-SEC-6: Insufficient Credential Hygiene
• CICD-SEC-7: Insecure System Configuration
• CICD-SEC-8: Ungoverned Usage of Third-Party Services
• CICD-SEC-9: Improper Artifact Integrity Validation
• CICD-SEC-10: Insufficient Logging and Visibility

(for more details about theses risks and guidance for mitigating them).

It's a losing battle for cybersecurity if it's not fully integrated into the DevOps process, so that it can reduce risk while not creating too much friction.

What does that mean ?

I would use my favorite triad to detail them, namely PEOPLE, PROCESS, and TECHNOLOGY.

PEOPLE

Vulnerability management and remediation cannot be centralized within security teams because the ratio between developers and security engineers is too imbalanced (beyond 1 to 100 in some organizations). To a lesser extent, the same issue arises with teams responsible for hybrid infrastructure (on-premises and cloud) during application deployment and operation.

Therefore, all these stakeholders must be involved, even responsible for securing a part of the process. They need to be made aware of risks, understand how they work, and be provided with solutions to address these risks as easily and quickly as possible.

PROCESS

DevSecOps is an extension of the DevOps approach that integrates security (Sec) throughout the software development lifecycle. The main goal of DevSecOps is to ensure that security is considered from the beginning of the development process, rather than added later when it becomes complicated, costly, and obstructive.

Key processes in the security part of DevSecOps include:

• Security Automation : Security practices such as static analysis, dynamic analysis, vulnerability management, and threat monitoring are automated and integrated into development and CI/CD pipelines.
• Collaboration between teams : Development, operations, and security teams work together from the early stages of the development process to identify, understand, and resolve security issues.
• Education and awareness : Team members are educated about security and receive training on security best practices to better understand the challenges and contribute to the security of the application.
• Vulnerability management : Vulnerabilities are identified, tracked, and proactively managed throughout the software lifecycle, and patches are applied promptly.
• Continuous security testing : Automated security tests are regularly conducted to detect vulnerabilities and weaknesses in the application, from design to operation in production.
• Compliance and audits : Development processes are designed to meet regulatory compliance requirements, and security audits are conducted to verify this alignment

TECHNOLOGY

The list of security controls is constantly increasing. These controls are carried out throughout the development cycle.

The diagram below is a good summary of the controls, and the associated steps.

In terms of the categories defined by the Gartner around the security within DevOps , there are at least :

• Application Security Testing - for solutions that perform static and dynamic code analysis (SAST/DAST), as well as Software composition analysis (SCA), (Best Application Security Testing Reviews 2024 | Gartner Peer Insights)
• Cloud-native application protection platforms (CNAPPs) are a unified and tightly integrated set of security and compliance capabilities designed to secure and protect cloud-native applications across development and production. CNAPPs consolidate a large number of previously siloed capabilities, including container scanning, cloud security posture management, infrastructure as code scanning, cloud infrastructure entitlement management, runtime cloud workload protection and runtime vulnerability/configuration scanning. (Best Cloud-Native Application Protection Platforms Reviews 2024 | Gartner Peer Insights)

There has clearly been a convergence over the past few years, with the demise of the CWPP and CSPM, gradually replaced by the CNAPP, as well as the AST players addressing container and API security, areas also covered by several CNAPP vendors.

What is included today as part of the DevSecOps may evolve as the market is highly dynamic and continues to experience rapid evolution in response to changing application architectures and enabling technologies.

So, where do we start ?

Like any self-respecting former consultant, I would say it depends !

Several factors need to be considered, namely :

• The Technical debt,
• The level of outsourcing in the DevOps process,
• The level of digitization of key processes,
• The fragmentation of the information system,
• The Industry or Legal obligations,
• The Risk Appetite.

One thing is sure - without an assessment of the current situation, any initiative to secure the process could be of limited effectiveness, or even counterproductive, especially if it only involves pointing out the presence of numerous vulnerabilities, implying incompetence from the DevOps teams.

Therefore, visibility would be, in my opinion, the first necessary step for any improvement in the DevSecOps process.

The second step is defined a plan agreed by the different stakeholders ( Dev, Sec and Ops)

Conclusion

Like the CI/CD process, this article will evolve to incorporate improvements or correct any potential mistakes.

Feel free to comment

Sources and additional materials :

Infrastructure as Code - comparing tools (microsoft.com)

12 Things to Get Right for Successful DevSecOps (gartner.com)

Best Cloud-Native Application Protection Platforms Reviews 2024 | Gartner Peer Insights

DevOps security risks and how to overcome them (itoutposts.com)

DevSecOps: Strategies, Organizational Benefits and Challenges | Gartner Peer Community

Outils?DevSecOps | Atlassian

The 12 best IDEs for programming | TechRepublic

What is Infrastructure as Code (IaC) Scanning? | Wiz

[MaJ] Vulnérabilité dans Apache Log4j – CERT-FR (ssi.gouv.fr)

What Is DevOps? | the agile admin