Testing the Security of a Kubernetes Cluster

Containers and Kubernetes are everywhere. The adoption of Kubernetes in production has increased to 83% to a survey by CNCF. Still, most security teams struggle to understand these modern technologies.

From an attacker's perspective, they can assess and attack Kubernetes Cluster environments to gain access to microservices, sensitive data, escaping containers, escalating to cluster privileges, and even its underlying cloud environments. Also, this article describes the offensive knowledge to build and design secured cluster environments.?

What are some of the most common concerns of people using Kubernetes?

• “Primary focus is to ensure systems and infrastructure remain “up”
• “We have backups, firewalls, and antivirus software”
• “We do annual assessments and no major security issues are discovered”
• “We use expensive ‘X’ software/tools to generate a report for management every week”
• “Our environment is too fast to keep up with security trends”
• “We use the latest images for our containers in Kubernetes”
• “Our public cloud footprint is too small to be noticed by attackers”

Why are complex systems challenging to secure?

• Weaknesses in implementation.
• Misconfigurations with the underlying transparent cloud infrastructure.
• Vulnerabilities in code running on top of the implementation.
• User privileges, boundary conflicts, and authorization issues.
• 0 days in any of the moving parts.?
• Poor/ difficult maintenance/ lack of security automation.

What gives an attacker a chance to attack?

• Kubernetes environments, multi-cloud architecture, and even multi-service infrastructure within a cloud can all become difficult to maintain from a security standpoint.
• Attackers use standard processes to detect, identify and exploit weaknesses.
• Most attackers are driven by a purpose when attempting to compromise systems-data theft for black market sell, compromise infra for resource reuse (crypto mining, malware hosting, and C&C bots), ransomware for extortion, damage reputation or bug bounty, etc.

Let’s take a look at some of the ways Kubernetes can be attacked and the ramifications of such attacks on data and the business in general.?

All attackers, irrespective of how unstructured they are, follow a set of tactics to reach their end goal. Each of these tactics (consider them as milestones) in an attack have various techniques by which the tactic is fulfilled. Depending on the technology being targeted, the tactic may differ but often align with: Initial discovery, code execution, privilege escalation, persistence (optional), evasion/evidence removal (optional), and Abuse/Impact.?

Mitre Attack & Threat Matrix:

Mitre-Attack is a globally accessible knowledge base of adversary tactics and techniques based on real-world observations. The threat matrix describes tactics, techniques and procedures (TTPs) for these technologies and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take. Microsoft created a similar threat matrix for kubernetes in 2020 and released an updated version in 2021.

The first vector, or category of attacker tactic, in the Kubernetes attack matrix is Initial Access. This tactic references the attacker’s objective of gaining access to Kubernetes clusters by either compromising a component within the Kubernetes control plane that handles cluster management or various resources within the cluster – for example, a running application component on a worker node.

(SSRF in Exchange leads to ROOT access in all instances )

Exposing a sensitive interface to the internet or within a cluster without strong authentication poses a security risk. Some popular cluster management services were not intended to be exposed to the internet, and therefore don’t require authentication by default. Thus, exposing such services to the internet allows unauthenticated access to a sensitive interface which might enable running code or deploying containers in the cluster by a malicious actor. Examples of such interfaces that were seen exploited include Apache NiFi, Kubeflow, Argo Workflows, Weave Scope, and the Kubernetes dashboard.

Once an exposed sensitive interface in Kubernetes has been identified, the first step is to take steps to secure the interface. This could include patching any vulnerabilities, implementing authentication and authorization controls, or disabling the interface if it is not needed. Additionally, it is important to monitor the exposed interface for any suspicious activity or unauthorized access attempts. By taking these steps, organizations can ensure that their Kubernetes infrastructure remains secure and protected from malicious actors.

Initial Access:

Multiple large companies have in the past exposed their kubernetes dashboard without authentication to the internet. Other sensitive interfaces that contain the ability to deploy containers or run code include Apache Nifi, Kubeflow, Argo Workflows, Weave Scope etc. can be exposed. App vulnerabilities that can be used to make network requests or read files like SSRF, XXEs, LFIs, Path Traversal, arbitrary file downloads, SQL injections or plain RCEs can be used to gain access to environment variables or files containing credentials. Stealing the config file while an ongoing attack and using cloud credentials are other popular techniques (dev machine compromise)?

Once the ability to interact with the cluster is obtained-either via an exposed dashboard, kubeconfig file taken off a developer laptop or secrets leaked via an application vulnerability, attackers will attempt to gain proper shell access or run custom code within the cluster. Attackers can also inject a sidecar container in an already running pod to hide their presence as an explicit container may be spotted by an admin.

Rarely, but not unheard of, cluster users may choose to run SSH servers within containers to provide for exec capabilities to devs without providing the kubeconfig. Attackers can target this to gain code exec capabilities.

Kubernetes Cronjob:

A CronJob runs within a Pod. By default, Kubernetes preserves the logs for terminated Pods representing the last three successful runs of a CronJob and the most recent failed Job. You can change or disable these defaults by changing the CronJob history limits.?

Why persistence?

Attackers can persist within the kubernetes infrastructure and maintain access in various different ways including but not limited to:

• Adding SSH keys to the underlying node.
• Creating jobs/cronjobs that cause a reverse shell to be connected.
• Create a shadow admin cluster role or reuse a cluster-admin role.
• Use a writable hostPath mount to add custom code to apps or config files.
• Read and reuse secrets configMaps from the cluster.
• Add a backdoor binary to a running container after gaining exec.

Some attackers like to persist in most networks for the thrill of seeing how long it takes owners to boot them out, to use the network as a means of further attack/abuse or as a symbolic trophy in their collection.

Privilege Escalation:

Escalation of privileges is a key step to gain additional access or to ensure the attacker remains hidden within the infrastructure.

Some ways by which an attacker can escalate privileges include:

• Reading secrets, tokens, or credentials from Kubernetes or application configuration.
• For managed Kubernetes on any cloud provider, a service token account is mapped to the IAM of the platform and is tied to the cluster as well. This is usually accessible as a mount point within the pod.
• Detecting which pods are running as privileged pods and then using exec to gain access to them and from there the underlying node if possible.
• Accessing the metadata endpoints to read IAM credentials or environment variables containing higher privilege credential material.?

Defense Evasion:

As privileges increase within the cluster, attackers find novel ways to remove their traces and remain undetected?

• Clearing logs by accessing the var/logs/pods directory on the underlying node.
• Clearing Kubernetes events using kubectl delete events- all
• Launch pods in reversed namespaces to match other workloads that are running
• Hide origin IP addresses using proxies, TOR etc.

Credential Access:?

Most attackers will try to identify alternate regions, resources, and networks that they can gain access to from their vantage point.

To reach protected regions, they may require credentials that would unlock doors for them. Here are some techniques that attackers use to get their hands on credentials?

• The Kubernetes secrets store.
• Using the service principal mounted within the underlying node to access Cloud resources (this allows the attacker to escape the Kubernetes environment into the managed cloud provider platform)
• Pod mounted container service account tokens.
• App creds in configMaps and pod descriptions or hard coded within app files.
• Malicious admission controllers.
• Cloud credentials to generate the kubeconfig for the cluster.

Lateral Movement:

• Lateral movement tactic is used to capture techniques that can be used to move between resources within the cluster or even use tokens to escape to the underlying cloud infrastructure.
• Accessing the internal networking of the cluster to see what other services are accessible/exploitable.
• A container that allows access to the host via a volume mount hostPath can compromise all containers, even the ones belonging to other tenants as the volume mount gives shell access to the underlying node and subsequently to all containers.
• In a somewhat even more complex approach, attackers with access to updating the ConfigMap may update CoreDNS configuration resulting in DNS hijacking to steal service account tokens.

Impact:

Impact Tactic is used to identify potential endgame goals of the attacker. Some documented techniques include:

• Data destruction by rolling back deployments and removing volumes and claims.
• Hijacking resources to perform alternate tasks. Most commonly heard technique examples include utilizing the cluster to perform cryptocurrency mining.
• Altering any state within the cluster can result in a Denial of Service. For example, rolling back a deployment, removal of secrets or altering configMaps, updating a NetworkPolicy etc. can all restrict application functionality and flow of network traffic.

Tips for cluster security:

• Keep Kubernetes updated, including core components, API server, etcd etc.
• Restrict SSH access to the nodes to only trusted IP addresses.
• Use Kubernetes Network Policies judiciously, allow only what is required with a default deny otherwise.
• Audit roles and bindings, service accounts, IAM access, look for shadow admins, use RBAC judiciously.
• Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards.
• Remove deployments that are no longer required.
• Run pods with a security context and a defined user, say no to root.
• Use namespaces to create logical segregation of resources.
• Do not store secrets in environment variables, in pod description or config Maps.
• Ensure minimum visibility from the outside.
• Take application security into consideration, and practice defense in depth.
• Use PodSecurity Admission controller by updating kube-apiserver.yaml.
• Use hardening guidelines like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations.

We hope you are now able to get an in depth insight to secure your Kubernetes cluster and what can be practices followed by a hacker. Want a handbook on end-to-end Kubernetes deployment? Read this article to know more:?www.e2enetworks.com/blog/kubernetes-deployment-handbook-configurations-checklist-error-handling-and-best-practices .

Also, are you looking for Kubernetes as a service? EKS, E2E Kubernetes Service: With the E2E Cloud platform, You can quickly launch the Kubernetes master, worker nodes and get started working with your kubernetes cluster in a minute.?

https://www.e2enetworks.com/product/eks

Connect with us for any query you may have:[email protected]