Containing Containers: They do overflow

Containers

With the evolution and advancements in technology, the importance of every functionality and User-Experience is of prime relevance. Aligning to this, current development methodologies have transformed from production of a single thick/giant software to “Lego” type micro-services and inter-communicating service-meshes. The approach has enabled activations of specialized teams to focus on their expertise areas. At the same time the upgrades and activation for every Lego piece can be done in its own atomicity. With all of these going and growing around, the area under the shadow (darker side) also keeps growing. With rapid increase of deployment needs of micro-services and service-mesh, the focus has primarily been on implementing and activating containers, overlooking much needed security aspects. Let's examine the popular assumption "containers are lightweight and isolated".

Blind Spot

As the adoption of containerisation keeps moving anti-gravity, so does the security incidents and exploit reporting. As much as 60% organisations hosting containers, encountered security incidents in 2018. Another report by SNYK mentioned top 10 docker containers were uncovered with 8000 exploitable vulnerability paths. Another prophecy - Container Security Incidents to Rise in 2019 as Companies Knowingly Deploy Vulnerable Containers by Security Boulevard..

Dockyard

This started with launch of docker - widely useable runC based linux container - in year 2013 and since then 20% of organizations on cloud hosting adopted docker as a container solution. This trend had been catching up even faster, and post 2017 adoption trend shows a steeper growth - projections hovering to 50% or more by year 2020.

In the midst of all these happenings, incidents like Docker snafu surfaced which led to millions of end-users download an incomplete Java runtime patch (in this case vulnerable). This development was supposed to fix an earlier uncovered security issue, though it was due to an innocent tagging problem, but the impact was no sane thing.

Needle in the haystack

Now the question is where to start and how to address this? Do we take a U-turn Did the technologist never saw this coming all the way?

Really, I don't think a U- turn is ever a solution. In further discussions and conversations with developers & implementing Architects, I found this to be a problem surfacing on places where a wrong perception prevails. The biggest misconception that "containers will have their own isolation and run alone to optimally utilize resources". The other side of container capabilities of being able to creep out and do unwanted has not been believed/explored.

And yes, Containers can go beyond their boundaries and expose things more than the developers wanted them to. So, the solution is to Contain, Insulate and Guard the containers from developments till this execution stages.

Apart from runtime attacks, observation is these issues creeps in mostly from developers’ machines during adoption/acquisition of re-useable container images, and/or as a build output with sensitive information getting packaged in final image. Last but not the least, from the usage of third-parties which could be vulnerable or from poisoned repositories.

Nip it in the bud

With the focus to address the above stated and prevent occurrence of security exploits, static image scanners must be put in place, this will help ensure the usage of right images. Secondarily, all the software development should undergo security scan to uncover any unsecure development practice and stop any vulnerable releases. On the build process there should be integration and static scanning of the output image to ensure the image doesn't contain any sensitive information and everything is safe to run in Production. Last but most important is to run, the Production environment guarded by Container Security Runtime monitoring agents, which can pre-empt any suspicious container behaviour flag and alert the squad to action.

Don't let it ripen, even if it leaked the checks

Proactive Security Implementation to prevent/predict attacks and exploits are a regular practice while hosting production systems. There are couple of good providers covering similar risks, available for containers as well. The runtime container monitoring agents like Twistlock, Aqua and coreos agents do this job pretty well. Its recommended to pump these agent logs into log aggregation & learning systems to derive trends that indicate the possibility of an intrusion and/or unusual activity; hence it can be contained.

So To Conclude

Finally putting all the above together, there are defined best practices, tools available for static scanning and security agents for container runtime monitoring. With the help of all three above, the development to production process cycle can be re-orchestrated such that the process includes these tools. This in turn enables developing, building and hosting secure container-based solutions.

Final Thoughts

I hope this write-up was successful in creating some awareness around a serious and growing concern in the containerisation solutions and hosting space. I'd be glad to know if there are learnings / stories with you which you can share.

Digital Transformation banks on

- Technology -

Adopt and Adhere to secure Practices and Lead the Transformation

References

Dzone. "Docker Snafu Leads to Millions of Downloads of Vulnerable JDK", https://dzone.com/articles/docker-snafu-vulnerable-jdk

Tripwire. "60% of Organizations Suffered a Container Security", www.tripwire.com/state-of-security/devops/organizations-container-security-incident/

Snyk. "Docker Image Security Best Practices", https://snyk.io/blog/10-docker-image-security-best-practices/

Security Boulevard, "Container Security Incidents to Rise in 2019 as Companies Knowingly Deploy Vulnerable Containers", https://securityboulevard.com/2019/01/container-security-incidents-to-rise-in-2019-as-companies-knowingly-deploy-vulnerable-containers/