Containerisation is a solution to isolation, not to security.
Certainly, running application in containers isolate it in different kernel namespaces (eg pid, user etc). But is that sufficient from security point of view?
Have you ever thought that, since these containers use the same kernel as of host, can the application control (maliciously) the host (by its kernel) itself?
Yes you got it right. Containerisation is not sufficient. And indeed containerisation is a solution to isolation and not to security.
So what else you need?
Let's try to see what malicious activities (from the compromised process) we want to prevent. Let's list them out.
3 major security concerns on process
Kernel vulnerbility
Newbies might think, kernel is bug free. Not at all. So yeah, we want to ensure that even if kernel has some vulnerbility, we shouldn't make the attacker successful. We want to ensure what system calls a process can make.
File system accessibilty
Say even if you use container, just because process resides in one of mnt (mount) namespaces doesn't mean that it will secure the file system. Just because that done with chroot, it doesn't considered as solution to security. A privileged program can often escape a chroot. And also don't forget to think about corner cases when multiple process have shared mounts for some reason.
Compromised process with privilages
Say you need to a process with some administrative privilages, (may be root). You need to be very sure that the process should just do, what its asked to do. It need to have a boundary of permissions
The solutions to these
Seccomp (secure computing)
Its a linux feature that program to setup system call filters. These specify what system calls are permitted, and also which arguments as well. It sort of very low level filters, that used to require sometime application changes, but now we have many lib (libseccomp) and wrapper tool (sandboxify and libsandbox.so) that requires no code changes at all, around it.
It uses BPF (Berkeley Packet Filter) and eBPF. These are often called mode 1 (strict) and mode 2 (extended BPF) respectively.
This post gives really nice explainations and examples around it. https://blog.cloudflare.com/sandboxing-in-linux-with-zero-lines-of-code
On side note, on OpenBSD you would need to use "pledge" instead of seccomp.
AppArmor
AppArmor is a Mandatory Access Control framework that act as LSM (Linux security module). In simple words, it whitelists or blacklists access to different subjects eg file, path etc. Good example would be that AppArmor would allow process to read `/etc/passwd` but not `/etc/shadow`.
The policies can also be used to restrict capabilities, eg to limit access to network etc.
Read more by googling it and here https://apparmor.wiki.kernel.org/
Linux Capabilities
With using this, privileged process revokes a subset of the privileges it is endowed with. Say if a process need to read some root only accessible file system, it doesn't necessarily need to connect to network. The problem however is, as of today, its not highly fine grained. But it provides a very very simple way to reduce the damage, a compromised process can do.
So we can run the process as root and yet we can restrict what root user can do. Sound interesting? Read more here https://man7.org/linux/man-pages/man7/capabilities.7.html
Why shouldn't i use VM instead?
Yes, if you want highest level of kernel isolation, then you may go for VM instead of container. But that doesn't mean you are 100% secured. Remember that if OS kernel can have bugs, virtualization layers can also have. Also with VM you will have different other issues, like high resource consumption, slow boot, hence scalability will be impacted.
Good practices
One should use cover practices to address of of above concerns. And there are palenty of lib and solutions around it. And almost all does not requires application code to change. So why not.
And yeah, addressing those all won't impact much on your application performance. It's already really lightweight.
Use links above to find how to address those. And you may read this as well https://security.stackexchange.com/questions/168452/how-is-sandboxing-implemented