Tracking bandwidth usage per process with eBPF and CGroups

Finding which process are sending receiving data with a given host appears to be a common need and here are some of the reasons I have seen for it:

• Preventing data usage charges on cellular networks by finding which application makes requests.
• Detecting malicious applications by tracking whether the data usage of applications aligns with their stated purpose.
• Debugging applications or the Linux kernel itself, by measuring the actual data usage against the expected amount.

So, for example, you may see on your firewall that a particular host downloads a lot of data from Hugging Face. It's normal for it to download some data from Hugging Face, but the current level is saturating your bandwidth.

You could setup a HTTP proxy to cache the downloads, but it's still natural to want to find which application is using the bandwidth and see if it can be avoided altogether.

By default the kernel has no user friendly interface to present this information, the closest is a trace point which shows how much data is sent or received on a per process basis. If you want to know where the data is coming from or going to then this needs to be correlated with other sources.

Enter pktstat-bpf

The kernel can be extended with eBPF byte code, this can be used to hook into various points within the kernel and provide the necessary functionality. That is you can register a small eBPF program with a particular hook point, so that when the hook point is hit, the eBPF program runs. There are multiple types of eBPF program with different restrictions and capabilities.

The eBPF based utility pktstat-bpf tracks the bandwidth usage of source-destination IP address pairs using a number of techniques. It can also track the process when using kprobes or the newly added (by me) CGroups hook points.

So far so good, but now let's get further into the technical details, which in the interest of keeping the article short I won't try to expand on all the jargon.

As I have mentioned previously in my newsletter there don't appear to be any convenient hook points to measure both the bandwidth and record the responsible process at the same time.

That is, there don't appear to be any stable hook points that have access to the packet data and only execute in the context of the sending or receiving process. To overcome this pktstat-bpf's author, Dinko, used a kprobe based approach that attaches probes to several internal kernel functions.

This provides the necessary information, however the internal functions that kprobes hook can change or disappear between kernel version. In fact just recompiling the kernel with different compiler settings can cause some function call sites to be inlined and a kprobe to disappear. Kprobes are also more challenging for security and can't ever be attached by a non-root user. Due to this I decided to investigate some stable hook points, the most fruitful of which so far have been the CGroup hook points.

Taking advantage of CGroups

CGroups can be used to organise processes into a hierarchy of groups and place various controls on them. Including memory limits, CPU limits, I/O limits and a whole host of other things. A big advantage of CGroups is that you can prevent one type of process from using up all of your system's resources.

It's also possible to attach eBPF programs to CGroups which hook into a number of different actions.

In particular there are CGroup hook points for sending and receiving packets and for creating sockets. Unfortunately the hook points for sending and receiving packets are not inside process context. However the hook point for socket creation is always executed inside the relevant proc's task. So what we can do is track which process creates a socket, then when we see a packet being sent or received on a particular socket, we can look up which process created that socket.

This has the disadvantage that sockets which were created before we started pktstat-bpf won't have any process associated with them. Also there is some cost associated with tracking which processes created a socket. However if we start pktstat-bpf before the workload then we'll get the full information. The only other issue is that a process may create a socket then transfer it to a different process. Commonly this happens when a process forks and its children inherit its sockets. However processes can also send sockets (actually file descriptors) using UNIX control messages.

On the plus side the CGroup hook points are stable and provide almost the same level of information as the kprobes. Also you can restrict monitoring to just one group of processes which can avoid spending resources on monitoring irrelevant traffic. On the other hand if you want to track all processes then it's simply a case of attaching to the root CGroup.

There are more CGroup hook points that my be useful for getting the process as well as other types of hook point that I haven't investigated. In conclusion there are a number of options for tracking process bandwidth usage in eBPF and pktstat-bpf is a good tool try this out with.