Since the advent of Docker and Kubernetes nearly a decade ago, one of the biggest drawbacks of containers has been the difficulty monitoring them. The ephemeral nature of containers, combined with their abstraction from the servers that host them, has made it difficult to collect monitoring data from containers in an efficient and effective way.
Fortunately, there's a promising, newish technology that offers a solution to the container monitoring conundrum. It's called eBPF, and there's a good chance that it will upend conventional approaches to monitoring Docker and Kubernetes workloads.
Container Monitoring: The Traditional Approach
Traditionally, developers and IT engineers who wanted to monitor containers faced a number of challenges:
- A single application could consist of dozens — and possibly even hundreds — of individual containers. Each container has to be monitored separately, which increases the work required to deploy monitoring agents and collect the necessary data from each container.
- Data stored inside containers disappears when the containers shut down, and it's often impossible to predict exactly when a container might shut down. For that reason, you can't pull monitoring data periodically. You need a way of collecting it from every container in real time.
- Because containers are abstracted from the operating system of the servers that host them, and because they may move around servers, host-based monitoring approaches don't work well. You can't easily run an agent on each server and use it to monitor all of your containers.
There are different ways to solve these challenges, but the most popular is using what has become known as the sidecar pattern for deploying container monitoring agents. Under the sidecar pattern, monitoring agents run inside special containers, which run alongside the containers they monitor. This approach is more effective than trying to deploy monitoring agents on the host. It also eliminates the need to expose monitoring data from directly within an application's logic, which would require complex changes to source code.
However, the sidecar pattern comes with a major downside: It doesn't use resources very efficiently. Having to deploy a sidecar container alongside each container that hosts your actual workload means you end up running many more containers. Since all of those additional containers require CPU and memory resources, they leave fewer resources available for your primary workloads.
A Better Approach to Container Monitoring: eBPF
eBPF offers a way to square this circle by monitoring each container without high resource consumption.
Introduced in 2015, eBPF is a Linux feature that makes it possible to run programs directly in the Linux kernel — as opposed to running them in "userland," where they don't have direct access to kernel resources.
Because they run in the kernel, eBPF programs consume minimal resources. They also have access to the data produced by any process running on the server where they operate.
If you want to monitor containers, then, you could write an eBPF program that intercepts the processes associated with each container and use it to collect monitoring data. You'd end up with a monitoring solution that is much less resource-hungry than traditional sidecar containers.
At the same time, you would not have to compromise on the amount of data you could collect for monitoring purposes. Virtually every piece of information you'd want about the status and performance of each container is available through the kernel.
Even deployment and management is simpler using an eBPF-based approach to monitoring. Instead of having to deploy and orchestrate a bunch of sidecar containers, you can simply run an eBPF program on each node in your cluster.
The State of eBPF for Container Monitoring
If eBPF is such a great solution for container monitoring, why isn't everyone already using it?
The probable reason is that eBPF remains relatively new, and it wasn't mature at the time that containers entered into widespread use circa five years ago. For that reason, most existing container monitoring tools were designed to use a sidecar pattern, instead of taking advantage of eBPF.
But this is already changing. Tools like Cilium are already using eBPF to improve efficiency and increase visibility. Plenty of observability vendors — like VMware, Splunk, and New Relic, to name a few — are also talking about the potential of eBPF.
So, if you're tired of settling for approaches to container monitoring that starve your applications or resources and are difficult to manage, a better world is coming. eBPF is poised to revolutionize (among other things) the way we monitor containers.
About the authorChristopher Tozzi is a technology analyst with subject matter expertise in cloud computing, application development, open source software, virtualization, containers and more. He also lectures at a major university in the Albany, New York, area. His book, “For Fun and Profit: A History of the Free and Open Source Software Revolution,” was published by MIT Press.