BPF for security—and chaos—in Kubernetes

BPF is probably familiar to many LWN readers, though it's likely not yet quite as well known in the Kubernetes community — but that could soon change. At KubeCon + CloudNativeCon Europe 2019 there were multiple sessions with BPF in the title where developers talked about how BPF can be used to help with Kubernetes security, monitoring, and even chaos engineering testing. We will look at two of those talks that were led by engineers closely aligned with the open-source Cilium project, which is all about bringing BPF to Kubernetes container environments. Thomas Graf, who contributes to BPF development in the Linux kernel, led a session on transparent chaos testing with Envoy, Cilium, and BPF, while his counterpart Dan Wendlandt, who is well known in the OpenStack community for helping to start the Neutron networking project, spoke about using the kernel's BPF capabilities to add visibility and security in a Kubernetes-aware manner.

The Cilium GitHub project page defines the technology as a way to provide network security that understands the APIs used between microservices by using BPF and eXpress Data Path (XDP). Cilium also makes use of BPF programs to extract data from running containers for visibility purposes.

Chaos engineering

A core architectural capability of Kubernetes is that it is resilient and, when properly configured, can recover from failure by bringing up new pods as needed. Configuration optimization for resilience can often benefit from a type of testing known as chaos testing. That form of testing comes from chaos engineering; Graf showed the definition from that web site in his slides. It is the discipline of experimenting on a software system in production in order to build confidence in the system's capability to withstand turbulent and unexpected conditions. "In a nutshell it means you're introducing chaos into your infrastructure to better understand failure modes," he said.

In particular, Graf identified fault injection as a subset of chaos testing that can be beneficial. He explained that, as the name implies, intentional faults are injected into software where a fault is not occurring normally. Fault injection can be used to simulate an outage or a service failure; it can also be used to delay an application response.

In order to do fault injection, Graf said there is a need to have some form of man-in-the-middle function to get control of communications between services. There is also a need for that function to inject the fault in only a certain percentage of the operations, rather than simply having a service failing all the time. Lastly, there is also a need for visibility to make sure that the fault injection works and that an observed failure was in fact the result of chaos testing, rather than the application crashing on its own.

Graf's chaos engineering testing stack includes using the Envoy proxy, which is an open-source service and edge-proxy project, hosted by the Cloud Native Computing Foundation (CNCF); it has Go language extensions for chaos testing with fault injections. The final piece of the stack is Cilium, which is based on BPF and implements Kubernetes network policies and identity-based enforcement. "Cilium will bring up Envoy, run it, and then, using BPF magically, or really it's not magic it's networking but it will look like it's magic, redirect all traffic subject to the fault-injection policy," he said.

The fault-injection code testing approach demonstrated by Graf is publicly available in a GitHub repository, with fault-injection testing options for delayed response and service failure conditions.

Securing Kubernetes with BPF

Wendlandt used his session to explain how BPF and Cilium can be used to improve the security of applications running in a Kubernetes cluster. He explained that, from a security standpoint, Kubernetes needs to be thought of in a different way than a general-purpose operating system. With microservices running in a Kubernetes cluster, he said it is possible to better reduce and control the attack surfaces as each specific microservice is only supposed to perform a discrete set of operations.

Wendlandt said that he is focused on runtime security attacks, those that happen as the application is running. "Attacks happen when a malicious actor finds an alternative execution path or data flow that's different than what the application team intended, but it's still permitted by the infrastructure," he said. Security controls should prevent these invalid execution and data flows, though Wendlandt cautioned that it's easier said than done, since the security controls should not limit legitimate actions and use. "Ultimately what I think is so cool about BPF and the potential of BPF for Kubernetes security is that it's not just about being more secure, it's about being more secure in a way that's very low friction and doesn't get in the way of your application developers," he said.

He explained how BPF works and what it does, noting that it can be used to provide a functions-as-a-service capability for kernel events; based on a particular event, some function can be triggered. Each of the triggered functions runs within the BPF sandbox, which isolates the function to limit unauthorized actions. With BPF, Wendlandt said that it's possible to dynamically put in hooks that enable Kubernetes and microservices intelligence.

BPF maps were described by Wendlandt as efficient data structures that persist across function invocation. In his view, BPF maps can also be thought of as arrays and hash tables. He added that BPF programs can read and write to BPF maps and that they persist beyond the lifetime of an individual execution of a BPF program. For further reference on persistent BPF objects, Wendlandt suggested an LWN article detailing what it's all about. "BPF maps can both be written by the BPF programs that are running in the kernel but also by BPF-aware tools that are running in user space," he said.

Beyond iptables

Iptables provides firewall rules based on IP addresses, which can work well for externally facing resources like web servers, though iptables is not an ideal approach for Kubernetes. Wendlandt said that in Kubernetes, pods come and go; there can be thousands of services running in a cluster. "Iptables was not really designed for that type of scale, it was fundamentally designed around IPs and ports and, if you know anything about Kubernetes networking, IPs are very ephemeral, they're coming and going and they're not meaningful for long-term identity."

What matters for identity in Kubernetes is labels. He added that with BPF and Cilium there is an understanding of Kubernetes identity that is built into the kernel processing layer both from an enforcement perspective and from a visibility perspective. "We do a lot to elevate the notion of identity from an IP address to things that are more meaningful in a cloud-native world," Wendlandt said. "Cilium is fully transparent to the apps, there are no side cars or anything, that's part of the power of BPF right? The traffic is just going through the kernel, we can grab it with BPF and do whatever we need to it without requiring the application to even be aware that we're doing that."

During his talk, he used an example of a server-side request forgery that could compromise an entire Kubernetes cluster. A compromise could come from any number of different attack vectors, even the smallest misconfiguration could represent a risk. "There's no amount of code that's too small to potentially have a security relevant bug in it," he said. "You just have to assume that at some point, some app in your environment is going to be compromised and you have to decide how you will contain that blast radius."

With BPF, it's possible to better understand what an application is doing, and constrain it to the exact space it requires, so that attackers don't have extra degrees of freedom to go down paths they they shouldn't. Wendlandt said that the nature of Kubernetes microservices make it possible to lock down access in a way that isn't as easily possible for any generic application running on Linux. While the term microservices is often just used as a buzzword, he said that it has a specific meaning for Kubernetes. "The micro in microservices means you're not running a single container with lots of stuff in it, rather you're trying to strip it down to do some basic unit of work, and it's actually a very sane and clean environment," he said.

He added that, for debugging microservices, it's possible to tell the difference between an individual coming in with the kubectl exec command and making a network call and the main process making the network call. As such, in Kubernetes it's possible to give the different access methods their own permissions. On the services side of microservices, Wendlandt said that IPs are not meaningful, rather an understanding of what services a given pod has access to has more value for security and policy. He emphasized that, in Kubernetes, identity is tied to the service API being offered via Kubernetes labels and not IP addresses. That's where Cilium comes into play, providing visibility into gRPC, HTTP, and service API requests.

Identifying and stopping runtime attacks

Limiting the risk of runtime attacks starts with having an understanding of what the expected behavior is for a given microservice. With that understanding, Wendlandt said that it's possible to monitor for deviations that are outside of the normal operating parameters. "We can take the measuring and monitoring information on how the application is supposed to behave, using information extracted via BPF, and use it to enforce security policies to limit a pod to only perform those actions."

Wendlandt demonstrated how using Cilium, along with tools from the BCC tools project, makes it possible to understand when a given system call runs (such as connect(), fork(), or exec()) and then extract additional metadata that can be fed into Cilium for making enforcement decisions.

He explained that everything in a Kubernetes pod is in a single network namespace, with one IP address, so a traditional firewall like iptables can't tell the difference between different events coming from a given pod, because it all comes with the same IP address. "With BPF, we can pull in operating system granularity, so that we can lock each one of the bits of code execution down to exactly the minimum workflow that they need and nothing else."

YouTube video of the Chaos Testing and eBPF Security sessions are available.