Could eBPF Save Us From CrowdStrike-Style Disasters?
Brendan Gregg, Intel Fellow and system expert, argues that eBPF can prevent future-Crowdstrike-like disasters. Others aren’t so sure.
In the aftermath of the CrowdStrike Windows security fiasco, security experts and developers alike are looking for a safer way to run low-level security programs. In a recent blog post, Brendan Gregg, a well-regarded system performance expert and Intel Fellow, suggests we can keep computers from crashing due to bad software updates, even those updates that involve kernel code,” thanks to eBPF.
Now, I like eBPF — the Swiss army knife program, which enables you to run software in a virtual machine (VM) in the Linux kernel — a lot. As Thomas Graf, Isovalent‘s CTO and co-founder, said in a speech at CloudNativeSecurityCon, “By allowing sandboxed programs to run within the operating system, eBPF enables developers to create programs that add capabilities to the operating system at runtime. The operating system then guarantees safety and execution efficiency as if natively compiled with the aid of a Just-In-Time (JIT) compiler and verification engine.
Diving deeper into security, Gregg wrote, “eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox. If the verifier finds any unsafe code, the program is rejected and not executed.”
He continued that Cisco recently acquired Isovalent and has announced a new eBPF security product: Cisco Hypershield, a fabric for security enforcement and monitoring. Gregg added that “Google and Meta already rely on eBPF to detect and stop bad actors in their fleets.” So, clearly, eBPF isn’t just an attractive deep-tech platform. It’s already being used in production by major tech players.
But, is eBPF really the answer for anyone needing commercial software that includes kernel drivers or kernel modules? Of course, eBPF isn’t production-ready for Windows yet, but Gregg appears certain it won’t be too much longer. Others aren’t so sure that eBPF is the perfect security platform for either operating system.
In an e-mail interview, Yashin Manraj, CEO of Pivotal Technologies, a low-ops development company, told me, “Gregg’s optimistic view of eBPF’s potential to eliminate kernel crashes, while compelling, requires careful consideration. While eBPF offers a safer sandbox for running code within the kernel, it is not a magic bullet.”
Manraj listed his concerns:
- As BPF programs become more complex, the potential for unforeseen errors increases. Careful testing and thorough code review are essential to mitigate this risk, not leading to system crashes but specific services going down while the rest of the system remains functional.
- Since eBPF programs interact directly with the kernel, even minor errors can have cascading effects, potentially leading to service instability.
- Like any software, eBPF programs can be vulnerable to exploits. Developers must prioritize security considerations, including input validation, memory management, and access control.
- Debugging eBPF programs can be challenging. Robust logging and tracing mechanisms are crucial for identifying and resolving issues.
Manraj concluded, “Ultimately, the success of eBPF in preventing kernel crashes and services from becoming unavailable hinges not only on the technology itself but also on the commitment of developers and security professionals to adopt robust coding practices and prioritize security throughout the development lifecycle.”
We’re not there yet.
Delving deeper, Tomer Filiba, CTO of Sweet Security, a cloud runtime security startup, warned in an e-mail interview that eBPF has its own security concerns. First, eBPF requires high privileges (CAP_SYS_ADMIN or “root”), and a program that has these privileges can also delete important operating system files or mess up the server’s configuration.” These foul-ups may be due to bugs and not malicious intent, but they’re still a real concern.
Second, Filiba continued that since eBPF can write to userspace memory, it can mess up “normal programs.” True, this wouldn’t “crash the kernel, as a driver can, but it can cause programs to crash.” Of course, that’s better than manually rebooting Windows systems into “safe mode” and fixing issues, but it will still mess up your production workloads.
Still, “Bottom line, any high privilege program can cause harm to your environment, but in terms of risk reduction, eBPF is by far superior. For instance, if your eBPF agent goes rogue, the system will likely still be operational enough to allow you to remove/upgrade the agent.
Is eBPF the answer to your security woes going forward? Well, it may not be the answer, especially in Windows. Still, between optimism and pessimism about eBPF, it’s obvious to me that eBPF-based security systems will be an important part of low-level security defenses and monitoring platforms.
