Attack (or Penetrate Test) Cloud Native the Easy Way
Meat-and-potatoes talks on just how easy it is to illicitly access cloud native infrastructure engendered some of the more interesting discussions during Rejekts EU 2024, the conference that precedes KubeCon + CloudNativeCon Europe, which featured talks that did not make the KubeCon cut. Due to weak security defenses, distributed networks on Kubernetes, in GitHub repositories and anywhere on the cloud were shown to be vulnerable to an attacker with simple tools, or via ongoing security holes largely left unchecked.
It has been a while since I’ve looked at this, but the different tools involved for exploits are fairly accessible. Some of them are used for penetration tests, so there are good and bad use cases. A tool that can be used to trace source code by leveraging even the most immediately available secrets can be used for penetration tests for good and noble purposes, but attackers can also use such tools.
@advocatemack at @rejektsio in #Paris: The number of secrets on @GitHub repositories has exploded. @thenewstack pic.twitter.com/4AwSd5KNDx
— BC Gain (@bcamerongain) March 18, 2024
At the same time, there’s a growing number of secrets available in repositories, as was shown during the talk, “The Attacker’s Guide to Exploiting Secrets in the Universe,” by Mackenzie Jackson, developer and security advocate for GitGuardian. However, there’s another route involving even-lower-hanging fruit and a slightly larger investment, which is just buying certain tools on the dark web. I had not looked at these tools in a while to review the marketplace, but as Michael Cade, a global technologist for Veeam Software, noted during his talk, “The Bang! When bad things happen to your data,” with Anais Urlichs, developer advocate at Aqua Security, it is now possible on the dark web to pay $300 for an SSH into a random company, or $10,000 for a more targeted entry point and attack for a more lucrative illicit access to target a specific organization.
Easy Access
During his talk, “Beyond the Surface – Exploring Attacker Persistence Strategies in Kubernetes,” Rory McCune, a senior advocate for Datadog, described common attacks due to not-so-well-known access points, and challenges in finding when and where attackers have gained access. An example of an accessible access point is the kubectl debug node for Kubernetes. Kubectl debug node “is literally a manifest that does the exact same thing as I did with the pen tester: It drops you into a privileged shell on the node,” McCune said. “If I’m a cluster admin, I can just do kubectl debug node and I have a privileged container with access to host namespaces.”
When reviewing clusters, some organizations thought they had locked down the access to a cluster by removing all SSH access and only providing kubectl access. “I had to give them the kind of bad news that that’s basically the same thing. In a lot of ways that’s actually more powerful than just SSH access,” McCune said. “But that’s what our attackers are going to do. They’re going to use kubectl debug node, getting them a privileged container.”
Log data exists for Kubernetes obviously. The problem is that the logs are too often underutilized and accessed. This explains why certain major attacks can occur, but it takes days before they are reported because logs and metrics were not properly monitored. Otherwise, the telemetry data could have immediately revealed that an attack had occurred.
While Kubernetes audit logs play a crucial role, ensuring that Kubernetes audit logs are activated across all clusters is essential — and again, they are not always utilized, McCune said. Without them, numerous activities will go unnoticed, such as API requests and token usage. Since Kubernetes lacks user accounts, tracking the actions of specific users becomes challenging without logging.
“While audit logging is becoming more prevalent, its effective utilization varies among users and organizations,” McCune said. “Despite its importance, many may not prioritize configuring and monitoring audit logs thoroughly, representing a significant challenge in current Kubernetes practices.”
Therefore, enabling logging is imperative, with retention being a key aspect. “However, identifying unauthorized activities can be complex due to the diverse nature of Kubernetes distributions and their default configurations,” McCune said.
Secrets Everywhere
As GitGuardian’s Jackson pointed out, the number of secrets has continued to proliferate (hence the sought-after target by attackers), especially on the internet and in public repositories. They number over 10 million alone on Docker Hub, according to statistics Jackson presented, and they continue to proliferate on GitHub, with approximately 12.8 million secrets detected on GitHub in 2023, according to GitGuardian, Jackson said. At the same time, in addition to GitHub and Docker Hub, networks are scanned for exposed Git repositories, classic phishing schemes and other attacks that are used, such as accessing secrets via Android and iOS mobile applications.
With Git alone, “the biggest way we see is people just kind of randomly sharing them… because it’s easy right? It’s a central place where lots of developers pull the secrets… so it remains to be a massive problem,” Jackson said. “But it’s not by itself — we’ve also got lots of secrets in different places like containers.”
Vulnerability Grief
@armosec’s @oshratn: “Don’t waste your time on security fixes that don’t need to be fixed.” @Rejektsio #paris @thenewstack pic.twitter.com/9AobnISJXM
— BC Gain (@bcamerongain) March 17, 2024
Code is generally rife with vulnerabilities, but this leads to vulnerability fatigue when trying to figure out what actions need to be taken immediately and what vulnerabilities can be classified as non-urgent. This is where security tools are needed to relativize and prioritize vulnerabilities that should be acted on immediately. This includes, of course, bad code used to instrument direct attacks as they happen.
In her talk, “Context-Based Security: What Your Cloud Native Apps Really Need,” Oshrat Nir, a developer advocate for the open source project Kubescape and Kubernetes security provider ARMO, discussed a more intelligent approach and mindset for security. This “context-based security” for cloud native involves applying concepts of relevancy of vulnerabilities, organized in order of their threat hierarchy. Thankfully, eBPF offers a consistent and thorough system to do this.
As it stands now, the ability to exploit vulnerabilities is in favor of bad people. The authors of President Joe Biden’s national cybersecurity strategy — “Back to the Building Blocks: A Path Toward Secure and Measurable Software,” published in February 2024 — did not mince their words:
“Mitigating known software vulnerabilities is a complex systems problem and the current ecosystem does not sufficiently incentivize the investments required to secure the foundations of cyberspace.”
Meanwhile, Kubernetes security is an especially difficult animal. As Nir described during her talk, it is complex because Kubernetes is dynamic: “It’s intricate, and it moves all the time,” she said. As an orchestration platform, Kubernetes manages numerous containers, each running different applications, often across various nodes, namespaces, and so on. This decentralized architecture offers the scalability and flexibility “we all know and love,” Nir noted.
However, this architecture also introduces multiple layers of potential security risks, from the workload down to the infrastructure. With Kubernetes, “To tell if a vulnerability or a suboptimal setting can be exploited to breach a system, you need to have a full picture of how the system is put together and how it is actually being used by the workloads,” she said. “If not, you may find yourself fixing security issues that don’t matter, and maybe not fixing the ones that do, and thus you’re wasting two of your most important resources: time and money.”
When Bad Stuff Goes Down
@MichaelCade1 at @rejektsio in #Paris on what happens when a PacMan game is compromised, hacked, encrypted and all the high scores are turned into gibberish. But with the right tools, everything is restored. This applies to large-scale data and apps on @kubernetesio. @thenewstack pic.twitter.com/Pqumr0wlHx
— BC Gain (@bcamerongain) March 18, 2024
And what happens when there’s an attack? That possibility was illustrated when Cade provided a fun demonstration of the classic arcade game “Pac-Man” being compromised and encrypted by a ransomware attack. Cade utilized a MongoDB backend and a Node.js frontend operating on Kubernetes as the infrastructure for his demo, “Pac-Man as a Service.”
Cade showed how, with Kasten K10 and the right tools, data could be rapidly restored. He demonstrated that the high scores were no longer encrypted on the screen and that people could download and play “Pac-Man” again from the demo repository.
@urlichsanais at @rejektsio in #Paris during her talk «The Bang! » When bad things happen to your data »with @MichaelCade1: Observability can help identify when things go wrong, « but when bad things happen, what do you do afterwards for remediation? » @thenewstack pic.twitter.com/YCeFBJpXf8
— BC Gain (@bcamerongain) March 18, 2024
“Luckily, working for a backup company, I had a regular cadence of backups that were happening, so I could restore it just in time to go on stage and do the talk,” Cade said. “It was a fun little project.”
So, for those interested in ethical hacking or those attackers, the good news is that there are a lot of opportunities out there to access data with off-the-shelf tools. But for those who are trying to protect a network, it shows how vulnerable we are.
This cat-and-mouse game is an ongoing saga, with organizations doing what is necessary — or as much as they can reasonably do — in order to protect their data and applications.
