The Blind Spots of Static Analysis
Deploying security scanners against a Kubernetes cluster is standard practice. Tools like Trivy for container images, kube-bench for CIS benchmarks, and Polaris for configuration checks offer a comforting layer of assurance. The author's homelab cluster, a typical cloud-native stack running on Talos Linux with ArgoCD and Cilium, passed every one of these tests with flying colors. Dashboards glowed green, suggesting a secure environment. However, this perceived security was an illusion, a critical blind spot exposed by the limitations of static analysis. Scanners audit declared configurations and known vulnerabilities. They don't test actual runtime behavior or detect subtle, hidden malicious activity. This gap is precisely where a purple team exercise can uncover critical security weaknesses.
Introducing the Stealthy Adversary
The experiment's core involved simulating a persistent attacker within the cluster. Instead of a noisy, easily detectable intrusion, the goal was stealth. The attacker's presence was masked by embedding malicious code within a seemingly legitimate component: a Kubernetes DaemonSet. DaemonSets ensure a copy of a pod runs on all (or some) nodes in the cluster, making them a powerful tool for system administrators and, crucially, for attackers seeking persistent, distributed access. By running the attacker's payload as part of a DaemonSet, the malicious activity was disguised as normal cluster operations. This technique bypasses traditional scanning methods that focus on the initial deployment or static configuration, as the compromise is embedded within the running workload itself.
The Purple Team Advantage
A purple team exercise blends the offensive tactics of a red team with the defensive insights of a blue team. The objective is not just to find vulnerabilities but to improve detection and response capabilities collaboratively. In this scenario, the author acted as both attacker and defender. The attacker, deployed via the DaemonSet, executed commands and gathered information without triggering any alerts from the existing security tools. The defender, leveraging the same tools and visibility, then attempted to detect this hidden presence. The surprising outcome was that the tools, which had previously declared the cluster secure, found no trace of the attacker. This highlights a fundamental limitation: scanners check for known bad, not for the unknown or the cleverly disguised.
Exploiting the Gaps
The attacker's payload, running within the DaemonSet, was designed to be as unobtrusive as possible. It didn't immediately try to exfiltrate data or cause disruption. Instead, it focused on reconnaissance, mapping the cluster from within, identifying other services, and looking for further privilege escalation opportunities. This low-and-slow approach is a classic attacker tactic, designed to evade automated detection systems that often look for sudden spikes in activity or known malicious signatures. The attacker could, for instance, use `kubectl exec` to run commands on other pods, or leverage the DaemonSet's node-level access to inspect host systems. The key was that these actions were indistinguishable from legitimate administrative tasks if viewed in isolation or through the lens of static configuration checks. The security scanners, focused on the manifest files and container images themselves, never analyzed the dynamic, runtime behavior of the compromised workload.
The Revelation: What Scanners Miss
The experiment revealed a critical truth about Kubernetes security: static analysis is insufficient for runtime security. While essential for policy enforcement and vulnerability management of known issues, it fails to detect novel threats or sophisticated evasions. The DaemonSet attacker was essentially invisible to Trivy, kube-bench, and Polaris because its malicious nature was not in its declared state but in its executed behavior. This is akin to a house alarm that only checks if the doors and windows are locked, but never detects someone who has already picked the lock and is quietly moving inside. The scanners audited the 'locked doors' but missed the 'inside job'.
Recommendations for Runtime Security
The implications for Kubernetes security are significant. Relying solely on pre-deployment scanning leaves clusters vulnerable to runtime attacks. To address this, organizations need to implement runtime security solutions. These tools monitor network traffic, process execution, and system calls within the cluster, looking for anomalous or malicious behavior. Examples include Falco, Aqua Security's runtime protection, or Sysdig Secure. These solutions can detect the kind of stealthy intrusions demonstrated by the DaemonSet attacker by observing deviations from normal operational patterns. Furthermore, continuous purple teaming exercises are crucial. Regularly simulating attacks, even simple ones like this DaemonSet example, provides invaluable feedback on the effectiveness of existing defenses and helps identify blind spots before they are exploited by real adversaries.
The Unanswered Question of Evasion
What remains to be seen is how quickly the security tooling vendors will adapt their static analysis products to better infer runtime risks, or if the market will fully embrace runtime security as a mandatory layer for Kubernetes. The current situation, where scanners can give a false sense of security while sophisticated attacks remain undetected, is untenable for organizations handling sensitive data. The challenge for defenders is to move beyond simply checking boxes and towards understanding the dynamic threat landscape within their clusters.