The Importance of Deep SIEM Visibility
Security Information and Event Management (SIEM) platforms are fundamental to modern cybersecurity operations. However, simply deploying a SIEM tool does not guarantee comprehensive visibility. Critical telemetry can be lost if the SIEM is not properly configured to ingest and correlate events from all relevant sources. This post details a hands-on approach to building, attacking, and auditing a Wazuh SIEM lab environment to uncover these potential blind spots. The objective is to move beyond dashboard watching and engineer true detection capabilities, understanding where and why logs might fail during an actual incident.
The author, Joseph, draws a parallel between cybersecurity and his previous career as a house painter, emphasizing that quality in both fields hinges on meticulous preparation. In painting, this means proper sanding and priming; in SIEM, it translates to robust log source configuration and validation. Skipping these foundational steps, whether in paint or security, leads to fragile systems that fail under pressure.
Building the Wazuh Lab Environment
The lab setup involves several key components designed to simulate a realistic attack surface and detection mechanism. The core of the SIEM is Wazuh, an open-source security monitoring platform. To provide a target for attacks and log generation, a Damn Vulnerable Web Application (DVWA) instance is deployed within a containerized environment. This setup allows for precise control over the attack vector and the logging capabilities of the target system.
A crucial aspect of this lab is configuring custom log ingestion. Web server access logs, for instance, can capture evidence of an attack, such as an attempted exploit. However, if the underlying host system or the container itself does not log the specific commands executed by an attacker, the SIEM will only see partial information. The goal is to ensure that the SIEM captures not just the web request, but also the system-level activity that results from it. This involves configuring Wazuh agents to monitor specific files, process executions, and network connections within the containerized environment.
The process begins with deploying Wazuh manager and agents. Agents are installed on the host system and configured to monitor the DVWA container. This monitoring includes standard system logs, but critically, also custom log files generated by the web server and potentially application-specific logs. The DVWA is intentionally configured to be vulnerable, providing a controlled environment to test detection rules and exploit pathways. The entire setup is managed using tools like Docker and potentially a container orchestration platform for more complex scenarios, though a single-server play environment is sufficient for demonstrating the core concepts.
Executing Attacks: Reconnaissance and Command Injection
With the lab environment established, the next phase involves simulating attacker actions. The first step is typically reconnaissance. Attackers will probe the web application to identify vulnerabilities, misconfigurations, and potential entry points. This can include scanning for open ports, identifying the web server software and version, and enumerating directories and files. In the Wazuh lab, these reconnaissance activities are logged by the web server access logs and, if configured correctly, by the Wazuh agent monitoring the host system.
Following reconnaissance, the lab simulates command injection attacks. This is a critical vulnerability where an attacker can execute arbitrary commands on the host operating system through the vulnerable web application. For example, an attacker might exploit a flaw in a web form that processes user input without proper sanitization, leading to the execution of shell commands. The DVWA provides specific modules designed to demonstrate these types of attacks.
The success of detection hinges on the SIEM's ability to correlate the web server's record of the malicious request with system-level events. If the web server logs an attempt to inject a command, but the Wazuh agent on the host doesn't log the execution of that command or any subsequent suspicious process activity, the SIEM's detection capability is incomplete. This is where the 'breaking' aspect of the lab comes into play – actively testing the limits of the logging configuration.
Auditing Post-Exploitation and Persistence
Once an attacker gains initial access through a vulnerability like command injection, they often attempt to establish persistence. This means finding ways to maintain access to the compromised system even if the initial vulnerability is patched or the system is rebooted. Common persistence techniques include creating new user accounts, installing backdoors, modifying system startup scripts, or leveraging scheduled tasks.
The auditing phase of the lab focuses on detecting these post-exploitation activities. This requires the SIEM to be configured to monitor for specific system changes: the creation of new user accounts, the modification of critical system files (like `/etc/passwd` or startup scripts), the execution of unusual processes, or unexpected network connections originating from the compromised host. Wazuh's File Integrity Monitoring (FIM) and process monitoring capabilities are crucial here.
By simulating these persistence techniques and observing whether Wazuh generates alerts and captures the necessary logs, the effectiveness of the SIEM's configuration can be audited. The surprising detail here is not the ease with which an attacker can establish persistence, but rather how easily these actions can go undetected if the SIEM's monitoring is not granular enough. A failure to log the creation of a new, suspicious user account, for instance, leaves a significant gap in the security posture.
Identifying Logging Breakdowns and Improving Detection
The primary takeaway from this hands-on exercise is the identification of logging breakdowns. These occur when an event is logged at one layer (e.g., web server access log) but not at a deeper layer (e.g., OS process execution log), or when the SIEM agent fails to ingest or parse critical events. For example, a web server might log a `GET` request containing a suspicious URL, but if the agent doesn't log the actual command that the web server process executed on the OS, the detection of a successful command injection is hampered.
Improving detection involves refining the Wazuh agent configuration. This might mean enabling more detailed logging levels on the host OS, configuring specific custom rules within Wazuh to parse and alert on unique log formats, or ensuring that agents are correctly deployed and configured to monitor all relevant file paths and processes within the containerized environment. It's about treating the SIEM setup not as a static deployment but as a dynamic, continuously validated system.
What nobody has addressed yet is the scalability challenge of this granular, custom log ingestion across thousands of diverse containerized applications in a production environment. While this lab demonstrates the principle, translating it to a large-scale deployment requires sophisticated automation and policy management.
Conclusion: Engineering Effective SIEM Detection
Building, breaking, and auditing a Wazuh lab environment provides invaluable insights into the practicalities of SIEM deployment. It highlights that effective detection is an engineering discipline, requiring careful planning, rigorous testing, and continuous refinement. By simulating real-world attacks and actively seeking out logging gaps, organizations can build more resilient security operations that provide true visibility into their systems, rather than just a superficial dashboard view.