Security Is Part of Reliability

Site Reliability Engineers (SREs) traditionally focus on availability, latency, and throughput. However, a security breach is fundamentally another type of incident—often the most catastrophic. Integrating security practices into the SRE workflow is not an add-on; it's a critical component of overall system reliability. This checklist provides actionable steps for SREs to enhance container security.

The Base Image Problem

The foundation of any containerized application is its base image. Insecure or bloated base images introduce vulnerabilities and increase the attack surface from the outset. SREs should aim for minimal, trusted base images.

Consider the difference between a large, general-purpose base image and a slim, purpose-built one. A bad example might be an 800MB Ubuntu image that includes development tools like GCC, which are unnecessary for a production runtime. Conversely, a good example uses a 50MB python:3.11-slim image, containing only the Python runtime and essential libraries. This drastically reduces the potential for embedded vulnerabilities and minimizes the attack footprint.

Key Actions:

  • Use minimal base images: Opt for -slim or -alpine variants where possible.
  • Scan base images: Integrate image scanning tools into your CI/CD pipeline to detect known vulnerabilities (CVEs) before deployment. Tools like Trivy, Clair, or Anchore can automate this.
  • Build your own base images: For critical applications, consider building hardened, minimal base images from scratch or from trusted sources.
  • Regularly update base images: Establish a process for updating base images to incorporate the latest security patches.

Securing the Build Process

The container build process itself can be a vector for compromise. Ensuring that build environments are secure and that only necessary artifacts are included in the final image is paramount.

Key Actions:

  • Minimize build dependencies: Only install build tools and dependencies when needed, and remove them before creating the final runtime image. Multi-stage builds in Dockerfiles are excellent for this.
  • Scan during build: Integrate vulnerability scanning into your build pipeline. This catches issues early, before images are pushed to a registry.
  • Use trusted build environments: Ensure your CI/CD runners and build agents are secure, patched, and have minimal unnecessary software.
  • Sign your images: Use tools like Notary or Docker Content Trust to cryptographically sign your container images. This verifies the image's origin and integrity.

Registry Security

Container registries are central repositories for your images. Securing them prevents unauthorized access, tampering, and the distribution of malicious images.

Key Actions:

  • Access control: Implement strong authentication and authorization for your container registry. Use role-based access control (RBAC) to grant the least privilege necessary.
  • Vulnerability scanning: Configure your registry to automatically scan images upon push or pull. Many managed registries (e.g., AWS ECR, Google GCR, Azure ACR) offer this functionality.
  • Image immutability: Configure your registry to prevent overwriting existing image tags. This ensures that a deployed image cannot be silently replaced with a malicious version.
  • Audit logs: Regularly review audit logs for suspicious activity, such as unauthorized access attempts or image deletions.

Runtime Security

Once containers are running, continuous monitoring and hardening are essential to protect against runtime threats.

Key Actions:

  • Run containers as non-root: Avoid running container processes as the root user. Define a specific, unprivileged user in your Dockerfile. This limits the blast radius if a process is compromised.
  • Least privilege principle: Grant containers only the necessary permissions and capabilities. Avoid giving containers excessive Linux capabilities (e.g., CAP_SYS_ADMIN) unless absolutely required.
  • Read-only root filesystem: Whenever possible, configure containers to run with a read-only root filesystem. This prevents attackers from modifying the container's filesystem.
  • Network segmentation: Use network policies (e.g., Kubernetes NetworkPolicies) to restrict network traffic between containers and to external services. Only allow necessary communication.
  • Runtime threat detection: Deploy runtime security tools (e.g., Falco, Aqua Security, Sysdig Secure) that monitor container behavior for anomalies and potential threats. These tools can detect suspicious process execution, file access, or network connections.
  • Secrets management: Never hardcode secrets (API keys, passwords, certificates) in container images or environment variables. Use dedicated secrets management solutions like HashiCorp Vault, Kubernetes Secrets, or cloud provider secret managers.

Orchestration Security (Kubernetes Focus)

For SREs managing containerized applications at scale, orchestration platforms like Kubernetes introduce their own security considerations.

Key Actions:

  • RBAC configuration: Ensure Kubernetes RBAC is meticulously configured. Limit the permissions of service accounts and users to the minimum required.
  • Pod Security Policies/Admission Controllers: Utilize Pod Security Policies (deprecated but still relevant for some) or Pod Security Admission (in newer Kubernetes versions) to enforce security standards on pods. This includes enforcing non-root users, disallowing privileged containers, and limiting host access.
  • Network Policies: As mentioned earlier, enforce network segmentation using Kubernetes NetworkPolicies.
  • Secrets management integration: Integrate your chosen secrets management solution with Kubernetes, leveraging custom resources or operators for seamless secret injection.
  • Audit logs: Enable and monitor Kubernetes audit logs for security-relevant events, such as API server requests, pod creation/deletion, and role binding changes.
  • Regularly update Kubernetes: Keep your Kubernetes control plane and nodes updated to patch known vulnerabilities.

By systematically addressing each of these areas, SREs can build and maintain a robust container security posture that complements their reliability objectives.