The Original Setup: Over-Engineering for Security

When setting up Prometheus to monitor a Spring Boot API hosted on a DigitalOcean droplet (jo4-server), the team faced a common dilemma: how to securely expose the API's Prometheus metrics endpoint (/actuator/prometheus) for scraping. The immediate, and seemingly obvious, solution was to route traffic through Cloudflare. This involved configuring Prometheus to scrape the API via Cloudflare's proxy. The logic was straightforward: leverage Cloudflare's security features, such as WAF, DDoS protection, and SSL/TLS termination, to shield the production API.

The production environment consisted of two DigitalOcean droplets. jo4-server ran the Spring Boot application behind an Nginx reverse proxy. The second droplet, jo4-impress, housed the internal observability stack: Prometheus, Grafana, and Loki. The team opted for a setup where Prometheus on jo4-impress would scrape metrics from jo4-server. However, instead of a direct connection, they routed this traffic through Cloudflare. This meant Prometheus would send requests to Cloudflare, which would then forward them to the API server. The API server, in turn, would respond to Cloudflare, which would then pass the metrics back to Prometheus. While functional, this architecture introduced several layers of unnecessary complexity and potential points of failure.

Diagram illustrating Prometheus scraping via Cloudflare proxy to the API server

The Problem with the Cloudflare Approach

The decision to use Cloudflare for internal service-to-service scraping, even for metrics, stemmed from a desire for robust security. However, this approach proved to be over-engineered for the specific use case. Cloudflare is an excellent tool for protecting public-facing web applications, but its benefits diminish significantly when used for internal network traffic between trusted servers within the same virtual private cloud (VPC) or cloud provider network.

Several issues arose from this configuration:

  • Increased Latency: Every scrape request had to travel from the Prometheus server to Cloudflare's edge, then to the API server, and the response had to make the same journey in reverse. This added latency to each scrape, potentially impacting the timeliness of monitoring data.
  • Unnecessary Cost: Cloudflare's services, while often free for basic usage, can incur costs at higher volumes or with advanced features. For internal traffic, these costs are entirely avoidable.
  • Complexity: Managing Cloudflare rules, DNS, and SSL certificates for internal scraping adds an overhead that doesn't provide commensurate value. Troubleshooting issues also became more difficult, as one had to consider Cloudflare's behavior as well as the application and Prometheus itself.
  • Limited Visibility: While Cloudflare provides logs, they are not as granular or as easily integrated into an internal observability stack as direct network traffic would be.

The core of the problem was treating internal infrastructure traffic with the same security posture as public-facing traffic. For internal communication, especially within a controlled VPC environment, a more direct and simpler approach is often more efficient and effective.

The Direct VPC Connection: A Simpler, More Efficient Solution

The realization dawned that the production API droplet (jo4-server) and the observability droplet (jo4-impress) were both within the same DigitalOcean project and, crucially, could communicate directly over the VPC. DigitalOcean's VPC networking allows resources within the same region and project to communicate privately using private IP addresses, bypassing the public internet entirely.

The revised architecture involved configuring Prometheus on jo4-impress to scrape the metrics endpoint of jo4-server using its private IP address. This eliminated Cloudflare from the scraping path. The steps to implement this were:

  1. Identify Private IP: Determine the private IP address assigned to the jo4-server droplet within the DigitalOcean VPC.
  2. Configure Prometheus Scrape Target: Update the Prometheus configuration (prometheus.yml) to point to the private IP address and port of the API's metrics endpoint. For example, if the private IP is 10.10.10.10 and Nginx is configured to listen on port 80 for internal traffic to the metrics endpoint, the scrape configuration would look something like:
scrape_configs:
  - job_name: 'jo4-api'
    static_configs:
      - targets: ['10.10.10.10:80']

Note: If Nginx is also handling SSL for internal traffic, Prometheus might need to be configured to use HTTPS, and the relevant certificates or settings would be applied. However, for internal metrics, often plain HTTP is sufficient and simpler.

This direct connection offers significant advantages:

  • Reduced Latency: Traffic travels directly between droplets within the DigitalOcean network, minimizing network hops and latency.
  • Lower Cost: No Cloudflare charges for this internal traffic.
  • Simplified Management: No need to manage Cloudflare configurations for internal metrics. The observability stack directly interacts with the application server.
  • Enhanced Security (for internal traffic): Traffic stays within the private VPC network, never exposed to the public internet. This is often more secure for internal services than routing through an external proxy.

What This Means for Your Infrastructure

This experience highlights a common pitfall: applying public-facing security patterns to internal infrastructure without critical evaluation. While Cloudflare is a powerful tool, its strengths lie in protecting services exposed to the internet. For inter-service communication within a secure cloud environment, leveraging the cloud provider's private networking capabilities (like DigitalOcean VPC or AWS VPC) is often the more performant, cost-effective, and simpler solution.

The lesson learned is to always question the necessity of external proxies for internal traffic. Can your monitoring tools reach your services directly and securely over private IPs? If so, that's almost always the better path. It simplifies your architecture, reduces costs, and often improves performance. The security benefits of keeping internal traffic within the VPC are substantial and shouldn't be overlooked. The team found the optimal solution was already provisioned within their DigitalOcean project, requiring only a configuration change, not the addition of external services.