The Evolution from Monitoring to Observability
The terms "monitoring" and "observability" are often used interchangeably, but they represent distinct approaches to understanding system behavior, particularly in complex network environments. While monitoring has historically focused on known unknowns – collecting metrics for predefined conditions and alerting when thresholds are breached – observability aims to address the unknown unknowns by providing deep, contextualized insights into system internals. This shift is not merely semantic; it reflects an evolution in how we manage and troubleshoot distributed systems.
In the context of network telemetry, the traditional approach of monitoring involves collecting specific, predetermined metrics. Think of it like having a dashboard in your car that shows you the speed, fuel level, and engine temperature. You know these are important metrics, and you expect to be alerted if the fuel is low or the engine is overheating. This is effective for detecting known failure modes or performance degradations. However, this approach falls short when faced with novel issues or complex interactions within a distributed system where the causes of failure are not immediately obvious or predictable.
Observability, on the other hand, is about enabling a system to be understood from the outside. It’s less about collecting specific metrics and more about collecting rich, high-cardinality data that allows engineers to explore and debug emergent behavior. This means capturing not just the state of the system but also the context surrounding that state. For network telemetry, this translates to capturing more than just packet counts or latency. It involves understanding the flow of traffic, the attributes of the devices involved, the configurations applied, and the relationships between different network components.
The distinction becomes clearer when considering the types of questions each approach can answer. Monitoring answers questions like: "Is the CPU utilization above 80%?" or "Are we experiencing high latency on this link?" These are questions that can be anticipated and for which specific metrics can be collected. Observability, however, is designed to answer questions like: "Why is this specific user experiencing slow performance on our web application, and what downstream services are contributing to the issue?" or "What is the impact of this configuration change on traffic patterns across our entire infrastructure?" These are questions that often arise from unexpected behavior and require a deeper, more flexible data foundation to investigate.
The O’Reilly book co-authored by Kentik CEO Avi Freedman highlights this evolution. The longer version of Chapter 3, which didn't make it into the eBook, emphasizes that "reason about" is a common phrase in discussions about handling vast amounts of data. This implies that the goal is to equip engineers with the tools and data to make sense of complex systems, not just to flag predefined problems. Observability provides the necessary data richness and context to enable this "reasoning" process.
The Role of Data in Observability
At its core, observability relies on three pillars of telemetry data: logs, metrics, and traces. While monitoring traditionally focused heavily on metrics, observability embraces all three, often with a greater emphasis on high-cardinality metrics and distributed tracing. Logs provide discrete events and detailed error messages. Metrics offer aggregated numerical data over time, useful for trend analysis and alerting. Traces provide a end-to-end view of a request as it traverses through various services, revealing latency bottlenecks and dependencies.
For network engineers, this translates to collecting a wider array of data points. Instead of just SNMP-based interface counters, observability requires richer data such as NetFlow, sFlow, IPFIX, BGP, streaming telemetry from network devices, and even application-layer logs that can be correlated with network events. This comprehensive data set allows for a more holistic understanding of network performance and behavior. For instance, correlating BGP route changes with observed traffic shifts or application performance degradation can pinpoint complex interdependencies that traditional monitoring would miss.
The concept of "reasoning" over data is central here. When a network issue arises, an engineer needs to be able to slice and dice the collected data in countless ways to uncover the root cause. They might need to filter traffic by source IP, destination port, application protocol, or even specific user identifiers. They might need to correlate network latency with application response times, or compare current traffic patterns to historical baselines under similar conditions. This requires a data platform that is not only capable of ingesting high volumes of diverse data but also of querying and analyzing it with flexibility and speed.
Bridging the Gap: Practical Implications
The transition from monitoring to observability is not about abandoning existing monitoring tools but about augmenting them with a more comprehensive data strategy. Many modern network telemetry solutions aim to provide both. They collect the traditional metrics that power monitoring dashboards and alerts, but they also capture the richer, context-aware data required for deep troubleshooting and emergent behavior analysis.
Consider a scenario where a web application suddenly becomes sluggish. A monitoring system might alert on high server CPU or network latency on a specific link. An observability approach would allow an engineer to not only see that alert but also to trace the specific user’s requests across the network path, identify which microservices were involved, check the performance of each hop, and examine application logs for errors that correlate with the network events. This level of detail is essential for quickly diagnosing and resolving complex, multi-faceted issues.
The challenge lies in the volume and variety of data. Network infrastructure, especially at scale, generates an enormous amount of telemetry. Effectively collecting, storing, and processing this data requires robust platforms. Furthermore, presenting this data in a way that facilitates "reasoning" requires sophisticated visualization and analysis tools. Dashboards are useful for known states, but interactive exploration and ad-hoc querying are key to uncovering the unknown unknowns that observability is designed to address.
Ultimately, the goal is to move from a reactive stance, where we wait for alerts to fire based on predefined conditions, to a proactive and investigative stance, where engineers can actively explore their systems to understand performance, identify potential issues before they impact users, and rapidly diagnose problems when they occur. This is the promise of observability in the realm of network management.