In a modern distributed system, a single user request can pass through twenty microservices, three message queues and two databases before it returns a response. When that request fails or takes too long, the operations team needs to respond in minutes, not hours. Observability is the discipline that makes such a response possible: the ability to understand the internal state of a system from the signals it emits to the outside world. This article develops the three pillars of observability (logs, metrics and traces), the reference tools for implementing them, and the mistakes that separate an instrumented system from a truly observable one.
The term comes from control theory: a system is observable if its internal state can be inferred from its outputs. Applied to software, it means that, faced with unexpected behaviour, we should be able to explain the cause without adding new instrumentation or reproducing the problem in a controlled environment. That is the key difference from traditional debugging, which assumes the failure can be reproduced at will: in distributed production, many incidents are unrepeatable and only leave a trace in the telemetry captured at the exact moment they occurred.
The three pillars of observability
It is worth distinguishing monitoring from observability. Classic monitoring answers questions you already knew how to ask: is the CPU above 80%? Are there HTTP 500 errors? Observability goes further and lets you answer questions you never anticipated during design, by exploring telemetry data without deploying new code. That property rests on three complementary types of signal.
Metrics are numerical values aggregated over time (requests per second, 99th percentile latency, memory usage). They are cheap to store and enable real-time alerting, but they lose the detail of each individual event. Logs are discrete records of events, ideally structured as JSON with queryable fields; they offer maximum detail, but their volume and cost grow quickly. Distributed traces follow the complete path of a request across every service, assigning a common trace_id and measuring the time spent in each segment, or span. Combining all three lets you move from "something is wrong" (metric) to "this specific request failed here" (trace) and "this is the exact error message" (log).
Log stack: the Elastic family (ELK)
The ELK Stack remains the reference for centralised log management. It comprises Elasticsearch (a search and storage engine based on inverted indices), Logstash or the lighter Beats agents for ingestion, and Kibana for visualisation. A common pattern ships each container's logs through Filebeat to a Logstash pipeline that parses them with grok expressions, normalises timestamps and enriches them with metadata (service name, version, environment) before indexing them.
The key to a good log system is structured logging. A plain-text log such as Error processing order 4521 is hard to query; the structured equivalent {"level":"error","event":"order_failed","order_id":4521,"service":"checkout"} lets you filter, aggregate and correlate. It is advisable to define consistent levels (DEBUG, INFO, WARN, ERROR), apply per-index retention policies through Index Lifecycle Management (hot data on SSD for the first few days, cold data on cheap storage afterwards) and, above all, avoid logging personal data in plain text: the General Data Protection Regulation requires minimisation and pseudonymisation, and logs are one of the most frequent and most overlooked sources of personal-data leaks.
Metrics and alerting with Prometheus
Prometheus dominates the metrics space in cloud-native environments. Its model is pull-based: the server periodically scrapes HTTP /metrics endpoints exposed by each service. It stores time series identified by a name and a set of key-value labels, queried with PromQL, a language designed for time-based aggregations. An expression such as histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) returns the 99th percentile latency over the last five minutes.
Visualisation is usually delegated to Grafana, which combines data from Prometheus, Elasticsearch and other sources into unified dashboards. Alerts are managed with Alertmanager, which groups, silences and routes notifications to avoid alert fatigue: the phenomenon by which a team receives so many irrelevant alerts that it ends up ignoring the important ones too. A good alerting rule is not based on arbitrary CPU thresholds, but on symptoms the user actually perceives, formalised as SLOs (Service Level Objectives) and their error budget consumption, following the reliability engineering practice described in Google's SRE body of knowledge.
Distributed tracing and the OpenTelemetry standard
The piece that closes the loop is the distributed trace, and here the de facto standard is OpenTelemetry (OTel), a Cloud Native Computing Foundation project. OpenTelemetry unifies the instrumentation of logs, metrics and traces under a single API and a common protocol (OTLP), so that application code is not tied to a specific vendor. Back ends such as Jaeger, Tempo or commercial solutions consume that telemetry without any need to re-instrument. This neutrality is strategic: switching observability vendor without OpenTelemetry means rewriting the instrumentation of hundreds of services; with OpenTelemetry, you simply re-point the exporter.
The central concept is context propagation: when service A calls service B, it passes the trace_id in the headers (the W3C Trace Context standard). The back end can then reconstruct the full tree of spans and show exactly where the time was spent. This turns latency debugging in systems with dozens of services into a visual task rather than an exercise in log archaeology. Good instrumentation also adds semantic attributes to each span (response code, pseudonymised user identifier, deployment version), which lets you filter and compare traces by business characteristics and not just by service.
Cardinality, volume and cost control
The operational mistake that drives up the most observability bills is cardinality explosion. In Prometheus, each unique combination of labels creates an independent time series; adding a high-variability identifier as a label (a user ID, a URL with parameters, a UUID) can generate millions of series and bring down the server. The rule of thumb is to reserve labels for low-cardinality, bounded dimensions (service, environment, status code, region) and to leave high-variability information for logs or traces, where it is stored differently.
With logs, the equivalent happens with raw volume: logging at DEBUG level in production or dumping full request bodies can multiply ingestion and indexing cost tenfold. The levers to control it are sampling (keeping 100% of errors but only a fraction of normal events), tiered retention policies by age, and the separation between storage for searchable logs and cheap cold archives for compliance. Measuring cost per service and per ingested gigabyte, and reviewing it periodically, prevents the surprise of a cloud bill that grows faster than the system itself.
Comparison of the three pillars
| Dimension | Metrics | Logs | Traces |
|---|---|---|---|
| Question it answers | What is happening? | Why did it happen? | Where did it happen? |
| Storage cost | Low | High | Medium (with sampling) |
| Granularity | Aggregated | Per event | Per request |
| Suitable for alerting | Yes | Limited | Not directly |
| Reference tool | Prometheus + Grafana | ELK Stack | Jaeger / Tempo + OTel |
Phased implementation
An orderly rollout avoids the common trap of instrumenting everything at once and drowning in data. A sensible sequence is: (1) centralise the structured logs of every service into a single queryable destination; (2) expose the four golden signals (latency, traffic, errors and saturation) in each service; (3) define realistic SLOs from real data and configure error-budget-based alerts; (4) instrument traces with OpenTelemetry in critical business flows; and (5) correlate the three pillars through common identifiers so you can jump from an anomalous metric to the exact trace and log in two clicks.
Common mistakes to avoid
The first is alerting on causes rather than symptoms: a CPU-at-90% alert means nothing if the user is still getting fast responses, and it overwhelms the on-call team. The second is logging without structure or retention, which inflates the storage bill and leaves personal data exposed. The third is instrumenting without sampling on high-volume traces, which generates cost and noise; tail-based sampling, which always keeps traces with errors or high latency, solves the problem. The fourth is relying on dashboards nobody looks at: observability only adds value if it is built into incident runbooks and into the team's culture.
Frequently asked questions
What is the difference between monitoring and observability?
Monitoring checks predefined conditions (thresholds, health checks). Observability lets you explore the system and answer new questions without deploying code, thanks to the richness of correlated logs, metrics and traces.
Do I need both Prometheus and ELK, or can I choose just one?
They solve different problems: Prometheus for metrics and real-time alerting, ELK for detailed log analysis. In most serious architectures they coexist, usually joined by Grafana as a common visualisation layer.
What are the "four golden signals"?
Latency, traffic, error rate and saturation. They are the minimum set recommended by reliability engineering to understand the health of any user-facing service.
Does OpenTelemetry replace Prometheus and the ELK Stack?
It does not replace them; it unifies them at the instrumentation layer. OpenTelemetry collects the signals in a neutral way and sends them to the back ends you prefer (Prometheus, Tempo, Elasticsearch), avoiding lock-in with a specific vendor.
Conclusion
Observability is not a pretty dashboard or a tool you buy: it is a property of the system that is designed in from the very first commit. A team with structured logs, metrics aligned with SLOs and distributed traces using OpenTelemetry drastically reduces the mean time to detect and resolve incidents (MTTD and MTTR), which are the metrics that truly determine perceived reliability. The investment is not justified by generic savings, but by something concrete: when the incident hits at three in the morning, the difference between five minutes and five hours of diagnosis comes down to having instrumented well before you needed it. At Summum Marketing we design observability architectures starting from critical business flows, not from the tool catalogue, so that every signal collected has a concrete question to answer.