Observability that answers useful questions

Collecting more telemetry does not automatically make a system easier to understand. A useful observability setup starts with the questions operators need to answer and captures enough connected evidence to answer them quickly.

The goal is not to build the largest dashboard. It is to shorten the path from “users are affected” to “we know where and why.”

Begin with service behavior

For a request-driven service, four signals provide a strong starting point:

• Traffic: how much demand the service receives
• Errors: how many operations fail from the user's perspective
• Latency: how long successful and failed operations take
• Saturation: how close limited resources are to exhaustion

Track these by meaningful operation such as create_order or search_products, not only by server hostname. Infrastructure health matters, but a healthy CPU graph cannot prove that checkout works.

Metrics show shape

Metrics are efficient for rates, distributions and long-term trends. Use counters for events, gauges for current state and histograms for distributions such as request duration.

Prefer ratios and percentiles over isolated totals or averages:

error rate = failed requests / total requests
capacity use = active database connections / connection limit
p95 latency = duration below which 95% of requests complete

An average can hide a slow minority of users. A percentile exposes the tail, but it still needs traffic volume and error rate for context.

Control label cardinality. Route names such as /orders/:id are bounded; raw URLs such as /orders/7fd8... can create a new time series for every order and overwhelm the metrics system.

Logs explain events

Logs should be structured, searchable and consistent across services.

{
  "timestamp": "2026-09-03T10:42:18Z",
  "level": "error",
  "service": "orders-api",
  "environment": "production",
  "operation": "create_order",
  "request_id": "req_81f2",
  "tenant_id": "tenant_42",
  "error_code": "inventory_timeout",
  "duration_ms": 1830
}

Use stable fields instead of embedding all context in a prose message. Include identifiers that connect related work, but never log passwords, access tokens, full payment details or unnecessary personal information.

Log at decision boundaries: request accepted, dependency failed, retry scheduled, state changed. Logging every function entry creates cost and noise without explaining what the system decided.

Traces connect the path

A distributed trace follows one operation across services. It can reveal that an API's latency comes from a database query, a downstream call or time waiting for a queue.

Propagate trace context through HTTP calls and asynchronous messages. Name spans after stable operations rather than values, and attach bounded attributes such as service, outcome and dependency.

Sampling is necessary at scale, but purely random sampling can discard the rare failures you need most. Keep a higher proportion of errors and unusually slow traces when the tracing system supports outcome-aware sampling.

Connect all three signals

Telemetry becomes powerful when an alert links to a dashboard, the dashboard links to representative traces, and traces link to correlated logs through trace and request identifiers.

alert: checkout error budget is burning
  → dashboard: failures concentrated in create_payment
  → trace: most time spent waiting for payment provider
  → logs: upstream_timeout after second retry

Without correlation, engineers manually align timestamps across several tools while the incident continues.

Alert on user impact

An alert should be actionable and urgent enough to justify interruption. Page for symptoms that threaten a service objective: sustained error rate, severe latency, unavailable capacity or fast error-budget consumption.

Use lower-urgency channels for conditions that require attention but not immediate interruption, such as disk growth forecasts or a single failed background job that will retry safely.

Every paging alert should answer:

1. What user-facing behavior is at risk?
2. How severe and widespread is it?
3. Which dashboard and runbook should the responder open?
4. Who owns the affected service?

If an alert fires frequently and requires no action, fix or remove it. Repeated noise teaches responders to ignore the system.

Design dashboards around decisions

A service overview should make the current state obvious before showing implementation detail.

Start with traffic, error rate and latency against their objectives. Then show dependency performance, saturation and deployment markers. Put diagnostic panels below the high-level health signals.

Deployment annotations are particularly valuable. A sharp change beginning at a release boundary immediately produces a testable hypothesis without claiming that the deployment must be the cause.

Make telemetry part of feature delivery

Observability added after an incident is usually incomplete. During feature design, decide:

• Which operation represents success or failure
• Which dimensions are needed to isolate affected users safely
• How work will be correlated across boundaries
• What normal and unacceptable behavior look like
• Which runbook will explain the first response

Test telemetry in staging and during controlled failure. A dashboard that has never displayed an error path may fail precisely when it becomes important.

A practical minimum

For each production service, establish:

1. A clear owner and service description
2. User-centered success, latency and availability indicators
3. Structured logs with request or trace correlation
4. Dependency timing and error visibility
5. Alerts tied to actionable thresholds
6. Deployment and configuration-change markers
7. Retention and access rules appropriate for the data
8. A runbook tested by someone other than its author

The useful mental model

Metrics tell you that behavior changed. Traces show where time and failure moved. Logs explain the events and decisions around that point.

Collect telemetry as a connected explanation of the system, not as separate piles of data.