The six dashboard rows that matter during an incident

Most dashboards are built for demos. These six rows, built on OpenTelemetry, Prometheus, Grafana and Loki, answer the questions people actually ask when production is on fire.

During an incident nobody reads a wall of 40 charts. People ask the same few questions in the same order: is it broken, for whom, since when, and where. Build the dashboard around those questions.

1. Errors and exceptions

Error rate by endpoint or message type, plus the top exception types over the last hour. A sudden new exception type is often the fastest pointer to the cause.

2. Latency

Percentiles, not averages: p50, p95 and p99 per endpoint. An average can stay flat while a tenth of your users time out.

3. Queue health

For every queue or topic: backlog size, the age of the oldest unprocessed message, and the number of active consumers. Backlog alone is misleading. A large backlog that's draining is fine. A small one whose oldest message is 20 minutes old is not.

4. Throughput

Messages or requests processed per second, next to the same time last week. Drops are often the first sign of trouble, before errors appear.

5. Database performance

Query duration for your top queries, connection pool usage, and deadlocks or timeouts. Many "application" incidents turn out to be one slow query.

6. Runtime: CPU, memory and garbage collection

CPU, working set, GC heap size, GC count and pause time, and thread pool queue length. For .NET, the OpenTelemetry runtime instrumentation publishes these without custom code. A rising heap with growing GC pauses points to memory pressure. A growing thread pool queue points to blocking calls in async code.

Connect metrics to logs

Write structured logs with Serilog, include the trace ID on every entry, and ship them to Loki. From any spike on a Grafana panel you can then jump straight to the log lines for the same requests, instead of searching by timestamp.

Alert on symptoms

Alert on what users feel, such as error rate, latency and queue age, and keep the cause-level charts on the dashboard for diagnosis. Fewer, better alerts are the ones people still respond to at night.

Dealing with this in your own system? We do this work for a living. The first call is free.

Book a call