SRE Metrics and KPIs: Why Measurement Matters
Site Reliability Engineering is built on a simple premise: you cannot improve what you do not measure. Yet many teams struggle to identify which metrics actually matter, how to calculate them, and how to use them to drive decisions.
This guide covers every SRE metric and KPI that production teams should track in 2026 — from the foundational Four Golden Signals to advanced error budget policies. For each metric, we explain what it measures, how to calculate it, what "good" looks like, and how to automate tracking.
Whether you are building your first sre dashboard metrics or refining an existing program, this is your reference.
The Four Golden Signals
Google's SRE book introduced the Four Golden Signals as the minimum set of metrics every service should track. If you measure nothing else, measure these.
1. Latency
What it measures: The time it takes to serve a request.
Why it matters: High latency degrades user experience even when the service is technically "up." A service that responds in 10 seconds is functionally broken for most users.
How to measure it:
- Track the p50 (median), p95, and p99 latency for every service endpoint.
- Separate successful requests from failed requests. A fast 500 error should not bring your average down — it masks the problem.
- Measure latency at the edge (what users experience), not just at the application layer.
What good looks like:
For web APIs, typical targets are p50 < 100ms, p95 < 300ms, p99 < 1s. But "good" depends entirely on your use case — a batch processing endpoint has different expectations than a real-time search API.
In Kubernetes: Latency often spikes during pod scheduling, HPA scale-up events, or node pressure. Tracking latency alongside cluster events helps identify infrastructure-caused regressions. SRExpert correlates API latency with Kubernetes events automatically, so you can see when a latency spike corresponds to a node going NotReady or an HPA scaling event.
2. Traffic
What it measures: The demand on your system — requests per second, transactions per minute, or whatever unit is natural for your service.
Why it matters: Traffic is the baseline that gives context to all other metrics. A 5% error rate during 10 req/s is very different from 5% during 10,000 req/s.
How to measure it:
- Track requests per second (RPS) at the service level and the endpoint level.
- For non-HTTP services, use the natural unit: messages per second for queues, queries per second for databases.
- Track both inbound (user-facing) and internal (service-to-service) traffic.
What good looks like:
There is no universal "good" traffic number — the goal is to understand your baseline and detect anomalies. A sudden 10x spike might mean a viral moment or a DDoS attack. A sudden drop might mean your load balancer is misconfigured.
In Kubernetes: Traffic patterns determine HPA behavior. If your autoscaler is misconfigured, you will see traffic patterns that do not match pod counts. SRExpert's dashboard visualizes traffic alongside pod replica counts, making autoscaler tuning straightforward.
3. Errors
What it measures: The rate of failed requests — HTTP 5xx responses, gRPC errors, timeout exceptions, or any result that is not what the user expected.
Why it matters: Error rate is the most direct measure of user-facing reliability.
How to measure it:
- Error rate = (failed requests / total requests) * 100
- Track both explicit errors (5xx status codes) and implicit errors (successful response codes with wrong data, or responses that exceed latency SLOs).
- Break errors down by type: are they timeouts, null pointer exceptions, database connection failures, or rate limits?
What good looks like:
For most services, targeting an error rate below 0.1% (99.9% success) is a reasonable starting point. Mission-critical services may target 0.01%.
In Kubernetes: Container crashes (CrashLoopBackOff), OOMKills, and readiness probe failures all contribute to error rates. SRExpert tracks these Kubernetes-native error signals alongside application-level errors, giving you a unified error picture.
4. Saturation
What it measures: How "full" your service is — the utilization of constrained resources like CPU, memory, disk, and network bandwidth.
Why it matters: Most systems degrade gracefully under load until they hit a saturation point, then they fall off a cliff. Tracking saturation lets you act before the cliff.
How to measure it:
- CPU utilization at the pod and node level.
- Memory utilization and whether pods are approaching their resource limits.
- Disk I/O for stateful workloads.
- Network bandwidth for data-intensive services.
- Custom saturation signals like thread pool usage, connection pool utilization, or queue depth.
What good looks like:
A common rule of thumb is to start alerting at 70-80% saturation to give yourself time to scale. But this varies — some workloads are CPU-bound, others are memory-bound, and the "safe" threshold differs.
In Kubernetes: Saturation is closely tied to resource requests and limits. If your pods request too little, they get throttled. If they have no limits, a single pod can starve its neighbors. SRExpert monitors resource utilization against requests and limits, flagging pods that are consistently over-provisioned (wasting money) or under-provisioned (risking OOMKills).
SLIs, SLOs, and SLAs
The golden signals kubernetes monitoring gives you raw data. SLIs, SLOs, and SLAs turn that data into actionable reliability targets.
Service Level Indicators (SLIs)
An SLI is a quantitative measure of a specific aspect of your service. It is the raw metric.
Examples:
- Proportion of requests served in under 300ms.
- Proportion of requests that return a non-error response.
- Proportion of time the service is reachable.
Best practices:
- Choose SLIs that reflect what users care about, not what is easy to measure.
- Express SLIs as ratios (good events / total events) so they map cleanly to SLOs.
Service Level Objectives (SLOs)
An SLO is a target value for an SLI. It defines "good enough."
Examples:
- 99.9% of requests will be served in under 300ms (latency SLO).
- 99.95% of requests will succeed (availability SLO).
Best practices:
- SLOs should be achievable. Setting 99.999% when your infrastructure supports 99.9% just guarantees failure.
- Start with a loose SLO and tighten it as you improve.
- Make SLOs visible to the entire team — they drive prioritization decisions.
Service Level Agreements (SLAs)
An SLA is a contract with your users that includes consequences for missing the target — refunds, credits, or other penalties.
The key distinction: Not every SLO needs an SLA. Internal services often have SLOs without SLAs. SLAs should be less aggressive than your SLOs — if your SLO is 99.9%, your SLA might be 99.5%, giving you a buffer.
SRExpert lets you define SLIs, SLOs, and track them in real time. When an SLO is at risk, the platform alerts the responsible team before the error budget is exhausted. Configure this on the SRExpert dashboard.
MTTR, MTTF, and MTBF
These three metrics measure the reliability and recoverability of your systems over time.
Mean Time to Recovery (MTTR)
Formula: MTTR = Total downtime / Number of incidents
What it tells you: How quickly your team restores service after a failure.
Example: If your service was down for 90 minutes across 3 incidents this month, your MTTR is 30 minutes.
How to improve MTTR:
- Reduce detection time with better alerting (see alert fatigue strategies).
- Reduce diagnosis time with runbooks and AI-assisted troubleshooting.
- Reduce remediation time with automated rollbacks and self-healing.
SRExpert's AI diagnostics directly reduce the diagnosis phase of MTTR by correlating Kubernetes events, logs, and metrics to suggest probable root causes within seconds of an alert firing.
Mean Time to Failure (MTTF)
Formula: MTTF = Total uptime / Number of failures
What it tells you: How long your system runs before failing. Used for non-repairable components or as a planning metric.
Mean Time Between Failures (MTBF)
Formula: MTBF = Total time / Number of failures = MTTF + MTTR
What it tells you: The average time between consecutive failures, including the recovery period.
What good looks like:
For Kubernetes workloads, MTBF is heavily influenced by deployment frequency. Teams practicing continuous deployment may have a lower MTTF (because changes introduce risk) but a much lower MTTR (because they have strong rollback mechanisms). The net effect is usually positive.
Error Budgets
Error budgets are the mechanism that turns SLOs from aspirational targets into operational tools.
The concept: If your SLO is 99.9% availability over a 30-day window, your error budget is 0.1% — approximately 43 minutes of downtime.
How to use error budgets:
- Budget remaining > 50%: Ship features aggressively. You have reliability headroom.
- Budget remaining 20-50%: Proceed with caution. Increase testing for risky changes.
- Budget remaining < 20%: Slow down. Focus on reliability improvements before shipping new features.
- Budget exhausted: Freeze non-essential changes. Conduct a reliability sprint.
Error budget formula:
- Budget = 1 - SLO target (e.g., 1 - 0.999 = 0.001)
- Budget in minutes per month = Budget * 30 * 24 * 60 = 43.2 minutes
Error budget consumption:
- Consumption = (bad minutes / total minutes) * 100 as a percentage of the budget
SRExpert tracks error budget consumption in real time and can alert when consumption rate projects that the budget will be exhausted before the end of the window. This prevents the common problem of burning 80% of the budget in the first week and spending the rest of the month in a feature freeze.
Toil Measurement
Google defines toil as "the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows."
How to Measure Toil
- Track time spent on toil — Have your team log hours spent on repetitive operational tasks for two weeks.
- Categorize by type:
- Manual deployments
- Certificate rotations
- User access requests
- Scaling interventions
- Alert response that requires no thinking
- Calculate toil percentage = (toil hours / total engineering hours) * 100
Target: Google recommends keeping toil below 50% of an SRE team's time. Best-in-class teams target below 30%.
How to Reduce Toil
- Automate repetitive tasks (Kubernetes operators, Helm hooks, CronJobs).
- Build self-service tooling for common requests.
- Use platforms like SRExpert that automate monitoring setup, compliance checks, and alert configuration — tasks that would otherwise be recurring toil.
On-Call Metrics
On-call health directly impacts team sustainability. Burned-out on-call engineers make worse decisions, miss alerts, and eventually leave.
Key On-Call Metrics
| Metric | Formula | Healthy Target |
|---|---|---|
| Pages per shift | Total pages / number of shifts | < 2 per 12-hour shift |
| Actionable page rate | Actionable pages / total pages | > 80% |
| Escalation rate | Escalated incidents / total incidents | < 20% |
| Mean time to acknowledge | Sum of acknowledge times / incidents | < 5 minutes |
| Off-hours page rate | Off-hours pages / total pages | < 30% |
| On-call satisfaction | Survey score (1-5) | > 3.5 |
Improving On-Call Health
- Reduce noise. If more than 20% of pages are non-actionable, your alerting thresholds need tuning. SRExpert's smart alerting uses AI to reduce false positives.
- Build runbooks. Every alert should link to a runbook that tells the on-call engineer what to check and what to do.
- Distribute load fairly. Track the number of pages per person, not just per shift.
- Debrief after incidents. Use blameless post-mortems to identify systemic improvements. See our incident response guide.
Building an SRE Metrics Dashboard
With all these metrics defined, you need a single place to view them. Here is how to structure your sre dashboard metrics:
Dashboard Layout
Top level: Service overview
- Error budget remaining (percentage and time)
- SLO status for each service (meeting / at risk / violated)
- Active incidents count
Second level: Golden Signals
- Latency (p50, p95, p99) over time
- Traffic (RPS) over time
- Error rate over time
- Saturation (CPU, memory, disk) gauges
Third level: Operational Health
- MTTR trend (is it improving?)
- On-call pages this week
- Toil percentage this sprint
- Deployment frequency and change failure rate (these tie into DORA metrics)
Fourth level: Kubernetes-Specific
- Pod restart count by namespace
- Node health and capacity
- HPA scaling events
- Pending pods and scheduling failures
Tooling Options
You can build this dashboard manually using Grafana (see our observability guide), but that requires:
- Deploying Prometheus, Loki, and Grafana
- Writing PromQL queries for each panel
- Building and maintaining the dashboard JSON
- Setting up alerting rules separately
SRExpert provides this dashboard out of the box. When you connect a cluster, you immediately get golden signals, resource utilization, pod health, and SLO tracking — no PromQL required. The AI layer adds root cause analysis that a static Grafana dashboard cannot provide.
Putting It All Together: An SRE Metrics Program
Here is a phased approach to building a metrics program:
Phase 1: Foundation (Week 1-2)
- Instrument the Four Golden Signals for your top 3 services.
- Define one SLI and SLO for each service.
- Set up basic alerting.
Phase 2: Error Budgets (Week 3-4)
- Calculate error budgets for each SLO.
- Create a policy for what happens when budgets are low.
- Share error budget status with product and engineering leadership.
Phase 3: Operational Metrics (Month 2)
- Start tracking MTTR, MTTF, and MTBF.
- Measure toil for one sprint.
- Implement on-call health metrics.
Phase 4: Optimization (Month 3+)
- Refine SLOs based on actual data.
- Automate toil reduction based on measurement.
- Build executive dashboards showing reliability trends.
How SRExpert Automates SRE Metrics Tracking
Setting up a complete site reliability engineering metrics program manually takes weeks of dashboard building, alert configuration, and data pipeline work. SRExpert automates most of it.
With a single agent installation, SRExpert gives you:
- Golden signals tracked automatically for every workload in every connected cluster.
- SLO tracking with real-time error budget consumption and alerts.
- MTTR measurement from incident detection to resolution.
- Resource saturation monitoring with intelligent thresholds (not just static percentages).
- On-call integrations with PagerDuty, Slack, and email for alert routing.
- AI-powered anomaly detection that catches issues before they trigger SLO violations.
- Compliance checks that run alongside reliability metrics, covering SOC 2, HIPAA, and PCI-DSS.
The result: your team spends time acting on metrics instead of building infrastructure to collect them.
Start Tracking What Matters
The best SRE teams treat metrics as the foundation of every decision — from "should we ship this feature?" to "do we need to hire another on-call engineer?"
If you are starting from zero, pick the Four Golden Signals and one SLO per service. That alone will transform your reliability conversations.
If you want to skip the weeks of setup and get a production-grade sre metrics dashboard in minutes, try SRExpert free. One cluster, no credit card, full access to golden signals, SLO tracking, and AI diagnostics.
Explore the full feature set or see pricing plans that scale with your team.