Latency is a queue, not a number

Every slow-API postmortem I have read starts the same way: average response time was 60ms, everything looked healthy, and then the pager went off. The average was never wrong. It was answering a question nobody asked.

The average hides the queue

A request's latency is service time plus waiting time. Service time is what your code does — parsing, a database round trip, serialising a response. Waiting time is everything that happens before your code gets a worker at all.

Service time is roughly constant. Waiting time is not: it grows with utilisation, and past about 80% it stops growing linearly and starts growing like 1 / (1 - utilisation). That denominator is the whole story.

Watch it break

Drag the worker count. The blue line is p50, which stays flat and reassuring far past the point where the system is in trouble. The orange line is p99, which is the line your users actually experience.

Two things worth noticing. First, the knee moves right as you add workers, but it never disappears — you buy headroom, not immunity. Second, p50 barely moves across the entire range. If p50 is your alert threshold, you find out about the cliff from your users.

What to do instead

Concretely, three changes:

1. Alert on a high percentile. p99 for user-facing paths, p999 if you have the traffic to make it meaningful.
2. Export queue depth, not just latency. Depth rises before latency does, which is the difference between a warning and an incident.
3. Shed load at the edge. A fast 503 costs one connection. A request that waits 30 seconds and then times out costs a worker for 30 seconds, and takes the requests behind it down with it.

const MAX_QUEUE_DEPTH = 128;
 
export async function withLoadShedding<T>(work: () => Promise<T>) {
  if (queue.depth > MAX_QUEUE_DEPTH) {
    // Fail fast: a queued request that will time out anyway costs more
    // than one that never starts.
    throw new ServiceUnavailable({ retryAfter: 1 });
  }
  return queue.run(work);
}

None of this is new — it is queueing theory from the 1960s. What is new is how easy it has become to run at 90% utilisation without noticing, because the dashboard is showing you a mean.