The dashboard said failures are kept in full. It was wrong.

Recording every request in a database means the observability system scales with the thing it observes, and the first symptom of that is the dashboard making an outage worse. The way out is two stores answering two different questions.

A rollup counts everything and knows nothing about any individual request. One row per minute, surface, metric and label set, written by an upsert that adds to a counter. Its size is governed by how many distinct label combinations exist, not by traffic: five surfaces at twenty metric names is a few thousand rows a day whether you serve ten requests or ten million.

INSERT INTO metric_samples (bucket, surface, name, labels, count, sum)
VALUES (...)
ON CONFLICT (bucket, surface, name, labels) DO UPDATE
  SET count = metric_samples.count + excluded.count,
      sum   = metric_samples.sum   + excluded.sum;

Two processes flushing the same minute both add, so nothing needs coordinating and no sample is lost to a read-modify-write, which would drop counts silently and only under the load where they matter.

The other store is individual requests, kept for a short window, for the moment somebody asks what actually happened at 13:20. That one is proportional to traffic, so it is sampled and pruned.

Labels are where a metrics table dies

The cost model above only holds if the label set stays small. So the status class goes on the metric and the status code does not: five values instead of sixty. The path never does, because a path with identifiers in it is unbounded, and that is the label that turns a few thousand rows a day into a few million.

add("http.request", {
  method: event.request.method,
  status: `${Math.floor(event.response.status / 100)}xx`,
  outcome: event.outcome,
  device: event.client.deviceClass,
  ...(event.client.country ? { country: event.client.country } : {})
}, event.wallTimeMs);

The full path, the address and the user agent all go on the sampled row instead, where the volume is bounded by the sample rate rather than by cardinality.

The part I got wrong was the sentence

The dashboard shipped with a line above the request log: failures are kept in full, successful requests are sampled. It reads like a reasonable summary of the design. It is not what the code does.

const failed = event.response.status >= 500 || event.outcome !== "ok";
if (!failed && random() > sampleRate) return;

`outcome` is `ok` for any completed response, including a 401 and a 404. So the sink keeps 5xx and exceptions in full, and samples everything else at one in twenty, 4xx included.

That is a defensible design. A public API is refused constantly by scanners, and keeping every 401 makes the drill-down grow exactly where it is least informative. The problem was the sentence, not the sampling.

An operator reading a short list under a 4xx filter and concluding only three people were refused has been misled by the interface, which is the precise mistake the exact-versus-sampled split exists to prevent. The screen now says which is which, in the empty state and above the log.

Approximate, and honest about it

The same principle applies to the p95. It comes off a cumulative histogram with six rungs, so it is reported as the rung rather than interpolated between two of them.

for (const bound of bounds) {
  if ((ladder[String(bound)] ?? 0) >= target) return bound;
}
// Past the top of the ladder. Returning the largest bound would report a
// slow surface as exactly its own threshold, so the overflow is visible.
const largest = bounds[bounds.length - 1];
return largest === undefined ? 0 : largest * 2;

At or under 250ms is a claim a six-rung histogram can support. 247ms is not, and the second one is more persuasive, which is exactly the reason not to print it.