Measuring latency: percentiles, tails and honest numbers

Latency is a distribution. Most messages take about the same time; a few take much longer, because of a cache miss here, an interrupt there, a page fault, a burst queueing behind itself. Plotted, it has a sharp body and a long tail to the right.

The mean mixes the two. A handful of 100 µs outliers among millions of 2 µs messages moves the mean a little and tells you nothing about either the typical message or the bad ones. In trading, the bad ones matter disproportionately: the moments when the system is slowest are often the busiest, most volatile moments, which are when speed earns money and slowness loses it.

Report percentiles: the latency below which a given fraction of messages fall.

• p50 (median): the typical message.
• p90, p99: how bad it gets regularly. On a million messages a day, 1% is ten thousand messages.
• p99.9, p99.99: the rare events.
• max: the worst case, which is noisy but should never be ignored.

Always give the count and the period with them. "p99 of 4.1 µs over 18 million messages on Tuesday" means something; "p99 of 4.1 µs" alone does not.

Two rules about percentiles that interviewers like:

• You cannot average them. The mean of each hour's p99 is not the day's p99. Combine the underlying histograms, then compute.
• Tail percentiles need enough data. The p99.99 of ten thousand samples is the single worst sample.

Storing every latency for a day and sorting it works offline. In a live system, the standard tool is a histogram with logarithmically sized buckets, in the style of HdrHistogram: recording a value is incrementing one counter, the memory is fixed, and the relative precision is bounded (say 1%) across a range from nanoseconds to seconds. Histograms from many threads, processes or hours combine by adding counters.

The recording must be cheap and must not distort the hot path: take timestamps on the hot thread, but push them through a ring to another thread that updates histograms (Module 4).

For a trading system, the defining number is tick-to-trade: from the market-data packet that triggers a decision arriving at the network card to the resulting order leaving it, measured with hardware timestamps at both ends (Module 7). Internal stages (decode, book update, strategy, risk, encode) are measured with software timestamps from the same clock source, so a regression can be attributed to a stage. Only messages that lead to orders have a tick-to-trade; the time to process every message (whether or not it triggers anything) is tracked separately, because a slow book update on an unrelated message can delay the next one that matters.

A classic way load tests lie. A client sends a request, waits for the response, records the time, sends the next. If the system stalls for 10 ms, the client stalls with it and sends nothing; it records one slow response and then continues at normal speed.

But a real source does not wait. An exchange sends market data whether or not you are keeping up; during the stall, many messages arrive and queue, and each of them is late by the remaining stall time. The naive test recorded one bad sample where reality had hundreds. Its percentiles look far better than the system deserves.

The fixes: generate load on a fixed schedule independent of responses, and measure each request's latency from when it should have been sent; or correct the histogram for the missing samples, which HdrHistogram supports. In production, measure from the packet's arrival timestamp at the card, which captures queueing in front of your code.

Lab measurements on replayed data establish what the code can do; production measurements show what it does. The two disagree, and the production numbers win. Keep both, and alert on changes in production percentiles, because a regression in p99 after a release is often the first sign of a problem.