Drill: measuring and debugging

I would give a distribution rather than a number: p50, p99, p99.9 and the maximum of tick-to-trade, measured from the card's receive timestamp of the triggering packet to its transmit timestamp of the order, over a stated period and sample count. Alongside, per-stage percentiles from software timestamps on the same clock, so we can see where the time goes, and the processing time of all messages, not only those that trigger orders.

Lab replays tell us what the code can do; I would quote production numbers as the answer.

A sampling profile would mostly show the unchanged common case, so I would use the per-message stage timestamps: select the slowest 0.1% and see which stage grew compared with typical messages and with the previous release. Then look for what those messages share: message type, instrument, first touch of something, bursts, time of day.

In parallel I would correlate them with kernel events on that core, faults, switches and interrupts, using traces on the same clock. Once I have a candidate, I would reproduce it on a replay and diff the releases on exactly that path.

Equal instruction counts mean the work is the same, so the 30% is stalls: IPC fell. I would compare branch misses and cache misses at each level per thousand instructions between versions, and run top-down analysis to see whether the lost slots are bad speculation, memory bound or front-end bound.

That points at the mechanism: perhaps a field moved and a struct now spans two cache lines, or a branch became data-dependent, or code layout changed. Then I would confirm with perf mem or the assembly, fix it, and check the counters return.

First find the loads: perf mem sampling shows the instructions and data addresses of slow loads, which tells me which structure misses, and whether it is latency or bandwidth bound, which on a hot path is almost always latency.

Then reduce the number of distinct lines an update touches: flat arrays instead of nodes, hot fields packed together, cold fields moved out, price levels near the touch in a tick-indexed array. Verify with LLC misses per message, IPC and the book-update stage percentiles.

Low CPU and flat scaling mean threads are waiting rather than working, most often on a lock. An on-CPU profile would not show that, so I would use off-CPU profiling or futex tracing to find where threads block, and an instrumented mutex to get wait and hold times per call site.

Then fix the cause: shorten the long critical section, split or stripe the lock, replace read-mostly locking with snapshots, or move the data to a single owner fed by a queue, and confirm that throughput scales and wait times fall.