Finding memory bottlenecks

A memory-bound function spends its cycles waiting for loads to return from caches or main memory. The next questions are which loads, from which level, and whether the code is limited by latency (each load waits for the previous) or bandwidth (the memory system is delivering data as fast as it can).

Hot-path code in trading is almost always latency bound: it touches a few scattered cache lines per message, not gigabytes per second. Bandwidth limits show up in bulk work, such as replaying a day of data, recalculating risk across many positions, or copying large buffers.

• Miss counters by level (L1-dcache-load-misses, LLC-load-misses, and the more specific events like mem_load_retired.l3_miss on Intel) say how many loads went how far.
• perf mem record samples individual loads with their data addresses and latencies, using precise sampling features of the processor. perf mem report then shows which instructions and which data addresses produced the slow loads, which usually points straight at one data structure.
• perf c2c for loads that hit lines modified by other cores (Module 5).
• Top-down's sub-levels split memory-bound time into L1, L2, L3 and DRAM bound, and into latency and bandwidth.

The fixes are the themes of Modules 1, 3 and 5:

• Fewer lines per operation. Pack the fields the hot path reads together; shrink types; move cold fields out; replace node-based structures with flat ones.
• Keep the working set resident. If the hot data fits in L2, misses become rare. Splitting a large table into a small hot part and a large cold part often does it.
• Remove dependent chains. Index-based structures instead of pointer chains, so addresses can be computed in advance and misses overlap.
• Prefetch where addresses are known ahead but the hardware cannot predict them.
• Huge pages if TLB misses are a significant share.

When the memory system is saturated, the fix is to move less data: smaller representations, structure of arrays so that a pass reads only the fields it needs, blocking loops so data is reused while in cache, avoiding writes of data nobody reads, and non-temporal stores for write-once output. Adding threads does not help a bandwidth-bound loop beyond the point of saturation, and may make it worse.

Measured bandwidth close to the machine's achievable peak (from a tool such as Intel's Memory Latency Checker or STREAM-like benchmarks) means bandwidth bound. Low bandwidth use with many long-latency loads means latency bound. Also telling: if two independent instances of the loop on different cores each run as fast as one alone, it was latency bound; if they slow each other, they are fighting over bandwidth (or a shared cache).

The book update from the question: perf mem shows most slow loads in Book::apply reading Level objects, with addresses scattered across a large region. The levels are in a std::map, one node per level. The fix: keep levels near the touch in an array indexed by tick offset (Module 9), so the levels the feed updates most sit in a few contiguous, cache-resident lines. Verify: LLC misses per message fall, IPC rises, the stage timestamp for the book update falls at p50 and p99.