Page faults and how to avoid them

When a thread touches a virtual page that has no valid mapping with the needed permissions, the processor traps into the kernel. The kernel decides whether the access is legal and, if so, makes it valid and resumes the thread at the same instruction.

• Minor fault: the data is already in memory or needs none, and the kernel only has to set up the mapping. The first write to freshly allocated memory is the commonest case: the kernel takes a free physical page, zeroes it and maps it.
• Major fault: the data must be read from disk, from a swapped-out page or a memory-mapped file. Milliseconds.

A minor fault costs on the order of a microsecond or more including the trap and the zeroing. That is small once and enormous on a path budgeted in single-digit microseconds.

malloc or new for a large block, and mmap, return address ranges, not memory. Linux allocates physical pages lazily, on first touch. So a vector reserved at start-up has its capacity in virtual terms only; the first write to each 4 KB page faults. That explains the slow start in the question: every pool, ring buffer and table touched for the first time by early messages faults page by page.

Faults later on have other sources:

• Memory that was never touched until a rare path or a busy period reached it: the last part of a pool, a rarely used table.
• Memory returned and re-requested. free may give memory back to the kernel (glibc trims the heap and unmaps large blocks); the next allocation faults again.
• Copy-on-write. After fork, parent and child share pages read-only; the first write by either to each shared page faults and copies it. Forking a large process to write a snapshot is a classic source of mysterious stalls.
• Swapping and reclaim. Under memory pressure the kernel may evict pages, including file-backed code pages of your binary, which then fault back from disk. Major faults in the middle of the day point here.
• Transparent huge page activity (previous lesson).

1. Pre-fault everything at start-up. After allocating pools, rings and tables, write to every page (writing a zero to each page is enough; memset the whole region is simplest), so the faults happen before trading.
2. Lock memory in RAM. mlockall(MCL_CURRENT | MCL_FUTURE) stops the kernel from paging anything out, including later allocations, and with MCL_FUTURE new mappings are faulted in when created. It needs the right resource limits (RLIMIT_MEMLOCK) or privileges.
3. MAP_POPULATE on mmap faults the whole mapping at creation.
4. Do not return memory. Configure the allocator not to trim or unmap (glibc's mallopt(M_TRIM_THRESHOLD, ...) and M_MMAP_THRESHOLD, or the equivalents in other allocators), and avoid munmap on live systems.
5. No fork of the trading process while it is running.
6. Warm up the code paths too: running representative messages through the system before the open faults in code pages and trains caches and branch predictors.

perf stat -e page-faults,major-faults over a run counts them. perf record -e page-faults shows where they happen. In production, a per-thread fault counter (getrusage(RUSAGE_THREAD) reports minor and major faults) read periodically can be exported as a metric, so a change that reintroduces faults on the hot path is caught the day it ships.

void prefault(void* p, std::size_t n) {
    auto* b = static_cast<volatile char*>(p);
    for (std::size_t off = 0; off < n; off += 4096) b[off] = 0;   // one write per page
}