False Sharing in C++: Detect It and Fix It with Padding

The question

Each worker thread has its own counter in a struct array, so there is no shared state. Yet adding threads makes each one slower. What is happening?

Sharing without sharing

The code looks race-free and contention-free:

struct Stats { std::uint64_t msgs; std::uint64_t bytes; };
Stats stats[8];                         // one per thread, adjacent in memory

void worker(int id) {
    for (;;) { /* ... */ stats[id].msgs++; stats[id].bytes += n; }
}

Each thread writes only its own element. But each Stats is 16 bytes, so four of them share one 64-byte cache line. Coherence works on lines, not variables: when thread 0 writes stats[0], every other core's copy of the line is invalidated, including the cores running threads 1 to 3, which are writing their own elements of the same line. The line bounces between four cores although no byte is shared.

False sharing. Two counters owned by two threads happen to sit in one cache line, so the line moves between the cores on every increment although no data is actually shared.

That is false sharing: the cost of true sharing from the coherence protocol, without any actual sharing in the program. It is one of the most common reasons a multithreaded program fails to scale, because nothing in the source code reveals it.

The fix

Give each thread's data its own cache line:

struct alignas(64) Stats { std::uint64_t msgs; std::uint64_t bytes; };   // sizeof is now 64
Stats stats[8];

alignas(64) makes each element start on a line boundary and pads it to a full line. The standard name for the right value is std::hardware_destructive_interference_size in <new>, available in recent compilers (GCC warns that its value may change between targets, so many codebases define their own constant). Some Intel processors prefetch pairs of adjacent lines, so some teams pad to 128 bytes to stop a neighbour's line being dragged in as well.

Alternatives: keep per-thread data in each thread's own stack or thread_local storage, or allocate it separately per thread, which usually puts it far apart.

Where false sharing hides

Arrays of per-thread counters or flags, as above.
The head and tail indices of a queue, written by different threads (Module 4).
A struct holding a lock and the data it protects, plus an unrelated hot field written by another thread.
Adjacent objects from the same allocator: two small objects created one after the other by different threads can share a line.
Fields of one object written by different threads, such as a request's status written by the gateway next to its timestamp written by the strategy.

Finding it

Symptoms: throughput that falls as threads are added, high time on simple stores in a profile, and many cache misses that the data's size does not explain.

On Linux, perf c2c (cache-to-cache) records loads that hit a line modified in another core's cache and reports the hottest contended lines with the offsets within them and the code that touched each. Two different threads writing different offsets of one line is the signature of false sharing. Intel VTune's memory access analysis shows the same.

Don't overcorrect

Padding everything to 64 bytes wastes cache. A struct that is read together by one thread should stay compact; splitting it across lines makes every access touch more lines. Pad only data that different threads write, and keep data that one thread reads and writes packed.

Check yourself

In the interview

Say your own answer out loud first, then open the one we would give.

What is false sharing?

Different threads writing different variables that happen to share a cache line. Coherence works per line, so each write invalidates the other cores' copies and the line bounces between them, even though no data is actually shared. It shows up as threads slowing each other down for no visible reason.

How do you fix and prevent it?

Align and pad data written by different threads to separate cache lines, with alignas(64) or the hardware_destructive_interference_size constant, sometimes 128 to account for adjacent-line prefetching. Keep per-thread state in per-thread storage. Do not pad data a single thread uses together, since that only wastes cache.

How would you detect it?

Throughput that drops as threads are added is the first hint. perf c2c then shows the contended lines, the offsets each thread writes and the code responsible; different threads writing different offsets of the same line confirms it.