Branches, indirect calls and inlining

A modern core works on many instructions at once, each at a different stage: fetched, decoded, waiting for its inputs, executing, retiring. To keep that pipeline full it cannot wait for a branch to be decided before fetching the instructions after it, so it guesses. A branch predictor, trained on the history of each branch, picks a direction and the core carries on down that path speculatively.

When the guess is right, the branch costs almost nothing. When it is wrong, everything fetched after the branch is thrown away and the core restarts from the correct target. On current server cores that costs on the order of fifteen to twenty cycles, every time.

What matters is not whether a branch is taken but whether its direction follows a pattern the predictor can learn:

• A loop condition that is true a thousand times and false once: predicted correctly almost every time.
• An error check that never fires: free.
• if (q.instrument == wanted) where one quote in a hundred matches: nearly free.
• if (q.side == BID) on a feed where bids and asks alternate at random: wrong about half the time.

The same code can be fast on one input and slow on another. A classic demonstration is summing the values above a threshold in an array: on sorted data the branch is predictable and the loop is fast; on the same data shuffled it can be several times slower.

When a branch really is unpredictable and both sides are cheap, you can remove it by computing both and selecting:

// branchy
if (q.side == 0) bid_qty += q.qty; else ask_qty += q.qty;

// branch-free: the side selects which counter to add to
std::int64_t qty[2] = {0, 0};
qty[q.side] += q.qty;

Compilers often do this themselves with a conditional move. Check the generated code before rewriting; a branch-free version that does more work can be slower when the branch was predictable.

C++20's [[likely]] and [[unlikely]] tell the compiler which side to lay out as the straight-line path. They help the code layout, so the common case falls through; they do not change the hardware predictor.

A direct call to a function whose body the compiler can see is usually inlined in hot code: the body is pasted at the call site and optimised together with the caller. Inlining removes the call, but the real gain is what it enables. Constants propagate into the body, dead branches disappear, loops can be vectorised across what used to be a call boundary.

An indirect call (through a function pointer, a virtual function, or a std::function) cannot normally be inlined because the target is not known at compile time. That costs the jump itself, which the CPU must also predict, and it costs every optimisation inlining would have allowed. The next two lessons look at the two most common sources of indirect calls.

Indirect branches are predicted too. If the same call site always goes to the same target, the predictor learns it and the jump is cheap. A call site that dispatches across many targets in an irregular pattern pays for mispredictions as well.

For hot code: keep the common path straight, push error handling into separate functions marked cold or [[unlikely]], keep the functions the hot path calls visible to the compiler (in headers, or with link-time optimisation), and let measurements, not intuition, decide whether a branch is worth removing. Module 5 returns to the pipeline in more depth.