Context Switch Cost and the Linux Scheduler for Low Latency

The question

The strategy thread has its own process and does nothing but poll a queue. Its latency still has spikes of tens of microseconds. Where could they come from?

What a context switch is

A context switch is the kernel stopping one thread on a core and running another: saving registers and state, choosing the next thread, restoring its state, possibly switching address spaces. It happens when a thread blocks (on I/O, a lock, a sleep: a voluntary switch) or when the scheduler preempts it (its time slice is up, or a higher-priority thread became runnable: an involuntary switch). Interrupts also take the core away briefly, without a full switch.

What it costs

The direct cost, the switch itself, is a few microseconds or less on current hardware. The indirect cost is usually larger: whatever ran in between used the core's caches, TLB and branch predictor. When the thread comes back, its data and code have been partly evicted and its predictions are stale, so it runs slowly for a while.

A preemption. The direct cost is the switch itself; the larger cost is that the other task has evicted the strategy's data and code, so it runs slowly for a while after it returns.

Step 1 / 5

The strategy runs with warm caches.
Step 2 / 5

A timer tick arrives and the scheduler decides to run something else.
Step 3 / 5

Another task uses the core and fills its caches with its own data.
Step 4 / 5

The strategy comes back, but its data has been evicted: it runs slowly while the caches refill.
Step 5 / 5

Back to full speed.

For a thread whose job is to react to a message in a few microseconds, being switched out at the wrong moment costs the switch, the time the other task ran, and the cold-cache period afterwards. That is the tens of microseconds in the question.

Sources of switches for a thread that never blocks

Even a thread that only spins can lose its core:

Other runnable threads on the same core. Without pinning, the scheduler places threads wherever it likes and may share the strategy's core with a logger, a monitoring agent or a kernel worker.
Migration. The scheduler may move the thread to another core to balance load, leaving its warm caches behind.
The timer tick. By default each core takes a periodic timer interrupt (often 250 or 1000 times a second) for scheduling bookkeeping.
Interrupts from devices routed to that core: network cards, disks, timers.
Kernel housekeeping: work queues, RCU callbacks, TLB shootdowns from other threads of the process.

Seeing them

/proc/<pid>/task/<tid>/status shows voluntary_ctxt_switches and nonvoluntary_ctxt_switches per thread. For a spinning thread, the involuntary count should not move during the day.
perf sched and perf trace record scheduling events with timestamps.
/proc/interrupts shows which interrupts land on which core.
Tracing tools such as ftrace or bpftrace can record every switch and interrupt on a chosen core, which is how teams find the one stray kernel thread that wakes up every few seconds.

What to do about it

Pin each latency-critical thread to its own core: pthread_setaffinity_np or sched_setaffinity in code, taskset from outside.
Keep everything else off those cores: isolate them from the general scheduler and move interrupts away (lesson 6).
Never block on the hot thread: no sleeping, no blocking I/O, no mutexes that might be contended. Spin instead.
Avoid hyper-threading siblings. Two hardware threads on one physical core share its caches and execution units. A busy sibling slows the hot thread down; many setups disable SMT or leave the sibling of each hot core idle.

Busy-polling costs a whole core per hot thread and the power to keep it busy. That is the standard trade in this industry: cores are cheap compared with microseconds.

Check yourself

Question 1 of 3

What is usually the larger cost of a context switch for a latency-critical thread?

Question 2 of 3

A thread spins and never blocks. How can it still have involuntary context switches?

Question 3 of 3

Where can you see a thread's involuntary context switch count on Linux?

In the interview

Say your own answer out loud first, then open the one we would give.

What does a context switch cost?

A few microseconds directly, to save and restore state and choose the next task. Indirectly, more: whatever ran in between pollutes the caches, TLB and branch predictors, so the thread runs slowly for a while after it resumes.

How would you stop a latency-critical thread being descheduled?

Pin it to a core, isolate that core from the general scheduler, move interrupts and kernel housekeeping off it, keep its hyper-thread sibling idle, and make sure it never blocks. Then confirm with the per-thread involuntary context switch counter staying flat.

Why do trading threads busy-poll instead of sleeping?

Waking a sleeping thread goes through the kernel and the scheduler, adding microseconds and jitter, and its caches go cold while it sleeps. A thread spinning on its own core sees new work in tens of nanoseconds. The cost is a dedicated core per hot thread, which firms accept.

Context switches and the scheduler