CPU isolation, real-time scheduling and kernel bypass

A pinned, spinning thread can still be interrupted by the scheduler, the timer tick, device interrupts and kernel housekeeping (lesson 3). Isolation means arranging the machine so that on the hot thread's core, nothing else ever runs.

On the kernel command line:

• isolcpus= (or cpusets/cgroups). Removes the listed cores from the general scheduler's load balancing, so ordinary threads are not placed there. Threads must then be pinned to those cores explicitly. Newer setups often prefer cgroup v2 cpusets with isolated partitions, which can be changed without a reboot.
• nohz_full=. Stops the periodic timer tick on the listed cores while they run a single task. Without it, every isolated core is still interrupted hundreds or thousands of times a second.
• rcu_nocbs=. Moves RCU callback processing off those cores to housekeeping cores.
• IRQ affinity. Each device interrupt has an affinity mask in /proc/irq/<n>/smp_affinity. Point them all at the housekeeping cores, and stop irqbalance from moving them back. The network card's interrupts go where the network-handling thread wants them, or nowhere if the card is polled (below).
• Kernel worker threads and work queues can be confined with /sys/devices/virtual/workqueue/cpumask and similar settings.

The processor's own power management adds latency:

• C-states: idle cores enter sleep states and take microseconds to wake. Spinning threads keep their own core awake, but limiting deep C-states (in BIOS, or processor.max_cstate, intel_idle.max_cstate, or by holding /dev/cpu_dma_latency open at 0) avoids wake-up delays elsewhere.
• Frequency scaling: set the governor to performance and fix the frequency, so cores do not ramp up and down.
• Turbo: higher peak frequency, less predictable; teams decide either way and measure.
• Hyper-threading: often disabled, or the sibling of each hot core left idle.
• SMIs (system management interrupts) from firmware are invisible to the OS and steal time; vendors provide low-latency BIOS profiles that reduce them, and tools like hwlatdetect measure them.

Linux offers real-time policies, SCHED_FIFO and SCHED_RR, whose threads always run in preference to normal ones. It is tempting to make the hot thread SCHED_FIFO at high priority. On an isolated core, it matters little: nothing else is meant to run there. On a shared core, a spinning SCHED_FIFO thread starves everything below it, including kernel threads that the system needs, which can hang the machine; the kernel's real-time throttling (sched_rt_runtime_us) exists to prevent exactly that and introduces its own pauses. The common advice is: isolate and pin first; use real-time priority deliberately and sparingly, not as a substitute for isolation.

Even on an isolated core, receiving a packet through the Linux network stack involves an interrupt, kernel processing and a copy into the socket buffer, then a system call to read it. Kernel bypass removes the kernel from the data path: the network card's receive queues are mapped into the process, and a thread polls them directly in user space.

• Solarflare/AMD Onload intercepts ordinary socket calls and serves them from a user-space stack, so existing code can use bypass unchanged; its lower-level API, ef_vi, exposes the raw queues.
• DPDK gives full control of the card from user space, with its own packet buffers and poll-mode drivers, at the cost of writing your own protocol handling.
• Mellanox/NVIDIA VMA and RDMA-based approaches play similar roles on their hardware.

The thread that polls the card is the network-handling thread from the core diagram; there are no interrupts and no system calls on the receive path. Module 7 goes into the network side.

Configuration drifts. Teams verify isolation with measurements: a tool that spins on each isolated core and records any gap in its own execution longer than a threshold (the principle behind hwlatdetect, sysjitter and osnoise) run before the open, plus per-thread context switch and interrupt counts exported as metrics during the day.