Where the network stack spends its time

1. The card receives the frame, checks it, and writes it by DMA into a buffer from its receive ring in host memory. It may timestamp it in hardware.
2. The card signals the host. Traditionally with an interrupt. To avoid an interrupt per packet under load, cards coalesce: they wait a few microseconds or a few packets before interrupting. Coalescing saves CPU and adds latency.
3. NAPI polling. The interrupt handler schedules a software interrupt (softirq) which polls the ring and processes a batch of packets, re-enabling interrupts when the ring is empty.
4. Protocol processing. The driver builds socket buffers, the IP and UDP layers validate and route each packet, and the packet is matched to a socket and appended to its receive queue.
5. Waking the reader. If a thread is blocked in recv, it is woken and must be scheduled.
6. The copy. recv copies the payload from kernel memory into the application's buffer and returns.

On a well-tuned host this totals a few microseconds; on a busy, untuned host, tens of microseconds with a long tail, from coalescing timers, softirq scheduling, cache misses in the kernel's data structures and wake-up latency.

• Interrupt coalescing: ethtool -C to reduce or disable it for latency (rx-usecs 0), at the cost of more interrupts.
• Interrupt affinity: route the card's receive queue interrupts to a core close to the application's thread, on the same socket as the card (PCIe devices belong to a socket), and not to the application's own isolated core.
• Receive-side scaling: cards hash flows to several receive queues; steering the feed's flows to a dedicated queue, and that queue to a chosen core, keeps them away from other traffic.
• Busy polling: SO_BUSY_POLL (or the net.core.busy_read sysctl) makes a recv on an empty socket poll the card's ring for a while instead of sleeping, so the reading thread processes its own packets without waiting for an interrupt. It costs CPU and brings the kernel path closer to bypass latency.
• Avoid the wake-up: a thread spinning on non-blocking reads never sleeps, so step 5 disappears.

To know where the time goes, you need timestamps at known points:

• Hardware receive timestamps taken by the card when the frame arrives, retrieved with SO_TIMESTAMPING and the hardware flags (the card must support it and be enabled with the SIOCSHWTSTAMP ioctl).
• Software timestamps in the kernel when the packet is processed.
• Application timestamps when your code sees it and when it sends a response.
• Hardware transmit timestamps for outgoing packets, delivered back through the socket's error queue.

The difference between the card's receive timestamp and the application's is the host's receive latency. Between the card's receive and transmit timestamps lies the whole software response, tick-to-trade (Module 8).

Card clocks must be synchronised with the host clock (with phc2sys) and with the outside world (with PTP) for those comparisons to mean anything (Module 10).

Sending goes through the stack in reverse: the send system call copies the data into a socket buffer, the protocol layers build headers, the packet is queued to the card, the driver writes a descriptor and rings a doorbell register, and the card fetches the packet by DMA and sends it. The same tools apply: fewer system calls, no queuing disciplines that delay packets, and kernel bypass for the critical path.

Every step in the kernel path is general-purpose code designed for throughput and fairness across all processes. The lowest latencies come from removing it, which is the subject of the next lesson.