Quant developer HFT course · Module 6: The operating system
Drill: the operating system
Operating-system questions are usually framed as mysteries: something is slow at a particular time. Answer with the mechanism, the evidence that would confirm it, and the setting or design that removes it.
Prompt 1: Slow after start
The interviewer says
The first few thousand messages after start-up are much slower than the rest. Why?
Before you answer
Question · choose every correct option
Which explanations fit? Choose all that apply.
Lazy allocation, cold hardware state and lazy loading all hit the first use of each path.
Answers that lose the point
The JIT is warming up.
There is no JIT in C++. The equivalent effects are faults and cold caches.
A strong answer
Several first-time costs land together. Linux maps physical memory lazily, so the first write to each page of every pool and ring faults; code pages and dynamically linked symbols are also resolved on first use; and the caches and branch predictors are cold.
I would pre-fault all memory at start-up with writes or MAP_POPULATE, lock it with mlockall, link with immediate binding or statically, and run representative traffic through the system before the open to warm the code paths. Then confirm with page-fault counters during the first minutes of trading.
Prompt 2: The mid-day stall
The interviewer says
Once or twice a day, a message takes hundreds of microseconds. No errors. What would you check?
Before you answer
Question · choose every correct option
Which are plausible? Choose all that apply.
Faults, THP activity, fork and interference are all classic; C++ has no garbage collector.
Answers that lose the point
Garbage collection.
Not in C++; this answer ends the interview badly.
A strong answer
I would correlate the stalls with system events. Page-fault counters per thread would show whether memory is being faulted in mid-day, from allocator trimming, rarely touched regions, or copy-on-write after a fork for snapshots. THP compaction shows up in kernel stats. Interrupts or kernel threads on the hot core show up in /proc/interrupts and in scheduler tracing.
The fixes follow: pre-fault and lock memory, stop the allocator returning memory, disable THP or restrict it to madvise, never fork the trading process, and isolate the core.
Prompt 3: Spinning but jittery
The interviewer says
The strategy thread spins on its queue and never blocks, yet it shows spikes of tens of microseconds. Where are they coming from?
Before you answer
Question · choose every correct option
Which sources are possible? Choose all that apply.
Not blocking does not stop the scheduler, the tick or interrupts from taking the core.
Answers that lose the point
Use SCHED_FIFO.
May starve the system on a shared core and does nothing about interrupts.
A strong answer
A thread that never blocks can still be preempted, migrated or interrupted. I would check its involuntary context switches, which interrupts land on its core, and whether the timer tick is running there.
The fix is isolation: pin the thread to a core removed from the general scheduler, enable nohz_full on it, move every interrupt and kernel work queue to housekeeping cores, and keep its hyper-thread sibling idle. Then verify with a jitter detector and flat context-switch counters.
Prompt 4: Two processes, one ring
The interviewer says
The feed and the strategy are separate processes. How do they exchange messages fast, and what can go wrong?
Before you answer
Question · choose every correct option
Which statements are right? Choose all that apply.
A mutex held by a crashed process blocks the survivor unless it is robust; lock-free structures avoid the problem.
Answers that lose the point
A TCP socket on localhost.
Works, at the cost of system calls and copies on every message.
A strong answer
Shared memory: both processes map the same pages, preferably huge pages pre-faulted at start-up, and exchange messages through a single-producer single-consumer ring with release and acquire on the indices. No system calls or copies per message.
Everything in the region must be position-independent and fixed-layout, with a version header so mismatched builds refuse to attach. The feed must treat a full ring as a dead or slow consumer and drop with a count rather than block, and each side needs a heartbeat to detect a dead peer.
Prompt 5: Setting up the box
The interviewer says
You are given a new server for the trading system. What do you configure?
Before you answer
Question · choose every correct option
Which belong in the answer? Choose all that apply.
Swap is the opposite of what a latency-critical process wants; memory is locked instead.
Answers that lose the point
Install the latest kernel.
Possibly useful, but not a configuration.
A strong answer
I would start from the core layout: which threads are latency-critical, one isolated core each, plus housekeeping cores for the kernel, interrupts and slow threads, all on the socket local to the network card. On the kernel command line: isolcpus or an isolated cpuset, nohz_full and rcu_nocbs for those cores, huge pages reserved at boot, and THP disabled or set to madvise.
Then interrupt affinity to housekeeping cores with irqbalance off, a BIOS low-latency profile with deep C-states limited and the frequency fixed, and a decision on hyper-threading. The process locks its memory, pins its threads and uses kernel bypass for the network. Finally I would prove it with a jitter detector before trusting it.
After the drill
The OS questions share one answer shape: find the event that took the core or stalled the thread, prove it with a counter or a trace, and remove it by preparing memory early, isolating cores and never asking the kernel for anything on the hot path. Module 7 applies the same thinking to the network.