Mutexes, lock scope and contention

An uncontended std::mutex::lock() on Linux is an atomic compare-and-swap on a word in user space: the lock is free, the thread takes it, no system call. Unlock is another atomic operation. Together, tens of nanoseconds, most of it the cost of the atomic instructions and the cache line they touch.

Contention changes everything. When a thread finds the lock held, glibc's mutex spins briefly and then calls futex to sleep in the kernel. The unlocking thread must call into the kernel to wake it, and the woken thread waits for the scheduler to run it again. That is microseconds at best, and unbounded when the machine is busy. Busy days mean more contention, which matches the symptom.

The usual reason for contention is not the number of threads but how long each one holds the lock.

void Strategy::on_fill(const Fill& f) {
    std::lock_guard g(book_mutex_);
    book_.apply(f);
    log_ << "fill " << f.id << " px " << f.price << '\n';   // formatting and I/O under the lock
    risk_.update(book_);                                    // a long computation under the lock
}

Everything inside the braces holds the gateway off the book: formatting, a write that may block, a risk computation that does not need exclusive access to the book at all. The fix is to keep only the shared-state update inside the lock:

void Strategy::on_fill(const Fill& f) {
    BookSummary snap;
    {
        std::lock_guard g(book_mutex_);
        book_.apply(f);
        snap = book_.summary();          // copy what the slow part needs
    }
    risk_.update(snap);                  // outside the lock
    log_ring_.push(FillRec{f});          // to a logging thread, no formatting here
}

The questions for any critical section: what in here actually touches the shared state? Can the slow part work on a copy? Does anything in here allocate, log, or make a system call?

• std::lock_guard: locks for the lifetime of a scope. The default.
• std::unique_lock: movable, can be unlocked early and relocked, needed for condition variables (lesson 3).
• std::scoped_lock (C++17): locks several mutexes at once without deadlock.
• std::shared_mutex: many readers or one writer. Useful when reads dominate and are long; the bookkeeping makes it slower than a plain mutex for short critical sections.

Two threads, two mutexes, opposite order:

Thread 1 takes the book lock then wants the risk lock; thread 2 takes the risk lock then wants the book lock. Each waits for ever. The standard cures:

• A global lock order. Every thread that needs both takes the book lock first. Document it, and in debug builds assert it.
• std::scoped_lock(a, b), which takes several locks using a deadlock-avoidance algorithm.
• Never call unknown code while holding a lock: a callback could take another lock in the wrong order.
• Design out the need: if one thread owns the book and others send it messages, there is no lock to order.

A spinlock never sleeps: a thread that finds it held loops until it is free. That avoids the futex wake-up cost, which makes it attractive when critical sections are a few dozen nanoseconds and each thread has its own core.

class Spinlock {
    std::atomic<bool> held_{false};
public:
    void lock() {
        for (;;) {
            if (!held_.exchange(true, std::memory_order_acquire)) return;
            while (held_.load(std::memory_order_relaxed)) __builtin_ia32_pause();  // spin on a read
        }
    }
    void unlock() { held_.store(false, std::memory_order_release); }
};

The inner loop spins on a plain load so the cache line stays shared until it changes, and pause tells the core it is spinning. Spinlocks are a disaster if the holder can be descheduled, because waiters burn their whole time slice; they belong on isolated cores where threads are never preempted (Module 6).

In trading systems the best answer to "how do you reduce lock contention" is often to remove the shared state. The book belongs to one thread. Other threads that need its contents get snapshots or messages through a lock-free queue. The rest of this module builds that queue.